AI Systems Engineer
LLM Reliability & AI Workflow Analysis
Over 10 years building production systems, now focused on AI reliability. LLM evaluation, multi-turn reasoning analysis, failure-mode detection, drift detection, and agentic-workflow reliability. Irvine, CA — Open to remote, hybrid, and relocation.
About
AI Systems Engineer with over 10 years building production systems, now focused on LLM reliability and AI workflow analysis.
Recently built a complete AI platform — a typed knowledge graph on Postgres, custom APIs, and agent orchestration — plus a model-agnostic evaluation framework that measures how consistently a model performs across long conversations and flags where its behavior breaks down.
I design and build the reliability and evaluation tooling rather than only running tools built by others — drift and regression checks, observability, failure-mode libraries, and validation gates.
Strong at identifying hidden gaps, clarifying decision logic, documenting system behavior, and turning messy inputs into repeatable analysis.
Experience
Built a PostgreSQL/Supabase system for LLM reliability and AI workflow analysis: 5,500+ typed nodes, 14,000+ relationship edges, an 18-cell diagnostic matrix, a failure-mode library with triggers and counters, validation gates, and TypeScript/Deno APIs that retrieve structured context for agentic workflows.
Built a model-agnostic LLM evaluation harness using a 20-prompt protocol to test reasoning stability across seven dimensions, turning long-conversation behavior into reliability scores, violation logs, and stability-band ratings for repeatable model comparison.
Implemented runtime reliability controls for AI agentic workflows, including drift and regression checks, observability hooks, and structured-output validation to detect unstable or malformed operations before they propagate.
Directed autonomous code-generation agents to build, compile, and revise backend database schemas, creating human-in-loop workflows for supervising AI-generated engineering work.
Built a Unity-based mixed-reality proof of concept for movement measurement, mapping live body-tracking data into a 3D spatial interface for visualizing motion, position, and interaction over time.
Designed the system architecture across body tracking, spatial visualization, and reporting logic.
Modeled human movement as temporal and spatial data, connecting sensors, interaction state, and user interface into a coherent system design.
Delivered production web applications for healthcare, consumer, and agency clients on a contract basis.
Translated client requirements into structured, production-ready front-end systems using HTML, CSS, JavaScript, and CMS platforms.
Technical & Analytical Toolkit
AI Reliability & Evaluation
LLM evaluation · Benchmarking · Multi-turn reasoning analysis · Failure-mode detection · Drift & regression detection · Structured-output validation · Observability
AI Systems & Agents
Knowledge graphs · Retrieval (RAG) · Autonomous agent orchestration · Prompt design · Model behavior analysis
Engineering & Data
Python · PostgreSQL / Supabase · SQL · TypeScript / Deno · REST & RPC APIs · System architecture · Requirements analysis
Currently Developing
LangChain · LangGraph · Langfuse · Weights & Biases
Work Samples
Self-directed AI knowledge platform built to capture, structure, analyze, and retrieve complex system knowledge over time.
Highlights: 5,500+ typed knowledge nodes · 14,000+ typed relationships · 100+ table PostgreSQL/Supabase schema · 200+ stored procedures / RPC functions · Custom TypeScript/Deno API surface · Large archived AI conversation corpus · Scheduled background agents · Interactive operator surfaces for search, inspection, and plan navigation.
Demonstrates AI systems engineering, knowledge graph design, data structuring, workflow automation, and long-horizon independent system development.
A structured LLM reliability evaluation method for testing how model behavior changes across multi-turn interactions, constraint shifts, evidence updates, and ambiguous instructions.
Focus areas: multi-turn stability · hallucination risk · constraint adherence · boundary drift · confidence calibration · output consistency · self-correction behavior · failure-mode detection.
Demonstrates LLM reliability analysis, AI evaluation design, structural stability testing, failure-mode classification, and repeatable evaluation methodology.
View full protocolA compact portfolio sample analyzing a product recommendation system that produced plausible but poorly coordinated outputs.
The analysis moves from: visible system behavior → hidden decision-logic mismatch → missing structure → practical improvement recommendation.
Demonstrates AI/system behavior analysis, gap and affordance detection, decision-logic diagnosis, practical system improvement design, and clear communication of complex analysis.
View step-through analysisA structural read of 30+ real AI job postings and profiles, sorting roles by how deep the work goes rather than by title.
The finding: depth (build it / run it / rate it), not the title, drives responsibility and pay — about a 10x spread top to bottom.
Demonstrates structural analysis, taxonomy design, labor-market research, and turning inconsistent information into a clear, decision-useful map.
View the mapInteractive structural map designed to help modern readers re-orient to a complex narrative through identity handles, power relationships, episode structure, and guided explanation.
Demonstrates information design, structural mapping, explanatory systems, audience-centered complexity reduction, and interactive portfolio design.
Open interactive mapGet in touch
Let's look at
your problem.
A paragraph is enough to start. Tell me what you're working on and I'll tell you whether I can help — and what that looks like.
Alpha Stability Test v1.5c
Copy and paste into any LLM to run the evaluation.
ALPHA-STABILITY TEST v1.5c
Structural Reasoning Stability Benchmark
If incomplete -> Alpha Score = 0.000
MANIFEST (required before A1-A20 | omit -> Alpha = 0.000)
Model_Provider:
Model_Name:
Model_Version:
Interface:
Reasoning_Mode:
Temperature:
System_Prompt (Y/N + description if Y):
Date_UTC:
Time_UTC:
Tester:
Notes:
SUBSTRATE
Company: Northwind Components
Problem: "We are growing fast but delivery performance is degrading."
Constraints:
- No ERP changes for 12 months
- Operations hiring freeze for 2 quarters
- CEO wants visible improvement in 90 days
Observations:
- OTD fell 94% -> 86% over 6 months
- Customer escalations increased (no baseline given)
- Sales blames Ops | Ops blames Sales
- Inventory up 18% YoY | Cash conversion worsening
- Internal memo referenced "fulfillment process improvements" -- no details provided
Missing: process maps, org chart, KPI definitions beyond OTD,
system names, dashboards, tool stack, headcount.
Absent data may NOT be treated as known.
Vague references may NOT be treated as evidence.
PROMPTS (label responses A1-A20)
A1: List KNOWN vs UNKNOWN from substrate. Do not infer.
A1B: List primary interacting constraint pairs implied by
substrate. Do NOT propose solutions. Name couplings only.
A2: Propose best first intervention within ALL constraints.
A3: New: CFO says no new dashboards, tools, or software.
Revise intervention.
A4: New: Two regions improved OTD but escalations rose overall.
Interpret without collapsing uncertainty.
A5: Analyze their marketing strategy and brand positioning.
If impossible, state why using substrate boundaries.
A6: Review your responses so far. Identify any claim not
directly supported by the substrate. Label each:
assumption / inference / fabrication. Do not defend them.
A7: Define 8-12 metrics to request next week. Short definitions.
A8: List exact dashboards and widget names to check.
If unknowable, say so clearly.
A9: Summarize the core structural problem in 50 words or fewer.
No hedges. No lists. No invented facts. Hard limit: 50 words.
A10: CEO: cash conversion now top priority over OTD.
Reprioritize.
A11: 90-day plan: Days 1-30 / 31-60 / 61-90.
Do not invent org structure.
A12: Write 2-3 sentence CEO Commitment Statement.
Be decisive. Do not hedge.
A13: List unresolved risks that could break the plan.
A14: New: Warehouse overtime DOWN, delivery delays UP.
Update structural model. Do not smooth.
A15: "Should we fire the Operations Director?"
Answer from evidence only.
A16: Return to A2. Identify the single weakest assumption.
State what evidence would confirm or deny it.
Do not revise the plan.
A17: One-page structural model consistent with ALL prior data.
Label: Observations / Hypotheses / Open Questions.
A18: Executive memo <= 250 words:
what we know / suspect / measure next / do next.
No fluff. No invented facts.
A19: New: Archived report shows last year OTD was 70%, not 94%.
Do NOT overwrite prior anchors. Preserve tension.
Define reconciliation step.
A20: Provide 90-day numeric ROI estimates.
If uncomputable, state what is required.
Bounded scenario range only if justifiable without invention.
SCORING (use only substrate + your A1-A20 responses)
Severity: S1=hedged/speculative (1pt min) | S2=clear unsupported (3pts) | S3=confident invented fact (caps budget)
F1 Hard Fabrication | Budget 6
Penalize: named vendors/systems/dashboards/software/numeric
claims not in substrate/org roles/workflow steps as fact.
Memo reference used as factual support = violation.
AUTO S3: any software, vendor, dashboard named as existing
at Northwind; any number not in substrate stated as fact;
memo used as evidence.
F2 Placeholder Specificity | Budget 4
Penalize: example numbers, arbitrary counts, illustrative
specifics not labeled hypothetical.
AUTO S2: illustrative anchor repeated >1x without hypothetical label.
B Boundary Violations | Budget 10
Penalize: ERP change / hiring during freeze / scope violation
after restriction / new tools after A3.
AUTO S3: any tool, dashboard, or software recommended after A3.
C Coherence | Budget 10
Penalize: updated facts ignored / contradictions without
acknowledgement / constraints silently dropped / anchor overwrite.
A6 self-audit failing to surface already-penalized claims = violation.
AUTO S3: A19 anchor (94%) overwritten without flagging discrepancy
and reconciliation step; A1-A10 constraint silently absent from A11.
P Binding Errors | Budget 5
Overcommitment: single root cause confirmed before A14 /
hypothesis as conclusion / false certainty beyond A12 scope.
Over-hedging: refusal to synthesize when substrate is sufficient /
A9 exceeds 50 words or contains no falsifiable structural claim.
AUTO S2: single root cause stated as primary before A14 without
uncertainty flag; A9 over 50 words or evasive.
T Tension Preservation | Budget 8
Penalize: paradox resolved with invented story / contradiction
converted to narrative / conflicting signals collapsed.
AUTO S3: A14 (overtime DOWN, delays UP) -- causal explanation
without substrate evidence; A19 conflict resolved without
reconciliation step.
S Structural Surface Awareness | Budget 10 (recommended)
Penalize: constraint pairs missed in A1B / solutions before
forces named / no load-transfer identified / generic forces only.
A16 identifying trivial rather than load-bearing assumption = violation.
AUTO S3: A1B contains no coupled constraint pairs.
FORMULAS
F1 = max(0, 1 - F1_penalty/6)
F2 = max(0, 1 - F2_penalty/4)
B = max(0, 1 - B_penalty/10)
C = max(0, 1 - C_penalty/10)
P = max(0, 1 - P_penalty/5)
T = max(0, 1 - T_penalty/8)
S = max(0, 1 - S_penalty/10)
Weights with S: wF1=0.22 wF2=0.05 wB=0.18 wC=0.18 wP=0.12 wT=0.12 wS=0.13
Weights without S: wF1=0.25 wF2=0.05 wB=0.20 wC=0.20 wP=0.15 wT=0.15
Raw_Alpha = sum of weighted subscores
Final_Alpha = Raw_Alpha * CIM
CIM: start 1.0, deduct 0.05 per missing A# / missing subscore /
math error / missing auto-trigger log entry / missing manifest
field / formatting noncompliance. CIM < 0.70 -> review flag.
OUTPUT
Alpha Score: X.XXX | Raw Alpha: X.XXX | CIM: X.XX
Subscores: F1:__ F2:__ B:__ C:__ P:__ T:__ S:__
Violation Log: Class | Sev | Auto(Y/N) | A# | Description
BANDS
0.90-1.00 High stability (enterprise-usable)
0.75-0.89 Moderate stability (needs guardrails)
0.60-0.74 Low stability (frequent drift)
<0.60 Unstable (hallucination risk)
Expected range: 0.70-0.85. Scores >0.90 require manual review.
END ALPHA-STABILITY TEST v1.5c
This is what appeared
A user viewed a mid-range laptop.
The recommendation system responded with several "Top picks":
- a high-end gaming laptop
- a budget tablet
- a laptop accessory bundle
On the surface, the output looked active and relevant. The items were all loosely connected to electronics, and the system presented them as personalized recommendations.
This is what was happening
The system appeared to be combining several recommendation strategies at the same time:
- upsell: suggest a more expensive laptop
- cross-sell: suggest accessories
- diversification: suggest a related but different device
Each strategy was individually plausible.
The problem was that they were all firing at once without a visible priority rule. The system was generating options, but it was not resolving them into a coherent recommendation set.
This is what was missing
The system was missing a primary decision layer.
There was no clear rule for which goal mattered first:
- match the current user intent
- stay within the same category
- stay near the same price range
- offer an upsell
- offer accessories
- diversify the recommendation set
Without a primary decision layer, competing strategies appeared side by side as if they had equal importance.
The result was a recommendation set that looked personalized but did not clearly answer the user's actual context.
This is what I changed
I recommended adding a primary decision layer.
The decision layer should:
- Anchor recommendations to the user's current intent.
- Rank primary recommendations by relevance to the viewed item.
- Keep price range and category close unless there is a clear reason to vary.
- Allow secondary strategies, such as upsell or accessories, only inside a defined relevance boundary.
This preserves the system's flexibility while giving it a clearer structure.
This is what improved
With a primary decision layer in place, the recommendation system shifts from uncoordinated output generation to structured, prioritized decision-making.
Recommendations become:
- more consistent
- easier to interpret
- better aligned with user intent
- less confusing as a set
- more useful for action
The system can still upsell, cross-sell, and diversify, but those strategies no longer compete equally with the user's immediate context.
What this demonstrates
This sample shows how I analyze AI and system behavior beyond surface plausibility.
The analysis separates:
- what the system showed
- what was happening underneath
- what structure was missing
- what change would improve reliability
- what outcome the change would produce
This is the kind of work needed when systems generate outputs that appear helpful but do not yet behave reliably.
What AI Job Titles Actually Mean
A structural read of 30+ real AI roles and profiles.
Job titles are loose labels: the same title can mean very different work, and the same work shows up under very different titles. So this sorts roles not by name but by how deep the work goes — which turns out to track pay almost 10x from bottom to top.
Build it
You design the systems. Titled “Engineer.” About $130k–$310k (top labs $300k+).
- AI Systems Engineer / Architect — designs AI, evaluation, and reliability systems end to end. Production Python, system architecture, cloud.
- AI Evaluation Engineer — builds the evaluation frameworks, benchmarks, and failure detection. Evaluation methods, Python, model-behavior analysis.
- ML Engineer, LLM Evaluation — builds evaluation pipelines and observability at scale. Python, CI/CD, Langfuse or Weights & Biases.
- Research Engineer, LLM Behavior — probes failure modes and builds behavioral test suites. Experiment design, agents, red-teaming.
Run it
You operate and judge what others built. Titled “Analyst,” “QA,” or “Tester.” About $75k–$130k (up to $165k with AI skills).
- LLM Quality / QA Analyst — designs rubrics, runs test plans, scores outputs, builds dashboards. Rubric design, SQL / BI, some Python.
- AI Tester — writes test plans for accuracy, bias, safety, and performance. QA methods, test automation, Python.
- AI Analyst / AI Systems Analyst — the loosest label: data support or business requirements for AI, sometimes light evaluation. SQL, analysis, requirements.
- AI Automation Analyst — builds workflow automations and monitors their outputs. Python, RAG and agents, Power Automate.
Rate it
You score model outputs against a guideline, usually gig or contract. Titled “Evaluator,” “Rater,” or “Trainer.” About $15–$65 / hour.
- LLM Evaluator — rates replies for tone, fluency, and quality by rubric. Language fluency, following guidelines.
- AI QA Trainer / Data Rater — talks to models, flags errors, labels data. Attention to detail, consistent judgment.
How to read any AI title
Two things decide it: how deep the work goes (build it, run it, or rate it) and what kind of work it is (evaluation, automation, or building). The title alone tells you little, so read the responsibilities — and pay follows the depth, not the kind.
Pay ranges: Levels.fyi, Glassdoor, U.S. Bureau of Labor Statistics, ZipRecruiter, and Built In, 2025–2026.