01 / HUMAN-IN-THE-LOOP AI
Designing the human system around the model
DoorDash used an AI-assisted matching system to connect merchant items to standardized product records. High-confidence matches could move automatically; low-confidence cases required human review.
I owned the operational transition of that review process to an offshore BPO team—and discovered that the main challenge wasn’t effort or training. It was undocumented judgment embedded in the workflow.
MY ROLE / Operational transition · workflow redesign · documentation · QA + calibration · launch readiness
15+ points
Accuracy improvement
~80% → ~95%
Initial accuracy → graduation threshold
98%
Expected accuracy at live volume
Live production
Thousands to tens of thousands of potential matches/week
DoorDash built the matching technology. I focused on the operating system around the model—how people reviewed uncertain cases, applied criteria, learned from QA, and became ready to scale.
THE PROBLEM
The process assumed knowledge it never explained.
Low-confidence matches required human review before they could move forward. As that work transitioned to an offshore BPO team, the first calibration results landed around 80% accuracy—well below the quality bar required for production.
At first, that looked like an execution problem: a new team struggling to follow the process.
But repeated errors, QA disputes, and calibration conversations pointed to something deeper. Reviewers were being asked to make decisions that depended on product knowledge, category familiarity, and judgment that had never been made explicit in the SOP.
WHAT IT LOOKED LIKE
Vendor performance problem
"Why can't they follow the instructions?"
WHAT IT ACTUALLY WAS
System-design problem
"What knowledge are the instructions assuming but failing to explain?"
Before I could redesign the workflow, I needed to identify the judgment hiding inside it.
DISCOVERY
Making tacit knowledge visible.
I treated the accuracy problem as a diagnosis problem.
I reviewed the SOP, vendor outputs, QA results, disputes, and recurring error patterns, then used interviews and calibration conversations to understand how experienced reviewers were actually making decisions.
The goal was to separate what the process explicitly taught from what people had simply learned over time.
SUBJECTIVE JUDGMENT
Rules that sounded objective often still required interpretation in edge cases.
IMPLICIT PRODUCT KNOWLEDGE
Experienced reviewers relied on familiarity with categories, packaging, variants, and product relationships that the SOP never explained.
CULTURAL / CATEGORY FAMILIARITY
Some decisions depended on grocery conventions or culturally specific product knowledge that was unevenly distributed across teams.
QA INTERPRETATION
QA sometimes applied standards or distinctions that frontline reviewers could not infer from the written process alone.
The gap wasn’t between the team and the SOP.
The gap was between
the SOP and the real decision.
Once the hidden judgment became visible, it could be documented, structured, tested, and calibrated.
SYSTEM DESIGN
Turning subjective judgment into a scalable operating model.
Once the hidden knowledge was visible, I could redesign the review process around explicit rules instead of assumed expertise.
The goal wasn’t to eliminate human judgment. It was to make that judgment more consistent, teachable, testable, and easier to improve over time.
01 /
CLARIFY THE SOP
Rewrite ambiguous instructions so reviewers could understand what the rule actually required.
02 /
EXTERNALIZE TACIT KNOWLEDGE
Capture product knowledge, category conventions, examples, and edge cases that experienced reviewers had learned informally.
03 /
STRUCTURE JUDGMENT
Turn interpretive decisions into explicit criteria, decision trees, and more objective checks wherever possible.
04 /
ALIGN QA
Make sure reviewers and QA were evaluating work against the same documented standards rather than separate interpretations.
05 /
BUILD THE LEARNING LOOP
Use disputes, recurring errors, and calibration results to update guidance and improve the operating model over time.
BEFORE
Instructions → Human judgment → QA
Judgment lived mostly in people’s heads.
AFTER
Explicit criteria → Human review → QA + calibration → feedback
Judgment became part of the operating system.
The review process wasn’t outside the AI system.
It was part of the AI system.
The model determined where automation was appropriate. The operating model determined where human judgment was required—and how that judgment could scale.
HUMAN + SYSTEM
Automation handled confidence. People handled ambiguity.
The matching system could automate high-confidence cases. The harder question was what should happen when confidence dropped and human judgment became necessary.
I focused on making that handoff deliberate rather than treating human review as an exception outside the system.
SYSTEM HANDLED
HUMAN OWNED
Generate potential matches
Interpret ambiguous cases
Assign confidence
Apply explicit criteria
Route high-confidence cases automatically
Resolve edge cases
Send uncertain cases to review
Surface disputes
Preserve consistent routing logic
Calibrate against QA
HIGH CONFIDENCE → AUTOMATE
LOW CONFIDENCE → HUMAN REVIEW
The goal wasn’t to remove people from the loop.
It was to make the loop work.
Human review became a designed part of the system—with clearer inputs, shared rules, and a feedback mechanism that could improve over time.
TRADEOFF
We delayed launch to protect the system from bad matches.
The workflow was initially expected to move into production sooner. But with accuracy still below the required threshold, launching would have meant scaling ambiguity along with volume.
I supported delaying launch by several weeks so we could keep refining the guidance, calibrating reviewers, and aligning QA before moving the workflow live.
SHIP NOW
Faster launch
Scale the existing process while accepting unresolved ambiguity.
DELAY + FIX
Slower launch
Improve the operating model first, then scale a process that reviewers and QA could execute consistently.
Scaling a flawed process would have made the problem harder, not smaller.
The delay bought time to make the workflow teachable, measurable, and production-ready.
RESULTS
The outcome wasn’t just better accuracy. It was a review process that could scale.
Crossing the quality threshold allowed the workflow to move into production. More importantly, the review process no longer depended on undocumented expertise concentrated in a small group of experienced reviewers.
01 / TEACHABLE
A process a new team could actually learn
Product knowledge, judgment criteria, and edge cases became part of the documented operating model instead of something reviewers were expected to infer.
02 / REPEATABLE
A shared standard for reviewers and QA
Frontline reviewers and QA could calibrate against the same explicit rules, reducing the gap between how work was performed and how it was evaluated.
03 / SCALABLE
A human-review layer built for production volume
The workflow could support thousands to tens of thousands of potential matches per week without relying on the same small pool of implicit expertise.
04 / ADAPTIVE
A system that could learn from its own errors
Disputes, recurring mistakes, and calibration results became inputs for improving the guidance and review process over time.
The process scaled because the judgment became part of the system.
The model could scale its decisions because the human review layer could scale its judgment.
LESSONS
What this changed about how I think about AI-assisted operations.
01 /
Don’t confuse repeated failure with poor execution
When a capable team repeatedly struggles with the same decision, the problem may be in the system rather than the people. Before pushing harder on performance, I look for ambiguity, missing context, or judgment the process has failed to make explicit.
02 /
Expert intuition does not transfer automatically
Experienced reviewers often carry years of product knowledge and pattern recognition that never appears in the SOP. Scaling the work means turning that intuition into something another person can actually learn and apply.
03 /
Human judgment around AI needs design too
A model can determine when confidence is low, but that does not make the human-review layer self-explanatory. The handoff, criteria, QA standards, and feedback loop all need to be designed deliberately.
04 /
Aggregate metrics tell you where to look, not what is wrong
Accuracy can signal that a system is struggling, but the number alone does not explain why. The useful work starts when you connect the metric back to error patterns, edge cases, and the decisions underneath them.
The model wasn’t the only thing that needed to scale. The human judgment around it did too.
HUMAN-IN-THE-LOOP AI · OPERATING SYSTEM DESIGN · SCALE
NEXT PROJECT
Catalog Quality Triage
Turning manual coordination into a system that could run itself.