05 / DATA QUALITY + DECISION DESIGN
Automating the easy decisions.
Saving human judgment for the risky ones.
DoorDash’s catalog ingested product data from multiple third-party syndicators. Because those sources sometimes represented the same physical product differently, duplicate records could enter the catalog with different identifiers, names, imagery, attributes, and metadata.
I helped design a remediation strategy that used SQL and heuristics to narrow the catalog into likely duplicate pairs, human review to confirm which duplicates were real, source-priority logic to choose the strongest canonical record, and automated triage to determine which merges were safe versus which required deeper review to preserve valuable data.
MY ROLE / Duplicate identification strategy · SQL candidate generation · source-priority logic · canonical record selection · merge triage · vendor workflow · burndown tracking
>8% → <2%
Broader duplicate rate → target reached
Within weeks
Focused burndown reached the target
~2,000 → ~220
Potential pairs → confirmed duplicates in one representative test
~69
Confirmed duplicates qualified for the simpler merge path in that test
The goal wasn’t simply to remove duplicates. It was to create one canonical product record without losing the best information distributed across the duplicates.
THE PROBLEM
The duplicates weren’t identical. That made “just delete one” the wrong solution.
DoorDash’s catalog incorporated product data from multiple third-party syndicators, and those sources were not equally strong in every category or attribute.
Two records could represent the same physical product while carrying different identifiers, names, imagery, attributes, or metadata. Removing one record might reduce duplication while also discarding information the surviving record did not contain.
SAME PHYSICAL PRODUCT
RECORD A
· Preferred source
· Better title
· Stronger core metadata
DUPLICATE
RECORD B
· Additional product photos
· Missing attribute
· Supplemental metadata
DELETE ONE?
RISK: LOSING UNIQUE DATA
CREATE ONE CANONICAL RECORD
WITHOUT LOSING THE BEST INFORMATION
01 / DETECTION
Which records might actually represent the same physical product?
02 / CANONICALIZATION
If they are duplicates, which record should remain?
03 / DATA PRESERVATION
If the secondary record contains unique value, how do we keep it?
WHY IT MATTERED
DUPLICATE REPRESENTATIONS · INCONSISTENT PRODUCT DATA · FRAGMENTED METADATA · WEAKER IMAGERY · NO SINGLE AUTHORITATIVE RECORD
This wasn’t a deletion problem.
It was a decision-design problem.
The system had to decide where a duplicate might exist, which record was authoritative, what information could be preserved automatically, and where human review was still necessary.
DETECT CANDIDATES
Start by shrinking the catalog into a set worth investigating.
The earliest detection method focused on exact UPC repetition. I used SQL queries to find cases where the same UPC appeared across multiple catalog records, creating a strong initial set of duplicate candidates.
As the work evolved, I partnered with Engineering to expand detection beyond exact identifiers using additional heuristics such as item-name similarity. That allowed the system to surface likely duplicates even when two records did not share the same UPC.
ENTIRE CATALOG
millions of records
EXACT UPC MATCHES
strong duplicate candidates
HEURISTIC MATCHES
name similarity + other signals
POTENTIAL DUPLICATE PAIRS
ready for human validation
DETECTION NARROWS THE SEARCH SPACE
01 / EXACT SIGNAL
Same UPC, multiple records
Exact identifier repetition provided a high-confidence starting population.
02 / FUZZIER SIGNAL
Different identifiers, similar product
Heuristics such as item-name similarity helped surface cases that exact UPC matching would miss.
DESIGN BOUNDARY
A candidate is not a deletion decision.
The detection layer was optimized to find records worth reviewing, not to make irreversible catalog decisions automatically. Incorrectly merging two legitimate products would create a new quality problem.
Good detection finds where judgment is needed. It doesn’t pretend judgment is unnecessary.
Once the candidate set was narrow enough, the next step was determining which pairs were actually duplicates.
VALIDATE WHAT'S REAL
Finding a likely duplicate was not the same as proving one existed.
Potential duplicate pairs were sent to BPO reviewers, who evaluated them using defined criteria and determined whether the two records actually represented the same product.
Their decisions then went through internal QA before a pair was treated as a confirmed duplicate. This kept the irreversible catalog decision behind a human judgment layer even though automation had already narrowed the search space substantially.
VENDOR REVIEW
INTERNAL QA
HUMAN JUDGMENT
SQL / HEURISTIC DETECTION
POTENTIAL DUPLICATE PAIR
CONFIRMED DUPLICATE
AUTOMATION NARROWS · PEOPLE VALIDATE
~2,000 → ~220
potential duplicate pairs → confirmed duplicates
In one representative test, roughly 2,000 candidate pairs were surfaced for review, but only about 220 were ultimately confirmed as true duplicates.
BEFORE / CANDIDATE
These records look similar enough to investigate.
Detection optimized for recall: find plausible pairs without assuming the match is correct.
AFTER / CONFIRMED
A reviewer and QA agree these records represent the same product.
Only then did the workflow proceed to canonical-record selection and remediation.
WHY HUMAN REVIEW STAYED
The cost of a false positive was different from the cost of a missed candidate.
A missed candidate could remain in the backlog for later detection. An incorrect merge could combine two legitimate products and create a new catalog-quality problem. So the system used automation to find where to look, while people retained authority over the actual duplicate decision.
Automation reduced the population. Human judgment established the truth.
Once a pair was confirmed, the next question changed from “Are these duplicates?” to “Which record should survive?”
CHOOSE THE CANONICAL RECORD
Once a duplicate was confirmed, the next decision was which record should survive.
I developed prioritization logic that used known source quality as one signal for choosing the stronger canonical record.
Certain syndicators were more reliable for particular product categories, so source priority helped determine which record should remain active. But that decision was only the starting point—the lower-priority record could still contain useful imagery, metadata, or attributes the preferred record did not have.
RECORD A
PREFERRED SOURCE
· stronger source priority
· stronger core metadata
· preferred for category
RECORD B
LOWER-PRIORITY SOURCE
· additional imagery
· missing attribute
· supplemental metadata
WHICH RECORD BECOMES AUTHORITATIVE?
CANONICAL RECORD = A
SOURCE PRIORITY HELPS CHOOSE THE SURVIVING RECORD
THE RULE
Choose the strongest record first. Then ask what would be lost.
Canonical-record selection established which item should remain active, but it did not mean the secondary record could simply disappear. Before removing it, the system still had to determine whether it contained useful information absent from the preferred record.
01 / SOURCE QUALITY
Which syndicator is stronger for this category?
02 / RECORD COMPLETENESS
Which record has the better core product information?
03 / UNIQUE VALUE
Does the secondary record contain anything the preferred one lacks?
CONFIRMED DUPLICATE
CHOOSE CANONICAL RECORD
CHECK FOR INFORMATION LOSS
Choosing the survivor solved only half the problem. The system still had to protect what the other record knew.
That became the next decision: could the duplicate be merged safely, or did a person need to preserve something first?
TRIAGE THE MERGE
Not every confirmed duplicate deserved the same amount of human review.
Once the canonical record was selected, I built another query to compare the information contained in each confirmed duplicate pair.
If the preferred record already contained the important information from the secondary record, the pair could follow a simple merge path. If removing the secondary record would discard useful imagery, metadata, or attributes, the case was routed to human review instead.
EASY MERGE
retain preferred record
deactivate duplicate
NO
CONFIRMED DUPLICATE
CANONICAL RECORD CHOSEN
DOES THE SECONDARY RECORD CONTAIN UNIQUE VALUE?
AUTOMATE THE SAFE PATH · ESCALATE THE RISKY ONE
HUMAN REVIEW
preserve useful data first
then remove duplicate
YES
WHAT TRIGGERED HUMAN REVIEW
01 / IMAGERY
Additional product photos
02 / METADATA
Supplemental product information
03 / ATTRIBUTES
Fields missing from the preferred record
04 / OTHER CATALOG VALUE
Useful information that needed to be transferred
~220 confirmed duplicates → ~69 easy merges
In the representative test population, about 69 confirmed duplicates qualified for the simpler merge path. The remainder required additional review before remediation.
BEFORE / TREAT EVERY DUPLICATE THE SAME
Manually inspect every field before remediation.
Human effort scales with the entire confirmed-duplicate population, even when many cases are straightforward.
AFTER / REVIEW ONLY WHERE VALUE IS AT RISK
Let the system identify which cases actually need deeper judgment.
Safe cases move through a simpler path; people focus on the subset where information could be lost.
DESIGN PRINCIPLE
Use automation to reduce the amount of judgment required—not the quality of the judgment.
The system did not try to eliminate human review. It used data and rules to make human review scarce, targeted, and valuable.
Good automation made human review more valuable by reserving it for the cases where information could actually be lost.
Once the workflow knew which path each duplicate should take, the final challenge was making sure the cleanup stayed fixed after the next data ingestion.
MAKE THE FIX DURABLE
A cleanup that comes back next week isn’t really a cleanup.
The remediation needed to persist beyond the initial burndown.
Once duplicate records were identified, reconciled, and associated with the appropriate master item, the secondary record could remain dormant instead of being recreated as a new independent item during a later syndicator ingestion.
TEMPORARY CLEANUP
DUPLICATE MERGED
SECONDARY RECORD REMOVED
NEXT INGESTION
DUPLICATE REAPPEARS
DURABLE RESOLUTION
DUPLICATE CONFIRMED
ASSOCIATE WITH MASTER ITEM
SECONDARY STAYS DORMANT
FUTURE INGESTION PRESERVES RELATIONSHIP
RESOLVE THE RELATIONSHIP · NOT JUST THE SYMPTOM
DESIGN PRINCIPLE
Fix the relationship that created the duplicate, not just the duplicate itself.
The goal was to prevent the same underlying record relationship from generating another independent catalog item during the next import cycle.
The burndown only became durable when the system remembered the relationship after the cleanup was over.
That turned a one-time cleanup into a fix that could survive future ingestion cycles.
RESULTS
The duplicate rate fell without turning the cleanup into a permanent manual operation.
The effort was designed as a focused burndown rather than an ongoing high-labor workflow.
The broader catalog duplicate rate started above approximately 8%, with a target of bringing it below 2%. The team reached that target within a matter of weeks, allowing the intensive cleanup process to scale back quickly.
DUPLICATE RATE
>8% → <2%
Broader catalog duplicate rate
TIME TO TARGET
Within weeks
Focused burndown reached the goal
REVIEW MODEL
Broad → targeted
Manual effort concentrated on ambiguous cases
ENTIRE CATALOG
POTENTIAL DUPLICATES
CONFIRMED DUPLICATES
EASY MERGES
AMBIGUOUS CASES
LESS VOLUME · MORE JUDGMENT PER CASE
01 / DUPLICATION REDUCED
The burndown reached the target quickly
The broader duplicate rate moved from above ~8% to below ~2% within weeks.
02 / HUMAN EFFORT CONCENTRATED
Manual review moved toward the ambiguous cases
Progressive filtering meant deeper human judgment was reserved for the subset where it added the most value.
03 / DATA QUALITY PROTECTED
Reducing items did not mean throwing away useful information
The workflow preserved valuable metadata, imagery, and attributes from secondary records instead of treating every duplicate as disposable.
04 / FIX MADE DURABLE
The same duplicate relationship did not need to be solved repeatedly
Resolved records could remain associated with the appropriate master item across later ingestion cycles.
WHAT SUCCESS ACTUALLY MEANT
The goal wasn’t fewer items. It was fewer duplicate items without losing the best data.
The workflow reduced duplication while protecting the strongest metadata, imagery, and attributes available across the source records.
The scalable part wasn’t doing more review. It was designing the system so fewer cases needed review at all.
The workflow used data to narrow the problem, people to validate uncertainty, and business rules to automate the safe decisions.
LESSONS
What this changed about how I think about data-quality systems.
01 / DATA CLEANUP IS DECISION DESIGN
Finding the duplicate is only the beginning.
The harder work is deciding which record is authoritative, what each source uniquely contributes, what can be automated safely, and where a person still needs to intervene.
02 / DECOMPOSE AMBIGUITY
Large problems become manageable when they become smaller decisions.
Instead of asking someone to inspect the entire catalog, the workflow progressively narrowed the problem—from possible duplicates, to confirmed duplicates, to safe merges, to only the ambiguous cases requiring deeper review.
03 / AUTOMATE THE SAFE DECISIONS
Automation is strongest when the boundary is clear.
Data identified where a problem might exist. Humans validated uncertainty. Business rules handled the obvious cases. Deeper review stayed reserved for the situations where valuable information could actually be lost.
04 / PROTECT QUALITY WHILE MOVING FASTER
Efficiency is not a win if the cleanup damages the data.
The fastest solution would have been to keep one record and delete the rest. The better solution was to understand which record should survive, preserve useful information from the others, and make the remediation durable.
Good automation doesn’t eliminate human review everywhere. It makes human review scarce, targeted, and valuable.
DATA QUALITY · DECISION DESIGN · HUMAN-IN-THE-LOOP · SCALABLE REMEDIATION
NEXT PROJECT
CPG Ads SLA System
Using operational data to prove a new business model could scale.