Or: A Lookup Table, a Specification Document, and the Audacity to Disagree With Tesseract
🗑️ The Mess on Your Desk
In Part I, we built a system that takes a photograph of an identity document, preprocesses it four different ways, runs three OCR configurations, and picks the best result. We ended on an uncomfortable note: even after all that work, the raw OCR output is frequently wrong.
Not "complete garbage" wrong. Worse. "Almost right" wrong.
Here's what Tesseract might return from a perfectly valid Romanian passport MRZ:
P<ROUP0PESCU<<B0GDAN<<<<KGK<K<<GKG<<KK<K<KGK
12345678<9ROU90O1157M3012315<<<<<<<<<<<<<<O4Spot the errors? There are at least six. The 0 in P0PESCU should be an O. The 0 in B0GDAN should also be an O. That garbage at the end of line 1 — KGK<K<<GKG<<KK<K<KGK — should be nothing but < filler characters, but Tesseract read most of them as K and G. On line 2, the O in 90O115 (the date of birth field, positions 13-18) should be a 0 — it's a date, 900115, meaning 15 January 1990. And O4 at the very end should be 04. The same character confusion everywhere, but flipped depending on whether you're in a letter field or a digit field.
This is the daily reality of OCR on identity documents. The text is almost right, and "almost right" is the most dangerous kind of wrong in data extraction. An obviously broken result gets caught. A subtly broken result gets trusted. Gets written to a database. Gets used to approve a loan.
Part II is about building the second machine — the one that takes the OCR engine's best effort and systematically fixes its mistakes. Not by guessing. By knowing what each character position in the MRZ is supposed to contain and using that knowledge to correct errors with surgical precision.
🧩 Position-Aware Error Correction: Knowing What Should Be Where
The MRZ's greatest strength as a data format is its rigid positional structure. Every character has a defined position, and every position has a defined type. Position 13-18 in a TD3 passport's line 2 is always a date of birth in YYMMDD format — six characters, always digits. Position 10-12 is always a nationality code — three characters, always letters.
The OCR engine doesn't know any of this. It sees pixels and returns its best guess. But we know. And that knowledge is the foundation of the entire correction system.
The correction works through character translation maps — two of them, pointing in opposite directions:
Letter-to-digit: O→0, I→1, Z→2, S→5, G→6, B→8. These are the characters most commonly confused when a digit position contains a letter. The O/0 swap is the heavyweight champion of OCR errors. It happens on nearly every document where the date fields contain a zero, which is most of them.
Digit-to-letter: The same mapping in reverse. 0→O, 1→I, 2→Z, 5→S, 6→G, 8→B. For when a digit lands in a position that should only contain letters.
The correction functions walk through each MRZ line character by character, applying the appropriate translation based on what that position is supposed to contain. Date fields get letter-to-digit correction. Nationality codes get digit-to-letter. Check digit positions get forced to digits. Document type and issuing country get forced to letters.
The sex field gets its own special handler, because the OCR engine has a remarkable talent for misreading M and F. An M becomes W, H, or N (the vertical strokes confuse the recognition). An F becomes E or P (the horizontal crossbar shifts). The corrector maps these common misreads back to their intended values.
This isn't artificial intelligence. It's not machine learning. It's a lookup table and a specification document. And it fixes the majority of OCR errors because those errors are single-character confusions between visually similar characters — applied to positions where we know exactly what type of character belongs.
The beauty of this approach is that it's deterministic. No guessing. No probability distributions. It says: "Position 14 of TD3 line 2 is a digit. You gave me the letter S. The most common confusion that produces an S from a digit is 5→S. Therefore this character is 5." And it's right far more often than it's wrong, because the confusion patterns are consistent and well-documented.
🕵️ The Great Filler Conspiracy: When K Isn't K
Of all the OCR errors the system corrects, the filler character problem is the most entertaining and the most technically interesting.
In MRZ encoding, < serves as the filler character. It pads fields to their fixed width and separates name components. A typical passport MRZ line 1 might end with fifteen consecutive < characters filling the unused portion of the name field.
The OCR engine, faced with a row of < characters printed in OCR-B font on a holographic background, does its best. And its best frequently involves reading < as K. Or G. The angular shape of < in OCR-B bears a passing resemblance to these letters, especially when the image is slightly blurred or the contrast is uneven.
The result is that a name line that should read POPESCU<<BOGDAN<<<<<<<<<<<<<<<<<<<<<< comes back as POPESCU<<BOGDAN<<<KGKK<K<<GKG<KK<K<KG. Your name field now ends with what looks like someone fell asleep on the keyboard.
The filler correction is a three-phase operation, and it's worth understanding why it needs three phases rather than one.
Phase 1: Right-to-left cluster analysis. The algorithm walks through the name field from right to left, identifying clusters of consecutive non-< characters. For each cluster, it asks: is this a real name, or is it filler noise?
The distinction is surprisingly reliable. A real name cluster — like BOGDAN or MICHELLE — contains at least two distinct characters that are not K or G. A noise cluster — like KGKK or GKG — is made entirely or almost entirely of the characters the OCR engine substitutes for <. The algorithm replaces noise clusters with < characters until it hits a cluster that looks like a real name, then stops.
This right-to-left direction matters. Name data sits at the left side of the field; filler padding sits at the right. By starting from the right and working inward, we clean up the noise without touching the actual names.
Phase 2: Boundary cleanup. After Phase 1, there are often stray K or G characters at the boundary between name text and filler. The pattern [real letter][K or G][<<<] — a K or G sandwiched between a genuine letter and a run of fillers — is almost certainly a misread <. A regex pass cleans these up.
Phase 3: Specific pattern fixes. The <K< pattern — a K sandwiched between two fillers — never occurs in valid MRZ names. It's always noise. A simple replacement loop handles it. Similarly, a K or G immediately after the << name separator and immediately before a real name letter (like <<KERIKA where the K before ERIKA is noise) gets cleaned up.
Three phases because each phase handles a different failure mode of the same underlying problem. The cluster analysis handles bulk noise at the end of the field. The boundary cleanup handles individual characters at the transition point. The pattern fixes handle specific artifacts that survive the first two passes.
It's ugly. It's multiple passes over the same data with different heuristics. It works.
⬅️ The Left-Shift Problem: When OCR Eats Your Leading Zero
This one took me longer to figure out than I'd like to admit.
In a TD3 passport MRZ line 2, position 9 is a check digit — always a single digit. So when the system runs position-aware correction and finds a letter at position 9, something has gone more seriously wrong than a simple character confusion.
What's happened is that the OCR engine dropped a character from the beginning of the line. Usually a leading 0, because leading zeros at the start of document numbers are thin, often partially occluded by the edge of the crop region, and look enough like nothing that the engine simply doesn't report them.
When that leading character vanishes, every subsequent character shifts one position to the left. The check digit slot (position 9) now contains what was at position 10 — the start of the nationality code, a letter. The date of birth slot (position 13) now holds what was at position 14. The entire field alignment is destroyed, like pulling one card from the bottom of a house of cards.
The fix is elegant in that "obvious in hindsight" way: if position 9 is a letter, prepend a 0 to the beginning of the line, truncate to 44 characters, and re-run the position-aware correction on the now-realigned line.
This works because the most common dropped character is a leading 0, and even when it wasn't a 0, prepending one and re-parsing usually produces a result where the check digits validate correctly (because the checksum algorithm will catch it if the prepended character was wrong, and we can fall back to the original parse).
The TD2 format has the same vulnerability at the same position, and the same fix applies.
It's the kind of error that's invisible if you're only looking at individual character quality. Every character might be correctly recognised. The problem isn't recognition — it's alignment. And the only way to detect it is to know what type of character each position expects and notice when the types don't match.
🔀 Cross-Line Reconstruction: When Line 1 Disagrees With Line 2
TD3 passports encode the issuing country on line 1 (positions 2-4) and the nationality on line 2 (positions 10-12). For the vast majority of documents, these are the same. Bogdan Popescu's Romanian passport has ROU in both positions. A German passport has DEU in both. It's redundant on purpose — like writing your phone number on both sides of a napkin because you know the waiter is going to spill something.
When the OCR returns different values in these two positions, one of them is wrong. And since nationality on line 2 tends to be more reliably read (it's surrounded by digit fields, which are easier to OCR, and the correction pass has already cleaned it up), the system uses line 2's nationality to cross-validate and potentially reconstruct line 1.
The reconstruction logic tries inserting the nationality code at position 2 of line 1 and shifting the remaining characters to accommodate it. If this reconstructed line passes checksum validation, it's accepted. This handles the case where the country code on line 1 was mangled — perhaps ROU became R0U (with a zero), which then got "corrected" to ROU by the letter-only fixer but with the surrounding characters shifted.
If reconstruction doesn't validate either, there's a secondary heuristic. If the misread country code was 3 or more characters (like DEL instead of ROU), it probably absorbed part of the surname during OCR, so those characters get prepended back to the name field. If it's 1-2 characters (like O instead of RO), it's just a character-level misread and the name is left alone. Corrupting someone's name to fix a country code that was going to be overridden by the nationality anyway? That's a trade nobody should make.
This cross-field validation only works because the MRZ encodes the same country in two places. Dates have check digits. The overall MRZ has a composite check computed across several fields. Every scrap of redundancy is an opportunity to catch a mistake — and the system uses all of them.
🔨 The Regex Fallback: Break Glass in Case of Emergency
Sometimes there's no MRZ. The document is too damaged, too poorly photographed, or it's a document type that simply doesn't have one. Some national ID cards, especially older ones, predate the entire concept of a machine-readable zone. They were designed for a world where "scanning" meant a bored civil servant squinting at laminated cardboard.
The regex fallback runs general-purpose OCR on the full image and then rummages through the output for labelled fields. It's looking for patterns like NAME: JOHN SMITH, DOB: 15/01/1990, PASSPORT NO: 12345678, SEX: M, NATIONALITY: ROU.
It's a collection of regex patterns, each targeted at a specific field type, each trying to handle the dozen different ways that field might be labelled across different countries' documents. DOB, DATE OF BIRTH, BIRTH DATE, BORN — all valid, all actually printed on real documents somewhere in the world.
The results from this path are always marked LOW confidence, and they deserve it. There's no checksum validation. There's no positional guarantee. There's no redundancy to cross-reference. You're pattern-matching raw OCR text and hoping that NAME: is followed by an actual name and not the word NATIONAL from the next line that the OCR engine concatenated because the column alignment confused its page segmentation.
But LOW confidence beats no results. And for documents without an MRZ, this is the entire toolkit. The system doesn't pretend otherwise — the confidence flag exists so downstream consumers know the difference between "we parsed a validated MRZ and every checksum passed" and "we found something that vaguely resembled a name near a label that said NAME, and frankly we're not thrilled about it either."
🎯 Confidence: Knowing When to Trust Yourself
The confidence system is the final piece, and in some ways it's the most important because it's the system being honest about its own limitations.
HIGH confidence: MRZ detected, all ICAO 9303 checksums validated. Every check digit matched. The positional structure is intact. The data is almost certainly correct. This is the result you get when the image quality is decent and the preprocessing did its job.
MEDIUM confidence: MRZ detected, but checksums failed. We found something that looks like an MRZ, we parsed fields from the expected positions, we applied all the corrections — but the check digits still don't add up. There are OCR errors the system couldn't fix. The extracted data is probably mostly right, but at least one field has a wrong character somewhere.
LOW confidence: No MRZ found at all. Fields extracted via regex pattern matching on raw OCR text. Trust at your own risk.
This three-tier system is a design decision about what to communicate. A boolean would have been simpler — success or failure. But a boolean loses the distinction between "verified data" and "our best guess." And that distinction matters. In a KYC workflow, it determines whether you auto-approve a customer onboarding or flag it for manual review. In a data entry application, it determines whether the extracted data gets written directly to the database or presented to a human for confirmation.
The confidence level turns the scanner from a black box into a system that admits what it doesn't know. It doesn't just hand you data. It tells you how much faith to put in that data. And that's the gap between a weekend demo and something you'd actually connect to a production workflow.
🏁 Conclusion
Over these two articles, we've walked through a system that solves a deceptively hard problem: extracting structured data from photographs of identity documents. Part I was about seeing — getting usable text out of a photograph that physics is actively working against. Part II was about thinking — taking that imperfect text and systematically fixing it using the structure of the data itself.
The deepest lesson here isn't about OCR or image processing or MRZ standards. It's about dealing with unreliable input.
The entire correction system exists because our primary data source gets things wrong. Not randomly — predictably. The O/0 confusion happens every time. The K-for-filler substitution happens every time. The dropped leading zero happens often enough to warrant its own detection logic. Once you map those failure patterns, you can build targeted fixes around them.
That principle travels well beyond document scanning. Any system consuming data it doesn't fully control — API integrations, user input, sensor feeds, third-party data — has the same fundamental challenge. The data arrives imperfect, and "imperfect" is not the same as "unusable." It's a problem with structure, and problems with structure have solutions with structure.
If your data has structure, your errors can be fixed. If your errors can be characterised, your corrections can be systematic. If your corrections are systematic, your confidence can be quantified.
And if your confidence can be quantified, you've built something that knows what it doesn't know. Which is more than you can say for most software. Or, for that matter, most people.
The full source code — React frontend, FastAPI backend, every correction heuristic described in this series — is available on GitHub. Clone it, scan your own documents, and see how many of Bogdan Popescu's problems you can reproduce in under five minutes. I'm betting all of them.
"All models are wrong, but some are useful." — George E.P. Box
