> ## Content Index
> Fetch the complete content index at: https://www.codyssey.tech/llms.txt
> Use this file to discover other available public pages before exploring further.

# 🤖 The Machine That Reads Badly — Part I: Teaching Silicon to See
- URL: https://www.codyssey.tech/the-machine-that-reads-badly-part-i-teaching-silicon-to-see/
- Published: 2026-09-09T08:00:00.000Z
- Updated: 2026-09-09T08:00:00.000Z
- Description: Here's a fun weekend project idea: build an app that scans ID documents. The answer, which you will discover approximately forty-five minutes in, is unreasonably hard. Not because the pieces are complicated — because the moment you connect them, they conspire against you.
- Author: Robert Marcel Saveanu
- Tags: Emerging Tech, AI & ML, #format-tutorial, #series-machine-reads, #syntax-highlight

📚 **Series Navigation:**  
👉 **You are here:** Part 1 - Teaching Silicon to See  
**Next:** [Part 2 - Fixing What the Machine Broke](https://www.codyssey.tech/the-machine-that-reads-badly-part-ii-fixing-what-the-machine-broke/) →

---

*Or: How I Built a Document Scanner and Then Had to Build a Second One to Fix the First*

---

## 🚨 The Problem Nobody Warned You About

Here's a fun weekend project idea: build an app that scans ID documents and extracts the holder's personal information. Name, date of birth, nationality, document number. How hard could it be?

The answer, which you will discover approximately forty-five minutes into the project, is: *unreasonably hard*.

Not because the individual pieces are complicated. Image processing has libraries. OCR has engines. Passports follow international standards. Each of these things works tolerably well in isolation. The problem is that the moment you connect them into something resembling a working system, they begin conspiring against you with the enthusiasm of a departmental reorganisation.

You point a camera at a passport. The camera captures the image slightly tilted. The passport has a holographic overlay that throws light across the text. The MRZ — that strip of capital letters and angle brackets at the bottom — gets partially obscured by the laminate catching the desk lamp at exactly the wrong angle. You feed this into your OCR engine, which confidently returns a string where the letter O has been replaced by the number 0, every `<` has become a K, and someone's surname now contains a 5.

I know this because I built one. A full ID and passport scanner application, from image upload to extracted personal data. The entire codebase is [open source on GitHub](https://github.com/Saveanu-Robert/ID-Scanner-App?ref=codyssey.tech) if you want to poke at it, break it, or judge my variable names. And what I actually built was two systems: one that reads documents, and a second one that fixes the first one's mistakes.

This is Part I. The reading part. In Part II, we'll get to the fixing — which is where the real engineering lives.

## 🛂 What Your Passport Is Actually Saying

Before we talk about scanning, we need to talk about what we're scanning. Because the MRZ — the Machine Readable Zone — is one of those things that's been sitting at the bottom of your passport your entire life and you've probably never looked at it with any curiosity whatsoever.

The MRZ is defined by ICAO Document 9303, which is the International Civil Aviation Organization's specification for machine-readable travel documents. It's been around since the 1980s. It exists because border control agents needed a way to process travellers faster than manually reading a name field printed in fourteen different national typography traditions.

There are three formats, and they're named with the creativity you'd expect from an international standards body:

**TD3** is the passport format. Two lines, 44 characters each. Line 1 carries the document type, issuing country, and the holder's name. Line 2 carries the document number, nationality, date of birth, gender, expiry date, and a series of check digits that exist to catch exactly the kind of OCR errors we're about to spend an entire article discussing.

**TD1** is the ID card format. Three lines, 30 characters each. Same information, different layout, more compact. Most European national identity cards use this one.

**TD2** is the middle child. Two lines, 36 characters each. Used by some national ID cards and older travel documents. Exists primarily to keep parser developers from getting too comfortable.

All three formats encode data in fixed positions. Character 0-1 of TD3 line 1 is always the document type. Characters 2-4 are always the issuing country as a three-letter code. Characters 13-18 of line 2 are always the date of birth in YYMMDD format. This positional rigidity is simultaneously the MRZ's greatest strength and, as we'll see, the source of its most spectacular failure modes. Because if the OCR engine misreads one character and shifts everything left by a position, suddenly your date of birth is one digit of your document number plus five digits of what used to be your nationality code.

The `<` character serves as a filler — it's the MRZ equivalent of whitespace. A double `<<` separates surname from given names. Single `<` characters pad fields to their fixed width. This matters enormously when your OCR engine decides that `<` looks a lot like the letter K, which it does with alarming regularity.

And then there are the check digits. Specific positions in each MRZ format contain a single digit calculated from the preceding field using a weighted modulo-10 algorithm. If you extract the document number and compute its check digit and it doesn't match position 9, you know something went wrong. This is ICAO's way of telling you: "We knew OCR would mess this up. Here's how to detect it."

## 😤 Why This Is Harder Than It Looks

If you've never tried to OCR an identity document, you might think this is a solved problem. Point camera at text. Read text. Done.

Here's what actually happens.

**Holograms and security features.** Modern passports and ID cards are covered in holographic overlays, UV-reactive inks, and microprinting. These exist specifically to make the document hard to reproduce, and as a side effect, they make it hard to photograph. That shimmering rainbow across the MRZ zone? That's national security working exactly as intended, and it's destroying your contrast ratios.

**Lighting.** The document is sitting on a desk lit by whatever combination of overhead fluorescent, desk lamp, and window light the user happens to have. The MRZ region, being at the bottom of the document, often catches shadows from the user's hand holding the phone. One half of the text is washed out, the other half is too dark.

**Angle and perspective.** Nobody holds their phone perfectly parallel to the document. There's always a slight angle, which means the characters at the far end of the MRZ line are slightly smaller and slightly more distorted than the ones at the near end.

**Image resolution.** Some users upload a high-resolution scan. Others take a photo with a phone from 2019 in a dimly lit kitchen. Your system needs to handle both, and the difference between them is roughly the difference between reading a newspaper and reading a newspaper that someone left in the rain and then ironed.

**The OCR-B font.** MRZ text is printed in OCR-B, a font specifically designed to be readable by machines. Designed in 1968\. When "machines" meant optical character readers the size of a filing cabinet. Modern Tesseract handles it reasonably well on clean, high-contrast images. On images that are slightly blurry, slightly tilted, and partially covered by a holographic eagle? Less well.

This is why the system can't be "upload image → run Tesseract → parse output." That pipeline works about 30% of the time, and 30% is the kind of accuracy that gets you fired from a fintech company and possibly investigated by a regulator.

## 🏗️ Architecture: Why a Pipeline, Not a Prayer

The architecture of the scanner is a multi-stage pipeline with fallbacks. Not because pipelines are fashionable, but because the problem fundamentally requires trying multiple approaches and picking the best result.

The system has four service layers:

**Image Preprocessor** — takes raw image bytes and produces several processed variants optimised for different OCR scenarios. This is the "prepare the battlefield" stage.

**OCR Service** — wraps Tesseract with MRZ-specific configuration. Tries three different page segmentation modes and picks the one that produces the most MRZ-looking output. This is the "ask the question three different ways and see which answer makes the most sense" stage.

**MRZ Service** — detects MRZ lines in OCR text, applies position-aware error correction, parses fields, and validates checksums. This is the heavy lifter, and it's the subject of Part II.

**Field Extractor** — regex-based fallback that runs when no MRZ is found at all. It scans the full OCR text for labelled fields like `NAME:`, `DOB:`, `PASSPORT NO:`. This is the "break glass in case of emergency" stage, and results from this path are permanently stamped with LOW confidence because they deserve it.

The route handler orchestrates these in a specific order: first try the cropped MRZ region (bottom 30% of the image) with all four preprocessing variants. If any variant produces a valid MRZ with passing checksums, return immediately — that's your HIGH confidence result. If none of them validate, try the full image with standard preprocessing. If that doesn't work either, pick the best non-validating MRZ result you found along the way (MEDIUM confidence). If there's no MRZ at all, fall back to regex extraction (LOW confidence).

This "try everything, rank the results, return the best" pattern is the core architectural decision. It's not elegant. It's not minimal. But it reflects the reality that you're dealing with input that varies wildly in quality and no single processing path handles all of it.

## 🖼️ Image Preprocessing: Four Ways to Disagree With Reality

The image preprocessor does two things: prepare the full image for OCR, and separately crop and prepare the MRZ region with four distinct binarisation variants.

**Why crop the MRZ separately?** Because the MRZ lives in the bottom 30% of the document, and running OCR on the full image means all the printed text above it — name in decorative font, address, photo caption, government department header — bleeds into your output and confuses the MRZ detection. Cropping first means the OCR only sees the bit you care about.

**Why multiple variants?** Because different images respond differently to different thresholding methods. A well-lit scan might work perfectly with simple Otsu thresholding. A photo taken under warm tungsten light might need CLAHE (Contrast Limited Adaptive Histogram Equalisation) to recover the text. A document photographed through glass might need aggressive enhancement.

The four variants are:

**CLAHE + adaptive threshold.** Handles uneven lighting well. This is the workhorse variant — it'll give you decent results on most inputs.

**Otsu threshold.** Works best on clean, high-contrast images. When the lighting is good and the document is flat, Otsu's automatic thresholding picks a clean boundary between text and background.

**Aggressive CLAHE + Otsu.** Higher clip limit, smaller tile grid. Pulls out faint MRZ text that the gentler CLAHE missed. Useful when the MRZ is washed out by a hologram or shadow.

**Enhanced grayscale without binarisation.** This one's counterintuitive. Instead of converting to black-and-white, it hands Tesseract an enhanced but unbinarised image and lets Tesseract apply its own internal thresholding. Sometimes the OCR engine's built-in processing does a better job than ours, and the honest engineering decision is to let it try.

Each variant also gets the standard treatment: decode from bytes, normalise size (upscale small images so Tesseract has enough pixels to work with, downscale enormous images to keep processing time sane), convert to grayscale, denoise.

The deskew step deserves a mention. It finds the largest contour in the binarised image, calculates its rotation angle, and corrects tilts up to 15 degrees. Beyond 15 degrees, the image is probably not a slightly tilted document — it's something else entirely, and "correcting" it would make things worse.

## 🔤 OCR: Asking Tesseract Nicely, Then Asking It Again

Tesseract is the OCR engine. It's open source, it's been around since the mid-2000s in its current form, and it's the standard choice for projects that don't want to pay per-API-call to a cloud OCR service.

The OCR service has two modes:

**General-purpose extraction** for the regex fallback path. Standard configuration, no character restrictions, page segmentation mode 6 (assume a single uniform block of text). This gives you everything Tesseract can read from the image, which for a full document might be the holder's printed name, the issuing authority, various field labels, and some decorative text that was part of the background pattern.

**MRZ-optimised extraction** for the primary pipeline path. This is where it gets interesting.

The MRZ-optimised mode restricts the engine to a whitelist of characters: `A-Z`, `0-9`, and `<`. Nothing else. This single configuration choice eliminates an entire category of errors — Tesseract can no longer "see" lowercase letters, punctuation, or special characters in the MRZ region, because we've told it those characters don't exist. If a hologram reflection looks like a comma, it can't return a comma. It'll return whatever whitelisted character is closest, which is usually closer to correct.

But the page segmentation mode — the way the engine interprets the physical layout of text — matters too. And different document images respond better to different modes. So the system tries three:

**PSM 6** — assume a single uniform block of text. The default and usually correct assumption for a cropped MRZ region.

**PSM 4** — assume a single column of text of variable sizes. Sometimes works better when the MRZ lines have irregular spacing.

**PSM 12** — sparse text with OSD. The "I'm not sure what this layout is" option that occasionally outperforms the others on unusual documents.

For each configuration, the system scores the output by how "MRZ-like" it looks: lines composed entirely of valid MRZ characters that are close to the expected lengths (30, 36, or 44 characters) score highest. The best-scoring output wins.

This pattern — try several configurations, score the results, keep the best one — is the same pattern used at the pipeline level and for the same reason. When you can't predict which approach will work best for a given input, you try a few and let the results speak for themselves.

## 🎛️ The Orchestrator: Making the Pipeline Behave

The scan endpoint ties everything together. It receives an uploaded image, validates it (file type, content type, size, not empty — the kind of checks that feel pointless until the day someone uploads a 47MB BMP of their cat and your server quietly dies), and then runs the pipeline.

The orchestration logic has a scoring function that ranks non-validating MRZ results. When several preprocessing variants detect an MRZ but none pass checksum validation, you need to pick the least noisy one. The scoring is blunt: 10 points for each populated field, 5 bonus points for properly formatted dates (they contain a `/`, which means the date parser didn't choke), 5 points if gender resolved to `Male` or `Female` rather than `Unknown`, and a penalty for names longer than 40 characters (because if someone's full name is 47 characters, it's almost certainly OCR noise that leaked into the name field, not a very committed Hungarian aristocrat).

The three-step cascade — cropped MRZ region, full image MRZ, regex fallback — means the system always returns *something*. The confidence level tells you how much to trust it. HIGH means the checksums validated and the data is almost certainly correct. MEDIUM means we found an MRZ but the checksums didn't pass, so there are probably OCR errors in the extracted fields. LOW means we couldn't find an MRZ at all and resorted to pattern matching raw text, which is roughly as reliable as asking a stranger on a bus to read your passport from across the aisle.

## 🏁 Conclusion

We've built the "see" half of the system. An image comes in, gets preprocessed four different ways, gets OCR'd with three different configurations, and the best results bubble up through a ranking system.

But here's the uncomfortable truth: even with all this preprocessing and multi-path extraction, the raw OCR output from the MRZ region is wrong roughly half the time. Not catastrophically wrong — usually it's one or two characters. An O that became a 0\. A `<` that became a K. A leading zero that vanished entirely, shifting every subsequent field one position to the left.

And that's where Part II begins. Because the real engineering in this project isn't in the reading. It's in the fixing.

---

> *"The real problem is not whether machines think but whether men do."* — B.F. Skinner