From OCR to LLM and Back: Reading Receipts

You send a screenshot of a bank transfer to a LINE chat. A moment later it is a transaction in your budget: amount, date and, when it can tell, a category. Nobody typed anything.

That's the whole feature. Behind it are three readers working together: the QR code for a reference I can trust, Tesseract OCR for a fast first answer, and a self-hosted vision model that fills in the rest, on CPU in my own Kubernetes cluster, with no GPU and no third-party AI API. It took me several designs, a load test, a benchmark over 20 models and a lot of synthetic test slips to get there. Each step solved one problem and showed me the next.

The idea

JodRubJai is a personal finance app: web, Android and chat bots in LINE and Telegram, in English, Thai and Russian. I'm the only engineer in a two-person team. You can try it at jodrubjai.com.

In Thailand most everyday payments are PromptPay transfers, and every transfer ends with a slip: a screenshot from the banking app. People already have those screenshots. So the plan was simple. Upload the slip, get a transaction.

The chat mattered as much as the app. In Thailand, LINE is where bank notifications already arrive: banks have their own LINE accounts and send a message for every transaction. That was the example to follow, not something to parse: if banks talk to people in LINE, a finance app should too. People forward slips to each other in LINE as well. So the LINE bot shipped together with the web app and the first parser: send it a slip, get back the saved transaction. Telegram came later, for people who don't use LINE, many of them Russian speakers. By then the chat code was shared, so it was mostly a new adapter. A Messenger bot is built too, but not live yet: Meta's rules for bots keep getting stricter.

The requirements:

• Fast. A transaction on screen right after the upload. In LINE this is a hard rule, not a nicety. A reply is free but uses a single-use token that expires quickly. A push, a message the bot starts on its own, counts against a monthly quota: 300 on the free plan. So every slip gets one quick reply, and nothing is pushed later.
• Everywhere. Web and the LINE bot from the start, then Android, a watched folder, the phone's share menu and Telegram.
• Shared with a partner. Couples link accounts, so one person's slip can count in a shared budget.
• No manual work. No retyping, no "please check these 8 fields".

Step 1: what's inside the QR?

Every Thai bank slip has a small QR code. My first thought: decode it, done. No OCR, no AI.

The QR is an EMVCo TLV string: two-digit tag, two-digit length, value, and a CRC at the end. I didn't start from the spec alone. The open-source JavaScript library promptparse already reads and builds Thai QR payloads, so I used it to generate the first test vectors. My Python decoder had a known-good answer to match from day one. Here is a synthetic payload:

00 38                         template
   00 06 000001               slip-verify marker
   01 03 002                  sending bank code
   02 17 DEMOREF0000000001    transaction reference
51 02 TH                      country
91 04 511F                    CRC-16/CCITT-FALSE

Just a bank code and a transaction reference. No amount, no date, no names. The details live on the bank's side, behind a slip-verification API. I wanted the app to work from the screenshot alone, without an outside request per slip, so I left it out of scope.

A PromptPay payment QR does carry an amount, but it's a request to pay, not proof of a transfer. The parser rejects it.

So the QR is great for one thing: a unique, checksummed ID. Everything else I had to read from the picture.

Step 2: OCR for the data, QR for duplicates

The first image parser combined both. The QR (via pyzbar) gave the reference; Tesseract (Thai + English) read the amount and date. Missing any of the three meant a failed upload. The reference became the deduplication key, enforced by a unique constraint: upload the same slip twice, get the same transaction back.

Later, re-uploading a slip started to refresh the transaction from the newer, better parser. A note or category is filled only if empty, and a transaction the user locked is never touched.

Then the benchmark (Step 7) showed that the small vision model drops characters when it reads the printed reference: nine zeros became five. A wrong reference is worse than none. Now the reference comes only from a CRC-checked QR, and receipts without one are deduplicated by the file's SHA-256.

Step 3: testing OCR with synthetic receipts

Real slips contain names, account numbers and amounts. They can't go into a git repository, and I don't want them in CI logs either.

So I wrote a generator. It draws synthetic test slips in the style of each bank's layout: demo names like "MR. DEMO USER", a made-up reference, and a QR that decodes correctly but points at no real transfer:

Next to each image the generator writes the ground truth, from the same values it just drew:

amount: '1234.56'
currency: ''
note: Coffee with friends
occurred_at: 2026-01-15 09:00
recipient: MS. DEMO RECIPIENT
sender: MR. DEMO USER
slip_ref: DEMOREF0000000001

Because the label and the pixels come from one function call, they can't drift apart. There's no hand-editing: change the renderer, regenerate.

The corpus covers watermarks, Thai and English labels, fees and odd date formats. When a real slip failed to parse, the fix came with a synthetic variant of it.

Step 4: batch upload and the first load test

Once a single upload worked, the next feature was obvious. People don't have one slip. They have a folder full of them. So batch upload came next, then folder auto-upload: grant access to a folder once, and new screenshots in it get imported on their own.

That raised a question before any user did: what happens when someone uploads everything at once? I have more than 2,900 slips of my own, so I had a realistic test set.

The tool was k6. Each virtual user signs in to its own account and uploads my slips in a loop; the load ramps up, holds, and ramps down. It ran against my local kind cluster, which uses the same manifests as production, patched before each run to one candidate mix of replicas, gunicorn workers and CPU.

What I counted:

• Gateway errors: responses with 502, 503 or 504. This is the "it broke" signal: the gateway gave up before the backend answered.
• Upload latency of the parseReceipt request: median, p95, max.
• Other failures, as a cross-check, so a broken login can't hide as a "fast" run.

That last one mattered. By default k6 resets cookies between iterations, so my first runs uploaded unauthenticated and got 403s: quick, clean and meaningless. noCookiesReset fixed it.

With production's settings at the time, it failed. OCR is CPU-bound and ran inside the request, so uploads queued behind the workers until the gateway gave up.

Then I tried a few configurations, and the test confirmed what I expected: more gunicorn workers on the same CPU made it worse, because CPU-bound workers only fight over cores. Adding replicas alone wasn't enough either. It took both: more pods, and one worker per core in each.

Step 5: every bank prints its slip differently

Then came slips from other banks. In Thailand people often call a bank by its colour, not its name, and I did the same in the code: green, purple and yellow. No brand names anywhere, and the generator draws look-alikes, not copies.

• Green and purple print FROM and TO labels, in Thai or English.
• Yellow prints no labels at all: two name blocks and an arrow.
• A Thai-language purple slip writes the date as 15 มกราคม 2569 - 09:00. My regex didn't expect the dash and silently recorded midnight. No error, just a plausible wrong answer.
• Watermarks bled into the text. Once a bank icon was read as part of a recipient's name.
• Some slips have no QR at all. No QR means no reference, so OCR is the only source of data.
• And one slip in my collection carried a whole cinema ticket next to the transfer details: plenty of text that looks like data and isn't.

Each fix was a heuristic for one real slip, and they piled up. So I replaced the pile with one rule table: a field, its keyword anchors, and an extraction mode. A new layout became a new table entry, not a new code path.

It still wasn't enough. Tesseract needed two passes and ever more bank-specific tuning. The classic alternatives failed too: PaddleOCR had no wheel for our Python version, and EasyOCR read 0 of 17 real slips and was about 48 times slower. It turned TO into gto.

Step 6: switching to an LLM, and privacy

Tesseract was the best of the classic tools, and it still wasn't good enough. Anything smarter meant deep learning and PyTorch, and at that point it's almost an LLM anyway. So I changed strategy and went all the way: a vision language model that looks at the image and returns JSON.

The obvious way is a hosted API. I didn't take it. A slip shows who you pay, how much and when. For a finance app that's the most sensitive data it has. I didn't want every photo going to a third party, for the same reason I left out the slip-verification API. So the model runs inside my own Kubernetes cluster. The privacy notice says it plainly: receipts are "never sent to a third-party AI service", and they are not used to train any model.

The photo is kept for 15 days, and that's on purpose. If a better model comes out, or a prompt tweak fixes a field, recent receipts can be read a second time. After that the photo is deleted; only its hash stays, for deduplication.

The first LLM version used Gemma 4 E2B on Ollama, one call for all fields. It read the fields that Tesseract struggled with, and it opened a door: shop receipts from any country, not just Thai slips. The synthetic corpus grew with them, USD, EUR and JPY receipts included. But the call sat inside the upload request. On CPU in the cluster it took 20–30 seconds, and that became the latency of every upload. For the LINE bot that's long enough to lose the reply.

Step 7: tweaking the LLM

First I built a benchmark: the real parsing code over a labelled corpus, accuracy per field and latency. Two corpora: 20 synthetic slips, and 50 of my own real receipts from 2019–2025, which never leave my machine. It all ran on my laptop, in the same local cluster I develop in: about 20 hours of runs. Every number below is a rate over repeated runs, because even at temperature: 0 the same model on CPU doesn't always give the same answer twice.

What I learned:

• Ask a small model for less at once. Field descriptions in the JSON schema made the model shuffle values between fields, and one extra field threw off fields that had been stable. So the strategy became multi-pass extraction: a few short questions about the same receipt instead of one long one.
• Reasoning mode eats the budget. Gemma 4 "thinks" by default. That used up the output tokens before any answer and cost about 40 s per receipt. reasoning_effort: "none" is not tuning. It's required.
• Image resolution decides Thai names. At 280 image tokens the model read the Thai letter ด as ต on every Thai-name image. At 560 it read it correctly, for about 5% more latency.
• Prompt rules help, then stop. Prompt rework moved recipient from 60% to 83% on synthetic slips. Nothing in the prompt moved sender: that was a model limit, and only the bigger model below fixed it.

Moving from Ollama to llama.cpp's llama-server roughly halved memory and exposed the image-token setting. That made the bigger Gemma 4 E4B practical:

Amount, date and reference read 100% in both. The reference comes only from the QR; a receipt with no QR code correctly gets none. The model only transcribes; tested code turns its text into numbers and dates.

The client itself is deliberately boring. No agent framework, no tool calling, just one HTTP request per question. A small model has a small context, and every token of instructions, tool schemas or framework boilerplate competes with the image. Simplified:

def ask(image_png: bytes, prompt: str, keys: list[str]) -> dict:
    """One question about one receipt. Simplified from receipts/parsers/vlm.py."""
    data_uri = "data:image/png;base64," + base64.b64encode(image_png).decode()
    response = requests.post(
        f"{VLM_BASE_URL}/chat/completions",       # any OpenAI-compatible server
        json={
            "model": VLM_MODEL,
            "messages": [{"role": "user", "content": [
                {"type": "text", "text": f"{prompt} Reply with ONLY a JSON "
                                         f"object with these keys: {', '.join(keys)}."},
                {"type": "image_url", "image_url": {"url": data_uri}},
            ]}],
            "response_format": {"type": "json_object"},
            "reasoning_effort": "none",   # Gemma 4 "thinks" by default
            "temperature": 0,
            "max_tokens": 1024,
        },
        timeout=180,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Because it speaks the plain OpenAI-compatible API, moving from Ollama to llama-server was a change of URL and model name, not of code.

Step 8: self-hosting the model

Self-hosting sounds like "run a container". In practice it was a list of small traps.

There is no GPU. Everything runs on CPU, and the model is pinned with a nodeSelector to a node with enough memory for it.

What the deployment looks like, trimmed:

initContainers:
  - name: fetch-model          # download once onto a PVC, verify, skip next time
    image: docker.io/curlimages/curl:8.11.1
    env:
      - { name: HF_REPO,      value: unsloth/gemma-4-E4B-it-GGUF }
      - { name: HF_REVISION,  value: <pinned commit> }
      - { name: MODEL_FILE,   value: gemma-4-E4B-it-Q4_K_M.gguf }
      - { name: MODEL_SHA256, value: <sha256> }
    command: ["/bin/sh", "-c"]
    args:
      - |
        set -eu; cd /models
        # already there and intact? then this start costs one hash read
        if echo "$MODEL_SHA256  $MODEL_FILE" | sha256sum -c - >/dev/null 2>&1; then exit 0; fi
        curl -fL --retry 5 -o "$MODEL_FILE.part" \
          "https://huggingface.co/$HF_REPO/resolve/$HF_REVISION/$MODEL_FILE"
        echo "$MODEL_SHA256  $MODEL_FILE.part" | sha256sum -c -   # wrong file: pod fails
        mv "$MODEL_FILE.part" "$MODEL_FILE"
    volumeMounts:
      - { name: models, mountPath: /models }          # writes the weights here
containers:
  - name: llama-server
    image: ghcr.io/ggml-org/llama.cpp@sha256:<digest>   # official image, pinned
    args: [-m, /models/gemma-4-E4B-it-Q4_K_M.gguf, --mmproj, /models/mmproj-BF16.gguf,
           -np, "2", -c, "8192", --image-max-tokens, "560",
           --cache-ram, "0", --metrics]
    env:
      - name: LLAMA_ARG_THREADS                       # threads follow the CPU limit
        valueFrom: { resourceFieldRef: { resource: limits.cpu } }
    volumeMounts:
      - { name: models, mountPath: /models, readOnly: true }
    resources:
      requests: { cpu: "2", memory: 8Gi }
      limits:   { cpu: "4", memory: 12Gi }
    # /health answers 503 while the model loads; a busy CPU can starve it past 1 s
    startupProbe:
      httpGet: { path: /health, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 10
      failureThreshold: 60    # 60 × 10 s: allow up to 10 min to load (the download already ran in the init container)
    livenessProbe:
      httpGet: { path: /health, port: 8080 }
      periodSeconds: 20
      timeoutSeconds: 10      # default 1 s killed a healthy server mid-receipt
      failureThreshold: 6
volumes:
  - name: models
    persistentVolumeClaim: { claimName: llama-server-models }

The traps behind those lines:

• Don't bake the weights into the image. Building an image with about 6 GB of weights needed about 25 GB of temporary disk. An init container downloads them from a pinned Hugging Face commit, checks the SHA-256 and stores them on a volume. A truncated download fails loudly instead of serving a different model.
• A cache that is never used still grows. llama-server keeps old prompts in RAM for reuse by default. Every receipt is a new image, so nothing is reused. After a long benchmark run on my laptop, the pod sat at 11.8 GB. With --cache-ram 0 it stays at about 3.8 GB plus the weights.
• A busy model looks dead. Generation pins every core, and the health endpoint can take longer than one second to answer. With default probes Kubernetes killed a perfectly healthy model server in the middle of a receipt. The probes now wait 10 seconds.
• More slots don't mean more speed. On CPU, four parallel slots barely raised throughput and slowed each request. I run two, for a reason in the next step.
• The server has no authentication. A NetworkPolicy lets in only the app's namespace and Prometheus.

There's one replica and the rollout strategy is Recreate: two copies would need twice the memory. So a model update is a short outage, and the Celery retries absorb it. It's deployed through Flux like the rest of the cluster.

And the cost model is simple. No per-receipt or per-token bill, no GPU. What I pay for is a VPS, or a Kubernetes node, with enough RAM to hold the model. A thousand receipts or a hundred thousand, the bill is the same; only the queue gets longer. If the queue gets too long, I add another node with another copy of the model. And to make it cheaper, the plan is elastic nodes: started when the queue grows, removed when it's empty, so I pay for model time only when there's work.

Step 9: back to OCR, and the LLM moves to the background

The model was good now, but still 20–30 seconds on CPU. So I brought Tesseract back. Not as the parser, but as the fast first pass:

1. In the request: store the photo, decode the QR, run one Tesseract pass. It takes what it can (reference from the QR, amount, date, note), but only the amount is required. The transaction appears right away.
2. In a Celery task: the vision model reads the same stored photo and fills in sender, recipient, note, currency and a precise time. If the first pass found nothing, the task does a full parse itself.

A background pass has a catch: the user may edit the transaction while the model reads it. So the task writes with a conditional UPDATE that lands only if nothing changed since upload. Otherwise its reading becomes a suggestion the user reviews field by field.

The LINE reply carries the OCR result; the model's improvements show up in the app.

Then the bulk case. Someone imports a few years of slips at once, like my 2,900. Each is a 20–30 second model job on the default queue. A LINE message waiting behind them would outlive its reply token, and the user would get no answer at all. So chat tasks got their own chat queue and worker. The model server has two slots, shared by both workers: during a big import a chat question waits for at most one receipt read, not for the whole import.

The whole pipeline today:

flowchart LR
    subgraph In["Where slips come from"]
        App["Web / Android app<br/>(also watched folder, share menu)"]
        Bot["LINE / Telegram bot"]
    end

    App -- "parseReceipt" --> API["Backend<br/>(GraphQL)"]
    Bot -- "webhook" --> CQ[["chat queue"]]
    CQ --> CW["Chat worker"]

    API --> Fast
    CW --> Fast
    Fast["Fast pass<br/>store photo · QR · Tesseract<br/>amount required"] --> Tx[("Transaction<br/>source: OCR")]
    CW -- "one free reply" --> Bot

    Fast -- "enqueue" --> DQ[["default queue"]]
    DQ --> RW["Receipt worker"]
    subgraph VLM["llama-server · Gemma 4 E4B · CPU only<br/>2 slots, shared: a request takes any free one"]
        S1["slot 1"]
        S2["slot 2"]
    end
    RW -- "receipt reads, 20–30 s" --> VLM
    CW -. "chat questions, short" .-> VLM

    RW --> Changed{"Edited since<br/>upload?"}
    Changed -- "no" --> Tx2[("Transaction<br/>source: LLM")]
    Changed -- "yes" --> Review["Suggestion,<br/>reviewed field by field"]

Results

On 50 real receipts, amount, date and reference read 100% of the time (no reference when there's no QR code, as intended). The first model read of a real photo takes about 20 seconds, but nobody waits for it.

One problem is still open: Thai-script names. The sender reads 95% right in Latin script and 11% in Thai. A bigger image budget didn't help. Of 20 models I benchmarked, only two read any Thai sender, at 6–8 times the latency. The overall sender score, 83%, hid this completely. If your documents mix scripts, measure accuracy per script, not just per field.

What it taught me

Not a list of mistakes. More a way to get to a result that works.

• Balance what's possible with what you have. A hosted API or a GPU would have been easier. I had CPU, a privacy promise and one engineer. So: a small model, a short prompt, and OCR kept for the part that has to be fast.
• Iterate, don't wait for perfect. Every version shipped and taught me something: the QR, the OCR, the rule table, the model in the request, the two passes. The benchmark came when guessing stopped being enough, not before the first line of code.
• Step back and look around. Going back to OCR wasn't a defeat. The model was better at reading; OCR was better at answering fast. The best design used both, and I only saw it after stepping back from "replace OCR with an LLM".

I'm a backend tech lead with more than ten years of Python, Django and Kubernetes. On JodRubJai I own the whole technical side: the product decisions, the architecture, the infrastructure, and the AI coding agents that implement features from my specs under my review. If you need someone to take a product like this from idea to production, or a tech lead to join your team, contact me.