The goal: create an agentic harness that can operate on a PDF. It has to:
- Extract and record structured facts (NPC names and backgrounds, world history, locations, quest hooks, and so on)
- Use tool calling to guide constraint-based random generation (of towns, dungeons, battlemaps, and so on)
- Use tool calling to create deterministic entities (NPCs, quests triggers and stages, special items, encounters, and so on)
- Link locations in a cyclical graph
I've never done this before, so I bumbled around quite a bit.
The first question was how to feed the LLM the PDF in the first place. I knew that extracting data from a PDF is a famously hard problem, so I decided I'd first see if I can just render each PDF page as an image, and feed the raw images into the LLM. It's a totally unga-bunga method, but it sidesteps any problems with a lossy PDF -> structured markup conversion process. Sometimes simpler can be better. Each chapter is mostly self-contained, and sequentially ordered, which means that it's not necessary to blow up the context window by trying to feed the LLM every single page in the book all at once.
I tested this first version out using ChatGPT 5.5, as I wanted maximum intelligence at this stage, just while I was doing my initial experiments. It was incredibly slow... but more importantly, it would use about a dollar a minute of API credits (scream). The final output was great, but I quickly abandoned the experiment for fear of bankrupting myself.
At this point I moved on to other tasks, since my TODO list was a kilometer long. But my breakthrough came when I randomly came across Mistral OCR, which inspired me to search for an open-source alternative: Chandra OCR 2, which is small enough to run locally, even on mundane single-GPU setups.
With renewed enthusiasm, I refactored my pipeline so that the output from Chandra OCR 2 is fed into Gemma 4 31B running through vLLM on my 3x 5090s, for a fully locally-hosted experience.
The outputs from Chandra were better than expected. It has some understanding of positional relationships:
Although it's not without errors:
All-in-all, the OCR results were more than usable. Minor problems like the out-of-context numbers won't affect the LLM much. Most importantly, I wasn't able to spot any problems with missing text or images, which would be a hard blocker.
However, when I ran the next part of my refactored pipeline (where the output from Chandra OCR is sent into Gemma 4 31B), the results from Gemma were awful. The LLM would get stuck in an loop, and only stop once it hit a guard in my harness to protect against repeated tool call failures.
Was I wrong about the OCR output being good? Was Gemma 4 31B just not intelligent enough compared to ChatGPT 5.5? Was my VLLM config wrong? Did I introduce a bug in my agentic harness? I didn't have a clue. There were too many variables (in retrospect, changing the model at the same time as switching to OCR was a mistake).
After adding loads of logging inside the harness, I figured out the error messages for the tool calls were really bad. I was using the default serde ones, and it wasn't always clear where the syntax error was. It was as bad as bash shell in terms of outputting unhelpful error messages. I hadn't actually touched this part of the code, so it made me wonder how ChatGPT 5.5 was working at all. I can only conclude it was so intelligent relative to Gemma 4 31B that, when syntax problems did occur, it was brute-forcing correct syntax through repeated guesswork. That must have partially been why it was so slow (and expensive) when I was doing my earlier experiments.
After introducing serde_path_to_error, the infinite-loop problem went away. I was getting useful output! The tool calls were working! Using a fully locally hosted pipeline! It was an exciting moment since it proved the concept.
There's a lesson to be learned here: it helps to use a relatively stupid model when developing an agentic harness, because an intelligent one will hide problems with the harness behind its own intelligence.
Final Architecture
My initial plan was a pipeline like:
- Render the PDF pages as images
- Send the images to an LLM, which would extract and store (mostly free-form) facts about the world
- Send the stored facts to an LLM, which would output constraints for the random generation, in order to make:
- locations
- NPCS
- quests
- and so on
- During gameplay, the stored facts can be referenced when an LLM is running NPC dialog or doing quest updates
The reason for re-reading the facts when creaing locations/NPCs/quests/etc was that the rendered PDF pages are very expensive in terms of context usage, so it wouldn't have been feasible to continually re-read the PDF. But now that I'm running OCR on the PDF, the resulting HTML is only a few kilobytes large, so it removes the need for step 3 to operate on the outputs of step 2. Each step can always have the raw HTML in its context window.
I therefore ended up modify the above slightly:
- OCR the PDF
- Send the OCR output to an LLM, which would extract and store (mostly free-form) facts about the world
- Send the OCR output to an LLM, which would output constraints for the random generation, in order to make:
- locations
- NPCS
- quests
- and so on
- During gameplay, the stored facts can be referenced when an LLM is running NPC dialog or doing quest updates
As I write this, I've implemented the world fact storing, and the location generation settings. The full architecture is:
flowchart TD
subgraph S0["Step 0 — OCR the PDFs"]
PDF["Source PDF files"]
OCR["Chandra extracts text and images"]
OCR_MODEL["vLLM serving Chandra OCR model"]
PDF --> OCR
OCR <-->|"page analysis"| OCR_MODEL
end
OCR_DOCUMENTS["HTML and sibling images"]
OCR --> OCR_DOCUMENTS
subgraph S1["Step 1 — Read OCR and store world facts"]
INPUT1["Read OCR document,<br/>alongside world facts already stored"]
MODEL1["LLM identifies useful world facts"]
MODEL_SERVER1["vLLM serving Gemma 4 31B"]
ACTION1{"Next tool call"}
VALID1{"Fact-storage call valid?"}
STORE1["Create or update world facts such as places, people, quests, encounters, and maps"]
ERROR1["Return the problem to the model so it can correct and retry"]
FINISH1{"Can the finish tool be accepted without errors or inconsistencies?"}
MORE1{"More OCR documents?"}
CHECK1["Check the complete campaign for consistent, connected facts"]
OUTPUT1["Campaign facts file"]
INPUT1 --> MODEL1
MODEL1 <-->|"fact analysis"| MODEL_SERVER1
MODEL1 --> ACTION1
ACTION1 -->|"store facts"| VALID1
VALID1 -->|"yes"| STORE1
STORE1 -->|"confirm what was stored"| MODEL1
VALID1 -->|"no"| ERROR1
ERROR1 --> MODEL1
ACTION1 -->|"finish tool"| FINISH1
FINISH1 -->|"not yet"| ERROR1
FINISH1 -->|"yes"| MORE1
MORE1 -->|"yes"| INPUT1
MORE1 -->|"no"| CHECK1 --> OUTPUT1
end
OCR_DOCUMENTS --> INPUT1
subgraph S2["Step 2 — Read OCR and store map-generation settings"]
INPUT2["Read OCR document, alongside map-generation settings already stored"]
MODEL2["LLM turns each source map into practical generation settings"]
MODEL_SERVER2["vLLM serving Gemma 4 31B"]
ACTION2{"Next tool call"}
VALID2{"Map-storage call valid?"}
STORE2["Store settings for generating a town, dungeon, or battle map"]
ERROR2["Return the problem to the model so it can correct and retry"]
FINISH2{"Can the finish tool be accepted without tool errors?"}
MORE2{"More OCR documents?"}
OUTPUT2["Map-generation settings files"]
INPUT2 --> MODEL2
MODEL2 <-->|"map analysis"| MODEL_SERVER2
MODEL2 --> ACTION2
ACTION2 -->|"store map settings"| VALID2
VALID2 -->|"yes"| STORE2
STORE2 -->|"confirm what was stored"| MODEL2
VALID2 -->|"no"| ERROR2
ERROR2 --> MODEL2
ACTION2 -->|"finish tool"| FINISH2
FINISH2 -->|"not yet"| ERROR2
FINISH2 -->|"yes"| MORE2
MORE2 -->|"yes"| INPUT2
MORE2 -->|"no"| OUTPUT2
end
OCR_DOCUMENTS --> INPUT2