Spiral Energetics

Part 1: LLM-driven CRPG?

The Concept?

The idea springs forth from five facts:

  1. Paizo publishes the rules for Pathfinder 2e under the ORC License.
  2. This license allows you to create a game that implements the Pathfinder ruleset.
  3. There is a large library of Pathfinder campaigns, published as human-readable PDFs.
  4. Anything that is human-readable is also LLM-readable.
  5. CRPGs are fun. Pathfinder 2e is fun. The campaigns are fun.

The idea is to feed an LLM a campaign PDF, and have it run the campaign for the player. I'll implement a classical game engine, much like any other CRPG. There will be 3d visuals, movement, combat, an inventory, a character page, levelling up, deterministic rules, and so on. Anything that you're used to, from a CRPG:

Baldur's Gate 3
Baldur's Gate 3
Divinity: Original Sin 2
Divinity: Original Sin 2
Neverwinter Nights 2
Neverwinter Nights 2

The LLM will then use tool calling to manipulate the game world: store facts, generate NPCs and maps, and run dialog and quests.

The idea is to create an agentic harness that allows the LLM to play the role of a dungeon master. Just like how codex is an agentic harness that allows the LLM to play the role of a software engineer.

Using Pathfinder 2e is crucial here, because the ruleset is open-source. This also down-risks because the game is already fun. There is already brand recognition. There is already an established playerbase I can market to.

Another crucial thing about using a pre-existing TTRPG is that it comes with pre-existing campaign books. These books are written for human dungeon masters. Feeding this to an LLM and asking it to run the campaign will work vastly better than making an LLM to come up with everything itself. LLMs aren't creative in that sense, and the pre-written campaign keeps the LLM from going off the rails. Where it will shine will be filling in the gaps, and making the campaign come alive and be reactive to the player.

Prior art?

I did my due dilligence, and searched for other people attempting to make an "LLM RPG".

The first type of prior art I found is effectively just a clone of the ChatGPT web interface. The LLM describes the game world to you, and you roleplay. You can choose to type a response, ask the LLM to say something more, ask the LLM to regenerate the response, or return to a previous point in time. Under the hood, the LLM will be storing facts about what happens. But there's not much more than that going on. Essentially, it is a wholesale replacement of a game engine with an LLM. It doesn't work because it misses much of what makes a game fun (graphics, combat, deterministic rules, and so on). In the end, all you have is a textbox. It's surprisingly tiring if you try to "play", as you have to come up with the responses to the LLM, which drives much of the fun, so if you don't type good responses, you don't have fun. All of this is really quite different to my idea of injecting an LLM into a traditional game engine.

AI Dungeon
There are few "games" like this. It seems the biggest one is AI Dungeon.

I found someone using Gemini voice mode as a dungeon master.

The second type of prior art goes a bit further than just cloning the ChatGPT web interface, but still lacks from being a true agentic harness for a dungeon master inside a classical game engine. There's an abandonded experimental project where an LLM agent lives inside a village. And there's a slightly better looking demo project with an engine, although it's still text-based.

Untitled demo project

Concretely?

I'll implement the Pathfinder 2e ruleset. And I'll create graphics, a combat system, character management, inventory management, and so on -- all the traditional, time-tested, CRPG game systems.

I'll then feed the LLM a campaign book somehow (OCR? rendered pages?), and the LLM will use tool calling to store facts from the campaign book:

  • tool for generating a world map
  • tool for generating explorable maps (settlements, battlemaps, dungeons, and so on)
  • tool for generating NPCs and populating maps with them
  • tool for populating maps with monsters
  • tool for storing campaign/quest state
  • tool for storing campaign/quest updates
  • tool for getting context about how to roleplay an NPC

The tools can read and write JSON. It likely doesn't have to be fancy.

Map and NPC generation will have to be parametrically random. For obvious reasons, I can't hardcode these sorts of things, as the player can input any possible campaign. But it can't be fully randomly generated either, because then it doesn't match the campaign. So, for example, the tool to generate a town will have to accept arguments like "5 houses, 1 blacksmith, 2 taverns", and the the individual houses will have to be customisable ("derelict house, inhabited by NPC XYZ"), and so on.

I can possibly use a voice model for NPC dialog, depending on whether there are good models that can be run locally on the player's machine.

Combat will be run exactly like a traditional game would run combat. There's absolutely no need to reimagine this. Pathfinder already has a fun ruleset, and CRPG combat in general is already fun.

In short, the LLM describes the intent to the underlying game engine, which then deterministicly compiles the intent to a playable game state. The LLM then provides textual interactivity (quests, dialog) on top of that game state.