AI playtesting notes

Automated Self-Play for AI Card Game Balancing

Moonfall plays complete matches against itself and scores every finished one, so unclear cards get caught before players meet them. What ten batches showed.

By Diffused Dreams· July 9, 2026· Updated August 17, 2026· 3 min read

Moonfall cards have no attack or health numbers. Each play is a prompt, and an AI Game Master decides what it does in the moment. That makes a whole class of bug impossible to unit-test: the card said one thing, the story implied another, and the board ended up somewhere else. You only find it after a full match has tangled all three together — and the story is not disposable, since a finished match can become a manga storybook somebody shares.

Letting the game play itself

Automated self-play runs complete Moonfall matches with nobody in them. Two socket clients take the players' seats on the real server, a simulated player picks from the legal moves, and the storyteller resolves each one exactly as it would in a live game. Nothing is mocked, so nothing passes here and fails in production.

Matches run in parallel, in batches. One odd game tells you nothing; the same oddity in six games out of twenty-five is a defect you can go and find.

Every match leaves a trace: the starting state, each action, the storyteller's calls, the patches it emitted, and where the board actually ended up. Comparing those is how that felt wrong becomes the patch on turn seven removed a card the story never mentioned.

Diagram of parallel Moonfall matches flowing through the server and storyteller into trace capture, an LLM judge, a change queue, and guarded application. — Nothing here is mocked — the harness drives the same server path a live match does.

Judging what comes out

Finished matches go up a short ladder. Deterministic checks run first, since anything a program can settle should not be argued about. What is left goes to an LLM judge that answers binary questions instead of scoring out of ten, and quotes the turn behind each answer — both following published advice on LLM evaluation and reasoning before scoring.

The half that matters most is the part usually left to taste: whether a moment genuinely surprised, whether the prose earned its dread instead of announcing it, whether a resolution respected what the card's name promised. Repeated findings become small proposals; constrained changes are applied automatically under guardrails, with an audit trail left for monitoring and optional intervention.

Table of the Moonfall judging ladder, listing what runs at each tier, the question it answers, and the shape of its verdict. — Deterministic checks first, so a provable violation never reaches a model that could argue with it.

What ten batches showed

Line chart of estimated storyteller accuracy, surprising moments, and earned emotion across ten Moonfall self-play batches. — The surprise line races; the accuracy line, already high after earlier runs, climbs more slowly.

Each point is one batch of twenty-five matches. Only matches that ran to completion are scored.

The stories became far more surprising. In the first batch a judge found a genuinely surprising, fair moment in 44% of matches; in the last two it found one in 95% and then 100%. Accuracy is growing more slowly now — it held between 92.6% and 97.1% across the ten batches, with earlier runs having already lifted it significantly.

Two caveats. Accuracy here is an estimate: the share of storyteller calls never linked to a registered correctness failure, useful for comparing batches rather than judging any single match. And ten batches is a pilot — hundreds of matches, not hundreds of thousands.

Could AI decks ever battle inside the game?

The harness is already a kind of auto-battler: AI-controlled decks pressuring each other while a storyteller narrates the result. Some version of that could be interesting inside Moonfall, with humans focusing more on cards and deck choices than on clicking through every turn. That is not a feature announcement, only a reason the work is interesting beyond testing.

For now it does something narrower. Moonfall is meant to surprise you, and the loop exists so that the surprise comes from the match rather than from a card whose wording was unclear.