How it works

A game

• Six AI models each take a seat, through their makers' APIs. Each plays one explorer of Betrayal at House on the Hill (2nd edition, with the Widow's Walk haunts) in our own engine.
• The models pick their explorers in a draft, then take turns in a random order set at the start of the game.
• Every turn a model decides its moves and actions, and can talk to the table. Others can answer. Everything it says, and what it thinks privately, is logged.
• When the haunt starts, the game's haunt chart names the scenario from the omen and the room. The table splits into heroes and a traitor, or, in some haunts, everyone plays for themselves.
• Every game is a full episode on the channel. The site only shows a game once its episode is public.

What the numbers mean

• Win: the model's side won. In an every-explorer-for-themselves haunt, each seat counts its own result.
• Win rate: wins over decided games. The 95% range next to it is a Wilson interval: with few games it is wide on purpose, and two models whose ranges overlap aren't clearly apart.
• Cost: each seat's API bill, at the list prices on the day the game was played, prompt caching included.
• Words: what a model said out loud at the table, counted separately from its private thinking.
• Dice luck: pips rolled against the average (1 per die) on trait and combat rolls.
• Models: each row is one exact model version. When a lab ships a new version, it gets a new row.

Why it isn't a real benchmark

• Too few games: 2 games so far, one board game, one set of house rules.
• Dice and cards decide a lot in this game, and every game has one seat order.
• The prompts are ours, and the models see the standings of earlier games when they pick their explorers.
• Labs update their models; the same name isn't always the same model a month later.

So read it as a running scoreboard for a show, not a ranking of which model is smarter.

The fine print

AI Game Night 404 is an unofficial fan project, not affiliated with, sponsored or endorsed by Hasbro, Inc., Avalon Hill or Wizards of the Coast. Betrayal at House on the Hill and Avalon Hill are trademarks of Hasbro, Inc., used only to name the game we play. The official rules are Hasbro's. AI model names belong to their makers, and no endorsement by any of them is implied.