The short answer
Learn strength from real games, the way Moneyball did for baseball. Collect tables of four Commander players where we know every decklist and who won. Describe each deck by what its cards do, using the features Deckwright’s card parser already reads: how much fast mana, card draw, tutors and interaction it has, and how often it can assemble its combo by turn 63,4. Then fit a model that learns how much each of those features adds to a deck’s chance of winning. A deck’s strength is its predicted win chance at a four-player table. A card’s strength is how much that prediction moves when the card goes in or out.
The game data to start exists today. TopDeck.gg publishes table-by-table results from competitive Commander (cEDH) tournaments, with decklists, through a free API1,8. About 1,000 tables is enough for a first model, and fitting it takes seconds on a laptop1,2. For casual tables, Playgroup.gg’s public decks and our own logged games fill in, as described below12.
Full game simulation with self-learning AI players comes last in the plan. It costs 15 to 45 hours of computing per deck on this machine, and nobody has shown it working for four-player Magic5.
Why this is hard
A strength number is only useful if it predicts who wins, and checking that needs game results. Baseball analysts had a century of box scores. Commander has very little public game data. Four players share a table, decks vary enormously, and most casual games are never recorded anywhere1.
A method that only agrees with expert opinion has not been shown to predict winning.
The four ways to estimate strength
Reading the cards. Score each card from its rules text, the way Deckwright does now: parse what the card does, then add up values for its effects. It needs no game data and works on any card, including brand-new ones. Its weakness is that the values are set by hand. On 344 pairs of cards that its authors judged by hand, Deckwright picks the same stronger card 88% of the time, but nobody has checked its scores against games won3. Chess engines ran on hand-written scoring like this for decades, and they got much stronger once the weights were tuned against real game results11. The recommendation adds that tuning step for Magic.
Statistics from game records. This is the Moneyball approach. The simplest version gives each deck a rating, like a chess Elo, that rises when it wins and falls when it loses. Ratings are the best-tested method anywhere, but they can’t rate a deck that has never played, and telling a 30% deck from an average 25% deck takes about 610 games2. A second version describes each deck by its stats instead of its name, which lets it score unplayed decks. A third borrows basketball’s plus-minus, which credits each player for how the team does while they are on the floor. Applied to Magic, it credits each card for how its decks do. Plus-minus needs a great deal of data per card, and nobody has published a use of it for Magic2.
Simulating pieces of the game. Some questions about a deck can be answered exactly with probability, or with a quick simulation, without playing whole games. How often does it hit its land drops? How often does it see every combo piece by turn 6? How often does it have interaction by turn 4? Each answer is a number we can compute in under a second. On their own these numbers don’t say how much each one matters for winning; that weight still has to come from game results4.
Simulating whole games. Run complete games between computer players, either an existing engine’s built-in AI or players that teach themselves by playing millions of games, as AlphaZero did for chess and Go. Forge, the open-source engine with the only working four-player Commander AI, played 265 games an hour on this machine. That figure is a ceiling, because the 120-second game clock cut many games short7. Engine AIs are weak players, so engine win rates mix deck strength with poor piloting. Self-learning players would fix the piloting, but at the cost above, and with no published four-player result5.
The recommendation, in more detail
Deckwright’s card reading supplies the features. The piece simulations add consistency numbers, such as combo-by-turn-6 odds. Real game results set the weights.
The model predicts, for each deck at a table, its chance of being the winner. Each deck also gets a small personal adjustment that grows only when its results clearly beat or trail what its features predict. A heavily played deck can then earn a rating of its own, while a new deck is scored from its features alone.
It can score decks nobody has played, which plain ratings can’t. Commander has far more possible decks than recorded games. Deckwright’s existing scores become inputs the model can weigh.
How we would know it works
Hold back about 300 tournament tables the model never sees during fitting, split by date and by event. A good model predicts those winners better than a coin flip (25% each at four seats) and better than plain ratings on decks it has never seen.
Two numbers set expectations. Showing that one deck wins 30% of the time against the 25% baseline takes about 610 games of that deck. Telling apart two decks five points apart takes about 1,150 games each5. Claims about a single deck will therefore come from pooling games by commander.
Knowing the full finish order, not just the winner, cuts those numbers a lot. At a four-player table, the full finish order carries 2.56 times the information of the winner alone, so about 240 fully recorded games do the work of 610 winner-only games12.
Confidence today is medium-low. Every part works on made-up test data, and nothing has yet been scored on real Commander games2,4.
What it needs
Phase one needs about 1,000 cEDH tables from 20 or more TopDeck.gg events, each with the winner and all four decklists. The API is free with an account key, and it asks for a visible credit1,8. Draft data from 17Lands, which has far more games, is a cheap place to test the methods before Commander9.
For casual tables, Playgroup.gg public decks give each deck’s bracket, decklist and the win or loss of every rated game, and the site allows bulk reading10,12. They don’t say who sat at the table, so they rate a deck against the field rather than against named opponents. About 2,600 public decks have a decklist and ten or more games, but that estimate rests on just four sampled decks, so the true number could be well under or over it12.
Three things are missing from public data. No platform gives casual games with the full finish order. TopDeck.gg records no turn order or game length, and SpellTable exports nothing. Nothing separates a strong pilot from a strong deck1,12.
The plan
-
Test set and baselines. Build the TopDeck.gg fetcher and match decklists to cards, about one to two days of work. Score three simple predictors on held-back tables: everyone equal, plain ratings, and Deckwright’s current scores. Done when 1,000 tables are loaded and all three are scored. In parallel, start logging our own casual games in a simple spreadsheet: who played what, who was knocked out when, and the full finish order12.
-
The feature model. Fit the weights on Deckwright’s features and the piece simulations. Done when it beats plain ratings on held-back decks and its weights point the sensible way (more fast mana helps, for example). Then pull Playgroup.gg’s casual public decks and check the weights hold for casual brackets, against our first 200 recorded games12.
-
Card values for Deckwright. Turn the fitted weights into a value for each card, and compare them with Deckwright’s current rankings on held-back decks. Done when the new values predict winners better than Deckwright alone. Then they go to the Deckwright build.
-
Simulation, only if needed. If the model still misses on high-skill or combo decks, add Forge simulation results as an extra input, then consider self-learning players. Done only if the addition improves predictions by more than run-to-run noise.
What it means for Deckwright
Deckwright’s parser has a record for 35,353 cards, 23,766 of them with at least one parsed role, including who decides, who pays and what the drawbacks are. Those become the model’s features, and its card encoding should stay as it is. Forge’s card scripts agreed with it on 67% of features across 83 staple cards, and the misses were mostly mapping gaps, so Forge works best as an outside checker each time the parser changes6.
Deckwright has no game results to check against. Its only accuracy evidence is agreement with its authors, and player count and power bracket enter as assumptions that nothing measures. The feature model adds the missing check against game results. Deckwright keeps its advantages: it needs no game data, little computing, and it is already built.
Cards Deckwright can’t fully parse, such as Pendant of Prosperity, should get wider uncertainty, since a feature it couldn’t read is missing from the prediction.
What we learned from the existing engines
-
Forge is the only engine with a working four-player Commander AI. Its GPL license means we would talk to it over a network connection and never build it into our code7.
-
Forge’s game clock counts a game cut off by the clock as a win. Its win rates need the clock raised, or cut-off games flagged, before we use them7.
-
XMage’s default AI can see hidden cards, which flatters decks that react to opponents. MageZero, a fork of XMage, is the best existing template for self-learning players7.
-
Magarena has the simplest deck-strength test: play two decks against each other with a fixed AI, many times. It is two-player only and no longer maintained7.
How the methods ranked
Scores run from 1 to 5 on evidence of accuracy, a clear accuracy check, data needs, computing cost, fit for four-player games and build effort, weighted most toward evidence and data needs.
| Rank | Method | Score |
|---|---|---|
| 1 | Feature model (recommended) | 3.90 |
| 2 | Deck-stat model | 3.65 |
| 3 | Deck ratings | 3.65 |
| 4 | Deckwright as is | 3.35 |
| 5 | Plus-minus, Deckwright start | 3.15 |
| 6 | Card-field formula | 3.10 |
| 7 | Plus-minus | 3.10 |
| 8 | Forge simulation | 3.05 |
| 9 | Piece simulation | 2.95 |
| 10 | Self-learning, checked | 2.85 |
| 11 | Self-learning per deck | 2.75 |
Shifting any one weight by 0.1 leaves the feature model first. Plain ratings need many games per deck, and Commander has few2.
Sources
- Ground truth data (SP1 tab): what outcome data exists for Magic and how to validate an estimate.
- Statistical methods (SP2 tab): ratings, deck-stat models, plus-minus and the games each needs.
- Deterministic models (SP3 tab): Deckwright’s rules-text scoring and its hand-judged benchmark.
- Piece simulation (SP4 tab): mana, draw, combo and interaction pieces, and how to combine them.
- Self-learning players (SP5 tab): what AlphaZero-style training would cost for Commander.
- Card encoding (SP6 tab): Deckwright against Forge, XMage, MTGJSON and Scryfall, with the Forge cross-check.
- Engines and AI (SP7 tab): Forge, XMage, Magarena, MageZero and Argentum, and the measured Forge speed.
- TopDeck.gg tournament API. https://topdeck.gg/docs/tournaments-v2
- 17Lands public datasets. https://www.17lands.com/public_datasets
- Playgroup.gg public API, which allows bulk access. https://playgroup.gg/.well-known/openapi.yaml
- Texel’s tuning method, Chess Programming Wiki. https://www.chessprogramming.org/Texel%27s_Tuning_Method
- Game data sources (SP9 tab): Playgroup.gg public decks, SpellTable, the game-logging plan and the value of full finish order.
SP1 Ground truth: outcome data inventory
Machine-readable list: sources.yaml (checked by tests/sp1/sp1_gt.py). Protocol: validation.md.
Question
Which public data records who actually won a game of Magic, at what grain and volume? A card or deck
strength estimate (SP2-SP8) is only as trustworthy as the outcomes we can score it against. Popularity
data such as EDHREC (https://edhrec.com) measures choice, not results, so it is listed but marked outcome: none.
Commander data
- cEDH pods, game grain. The TopDeck.gg API returns each round’s tables with
players,winnerandstatus, plus decklists once public. It is free with a key and a visible credit (https://topdeck.gg/docs/tournaments-v2). This is the only public per-pod winner source found. - cEDH standings. EDHTop16 aggregates TopDeck.gg cEDH events into standings and win/loss/draw records per commander and pilot; all Command Tower events are added automatically (https://edhtop16.com/about).
- Casual Commander. Playgroup.gg ranks 794 commanders by win rate from 46,776 tracked paper games in 30 days, normalised to a 4-seat baseline, with a 40-game, 10-pilot floor (https://playgroup.gg/commanders/tier-list). Only aggregates are public; raw game logs are not.
- Arena Brawl. Untapped.gg shows Historic Brawl commander win rates over a rolling 3 months (https://mtga.untapped.gg/commanders). It is 1v1 Brawl, not 4-seat Commander.
- Popularity only. EDHREC and the cEDH Decklist Database (https://cedh-decklist-database.com) have no outcomes.
Limited and constructed data
- 17Lands publishes per-game CSVs per set and queue (deck, hand, mulligans, turns, won flag) under CC BY 4.0; the FDN PremierDraft game file is 60 MB gzipped (https://www.17lands.com/public_datasets). This is the richest public win/loss data in Magic, but it covers Arena Limited only.
- Arena constructed. Untapped.gg gives deck and card win rates from tracker users, as aggregates only (https://mtga.untapped.gg). Wizards holds every Arena game but does not release logs (https://magic.wizards.com/en/mtgarena).
- Tournaments. MTGO publishes Challenge standings and 5-0 league lists (https://www.mtgo.com/decklists); MTGTop8 archives top finishes across formats (https://www.mtgtop8.com). Both give placement, not games.
Gaps
- No public per-game log for casual 4-seat Commander. Playgroup.gg holds logs but publishes only aggregates (https://playgroup.gg).
- Turn order and game length are not among the TopDeck.gg table fields (table, players, winner, status) (https://topdeck.gg/docs/tournaments-v2).
- Draws and intentional draws are common in cEDH Swiss rounds and carry no winner (general knowledge).
- Pilot skill is confounded with deck strength in every tournament source (general knowledge).
- Scraped sources (MTGO, MTGTop8, EDHREC) can change layout or terms without notice (general knowledge).
Verdict
A Commander validation set can draw today on cEDH pod winners with decklists from the TopDeck.gg API (https://topdeck.gg/docs/tournaments-v2), with EDHTop16 standings as a coarser cross-check. For casual Commander only commander-level win rates exist (Playgroup.gg, about 47k games a month), which can check a ranking but not score per-game predictions. Missing: public casual per-game logs, turn order, and any way to separate pilot skill from deck strength. 17Lands remains the best place to test methods on plentiful game-level data before moving them to Commander (https://www.17lands.com/public_datasets).
SP1 Validation protocol
Every method in SP2-SP8 reports results in this form. Reference code: tests/sp1/sp1_gt.py.
Sources: sources.yaml and README.md.
Targets
An estimate is scored against one of three observed outcomes, from finest to coarsest:
1. Game winner: which seat won a pod or match. Commander: TopDeck.gg table winner
(https://topdeck.gg/docs/tournaments-v2). Limited: 17Lands won (https://www.17lands.com/public_datasets).
2. Placement: final standing in an event (EDHTop16, https://edhtop16.com/about; MTGO, https://www.mtgo.com/decklists).
3. Deck win rate: a deck’s or commander’s share of wins against its seat baseline
(Playgroup.gg, https://playgroup.gg/commanders/tier-list).
Draws, intentional draws and unfinished games are dropped by the caller before scoring; report the
drop rate next to every result (general knowledge).
Metrics
score_games(preds, winners) takes one dict per game (seat -> win probability) and the winner.
- Primary: log_loss, mean of -ln p(winner), and skill = 1 - log_loss / baseline_log_loss.
The baseline is the uniform guess, ln(seats), so 4-seat pods and 1v1 matches share one scale. Skill above 0 beats chance.
- Secondary: brier (sum over seats of squared error) and accuracy (winner alone has the top p).
Log loss is a strictly proper scoring rule, so honest probabilities score best (https://en.wikipedia.org/wiki/Scoring_rule).
Card-level methods turn card scores into deck win probabilities first, then go through the same scorer.
Holdout
- Split by time: fit on events before a cutoff date and score on later events, never at random. This keeps the metagame drift and ban-list changes the method will meet in use (general knowledge).
- Keep every game of one event on the same side, so pilots and decklists do not leak across.
- Report the cutoff, event count and game count. Fixing the holdout before tuning anything matters more than its exact size.
Sample size
min_games(p0, p1) gives the games one deck needs before its win rate p1 is told apart from
the seat baseline p0 (two-sided z test, alpha 0.05). For a 4-seat pod, p0 = 0.25:
| True win rate p1 | Games at power 0.8 | Games at power 0.9 |
|---|---|---|
| 0.27 | 3,736 | 5,025 |
| 0.30 | 610 | 825 |
| 0.35 | 157 | 214 |
| 0.40 | 71 | 97 |
A 1v1 deck at 55% against 50% needs 783 games. Few single cEDH decks reach 600 pods, so deck-level claims must pool by commander or be stated as game-level log loss over many decks (general knowledge).
Validation set
Smallest set worth building: about 1,000 cEDH pods with a known winner and all four decklists, pulled from the TopDeck.gg API (https://topdeck.gg/docs/tournaments-v2), from at least 20 events, split by time into roughly 700 for fitting and 300 for holdout. Estimate: a skill gain of 0.02 is a log-loss change of about 0.028 on a baseline of ln 4 = 1.386; with a per-game paired SD near 0.3 that needs about 900 games (general knowledge). Add a 17Lands set (one set, PremierDraft) as a high-volume control (https://www.17lands.com/public_datasets). Placement-aware metrics (rank of each seat, not just the winner) are future work.
Cost
- TopDeck.gg: free API key and a visible credit. 1,000 pods across 20 events is well under the 1,000,000 documents a day and 50 requests a minute on rounds endpoints (https://topdeck.gg/docs/tournaments-v2).
- 17Lands: one gzipped CSV per set, about 60 MB, free under CC BY 4.0 (https://www.17lands.com/public_datasets).
- Engineering: one fetcher and a decklist-to-card-ID mapper, about 1-2 days. No paid data is needed (general knowledge).
- Casual Commander per-game data would need a partnership with a tracker such as Playgroup.gg (https://playgroup.gg).
Statistical strength methods for Commander
Four families can turn game records into strength estimates: ratings, card win rates, plus-minus
(ridge regression on cards) and deck-stat regression. All four run in the spike
docs/research/sp2-statistical/spike/sp2stats.py (stdlib Python) on 4-player pods where only
the winner is known.
Tags: V = checked this session (page fetched, or number reproduced by the spike). GK = general knowledge, not re-checked.
Ratings
How it works: Give each deck one number. Elo updates a rating after each game in proportion to how surprising the result was GK (https://en.wikipedia.org/wiki/Elo_rating_system V). Glicko-2 adds a rating deviation and a volatility, so a deck with few games has wide uncertainty V (http://www.glicko.net/glicko/glicko2.pdf). TrueSkill handles teams and free-for-all games with Bayesian factor graphs GK (https://en.wikipedia.org/wiki/TrueSkill). Weng and Lin’s approximations (the basis of OpenSkill) give similar online updates for multiplayer rankings V (https://jmlr.org/papers/v12/weng11a.html, https://github.com/vivekjoshy/openskill.py). Plackett-Luce models a full finishing order as a chain of softmax choices V (https://cran.r-project.org/package=PlackettLuce). Bradley-Terry is its two-player case GK (https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model). Commander records usually keep only the winner. Cut Plackett-Luce to first place and you get the spike’s model: P(deck i wins) = exp(r_i) / sum over the pod of exp(r_j). This is a conditional logit (https://en.wikipedia.org/wiki/Conditional_logistic_regression GK). The spike fits it in one batch by maximum likelihood with a small L2 penalty, solved by Newton steps.
Where it has worked: Chess (Elo, Glicko), Xbox matchmaking (TrueSkill), and open-source game ladders (OpenSkill) GK. Arena and MTGO matchmaking use hidden ratings GK.
Cost: Needs pod membership and the winner per game, plus several games per deck. Gives no card-level answer. A rating describes a deck with its pilot, and pilots are confounded with decks.
Validation: Fit on earlier games and score later ones by log loss against the ln(4) baseline (see Validation). On the fixture world it recovers the true deck order with Spearman 0.98 V.
Card win rates
How it works: For each card, count the deck-seats whose deck holds it, and how many of
those seats won. The baseline is not 0.5: in a pod of n the fair share is 1/n, so the spike sums
expected = sum(1/seats) and shrinks toward it: rate = (wins + k*b) / (games + k) with
b = expected/games and k = 20 prior games. A card in every deck rates exactly 1/seats V.
17Lands’ GIH WR (win rate of games where the card was in hand: opening hand or drawn) and IWD
(improvement when drawn: GIH WR minus win rate when not drawn) refine this for Limited using
in-game draw data GK (https://www.17lands.com/metrics_definitions; page is script-rendered,
definitions not re-read).
Where it has worked: 17Lands draft data, where tens of thousands of games per card per set make GIH WR the community’s standard card-quality signal GK. EDHREC publishes inclusion rates, not win rates, which measure popularity rather than strength GK (https://edhrec.com).
Cost: Cheapest method: decklists plus winners. Its flaw is co-occurrence: a card that rides
in strong decks inherits their wins. In the fixture world card_passenger adds nothing yet rates
0.284 against a 0.25 baseline V, because it mostly sits beside card_strong (0.347 V).
Filler cards that happen to sit in strong decks reach 0.40 V.
Validation: Compare against a known-effect card (a synthetic world, or a card added to a fixed
set of decks). Needs games_needed deck-seats per card for a given effect size (see Spike results).
Plus-minus
How it works: Adjusted plus-minus from basketball: regress each stint’s point margin on indicators of who was on the floor, so each player is credited net of teammates V (https://www.82games.com/ilardi1.htm, Ilardi 2007). Regularised APM (RAPM) adds a ridge penalty because teammates are collinear GK (https://en.wikipedia.org/wiki/Ridge_regression, https://en.wikipedia.org/wiki/Plus%E2%80%93minus_(sports)). For Commander the spike uses one row per deck-seat, y = won - 1/seats, X = 0/1 per card in the deck, ridge with no intercept.
Where it has worked: NBA player evaluation (APM, RAPM and later RPM variants) GK. Not known to be used for Magic cards in public GK.
Cost: Same data as card win rates. Needs many distinct decks so cards are not perfectly collinear: two cards that always appear together cannot be separated by any amount of data. Shrinks rare cards to zero, which is safe but hides real effects until data piles up. No pair or synergy terms; adding them multiplies the parameter count.
Validation: In the fixture world it credits card_strong +0.133 per seat and gives
card_passenger +0.0003 V, where the raw win rate was fooled. A huge penalty (l2 = 1e6) shrinks
all effects toward zero, as it should V. Real use needs cross-validated choice of l2.
Deck-stat regression
How it works: Same conditional logit as Ratings, but each deck is described by a stat vector (fast mana count, average mana value, interaction count, and so on) instead of an identity. It returns one weight per stat, which is logistic regression on deck stats in a pod setting GK. Because weights are shared, it scores decks it has never seen.
Where it has worked: Win-probability models in sport and esports regress outcomes on pre-game features GK. Commander’s “power level” talk and the bracket system’s game-changer list are informal versions: they name stats (fast mana, tutors, extra turns) thought to drive winning V (https://magic.wizards.com/en/news/announcements/introducing-commander-brackets-beta).
Cost: Needs decklists turned into stats, which is a feature-engineering job (SP8 territory). Stats correlate with each other and with pilot skill. Omitted stats bias the included ones.
Validation: On the fixture world it finds fast_mana = 1.01 (true card effect 1.2, diluted by
per-deck noise) and avg_cmc = 0.18, which drives nothing V. Hold-out log loss on new decks is
the real test.
Validation
- Log loss: mean of -ln p(winner), p clipped at 1e-12. Baseline is mean ln(seats), ln 4 = 1.386 for 4-player pods. A proper scoring rule, so honest probabilities score best GK.
- Brier: mean over games of the sum over seats of (p - won)^2. Uniform guessing in a 4-pod scores 0.75 V (https://en.wikipedia.org/wiki/Brier_score).
- Top-1: share of games where the highest-probability deck won. Chance is 0.25.
- Time order: fit on earlier games, score later ones. Random splits leak deck and meta drift.
- Sample size:
games_needed(p0, p1)is the one-sample two-sided normal approximation (https://en.wikipedia.org/wiki/Sample_size_determination GK). It counts deck-seats, not pods. - Known-effect checks: synthetic worlds with planted cards (the fixture world) test whether a method can find a true effect and ignore a passenger. Every method here passes one.
Commander data
- Four-player pods: the outcome is one winner out of four, so each game carries less information than a 1v1 result. Draws and “everybody conceded” outcomes are common and skipped.
- 25% baseline: a card or deck is average at 1/seats, not 50%. Using 0.5 makes every card look bad. Pods of 3 or 5 need per-game baselines, which the spike computes.
- Power-level matching: players choose pods by stated power level or bracket, so strong decks meet strong decks. Win rates then compress toward 25% and understate strength. Ratings and conditional logit correct for opponents only if pods mix across levels often enough to link them.
- Selection bias: records come from players who log games: tournament players (cEDH), app users, or streamers. Their decks, pilots and metas differ from kitchen-table play, and they may log wins more than losses GK (https://en.wikipedia.org/wiki/Selection_bias).
- Few public records: cEDH tournament standings exist (https://edhtop16.com V), but casual game outcomes with full decklists are rare in public. Seat and turn order are rarely recorded, and the spike ignores them (bias named, not modelled). SP1 owns the data inventory.
Spike results
Command: python3 docs/research/sp2-statistical/spike/sp2stats.py report --games G --decks D
--features F on the acceptance fixture world (16 decks, 2,000 four-player games, 20 draws,
card_strong +1.2 log-strength, card_passenger +0). All numbers V, reproduced this session.
| Method | Result |
|---|---|
| Ratings | Spearman 0.98 vs true strength; strong minus weak mean rating about 1.1 (true 1.2) |
| Hold-out (first 1,600 games fit, 396 decided scored) | log loss 1.250 vs baseline 1.386; Brier 0.685; top-1 0.389 |
| Same hold-out with the true strengths | log loss 1.251; top-1 0.417 (ratings are near the ceiling) |
| Card win rate | card_strong 0.347; card_passenger 0.284 (fooled); best filler 0.404 (fooled) |
| Plus-minus (l2 = 1) | card_strong +0.133; card_passenger +0.0003; best filler +0.045 |
| Deck-stat model | fast_mana 1.01; avg_cmc 0.18 |
Sample size: telling a 30% card from the 25% baseline at alpha 0.05 and power 0.8 takes
games_needed(0.25, 0.30) = 610 deck-seats. A 27% card takes 3,736; a 35% card takes 157 V.
So a card needs hundreds of logged games before its win rate means anything, which only a few
hundred staples will reach in any plausible public dataset.
Verdict
Ranked for Commander, best first:
- Ratings (conditional logit or OpenSkill). Most reliable with winner-only pod data and the only one that adjusts for opponents. Needs: games linking decks across pods, several games per deck, and a pilot term if pilots vary. Gives deck strength only.
- Deck-stat regression. Scores new and unplayed decks, so it is the most useful bridge to a deckbuilder. Needs: good stat extraction from decklists, a few thousand games, and hold-out checks on unseen decks. Weights are only as good as the stats chosen.
- Plus-minus (ridge on cards). The right way to get card-level strength from outcomes, since
it separates passengers from winners. Needs: many distinct decklists so cards decorrelate,
hundreds of seats per card, and cross-validated
l2. Pair terms only with far more data. - Card win rates. Cheap and easy to explain, and fine as a screen with a 1/seats baseline and shrinkage. Needs: the same data as plus-minus. Never use alone: co-occurrence fools it, as the passenger card shows.
All four need time-ordered hold-out scoring against ln(4) before any number is trusted.
SP3 Deterministic strength models
Tags: [V] checked this session against the cited source; [GK] general knowledge, not re-checked.
Engine sources read at Forge 7c34365 (2026-11-03), XMage 5a570a6 (2026-11-02), Magarena efa0aba
(last commit 2023-04-24), shallow clones under ~/.cache/mtg-strength/src/. No engine code is copied here.
Q1 How deterministic models work
Three families. All map a card (or a board) to a number with fixed rules; none learn from play at run time.
rules-text. Parse oracle text into structured effects, then value the effects. Deckwright’s
packages/roles reads each ability line, types it (activated, triggered, mode, cost, static) and matches
effect rules, giving roles with a credit in [0, 1] and effect records with cost, speed, scope, repeat and
beneficiary V (packages/roles/src/deckwright_roles/roles.py). Its packages/strength turns those effects
into a value per job and context V. Engines also parse text, but into executable scripts rather than value:
Forge card scripts (wiki) V, and text-to-code
models such as Latent Predictor Networks on MTG and Hearthstone cards (https://arxiv.org/abs/1603.06744) V.
Full rules-text understanding is hard in principle: MTG is Turing complete (https://arxiv.org/abs/1904.09828) V.
heuristic. Hand-weighted formulas over card fields: rarity, type, mana value, power and toughness,
keywords, a removal flag. XMage’s draft bot uses this when a set has no rating file:
rarity base + 2 x type weight + mana-cost score + 40 x removal, scaled by 0.5 V
(Mage.Common/src/main/java/mage/cards/RateCard.java). The usual analogue is the Hearthstone “vanilla test”
(stats per mana) GK. A second kind is the lookup list: Forge ships 193 per-set draft rank files and
XMage 48 rating CSVs, many built from 17Lands data V (forge-gui/res/draft/rankings/,
Utils/gen-draft-rates-17Lands.py).
engine-eval. A static evaluation of a game state used by search: sum life, cards and per-permanent values for each side. Forge, XMage and Magarena all do this (see the engine lines below). Chess engines are the canonical case: material plus positional terms (https://www.chessprogramming.org/Evaluation) V.
Engine evaluation functions, from source:
- Forge forge-ai/src/main/java/forge/ai/CreatureEvaluator.java V: creature value = 80, +20 if not a
token, +15/power, +10/toughness, +5/mana value, then keyword terms (flying +10/power, deathtouch +25,
indestructible +70, hexproof +35, defender minus 9/power + 40, and so on), activated abilities, mana ability +10.
- Forge forge-ai/src/main/java/forge/ai/simulation/GameStateEvaluator.java V: board score = hand-card terms, 2 x team life minus 2 x opponent life, a mana-base term, plus each battlefield card’s value
(creatures via the evaluator above, lands by colours produced, other permanents 50 + 30 x mana value).
- XMage Mage.Server.Plugins/Mage.Player.AI/src/mage/player/ai/score/GameStateEvaluator2.java V: score =
life score difference + permanent score difference + hand-size difference; win = 100,000,000.
- XMage Mage.Server.Plugins/Mage.Player.AI/src/mage/player/ai/score/ArtificialScoringSystem.java V: life
maps through a concave table (0 to 10,000 over 0 to 20 life, +100 per point above 20); creatures score
300 x power + 200 x toughness + ability scores; the card-definition value is a constant 3 marked TODO.
- Magarena src/magic/ai/ArtificialScoringSystem.java V: the same life and poison tables and
300/200 power/toughness weights (XMage’s file is a port), but the card term uses a per-card value= field
from each script, e.g. release/Magarena/scripts/Grizzly_Bears.txt has value=3.745 V. Those values
came from Gatherer community ratings GK.
Q2 Where they have worked
MTG. All three open-source engines ship playable AIs on hand-tuned evaluators and still use them as of
November 2026 V (https://github.com/Card-Forge/forge, https://github.com/magefree/mage,
https://github.com/magarena/magarena). They are good enough for casual play and draft bots, not strong
play GK. XMage’s bot deliberately prefers 17Lands-derived file ratings over its own heuristic:
“any bonuses on top make bot’s picks worse, checked on real human picks from 17Lands” V (RateCard.java).
Deckwright uses a rules-text model in production for Commander role and strength advice V.
Chess. Hand-written evaluation functions with tuned weights powered top engines for decades; Texel’s method fits those weights by logistic regression on game results (https://www.chessprogramming.org/Texel%27s_Tuning_Method) V. Stockfish moved to the learned NNUE evaluation (https://www.chessprogramming.org/NNUE, https://github.com/official-stockfish/Stockfish) V, which is the lesson: deterministic structure plus fitted weights beat hand weights, and learned beat both GK. Classic piece values (1/3/3/5/9) are the material prior every evaluator starts from (https://en.wikipedia.org/wiki/Chess_piece_relative_value) V.
Hearthstone and others. Community stat-per-mana formulas (vanilla test) were used to value cards before deck-tracker win rates; once played-win-rate data existed (HSReplay), it replaced them for strength GK. In Limited MTG the same happened with 17Lands GIH WR (https://www.17lands.com/card_ratings, https://www.17lands.com/metrics_definitions) V.
Q3 Cost
Build. A heuristic is days of work: a formula over Scryfall fields (https://scryfall.com/docs/api) V;
this session’s spike took about 100 lines (spike/build_scores.py) V. An engine evaluator rides on a full
rules engine, which is years of work: Forge and XMage are large multi-module Java projects V. A rules-text
parser is between: Deckwright’s packages/roles is about 5,000 lines of Python at parser version 2.29.0 V.
Per new set. Heuristics need nothing unless new keywords appear. Rules-text parsers need new patterns for new
mechanics; Deckwright tracks this as a parse coverage score per card V (roles.py _coverage). Engines must
script every card before the evaluator sees it GK. Lookup lists need fresh 17Lands data each set V.
Maintain. Weights drift silently; nothing in Forge or XMage re-fits them V (constants in source).
Deckwright re-fits nine named priors by Bradley-Terry on its benchmark V
(packages/strength/src/deckwright_strength/fit.py).
Q4 Accuracy
Engine evaluators. Validated against nothing measurable. Forge has direction-only unit tests (an allied
permanent raises the score; a loss scores terminal) in
forge-gui-desktop/src/test/java/forge/ai/simulation/GameStateEvaluatorTest.java V; XMage and Magarena have no
evaluator tests by file name V. No repo compares evaluator output to outcomes. Accuracy unknown.
XMage heuristic. Its source states a regression over 8,622 cards from all 17Lands sets with
“spearman 0.46” against file ratings V (RateCard.java). The target is 17Lands-derived ratings, not
win rates, and the script is not in the repo, so the number is unverified.
Deckwright strength. Fitted file reports held-out pair agreement 0.881 and Kendall tau 0.642 on
344 hand-judged pairs over 153 cards V (packages/strength/src/deckwright_strength/data/fitted.json).
The pairs are hand-judged, not outcomes.
Spike (this session). 17Lands Premier Draft card ratings for FRA, fetched 2026-11-03, cards with at least 500 games in hand, joined to Scryfall V. Spearman against GIH WR, n = 112:
| Scorer | Spearman |
|---|---|
| rarity + type + removal heuristic (XMage-shaped, own code) | 0.132 |
| stats heuristic (Forge-shaped board value, own code) | -0.048 |
| rarity alone | 0.217 |
| human average pick position (17Lands ATA, negated) | 0.481 |
At 1,000+ games (n = 41) the heuristic reaches 0.210 and the stats heuristic -0.090 V. Sampling noise caps any scorer: estimated reliability of GIH WR here is 0.46 (binomial noise variance / observed variance), so the ceiling is about 0.68 V. A hand heuristic gets under a fifth of that ceiling; human pick order gets about 70%. Historical sets (FDN, TLA) returned zero counts from the public endpoint, so only the live set was usable.
Q5 Deckwright’s current method
Deckwright is a rules-text model in two layers. packages/roles (packages/roles/src/deckwright_roles/roles.py)
parses oracle text into 15 function roles and 14 plan roles with credits in [0, 1]; opponent-decided effects
earn half credit and a Scryfall-tag-only role earns 0.5 V. It predicts what a card does, not how strong it
is. It was checked against Scryfall tagger tags as a recall report (packages/roles/src/deckwright_roles/recall.py,
https://tagger.scryfall.com) and against anchor cards such as Sol Ring as ramp and Swords to Plowshares as
removal (packages/roles/src/deckwright_roles/check_real.py) V. packages/strength then predicts a value
and rank per card, job and context (bracket, players, strategy) from those effects only, never name, price or
popularity V. It was checked against a frozen benchmark of hand-judged pairs and reference cases
(packages/bench), with the numbers in Q4 V. Research note 14 found the earlier trunk ranked Pendant of
Prosperity above Sol Ring and Overkill above every removal spell V.
Gaps: - Never checked against outcome data; the benchmark is 344 hand-judged pairs, so it measures agreement with its authors. - Commander has no per-card win rates, so the Q4 spike cannot be repeated for its domain; SP1 must supply data. - Multiplayer and deck-interaction effects are priors, not measured; note 14 names per-event triggers firing once per turn. - Parse failures become value errors silently (Pendant, Overkill); coverage is tracked but not used to widen uncertainty. - No calibration: values are ranks per job, not win-probability units, so cross-job comparison is unvalidated.
Q6 Where deterministic models fit in a hybrid
Use them where they are strong: cheap, explainable, available on day one of a set. Three roles.
Prior. A deterministic score is a prior for a statistical model (SP2) that shrinks toward it when data is thin, as Texel tuning starts from hand values V. The spike says this prior is weak for raw heuristics (rho 0.13) and only rarity carries much signal, so prior strength should be low and fitted.
Features. Rules-text output (roles, effect scope, speed, cost) is the most useful part: it gives a statistical or learned model inputs that generalise to unseen cards. Deckwright’s effect records are already this V.
Pruning for simulation. Engine evaluators exist to cut search: rank moves and stop rollouts early (SP4, SP5). Here they need only rank boards roughly, which is what Forge, XMage and Magarena use them for V. Card value should come from outcomes (17Lands in Limited, SP1 data in Commander), with deterministic scores as fallback and as explanation GK.
SP4 Decomposed constrained simulation
Verdict: cheap, transparent and exact where it matters, but it measures consistency, not winning. It is a good feature layer for a statistical model (SP2) or a prior for RL (SP5). It cannot stand alone as a strength estimator until fitted to real outcomes (SP1).
How it works
Full Commander simulation needs a rules engine, four agents and politics. Decomposition skips that. It asks narrow questions a deck must answer, each cheap to compute (general knowledge):
- Can it cast its spells on time? (mana)
- Does it see the cards it needs? (draws)
- Can it assemble its win condition? (combos)
- Can it stop opponents? (interaction)
Each answer is a probability in [0, 1]. Questions with a closed form use the hypergeometric distribution. Questions with choices (which land to play, which spell to cast) use a constrained game: a goldfish (no opponent) played for a fixed number of turns, as in Karsten’s Commander curve model1 and Deckwright’s goldfish simulators3. A combiner, fitted on decks with known outcomes, maps the feature vector to one strength score9.
Pieces
Which pieces carry most of deck strength: speed of the win condition first, then interaction
density, then mana consistency. Power-level discussion and cEDH practice treat “turn the deck
can win” and “how much it can stop” as the main axes (general knowledge). In the spike’s
synthetic outcomes, combo_t6 alone reaches held-out Spearman 0.77, the other two under 0.25.
That is by construction of the fixture, so it only shows the method can find a dominant piece.
| Piece | Question | Exact math or simulation |
|---|---|---|
| Mana, generic | k lands by turn t | Exact: hypergeometric1,2 |
| Mana, colored | lands seen can pay pips, one drop a turn | Exact by enumerating land classes (multivariate hypergeometric); Monte Carlo is simpler2 |
| Mana with rocks, ramp, MDFCs | mana on turn t | Simulation: sequencing matters2,3 |
| Draws | cards seen by turn t | Exact under a fixed draw rule; draw engines need simulation1 |
| Combo assembly | every piece seen by turn t | Exact: inclusion-exclusion; tutors need simulation8 |
| Combo castable | pieces seen and mana to cast them | Simulation (joint mana and draw) |
| Interaction | at least one answer by turn t | Exact: hypergeometric |
| Interaction timing | answer held up when needed | Needs an opponent model; simulation or engine4,5 |
| Mulligans | effect on all of the above | Simulation with a mulligan policy6 |
Exact pieces: anything that only asks “is card X among the first n cards”. The draw count rule sets n. Simulation is needed once choices or state enter: sequencing land drops with rocks, casting order, mulligan decisions, tutors, and anything about the opponents (general knowledge).
Combining pieces
Options, from simplest:
- Hand weights. Points per feature, as power-level calculators do. No validation (general knowledge).
- Linear least squares on features against win rate. Used in the spike (D4). Signs are readable; a negative weight on a known-good feature flags a bad feature or confounding9.
- Logistic / Bradley-Terry on pairwise or pod results, if outcomes are game-level (general knowledge).
- Tree models catch interactions (combo speed only matters with protection), but need far more data than a 3-6 feature linear model (general knowledge).
The spike uses ordinary least squares with an intercept, solved by Gaussian elimination in pure Python. Fitted weights on the fixture: on_curve 0.149, interaction_t4 0.084, combo_t6 0.287; the generator’s true weights were 0.15, 0.10, 0.30.
Where it has worked
- Land counts and mana bases. Karsten’s hypergeometric source tables and Commander curve simulator are the standard reference; Deckwright reproduced his model game for game1,3.
- Available-mana simulation beyond the hypergeometric, including rocks2; landlord does the same for 60-card decks10.
- Deckwright goldfish simulators: seeded land-only goldfish for colored untapped mana, and a generic-mana goldfish with rocks, used as consistency checks in its strength report3.
- Constrained games in research: Monte Carlo search over determinized Magic games with simplified rules5; Forge’s AI plays full games but is weak for strength estimation4.
- Card-level outcome decomposition: 17lands “game in hand win rate” credits single cards with outcomes in Limited, which shows feature-to-outcome fitting at scale9.
Nowhere found: a published Commander strength model built from decomposed pieces and validated against pod outcomes. That gap is the question SP1 data would answer (general knowledge).
Cost
- Exact pieces: microseconds per feature. A full profile of a 99-card deck takes under 10 ms in the spike (exact formulas) (general knowledge, measured below).
- Monte Carlo: 20,000 trials of one piece take about 50 ms in pure Python. A full goldfish with rocks and sequencing is about 1 s per 200,000 games per deck in vectorized numpy3.
- Build: pieces are days of work each; a goldfish with ramp and tutors is weeks3.
- Data: the combiner needs labeled decks. That cost dominates (see Validation).
Validation
How the combined score is checked: fit weights on training decks, score held-out decks, report Spearman rank correlation between score and observed win rate, and compare with the best single feature. A combiner that does not beat its best feature adds nothing9.
How much outcome data: two noise sources (general knowledge, binomial arithmetic).
- Per-deck win rate noise. In 4-seat pods the base rate is 0.25. With g games the standard error is sqrt(0.1875 / g): 0.043 at 100 games, 0.019 at 500. Real deck-to-deck spread is likely 0.05-0.15, so each deck needs a few hundred games before its label ranks reliably.
- Number of decks. The standard error of Spearman is about 1/sqrt(n - 1) at rho near 0. Telling rho 0.7 from 0.5 needs about 60-100 held-out decks; the fixture uses 40.
- Confounding. Pilot skill and pod matching move win rate as much as the deck. Without pilot control, a fitted weight can reflect who plays a deck, not the deck (general knowledge).
So: roughly 100+ decks with 200+ logged games each, or a ranking signal (tiers, tournament results) used as an ordinal target. SP1 owns whether such data exists.
Spike results
Code: spike/decomp_sim.py (decomp_sim, stdlib only). Run on the six SP4 fixtures and the
synthetic outcome table (60 train / 40 holdout, 4 seats):
python3 docs/research/sp4-decomposed-sim/spike/decomp_sim.py validate <fixtures dir>
- held-out Spearman: 0.855 (best single feature, combo_t6: 0.768)
- max error vs exact: 0.0056 (11 Monte Carlo checks at 20,000 trials vs exact values)
- run time: 0.58 s (all checks, six profiles, fit and score)
- Fitted weights are all positive and within 0.02 of the generating weights.
- Profiles: mono_40 on_curve 0.788; combo_high_interaction interaction_t4 0.674, combo_t6 0.0136.
Limits:
- Outcomes are synthetic and linear in the features, so 0.855 shows the plumbing works, not that real decks behave this way.
- Only lands make mana (D3); no rocks, ramp, tutors, draw engines or mulligans (D2).
- Interaction is “seen by turn 4”; it ignores timing, targets and what opponents do.
- Draw rule follows the spec (first player skips the turn-1 draw). Comprehensive Rules 103.8 lets every player draw on turn 1 in most multiplayer games7, so on-play values run one card low.
- Exact formulas replace Monte Carlo inside
profile; Monte Carlo is kept for checks and for future pieces with choices.
Scorecard
| Criterion | Rating | Evidence |
|---|---|---|
| accuracy evidence | Low | Exact pieces are correct; no real outcome validation exists for the combined score |
| validation target | Held-out Spearman vs pod win rate | Must beat best single feature; 0.855 on synthetic data |
| data needs | High for combiner, none for pieces | ~100+ decks x 200+ games, or an ordinal tier target |
| compute cost | Very low | Under 1 s per deck for pieces; full goldfish about 1 s per 200k games |
| multiplayer fit | Medium-low | Pieces are solitaire; politics, threat assessment and stack fights are missing |
| build effort | Low to medium | Spike: one file; with ramp, tutors, mulligans: weeks |
Where pieces fail: politics (who gets attacked or answered), stack interaction (counter wars, protection), multiplayer threat dynamics, and synergy that is not a named combo. These need a two-player engine with scripted opponents at least, or self-play (SP5).
Constrained-game options, cheapest first: goldfish to turn N; goldfish with a scripted “answer on sight” opponent; two-player engine games for a fixed number of turns; four-player engine games. Each step adds opponent realism and cost by roughly 10-100x.
Sources
- Frank Karsten, MTG-Math,
optimal_curve_commander.pyand source-count tables. https://github.com/frankkarsten/MTG-Math - EDHREC, “Simulating Available Mana: Beyond the Hypergeometric Distribution”. https://edhrec.com/articles/simultaing-available-mana-beyond-the-hypergeometric-distribution
- Deckwright:
docs/research/10-simulation-study.md,packages/analysis/src/deckwright_analysis/goldfish.py,packages/manabase/src/deckwright_manabase/simulate.py(worktree pilot-lr4). - Forge AI documentation. https://github.com/Card-Forge/forge/wiki/ai
- P. Cowling, C. Ward, E. Powley, “Ensemble Determinization in Monte Carlo Tree Search for the Imperfect Information Card Game Magic: The Gathering”, IEEE TCIAIG 4(4), 2012.
- Mulligan rules (Commander free mulligan, London mulligan). https://mtg.wiki/page/Mulligan
- Magic: The Gathering Comprehensive Rules, rule 103.8. https://magic.wizards.com/en/rules
- Commander Spellbook, combo database. https://commanderspellbook.com
- 17lands public datasets and card metrics (game in hand win rate). https://www.17lands.com/public_datasets
- landlord, mana base simulator. https://github.com/mtgoncurve/landlord
Self-play RL for Commander strength (SP5)
Research only. No agent, environment or training run was built. Numbers come from
spike/rl_cost.py applied to SP7’s measured forge-speed.json10.
How it works
- An agent plays copies of itself (or a pool of past selves) and learns a policy (what to do) and a value (who will win) from the results1,8.
- For strength measurement, a well-trained agent piloting a deck gives a deck’s ceiling win rate, not the autopilot floor that scripted AI gives. MageZero showed this on XMage: one deck rose from 16% to 66% against minimax bots6.
- Card strength then follows by ablation: swap a card, retrain or fine-tune, re-measure (general knowledge).
Algorithms
- MCTS + network (AlphaZero family)1: search guided by a policy/value net. Best fit for Magic’s long turns and huge, structured action space. MageZero already runs it on XMage6.
- ISMCTS7: MCTS over information sets, sampling hidden cards per iteration. Ensemble determinization was studied on Magic itself11. Cheap, no training; a strong baseline and the search half of any learned agent.
- Self-play PPO8: model-free policy gradient. Simple and multiplayer-friendly (OpenAI Five12), but sample-hungry; with Forge at hundreds of games/hour it would not converge (see Compute cost).
- CFR-style search (ReBeL, Student of Games)3,4: sound for hidden information, with guarantees only in two-player zero-sum. Public belief states over a 99-card library and hands are far larger than poker’s; not practical here (general knowledge).
- Fit: AlphaZero-style MCTS with ISMCTS-style sampling of hidden cards, trained per deck against a pool, as MageZero does6,7. PPO only if a fast engine exists.
Hidden information
- Determinization: sample opponents’ hands and library orders consistent with what is known, search each sample, vote. Simple but suffers strategy fusion and leaks: MageZero’s search sees future draws, so its numbers flatter reactive decks6,7.
- Information sets (ISMCTS): one tree over what the acting player can observe; removes most leakage at some cost in strength per simulation7.
- Belief states: ReBeL and Student of Games search over public belief states3,4; the state for Commander (four 99-card libraries, hidden hands) is intractable without heavy abstraction.
- Recommendation: information-set sampling at search time, network inputs without hidden cards (MageZero already strips them from the net)6,7.
Multiplayer
- At 2 seats the game is zero-sum and self-play has equilibrium guarantees (ReBeL, Student of Games)3,4. Every extra seat breaks them.
- At 3 to 6 seats the game is non-zero-sum between subsets: no Nash guarantee holds, yet Pluribus still beat pros at 6-player poker with self-play plus search2.
- Kingmaking: a losing player decides who wins. Win-rate labels then depend on table behaviour, not deck strength alone (general knowledge).
- Politics: deals, threat assessment and “attack the leader”. CICERO needed human data and language to play Diplomacy’s 7-player cooperation well; pure self-play learned strategies humans would not follow5.
- As seats grow: games get longer (D2, linear in this model), the action space and hidden state grow, credit assignment gets noisier, and the base win rate falls to 1/seats, so evaluation needs more games per deck to see the same gap (see Validation)2 (general knowledge).
- MageZero is two-player only; its graph encoding keeps the player node generic for later6.
Basic guidance
Intake Q4: pure self-play or bootstrap from existing play? - Pure self-play from random play works in perfect-information two-player games (AlphaZero)1, but random Magic play rarely finishes a Commander game in a 120 s clock (general knowledge). - Bootstrap from Forge AI: imitate Forge’s scripted AI first, then improve by self-play. AlphaStar bootstrapped from human replays then used league self-play9; MageZero starts against minimax bots and adds an opponent pool6. - Human data: CICERO regularized toward human play (piKL) to stay compatible with people5. Commander logs with decisions are not public at scale (general knowledge). - Recommendation: bootstrap from Forge AI and keep a fixed Forge-AI opponent seat as an anchor; pure self-play at 4+ seats risks table conventions no human table shares5,9.
Engine and environment
- Wrap Forge: best rules coverage and four-player Commander. Measured: 264.5 games/hour
at 8 workers, 4 seats, clock 120 s (
forge-speed.json)10. Many games hit the clock, so real game time is longer and this rate is an upper bound10. - Wrap XMage: MageZero’s route, with a working JVM-to-Python bridge (HTTP + msgpack) and about 800 two-player games/hour on 16 threads6.
- Build a fast engine: AlphaZero-class training needs millions of games1; with Forge that is months (see Compute cost). A rules engine covering Commander is a multi-year effort; a 50x speedup is an assumption, not a measurement (D4, general knowledge).
- Learned agents add network inference per decision; an MCTS budget of hundreds of simulations per move multiplies engine cost again6 (general knowledge). The cost table ignores this, so it is a floor.
Compute cost
From forge-speed.json10 via rl_cost.py table. Throughput scales linearly with workers
and speedup; game length linearly with seats (D2, a stated limit). Workers capped at 20 threads.
| Scenario | Games | Seats | Workers | Speedup | Games/h | Hours | Days |
|---|---|---|---|---|---|---|---|
forge-2p-deck |
20,000 | 2 | 20 | 1 | 1,322.7 | 15.1 | 0.6 |
forge-4p-deck |
20,000 | 4 | 20 | 1 | 661.4 | 30.2 | 1.3 |
forge-6p-deck |
20,000 | 6 | 20 | 1 | 440.9 | 45.4 | 1.9 |
forge-4p-general |
1,000,000 | 4 | 20 | 1 | 661.4 | 1,512.1 | 63.0 |
fast-4p-general |
1,000,000 | 4 | 20 | 50 | 33,067.5 | 30.2 | 1.3 |
fast-6p-alphazero |
20,000,000 | 6 | 20 | 50 | 22,045.0 | 907.2 | 37.8 |
Rationale for game counts: - 20,000 per deck: MageZero runs about 1,000 games per generation for many generations per deck6; 20 generations is the scale its README reports. - 1,000,000 general: a general agent across decks needs orders of magnitude more; Pluribus used 8 days on 64 cores2; one million is a modest floor (general knowledge). - 20,000,000: AlphaZero used 21 to 44 million self-play games per game1. - Speedup 50 is an assumption for a lean engine without Forge’s GUI-era overhead (D4, general knowledge). - Excluded: MCTS simulations per move, network inference, retraining per card swap. Each multiplies the hours above6 (general knowledge).
Hardware
- This machine: 20 CPU threads, Intel Arc A770M 16 GB, no NVIDIA, so no CUDA (intake §10).
- Engines (Forge, XMage) are CPU-bound JVMs; the 20 threads cap workers10.
- Networks: PyTorch supports Intel GPUs through the XPU backend13; small policy/value nets (MageZero-sized) train and infer on the A770M. Some CUDA-only tooling will not run (general knowledge).
- The bottleneck is game generation on CPU, not the GPU; Pluribus trained with no GPU at all2.
Where it has worked
- AlphaZero1: chess, shogi, Go from pure self-play; perfect information, two players.
- Pluribus2: six-player no-limit poker beat pros; hidden information, multiplayer, CPU only.
- ReBeL / Student of Games3,4: one algorithm for perfect and imperfect information, two-player.
- CICERO5: human-level 7-player Diplomacy with negotiation; needed human data.
- AlphaStar / OpenAI Five9,12: huge action spaces and hidden state, at enormous compute.
- MageZero6: AlphaZero-style per-deck agents on XMage; two-player Magic, Commander weakly.
- Magic MCTS research11: determinized MCTS played Magic better than rule-based play.
- Nothing has shown learned agents for four-player Commander (general knowledge).
Validation
- Agent win rates become deck strength by playing each deck many times at a fixed table of reference opponents (a Forge-AI anchor plus agents) and reading its win rate against 1/seats2,6.
- Card strength: win-rate change when a card is swapped, under the same agents (general knowledge).
- Sample size:
games_for_ciwith the normal approximation;games_to_rankneeds each deck’s rate within half the gap (general knowledge). - games per deck to separate 5 points at 4 seats: 1153
- At 2 seats it is 1,537 and at 6 seats 854 games. At 661 games/h (4 seats, 20 workers) that is about 1.7 h per deck in Forge10.
- Check against SP1 outcome data (real tables) and SP2 statistics; agent ceilings should rank decks the same way human win rates do, or the method fails (general knowledge).
Verdict
Defer. Learned self-play agents are the only method that measures a deck’s ceiling under skilled play, and per-deck AlphaZero-style training is affordable on Forge (15-45 h per deck, 2 to 6 seats)6,10. But a general agent across decks is months on Forge, hidden information still leaks, kingmaking and politics make multiplayer labels noisy, and nothing has shown it for four-player Commander2,5. Build it only if (1) cheaper methods (SP2-SP4) leave a validated gap for high-skill decks, (2) SP1 provides outcome data to check agent rankings, and (3) either per-deck scope is accepted or a 50x-faster engine exists. Start from MageZero’s design on a duel, with ISMCTS sampling and a Forge-AI anchor seat6,7.
Scorecard
| Criterion | Rating | Note |
|---|---|---|
| accuracy evidence | medium | Strong in games like poker and Go1,2; none for 4-player Commander |
| validation target | 1,153 games/deck | 5-point gap at 4 seats, about 1.7 h per deck in Forge |
| data needs | low | Self-play needs no outcome data; SP1 needed only to validate |
| compute cost | high | 15-45 h per deck; general agent 63 days on Forge, 1.3 days at 50x |
| multiplayer fit | low-medium | No equilibrium past 2 seats; kingmaking, politics2,5 |
| build effort | high | Bridge, encoding, training loop; months; fast engine is years |
Sources
- Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play”, Science 2018. https://www.science.org/doi/10.1126/science.aar6404
- Brown and Sandholm, “Superhuman AI for multiplayer poker” (Pluribus), Science 2019. https://www.science.org/doi/10.1126/science.aay2400
- Brown et al., “Combining Deep Reinforcement Learning and Search for Imperfect-Information Games” (ReBeL), NeurIPS 2020. https://arxiv.org/abs/2007.13544
- Schmid et al., “Student of Games: A unified learning algorithm for both perfect and imperfect information games”, Science Advances 2023. https://www.science.org/doi/10.1126/sciadv.adg3256
- Meta FAIR Diplomacy Team, “Human-level play in the game of Diplomacy by combining language models with strategic reasoning” (CICERO), Science 2022. https://www.science.org/doi/10.1126/science.ade9097
- MageZero, AlphaZero-style agents on XMage. https://github.com/WillWroble/MageZero ; SP7 note
docs/research/sp7-engines-ai/magezero.md - Cowling, Powley and Whitehouse, “Information Set Monte Carlo Tree Search”, IEEE TCIAIG 2012. https://doi.org/10.1109/TCIAIG.2012.2200894
- Schulman et al., “Proximal Policy Optimization Algorithms”, 2017. https://arxiv.org/abs/1707.06347
- Vinyals et al., “Grandmaster level in StarCraft II using multi-agent reinforcement learning” (AlphaStar), Nature 2019. https://www.nature.com/articles/s41586-019-1724-z
- SP7 measured Forge speed:
docs/research/sp7-engines-ai/forge-speed.jsonandforge-speed.md(this repo). - Cowling, Ward and Powley, “Ensemble Determinization in Monte Carlo Tree Search for the Imperfect Information Card Game Magic: The Gathering”, IEEE TCIAIG 2012.
- OpenAI et al., “Dota 2 with Large Scale Deep Reinforcement Learning” (OpenAI Five), 2019. https://arxiv.org/abs/1912.06680
- PyTorch, “Getting Started on Intel GPU” (XPU backend). https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html
SP6 Card encoding comparison
Verdict: keep — Deckwright C21 stays the encoding; borrow Forge scripts as a read-only cross-check oracle, and MTGJSON/Scryfall for identity, oracle text and legality.
Comparison
Criteria are D1. “Strength features” means the six-feature vocabulary of D3 (spikes/sp6/features.py).
| Encoding | Executable | Coverage | Licence | Maintenance | Strength features | Extraction cost |
|---|---|---|---|---|---|---|
| Deckwright C21 | No; declarative effect rows read by C22 strength (pilot-lr4 packages/roles/src/deckwright_roles/store.py) |
35,353 cards, 23,766 with roles (roles.sqlite meta, parser 2.28.0) |
Private, ours (repo path) | Ours; parser 2.29.0 at pilot-lr4 3b5c002 |
All six named directly, plus decider, payer, drawbacks (store.py Effect) |
Already a table; one query |
| Forge | Yes; scripts drive the Forge rules engine | 33,980 cardsfolder scripts at forge-2.0.15 (GitHub tree count) |
GPL-3.0 (LICENSE); read as data only (D5) | Active community; new sets scripted fast (GK) | Five of six by API name; tutor and ramp blur into ChangeZone (cross-check) |
Low: line format, 76-line stdlib parser (spikes/sp6/forge_script.py) |
| XMage | Yes; one Java class per card (SolRing.java) | Near-complete card pool (GK) | MIT (LICENSE) | Active (GK) | Implicit in effect classes; needs Java reflection or AST work | High: Java build or source parsing (named, not built) |
| MTGJSON | No; data only | Every printed card, atomic and per-printing (card-atomic) | MIT, verified (LICENSE.txt, licence page) | Regular builds (GK) | None beyond keywords and oracle text | Low for identity; semantics need a parser like ours |
| Scryfall | No; data only | Every card, daily bulk exports (bulk data) | Free for software and research under the WotC Fan Content Policy, verified (API docs) | Daily (bulk-data page) | Keywords, oracle text, legality only (cards) | Low for identity; semantics need a parser like ours |
Not tabled: community grammars such as demystify parse oracle text but are partial and unmaintained (GK). The Ling et al. 2016 card-to-code corpus (paper, data) pairs MTG card text with XMage Java; it suits code generation, not strength features.
Cross-check
Report: spikes/sp6/out/crosscheck-report.json. Forge forge-2.0.15 scripts against Deckwright pilot-lr4 3b5c002 (parser 2.29.0), on 83 matched Commander staples (spikes/sp6/sample.txt). None unmatched.
Agreement is 67% (70 shared of 105 feature hits). By feature (both / Forge-only / Deckwright-only):
| Feature | Both | Forge only | Deckwright only | Cause of disagreement |
|---|---|---|---|---|
| mana | 28 | 0 | 10 | Mapping gap: Forge uses ManaReflected (Fellwar Stone, Chrome Mox), ChangeZone for land ramp, Token for Treasure, Untap (Treachery) |
| draw | 16 | 2 | 2 | Mapping gap: Brainstorm and Ponder are card_selection in C21. Forge-only concept: Necropotence and Sylvan Library draw by zone moves |
| tutor | 5 | 3 | 7 | Mapping gap: R2 counts only Library→Hand; top-of-library (Vampiric) and to-battlefield (Finale) tutors miss; Cultivate and Fact or Fiction hit |
| veto_cost | 8 | 1 | 0 | Forge-only concept: UnlessCost covers any payer; Sylvan Library’s owner pays 4 life, which C21 calls life_loss |
| enters_tapped | 13 | 0 | 0 | None |
| gain_control | 0 | 10 | 0 | Mapping gap: D3 reads opponent_controls; C21 encodes theft as kind steal. Deckwright parser miss: Donate and Harmless Offering get no effect |
No Forge parser misses: every Forge-side hit traced to a real script param. 35 of 105 hits disagree; 33 are mapping gaps or Forge-only concepts, and the only real Deckwright misses are two give-away cards.
Why keep
- C21 already names what strength needs (veto payer, decider, drawbacks); Forge leaves these implicit in params.
- Forge is executable but GPL; we may read it as data only, so it cannot be our encoding (D5).
- XMage needs Java tooling; MTGJSON and Scryfall carry no effect semantics.
- Forge earns its place as an oracle: a stdlib parser plus this cross-check finds C21 parser misses cheaply.
For SP8
Strength models can rely on C21 features as is; agreement gaps are mapping choices, not wrong data. Re-run the cross-check after each parser bump to catch misses like Donate before they skew strength. Treat tutor and steal effects with care: their C21 encoding is finer than the shared vocabulary.
Sources
- https://github.com/Card-Forge/forge/tree/forge-2.0.15/forge-gui/res/cardsfolder
- https://raw.githubusercontent.com/Card-Forge/forge/forge-2.0.15/LICENSE
- https://github.com/Card-Forge/forge/wiki/Card-scripting-API
- https://raw.githubusercontent.com/magefree/mage/master/LICENSE.txt
- https://raw.githubusercontent.com/magefree/mage/master/Mage.Sets/src/mage/cards/s/SolRing.java
- https://raw.githubusercontent.com/mtgjson/mtgjson/master/LICENSE.txt
- https://mtgjson.com/license/
- https://mtgjson.com/data-models/card/card-atomic/
- https://scryfall.com/docs/api
- https://scryfall.com/docs/api/bulk-data
- https://scryfall.com/docs/api/cards
- https://company.wizards.com/en/legal/fancontentpolicy
- https://github.com/Zannick/demystify
- https://arxiv.org/abs/1603.06744
- https://github.com/google-deepmind/card2code
- Repo: pilot-lr4
packages/roles/src/deckwright_roles/store.py,roles.py(Deckwright C21) - Repo:
spikes/sp6/(parser, features, cross-check, sample, report)
SP7 Open-source engines and AI
Notes, each read from pinned source: forge.md, xmage.md, magarena.md,
magezero.md, argentum.md. Speed: forge-speed.md;
record in forge-speed.json (measured on the host, 2026-11-03).
Tool: spikes/sp7/forge_speed.py.
Lessons
For a strength estimator (ranked):
1. Engine win rates measure deck x pilot. Every engine AI here is heuristic or shallow search, strongest with
linear aggro and weakest with combo and reactive decks (forge.md, magezero.md). Treat engine win rates as a biased feature, not ground truth.
2. Check what the pilot can see. XMage’s default bot and Argentum’s bot use the full hidden state; Forge’s sim
path has small per-card peeks. Peeking inflates interactive decks’ win rates (xmage.md, argentum.md).
3. Pin the build. Forge’s AI changed a lot after 2.0.15; numbers compare only within one tag (forge.md).
4. Forge scores clock-capped games as wins, not draws. Raise the clock or flag capped games, or slow decks get noisy wins (forge-speed.md).
5. Magarena’s DeckStrCal (two decks, fixed AI, N games) is the simplest estimator template; its honest/cheat
AI pairs show how to measure the peeking bias (magarena.md).
For an RL environment (ranked):
1. Immutable state with cheap fork (Argentum) beats whole-game copies (Forge, XMage) for search speed.
2. Mask hidden zones in the observation by default (Argentum gym). Determinized search still leaks (magezero.md).
3. Bridge over local HTTP with batched inference (MageZero) rather than linking into a GPL JVM.
4. Forge is the only engine with a working four-player Commander AI today; a PlayerController socket
bridge covers about 110 decision methods, with heuristic fallback per method.
5. Deck-local agents (MageZero) are proven in 1v1 at about 800 games/hour; four-player Commander costs more.
Open questions answered
Q1
Forge 2.0.15, four-player Commander, 8 workers, 120 s clock: 264.5 games/hour (32 games, one precon x4; median 104 s). This is an upper bound: capped games count as wins and many games hit the clock. About 3.8 h per 1,000 games (forge-speed.md).
Q2
PlayerController (abstract, about 110 methods) is the hook; PlayerControllerAi implements it. In sim,
simulation is off, so the heuristic path runs. It is mostly honest but peeks in ChangeZoneAi, chooseCardName and ClashAi.
At the tag the simulation path copies real hidden cards (PRUNE_HIDDEN_INFO = false).
Q3
MageZero’s Java side calls a Python Flask/waitress server on 127.0.0.1 over HTTP with msgpack and batched GPU inference.
It supports Commander only as two-player (game_mode: commander, player_a/player_b); multiplayer is unsupported per its FAQ.
Q4
XMage’s default bot is “Computer - mad” (ComputerPlayer7 proxy, minimax, 5,000-node cap) per config.xml.
It searches full game copies, so it sees hidden cards. Four-player behaviour was not tested (D6: read, not run).
Q5
Forge is GPL-3.0: an agent linked into the JVM becomes a GPL derived work. A separate process talking over a socket keeps our code under our own licence (general knowledge, not legal advice). XMage, MageZero and Argentum are MIT.
Hidden information compared
| Engine | Default AI sees hidden cards? | Honest option | Note |
|---|---|---|---|
| forge | Mostly no; small per-card peeks | Heuristic path | Simulation copies real hands at 2.0.15 |
| xmage | Yes (MAD bot, full copy) | MCTS bot determinizes | Default bot is MAD |
| magarena | No (default variants hide) | Built in; “(cheat)” variants peek | 1v1 only, dormant |
| magezero | Search leaks; net input masked | hiddenInfo toggle (unread) |
Author calls it unsolved |
| argentum | Yes (bot reads unmasked state) | Gym masks by default | RL interface is honest |
Forge AI
Source read: Card-Forge/forge@4ec5f1a2c32fa90ecb983a72b9eb47aa5c5d7676
Pinned tag forge-2.0.15 (peeled commit above), shallow clone read on 2026-11-03. No code copied.
Where the code lives
- Module
forge-ai, packageforge/ai: about 30 top-level classes plusability/(one*Aiclass per effect type) andsimulation/(11 files at this tag). Read:forge-ai/src/main/java/forge/ai/AiController.java,forge/ai/PlayerControllerAi.java,forge/ai/simulation/GameCopier.java,forge/ai/simulation/SimulationController.java. - Headless entry:
forge-gui-desktop/src/main/java/forge/view/SimulateMatch.java. - AI profiles:
forge-gui/res/ai/Default.ai,Cautious.ai,Reckless.ai,Experimental.ai. - The intake scan (03) listed
HiddenInfoShuffler,CommanderPlan,FlatMonteCarloPickerfrom master. None of them exist at the pinned tag; they are post-2.0.15 work (https://github.com/Card-Forge/forge/tree/master/forge-ai/src/main/java/forge/ai/simulation).
Decision method
- Default path is hand-written heuristics:
AiController.javaasks per-effect classes inforge/ai/ability/(viaSpellApiToAi.java) whether and how to play each spell; combat is inAiAttackController.javaandAiBlockController.java; card value comes fromCreatureEvaluator.javaandComputerUtilCard.java. - Profiles tune many switches (
AiProps.java), not a search budget. - Optional look-ahead:
AIOption.javahasUSE_HYBRID_SIMULATIONandUSE_FULL_SIMULATION.SimulationController.javasearches toDEFAULT_MAX_DEPTH = 3plays, scoring states withGameStateEvaluator.java. No time budget. SimulateMatch.javacreates AI players throughGamePlayerUtil.createAiPlayer(name, index, profile), which passes no AI options. Sosimruns the pure heuristic AI, simulation off.
Hidden information
- Simulation path peeks at this tag:
GameCopier.javahasPRUNE_HIDDEN_INFO = false, so the copied game holds the real opponent hands and libraries. A face-down-exile fix stops one leak, not the rest. - Heuristic path mostly reasons from visible state, with known leaks:
forge/ai/ability/ChangeZoneAi.java(“OppType” logic) scans opponents’ library and hand, justified in a comment as “deck list is known”;PlayerControllerAi.javachooseCardNamereads the strongest opponent’s library;forge/ai/ability/ClashAi.javareads opponents’ top card. Hand sizes are used openly (legal). - Net: Forge’s AI is mostly honest in
sim, with small per-card cheats. Settles intake Q2.
Multiplayer and Commander
SimulateMatch.javaseats one AI per-ddeck;-f Commandermakes each aRegisteredPlayer.forCommander. Four decks give a four-player free-for-all.- Commander awareness at the tag is light: commander checks inside
AiAttackController.java,AiBlockController.java,ComputerUtilCard.java,ComputerUtilCombat.java(commander damage, recasting). No archetype planner (that is master-onlyCommanderPlan). - Threat choice uses helpers like
getWeakestOpponent/getStrongestOpponent(PlayerControllerAi.java): a crude multiplayer politics model.
Headless use and speed
forge sim -d d1 d2 d3 d4 -f Commander -n N -c 120 -qprints oneGame Result:line per game (SimulateMatch.java). Default clock 120 s; a capped game is still scored as a win (measured run, forge-speed.md).- Telemetry:
forge-gui-desktop/src/main/java/forge/view/Main.javainitialises Sentry. Runs must have networking removed (deckwright note 11 conditions; our wrapper isunshare -rn). - Speed: 264.5 games/hour, 8 workers, 120 s clock, one precon x4 (
forge-speed.md). Upper bound, since capped games count. Needsres/in the working dir and-Duser.homeunderunshare -r.
Learning-agent hook
forge-game/src/main/java/forge/game/player/PlayerController.javais an abstract class with about 110 abstract methods (every choice the rules can ask).PlayerControllerAi.javaimplements it for the AI. An RL agent would be a third implementation that forwards decisions over a socket.- 110 decision types is the real cost: a bridge must cover each or fall back to the heuristic AI. Falling back per method is the cheap path (general knowledge).
- No existing RL wrapper found (intake scan 03).
Licence and activity
- GPL-3.0 (
LICENSEin repo). Linking our agent into the JVM makes a derived work; a separate process talking over a socket keeps our code separate (general knowledge, not legal advice). Settles intake Q5. - Very active: 2.0.15 released 2026-09-28, many commits per day on master (https://github.com/Card-Forge/forge/commits/master).
Lessons
- For a strength estimator: Forge win rates measure “how well the heuristic AI pilots this deck”, which favours linear aggro/midrange (https://github.com/Card-Forge/forge/wiki/AI). Treat as a biased signal.
- Pin the tag: AI behaviour changes fast (simulation package rewritten after 2.0.15), so results are only comparable within one build.
- The 120 s clock turns slow games into draws; count timeouts, or slow combo decks look like draw machines.
- Hook via a
PlayerControllersubclass in a separate GPL fork plus a socket, not by linking.
Forge four-player Commander speed
Measured 2026-11-03 on the host (the CEO ran the build and games; GAP-29). Record: forge-speed.json.
Result
| Field | Value |
|---|---|
| Build | forge-2.0.15 (commit 4ec5f1a2c32f), OpenJDK 17.0.21 |
| Setup | 8 workers, 32 games, Arcane Wizardry (C17 precon) x4, -c 120, unshare -rn |
| Throughput | 264.5 games/hour (wall 435.5 s) |
| Game time | mean 95.3 s, median 104.0 s, p90 120.2 s |
| Seat wins | 7 / 8 / 11 / 6; timeouts reported 0 |
Projection (project_hours): 1,000 games take about 3.8 h at 8 workers. Treat this as a lower bound on time.
Caveats
- Clock-capped games count as wins, not draws. Forge picks a winner when the 120 s clock runs out,
so
timeouts = 0undercounts. The median sits near the clock, so many games were capped. 265/hour is an upper bound for capped games, not a free-play rate. Capped wins also add seat noise. - One deck, one run, 32 games. JIT warm-up is included (two games per JVM). Other decks will differ: aggro ends fast, control and combo hit the clock.
games_per_houris whole-machine wall clock with start-up, as SP5 needs. Linear scaling past 20 host threads and about 1-2 GB per JVM is optimistic (general knowledge).- Intake estimate for comparison: MageZero on XMage, about 800 1v1 games/hour on 16 threads (https://github.com/WillWroble/MageZero).
Method (as run)
- Build at the tag:
mvn -pl forge-gui-desktop -am package -DskipTestswith JDK 17. Takeforge-gui-desktop/target/*-jar-with-dependencies.jar. - Run folder
forge-run/: copy the jar there asforge.jarand symlink Forge’sres/into it. Forge needsres/in its working directory. - Under
unshare -rForge runs as uid 0, so setJAVA_TOOL_OPTIONS=-Duser.home=<forge-run>/homeand put the decks inhome/.forge/decks/commander/. - Measure (8 workers, 2 batches of 2 games each per worker = 32 games):
python3 spikes/sp7/forge_speed.py measure --jar forge-run/forge.jar --java <jdk>/bin/java --workers 8 --checkpoint-dir ck D D D D > run.json - Record:
python3 spikes/sp7/forge_speed.py record --out docs/research/sp7-engines-ai/forge-speed.json run.json(validates before writing).
Each process is unshare -rn java -jar forge.jar sim -f Commander -n 2 -q -d D D D D -c 120
(build_sim_command). Each batch is killed after 900 s (R3). A cut run resumes from ck/w*.jsonl.
How SP5 uses it
project_hours(run, target_games, workers) = target / (games_per_hour x workers / run.workers).
For free-play rates, rerun with a longer clock (or none) and compare the median against 120 s.
XMage AI
Source read: magefree/mage@5a570a6fc9bed8f671ce7395047f04e19882f400
Master as of 2026-11-02, read with git show on a sparse clone. No code copied.
Where the code lives
Mage.Server.Plugins/Mage.Player.AI/src/mage/player/ai/ComputerPlayer.java: base rule-based AI, plusPermanentEvaluator.java,CombatEvaluator.java,score/GameStateEvaluator2.java.Mage.Server.Plugins/Mage.Player.AI.MAD/src/mage/player/ai/ComputerPlayer6.javaandComputerPlayer7.java: tree search over simulated games (SimulationNode2.java,SimulatedPlayer2.java).Mage.Server.Plugins/Mage.Player.AIMCTS/src/mage/player/ai/ComputerPlayerMCTS.java,MCTSNode.java: Monte Carlo tree search.- Bot registry:
Mage.Server/config/config.xmllists “Computer - mad” (ComputerPlayerControllableProxy) and “Computer - monte carlo” (ComputerPlayerMCTS).
Decision method
- MAD (ComputerPlayer6/7): minimax-style search over copies of the game. In
ComputerPlayer6.javadepth is 4 below skill 4, else equal to skill; think time is skill x 3 s; nodes capped atMAX_SIMULATED_NODES_PER_CALC = 5000. This confirms the 5,000-node figure intake scan 03 left unverified. - Leaves scored by
score/GameStateEvaluator2.java(life, permanents, hand) - heuristic, hand-tuned. - MCTS: UCT selection, random playouts (
MCTSNode.java). A 2015 developer post says it has no chance nodes (https://www.slightlymagic.net/forum/viewtopic.php?p=173969).
Hidden information
- MAD peeks:
ComputerPlayer6.javacreateSimulationcallsgame.createSimulationForAI(), which inMage/src/main/java/mage/game/GameImpl.javais a plain full copy. Opponent hands and library order are real. - MCTS is honest-ish:
MCTSNode.javarandomizePlayersshuffles every library and swaps opponents’ hands for random library cards before each simulation (determinization, general knowledge). - So XMage’s default bot plays with perfect information.
Multiplayer and Commander
- Game types as plugins:
Mage.Server.Plugins/Mage.Game.CommanderFreeForAll/src/mage/game/CommanderFreeForAll.java, plus Commander Duel, Brawl, Oathbreaker. - The AI runs in multiplayer (it is the server’s bot), but its search treats opponents as one side (general knowledge from reading the node loop; not measured). No Commander-specific AI code found.
- Settles intake Q4: default bot is “Computer - mad” (first AI in
config.xml, a ComputerPlayer7 proxy). Four-player Commander behaviour was not tested (D6: read, not run).
Headless use and speed
- No CLI. AI-vs-AI runs through JUnit:
Mage.Tests/src/test/java/org/mage/test/load/LoadTest.java. - Speed: none published for upstream. MageZero’s fork reports about 800 games/hour on 16 threads, 60-card 1v1 with its own MCTS (https://github.com/WillWroble/MageZero).
Learning-agent hook
- Each bot is a
Playerimplementation loaded fromconfig.xml; a new player type in a plugin jar is the clean hook. MageZero did exactly that (Player.AI.RL,Player.AI.MCTSin https://github.com/WillWroble/mage). - Game state copies (
createSimulationForAI) give search for free but are slow; MageZero reports about 300 simulations/s per thread.
Licence and activity
- MIT (
LICENSE.txt, https://github.com/magefree/mage). Easiest licence for a linked agent. - Very active: release xmage_1.4.63V1 on 2026-10-31 (https://github.com/magefree/mage/releases).
Lessons
- Do not use the MAD bot as an honest benchmark: it sees hidden cards (
ComputerPlayer6.java). Its win rates overstate interactive decks. - The MCTS bot’s determinization is the right pattern but crude (uniform resampling ignores what was revealed).
- XMage is the best RL host by licence and precedent, but has no Commander evidence and no CLI.
- Node-capped search (5,000) makes play quality depend on how branchy a board is: big Commander boards hit the cap fast.
Magarena AI
Source read: magarena/magarena@efa0aba85e681816a92b4938b28741d540384e35
Master, last commit 2023-04-24, read with git show. No code copied.
Where the code lives
src/magic/ai/: 16 files.MCTSAI.java,MMAB.java(minimax alpha-beta),MTDF.java,VegasAI.javawithVegasWorker.java, scoring inArtificialScoringSystem.javaandArtificialScoreBoard.java.- AI menu:
src/magic/ai/MagicAIImpl.java. Headless runner:src/magic/DeckStrCal.java.
Decision method
- Four searchers: minimax with alpha-beta (
MMAB.java), MTD(f) (MTDF.java), Monte Carlo tree search (MCTSAI.java, UCT with uniform random playouts, time budget of one second per AI level), and “Vegas” (VegasAI.java, flat multithreaded random playouts). MagicAIImpl.javaalso defines a fast minimax used only for deck-strength runs.- Board scoring is hand-tuned (
ArtificialScoringSystem.java).
Hidden information
- Every searcher has an honest and a cheat variant in
MagicAIImpl.java(for example “monte carlo tree search” and “monte carlo tree search (cheat)”). - Honest mode:
MCTSAI.javacallshideHiddenCards()when not cheating;src/magic/model/MagicGame.javamarks the opponent hand unknown and libraries unknown, andshowRandomizedHiddenCards()deals random cards in. That is determinization (general knowledge). - Intake scan 03 said the Monte Carlo AI cheats. Only the “(cheat)” variants do; the default variants hide. Forum posts from 2010-2013 (https://www.slightlymagic.net/forum/viewtopic.php?p=116519) predate the split.
Multiplayer and Commander
- Two-player only:
MagicGame.javausesgetOpponent(...)(a single opponent) throughout. - No Commander format (deckwright note 11). Useless for four-player Commander.
Headless use and speed
src/magic/DeckStrCal.javaruns N games between two decks with chosen AIs (--deck1,--deck2,--ai1,--ai2,--games,--str1,--str2): a built-in deck-strength calculator.- No published speed. Not run (D6).
Learning-agent hook
src/magic/ai/MagicAI.javais the small AI interface (choose a move for the game); a learned agent would implement it. Small surface, but the engine is Java-only and dormant (general knowledge).
Licence and activity
- GPL-3.0 (https://github.com/magarena/magarena). Last commit 2023-04-24; last release 1.96 in 2019 (https://github.com/magarena/magarena/releases). Dormant.
Lessons
DeckStrCal.javais the only engine tool built to rate decks: two decks, fixed AI, N games, a win rate. That is the simplest strength estimator design and our baseline idea for Commander self-play.- Shipping honest and cheat variants side by side is cheap and lets you measure how much peeking inflates a deck’s win rate. Worth copying as a measurement idea.
- Too dead and too 1v1 to host anything.
MageZero
Source read: WillWroble/MageZero@fb6ee21da77d73fe460984a23aa65c88f60a9841
Python repo read in full on 2026-11-03. The Java side lives in the fork WillWroble/mage (HEAD cb7e9c6fe2dc, not cloned; module names from the README). No code copied.
Where the code lives
- Python:
src/magezero/cli.py(mzcommand),runner.py(launches XMage JVMs per generation),server.py(inference server),model.py(graph transformer policy/value net),train.py,dataset.py. - Config:
configs/game.yml,configs/run.yml,configs/curriculum.yml. - Java: XMage fork modules
Player.AI.RL,Player.AI.MCTS,Mage.MageZero, withStateEncoder.java(README.md, https://github.com/WillWroble/mage).
Decision method
- AlphaZero-style: MCTS in Java guided by a neural policy/value network served from Python (
README.md). - Search budget per decision in
configs/game.yml(search_budget: 4000,timeout_ms: 4000), optional Dirichlet noise and policy priors. - Deck-local: one agent per deck, trained against an opponent pool (README.md section 1). Not a general player.
Hidden information
- The network input excludes hidden info, but the tree search is deterministic, so “both agents play with
information leakage about their future draws, opponent’s hands” (
faq_goals.md). configs/game.ymlhas ahiddenInfotoggle per player; what it changes on the Java side was not read.- Author lists hidden-information MCTS and a MuZero-style model as open goals (
faq_goals.md).
Multiplayer and Commander
faq_goals.md: Commander works “but not as well as 60 card constructed”; “Multiplayer not supported”.configs/game.ymlhasgame_mode: normal # normal, commanderwith exactlyplayer_aandplayer_b: two-player Commander only. Settles intake Q3 (Commander: duel only, no four-player).
Headless use and speed
- Fully headless:
mz traindrives JVM self-play and training (setup_guide.md); a precompiled XMage distribution ships in releases. - Reported: about 800 games/hour on 16 threads at a 300-simulation budget; about 300 MCTS simulations/s
single-threaded (
README.md). Not reproduced here (D6).
Learning-agent hook
- Settles intake Q3 (transport): Java calls Python over local HTTP.
server.pyruns Flask under waitress on 127.0.0.1 (default port 50052), batches requests on a queue for the GPU and exchanges msgpack payloads. - State encoding is a typed graph (root, players, zones, objects, abilities, attributes), README.md section 4. The PLAYER node is kept generic “so the encoding can be adopted to multiplayer”.
Licence and activity
- MIT (
LICENSE, copyright 2025 Will Wroble). The intake doubt (MIT vs unnamed) is settled: MIT. - Active: v0.3.0-alpha in October 2026, pushed 2026-11-02, 66 stars (https://github.com/WillWroble/MageZero).
Lessons
- Deck-local RL works on a hard engine: a tempo deck went from 16% to 66% win rate against minimax bots
(
README.md). Per-deck agents are a credible way to measure a deck’s ceiling, not just its autopilot floor. - Cost: about 1,000 games per generation and many generations per deck. At 800 games/hour that is hours per deck, in 1v1. Four-player Commander would be slower.
- HTTP plus msgpack plus server-side batching is a proven, simple bridge pattern for a JVM engine.
- Determinized search still leaks; a strength number from such agents is optimistic for reactive decks.
Argentum Engine AI
Source read: rwaldron/argentum-engine@de90110cddfdf85ad4b5efce7d0f35bcd49c780e
Fork named by intake scan 01; last pushed 2026-07-29. Upstream is wingedsheep/argentum-engine (HEAD 1b6e9aed78ce, pushed 2026-11-03, 83 stars), which is where work continues. Read only. No code copied.
Where the code lives
- Kotlin, Gradle.
rules-engine/(pure rules),ai/(bots),gym/,gym-server/,gym-trainer/(RL),game-server/(Spring Boot),web-client/(README.md). - AI:
ai/src/main/kotlin/com/wingedsheep/ai/engine/AIPlayer.kt,Searcher.kt,Strategist.kt,CombatAdvisor.kt,evaluation/BoardEvaluator.kt, set-specificadvisor/modules/*AdvisorModule.kt. - LLM bot:
ai/src/main/kotlin/com/wingedsheep/ai/llm/LlmAiPlayerController.kt.
Decision method
Searcher.kt: alpha-beta search with adaptive depth measured in priority passes; close decisions search deeper, obvious ones stay 1-ply; node cap from config. Leaves scored byBoardEvaluator.kt.- Per-card and per-set hints via advisor modules (Bloomburrow, Onslaught in
EngineAiPlayerController.kt). - An LLM-driven controller exists as an alternative bot (
llm/package). gym-trainer/src/main/kotlin/com/wingedsheep/gym/trainer/search/AlphaZeroSearch.ktandMctsNode.kt: JVM-side AlphaZero MCTS for self-play.
Hidden information
- The built-in bot peeks:
EngineAiPlayerController.ktsays it needs “the real (unmasked) GameState” to simulate. - The gym is honest by default:
TrainingObservationmasks opponent hand and libraries and reports only zone sizes for hidden zones (gym/README.md). - So the RL interface is clean; the heuristic baseline is not.
Multiplayer and Commander
- Free-for-all for 2-6 players (CR 806) in the server (
README.md). - Commander rules in the engine:
rules-engine/.../sba/player/CommanderDamageLossCheck.kt,CommanderZoneChoiceCheck.kt,state/components/identity/CommanderComponent.kt. - The searcher only models a reply when there is a sole opponent (
Searcher.ktusessoleOpponent); with three opponents it falls back to 1-ply evaluation. Weak multiplayer AI. - Card pool is set-by-set (
mtg-sets/,card-implementation-progress.html); a full precon may not be legal yet.
Headless use and speed
gym/README.md:GameEnvironmentwithreset/step/observe/fork; O(1) fork becauseGameStateis immutable;MultiEnvServiceruns many envs and steps them in parallel.gym-server/README.md: HTTP transport for Python.- Speed harness exists (
ai/src/test/kotlin/com/wingedsheep/ai/engine/GameBenchmark.ktprints games/sec); no published numbers. Not run (D6).
Learning-agent hook
- Best-designed hook of the five: a Gymnasium-style env with legal-action enumeration, masking, snapshot/restore,
a schema hash, and pluggable featurizers and evaluators in
gym-trainer/.../spi/(Evaluator.kt,StateFeaturizer.kt,RemoteHttpEvaluator.kt). - Python drives it over HTTP (
gym-server/README.md).
Licence and activity
- MIT (
LICENSE, with a trademark notice appended; GitHub reports NOASSERTION because of that addendum). - Upstream very active (pushed 2026-11-03, https://github.com/wingedsheep/argentum-engine); the named fork is stale.
Lessons
- Immutable state plus O(1) fork is the property that makes search and RL cheap; Forge and XMage copy whole games.
- Masked observations by default is the right contract for an honest agent (
gym/README.md); copy the idea, not the code. - Card coverage, not AI, is the limit for Commander: check precon legality before relying on it.
- Track upstream wingedsheep, not the rwaldron fork.
SP9 Commander game data sources
Inventory: platforms.yaml (checked by tests/sp9/sp9_data.py). Record format and capture plan:
capture.md. Example records: sample_games.json. Value spike: spikes/sp9/finish_value.py.
Question
Jaime, 2026-11-03: can SpellTable or any other Commander platform export turn data, and how do we get
casual (bracket 1-3) games with every seat’s finish order and turn events? Is turn-level data useful?
Brackets are the WotC tiers 1-4 plus cEDH as 5 (https://magic.wizards.com/en/news/announcements/introducing-commander-brackets-beta).
Start point: the run’s intake scan intake-research/04-online-platform-game-data.md (general knowledge
of this run) and SP1’s outcome inventory docs/research/sp1-ground-truth/README.md.
Platforms
Fourteen entries in platforms.yaml, each checked 2026-11-03. Casual play is spread over webcam tables,
trackers and clients; none tags a game with a bracket, so bracket comes from the decklist.
-
Playgroup.gg (tracker, largest casual corpus). Probed in depth, see
playgroup.md. With no key, public decks give bracket, power level, decklist URL and a per-game Elo history (win or loss, game id, time); “Scraping and bulk access are fine” (https://playgroup.gg/.well-known/openapi.yaml). Game endpoints need a key and membership. They return per-seatwinnerbut no place, and damage, poison and life events but no Kill, sofinish: winner_only. Full order needs a life replay. -
TopDeck.gg (tournaments). “Casual EDH” is a real, separate format value in the API format list, checked not assumed (https://topdeck.gg/docs/tournaments-v2). The docs do not define it; I set brackets 3-5. Multiplayer pods return
winneronly (winner_gamesis null). Decklists yes, free key, visible credit required. -
Cockatrice, MTGO, MTG Arena: full event logs, but only as local files of a player’s own games (https://github.com/BigPeet/mtgo_utils, https://mtgarena-support.wizards.com/hc/en-us/articles/360000726823-Creating-Log-Files). A corpus means volunteers sending files (
export: manual). Arena has Brawl, not 4-seat Commander. -
Forge, XMage (engines, https://github.com/Card-Forge/forge, https://github.com/magefree/mage): every field, any bracket, but AI pilots, not people. A labelled stand-in only.
-
SpellBot (Discord bot, https://github.com/lexicalunit/spellbot/blob/main/API.md): public API for queues; its record endpoint stores players and commanders, no results. It now pairs players for Convoke, Girudo and Playgroup Live, not SpellTable.
-
Convoke sends finished games with events to Playgroup by partner key (https://www.convoke.games/en/partners). Lifetap and similar trackers keep games on the phone, no API (https://apps.apple.com/app/id1508241754). EDHTop16 is cEDH only (https://edhtop16.com/about). Moxfield has decks, no games. Untap.in: chat log only.
SpellTable
No. SpellTable has no public API, no game history and no export: inventory value export: none, with
finish, turn_events and decklists all none. Its site is a webcam client only
(https://spelltable.wizards.com), and the life totals it tracks live are not saved where a third party
can read them. The old SpellBot token route is gone: SpellBot’s README now lists Convoke, Girudo,
Playgroup Live and EDHLAB as its tables, not SpellTable
(https://github.com/lexicalunit/spellbot). Turn data from SpellTable games needs a separate tracker.
Data value
Is turn-level data useful? First, full finish order alone is worth a lot. Under Plackett-Luce, the
Fisher information about a deck’s strength from one game is p(1-p) for the winner-only outcome, and the
sum of that term over every elimination stage the deck survives for full order
(Plackett-Luce: https://en.wikipedia.org/wiki/Plackett%E2%80%93Luce_model; spike spikes/sp9/finish_value.py).
For an equal 4-seat pod the full/winner ratio is 2.6 (exactly 2.56): full order needs about
2.6 times fewer games than winner only. By pod size: 2 seats 1.0, 3 seats 1.75, 5 seats 3.40,
6 seats 4.26. The gain is larger for weak decks (3.36 when the focus deck is half as strong) and smaller
for strong ones (1.90 when twice as strong), because weak decks rarely win but often place.
Turn events add a second layer. They let a model see when and how a deck kills or dies (turn of first
elimination, commander damage, key casts), which feeds the decomposed simulation in SP4
(docs/research/sp4-decomposed-sim/note.md) and the covariates of the SP2 statistical models
(docs/research/sp2-statistical/sp2-statistical-methods.md). They do not replace outcomes: rating still
needs finish order, so capture finish order first and events second.
Gaps
- No platform gives human bracket 1-3 games with full finish and turn events as an open download.
-
Playgroup member games were not pulled. The signup has a captcha, so Jaime must make the account and key, and full cross-group games need a data partnership (both NEEDS-JAIME, steps in
playgroup.md). TopDeck Casual EDH counts need a key (https://topdeck.gg/docs/tournaments-v2). -
Terms text for SpellTable, Moxfield, Untap and EDHTop16 was not reviewed in full.
-
Bracket per game must be inferred from decklists, for example with Playgroup’s bracket calculator (https://playgroup.gg/commander/bracket-calculator).
-
Forge and XMage log mapping to the record is described in
capture.md, not coded.
Verdict
For brackets 1-3 with full finish placement, rank_platforms(real, {"brackets": [1, 2, 3], "finish": "full"})
returns forge, xmage, cockatrice, mtgo. These are the only sources with full placement, and forge is an
AI stand-in. Ranked sources for bracket 1-3 work:
-
Playgroup.gg public decks, the best human source today: bracket, decklist and per-game win or loss per deck, free to bulk-read (https://playgroup.gg/.well-known/openapi.yaml). It is winner-only, with no pods, and only about 2,600 decks have a decklist plus 10+ games (
playgroup.md). Member games or a partnership add more. -
Our own capture with the CSV in
capture.md: human games with full finish and key turn events, at the pace of the pods that log. This is the only full-finish human ground truth. -
forge (then xmage): every field at volume, but as a labelled AI stand-in for testing methods, not ground truth (https://github.com/Card-Forge/forge).
-
cockatrice and mtgo volunteer logs: full data, slow to gather.
- TopDeck.gg Casual EDH: decklists and winners at scale, no finish order or turns. SpellTable is not a source.
Playgroup.gg game data
Spikes: spikes/sp9/playgroup_probe.py (public, run 2026-11-03), spikes/sp9/playgroup_games.py
(members-only, ready, not yet run: needs a key). Output: spikes/sp9/playgroup_probe.json, playgroup_sample.json.
Access
Base https://playgroup.gg/api/public/v1. Spec: “Scraping and bulk access are fine. We just want to know
who is calling”; 400 req/min with a User-Agent, 10 without (https://playgroup.gg/.well-known/openapi.yaml).
No key needed for public decks, public users and commanders. Games need a key and membership of the
playgroup, so a key only unlocks our own groups, never the corpus. Signup
(https://playgroup.gg/users/sign_up) has reCAPTCHA, and login offers Discord and Google OAuth, so I could
not make the account. Filed as NEEDS-JAIME.
Public data
-
/decks/{id}:bracket,power_level,competitiveness_rating,decklist_url(Archidekt/Moxfield),games_won,games_lost,win_rate_percentage,average_mulligans,most_popular_wincon,average_wins_by_round, commander, partner, colours,last_game_played_at. -
/decks/{id}/elo_history: one entry per rated game:rating,delta,game_id,played_at. The sign ofdeltamatches wins: positive entries equalgames_wonin 3 of 5 decks, and run short by 2 in the other two (unrated games). No opponents, no seat, no place. -
/commanders/{id}: stats and 5top_decks(all 26 sampled were public)./commanders/turn_damage: 5,334 commanders with mean damage, commander damage, poison and healing per turn: an aggregate, not games. -
/users/{id}/decks: a user’s decks, but many profiles are private (user 1234: 401). Field lists are from the spec and live calls (https://playgroup.gg/api/public/v1/decks/431905).
Measured coverage
Random scan of 1,600 deck ids in 1..1.05M (live ids stop near 1.0M), 2026-11-03
(spikes/sp9/playgroup_probe.json; an earlier 400-id run agrees):
- 198 public (200), 1,004 private (401), 398 missing (404). About 790k decks exist and about 130k are public.
-
Of the 198 public decks: 124 have a
decklist_urland 103 a bracket (1: 2, 2: 30, 3: 47, 4: 21, 5: 3), so 77% of bracketed public decks are bracket 1-3. All have a power level. -
Only 10 public decks have 10+ games, and 4 have a decklist plus 10+ games. Scaled up that is about 2,600 such decks (95% Poisson range about 700 to 6,700). Most decks are played a handful of times.
-
Commander pages: 14 of 30 random ids exist (about 4,700 commanders), averaging 81 to 176 decks and 270 to 590 seat-games each across two runs. That fits Playgroup’s own “760,000+ games” (https://playgroup.gg/about).
-
Pod rebuild: 124 Elo entries from 7 decks shared 0 game ids. Public data cannot rebuild who sat together, because there is no public route from a game id to its seats.
-
Sample: 229 decks (198 scan, 31 top decks), 7 with full Elo history, in
spikes/sp9/playgroup_sample.json. A full bracket-1-3 pull of qualifying decks means scanning about 1.05M ids: about 45 hours at 400/min.
Members-only games
/playgroups/{id}/games?include_events=true (up to 100 per page) returns total_rounds, started_at,
ended_at, life_amount, win_con, and per seat winner, deck_id, user_id, deck_name,
mulligans_taken, salt_rating, fun_rating. There is no place field. Event kind is only
normal_damage, commander_damage, poison_tokens, healing, mill and life_loss, with turn, amount, source
and receiver (https://playgroup.gg/.well-known/openapi.yaml). Kill and PassTurn appear only on the
batch import endpoint. So finish order is not given. playgroup_games.py rebuilds it by replaying
life (0 life, 21 from one commander, or 10 poison), tested on a synthetic game in
tests/sp9/test_sp9_playgroup.py. No casts, no bracket per game: the bracket comes from each deck.
Verdict
For bracket 1-3 human data, Playgroup is the best source at scale: per-deck bracket, decklist, and per-game win or loss with time (https://playgroup.gg/.well-known/openapi.yaml). It is winner-only and per deck. It has no pods, so it can rate decks against the field but not against each other. Full finish order needs member games plus the life replay, or a data partnership.
Gaps
-
NEEDS-JAIME (account): sign up at https://playgroup.gg/users/sign_up with jaime@pvragon.com (solve the captcha, confirm the email). Create an API key in account settings and keep it outside the repo as
PLAYGROUP_API_KEY. Join or create a playgroup that logs games. Then runpython -I spikes/sp9/playgroup_games.py 50. -
NEEDS-JAIME (partnership, draft only, D7): ask Playgroup for a research export of cross-group games with every seat’s deck id, bracket,
winner, the elimination turn or place per seat, and the event log including Kill and PassTurn, with user ids hashed. Credit is offered (https://playgroup.gg/about). -
The coverage figure rests on 4 qualifying decks. A wider scan would narrow it. The
top_decksof each commander are a cheaper route to the most-played decks (about 4,700 calls).
SP9 capture plan
Code: tests/sp9/sp9_data.py (validate_game, finish_order, parse_capture_csv). Sources: README.md.
Record format
One game is a dict: game_id, source, date (YYYY-MM-DD), bracket (1-5, 5 = cEDH, or null),
seats (2-6 of {seat, commander}, optional deck_url, pilot), events ({turn, seat, type},
optional detail, same_time) and finish (seat -> place). Event types: cast, attack, damage, life,
eliminated, conceded, win, other. Finish order is derived from eliminated, conceded and win events;
seats out at the same time share the best place they span (standard competition ranking,
https://en.wikipedia.org/wiki/Ranking#Standard_competition_ranking_(%221224%22_ranking)). A win event (alt
win) gives the winner 1 and every seat still in place 2. Examples: sample_games.json.
Capture paths
-
Manual tracker or form: one person per pod logs joins, eliminations and a few key casts into the capture CSV, by hand, a shared sheet or a web form. Human games, full finish, our own data. Bracket from each decklist (https://playgroup.gg/commander/bracket-calculator).
-
Platform partnership or data request: ask Playgroup.gg for a research export. Member games carry no place field and no Kill events (https://playgroup.gg/.well-known/openapi.yaml; see
playgroup.md). Draft only; Jaime sends it (D7):Hello, we study Commander deck strength. Could we get a research export of games across playgroups: each seat’s deck id, bracket, winner flag, elimination turn or place, and the event log including Kill and PassTurn, with user ids hashed? Credit given, results shared. Jaime Meanwhile,
spikes/sp9/playgroup_games.pypulls our own playgroups’ games with a key. -
Engine self-play logs (Forge/XMage), labelled stand-in: run AI pods of our decklists and map each game log to the record (seat per player, eliminated on loss, cast lines as events). Not human play; use it to test methods and pipelines, never as ground truth (https://github.com/Card-Forge/forge, https://github.com/magefree/mage).
-
Volunteer Cockatrice or MTGO logs: full data, slow, needs consent per player (https://github.com/BigPeet/mtgo_utils).
Logger design
The smallest self-run logger is one CSV file per night with header
game_id,date,bracket,turn,seat,type,detail,same_time. Before play, one join row per seat with the
commander in detail. During play, add a row when a seat casts a key card, is eliminated or concedes;
put 1 in same_time on every row of a tie: two or more eliminated/conceded rows, back to back,
same game and turn (e.g. one board wipe). Tied seats share the best open place. A lone mark is refused.
An alt win is a win row. Example: B out alone turn 6, C and D tied turn 9, so A 1, C 2, D 2, B 4.
csv
game_id,date,bracket,turn,seat,type,detail,same_time
n1,2026-11-03,2,0,A,join,Atraxa,
n1,2026-11-03,2,0,B,join,Krenko,
n1,2026-11-03,2,0,C,join,Talrand,
n1,2026-11-03,2,0,D,join,Yuriko,
n1,2026-11-03,2,6,B,eliminated,,
n1,2026-11-03,2,9,C,eliminated,Blasphemous Act,1
n1,2026-11-03,2,9,D,eliminated,Blasphemous Act,1
parse_capture_csv turns the file into records and refuses a bad header, a non-int turn or bracket,
an event before its seat joins, or a lone tie mark, so mistakes surface the same night. A spreadsheet with these eight
columns, exported as CSV, is enough; a phone form can come later (general knowledge).
Volume
SP1’s min_games(0.25, 0.30) gives 610 games to tell a 30% deck from the 25% base rate in 4-seat pods
with winner only (alpha 0.05, power 0.8; tests/sp1/sp1_gt.py). With full order the same precision
needs about 610 / 2.56 = 240 games (README.md, Data value). To reach 1,000 games: one weekly pod
playing 3 games a night logs about 150 a year, so about 7 years; 10 pods logging take about 8 months;
Forge AI pods at a few minutes a game reach 1,000 in about a day on one machine, untimed here
(general knowledge). Human volume is the bottleneck, which is why full finish order matters.
Terms and privacy
Our own logs: ask each player before logging, store a seat label or nickname, not real names, and keep
pilot optional. Playgroup allows bulk API access with a User-Agent but game data is members-only
(https://playgroup.gg/.well-known/openapi.yaml). TopDeck needs a key and visible credit
(https://topdeck.gg/docs/tournaments-v2). Cockatrice replays can expose personal data, so volunteer
files need consent (https://www.github.com/djpiper28/Cockatrice-Replay-Tool). No login scraping (D5).
Recommended
Start the CSV logger in Jaime’s own pods now, bracket 1-3, full finish plus eliminations and key casts. In parallel, generate Forge pods as a labelled stand-in to exercise SP2/SP4 methods, and let Jaime send the Playgroup request above. Bulk-read Playgroup public decks for winner-only bracket 1-3 outcomes. Revisit after 200 logged human games (general knowledge).
Deckwright deck strength: the plan
Files here: signals.yaml (every signal), calculators.yaml (calculator survey), thesis.json
(popularity test). Reference code: tests/sp10/sp10_plan.py. Spike: spikes/sp10/thesis_spike.py.
Answer
Deckwright should estimate a deck’s strength as its predicted chance to win a four-player table at a given bracket, learned from game results, with every other signal as an input that the results weigh. No single source is the truth. Real game results are the label. Deck popularity, declared brackets, calculators like edhpowerlevel.com, Deckwright’s card features and simulations are inputs. Each has a known bias, and the model learns how much to trust each one.
This updates SP8’s answer (docs/research/sp8-synthesis/answer.md) in three ways:
-
SP8 fitted on cEDH tables only. Playgroup.gg’s public casual decks (SP9) now give labels for brackets 1-4, and our own logged games give full finish order (https://playgroup.gg/.well-known/openapi.yaml).
-
SP8 used card features and piece simulation. This plan adds deck popularity, declared bracket, outside calculators and a pod race model as extra inputs, and says how to handle each one missing.
-
Jaime’s deck-popularity thesis got a first test: on 97 commanders it is a weak positive signal (rho 0.12, not significant), so popularity stays in as a low-weight input, not a label.
Until enough games exist, combine() gives a hand-weighted baseline score (general knowledge).
Terms
- Deck strength: the chance a deck wins a four-seat table of its bracket, compared with 1 in 4 (general knowledge).
- Bracket: Wizards’ 1-5 Commander tiers; 5 is cEDH, competitive play (https://magic.wizards.com/en/news/announcements/introducing-commander-brackets-beta).
- Game Changer: a card on Wizards’ list that caps how low a deck’s bracket can be.
-
Signal: any measurement that may carry strength information. Label: what the model learns to predict (who won). Feature: an input. Prior: a guess used before data. Referee: settles disputes.
-
Goldfish: playing a deck alone, with no opponents, to see how fast it can win.
- Disruption: an opponent stopping a win attempt. q: the chance one attempt is stopped.
- Pod race: four decks race to win; the first to win takes the table.
- Popularity rate: copies, likes and views of a deck per day it has existed (
popularity_rate). - Spearman rho: a rank correlation from -1 to 1; 0 means no link.
- Win-rate excess: win rate minus 1 / seats (0.25 at four seats).
-
Log loss: how surprised the predictions are by the actual winners; lower is better. Skill: 1 minus log loss over a coin-flip’s log loss; above 0 beats guessing (
docs/research/sp1-ground-truth/validation.md). -
Ordinal label: an ordered category (1 < 2 < 3) whose gaps are unknown.
- Coverage: the share of signal weight present for a deck.
Signals
Full fields in signals.yaml. Weight is the baseline combine() weight; the fitted model replaces it.
Outcomes (labels, not inputs):
topdeck_pods: cEDH tables with all four decklists. Bias: cEDH and strong pilots only. Label for bracket 5 (https://topdeck.gg/docs/tournaments-v2).playgroup_decks: public casual decks with win and loss counts. Bias: winner only, no pods, self-selected. Label for brackets 1-4 (docs/research/sp9-game-data/playgroup.md).-
own_logged_games: our pods via the capture CSV. Bias: small, our pilots. Full finish order; the final test set (docs/research/sp9-game-data/capture.md). Inputs: -
edhrec_commander_decks: EDHREC decks per commander over two years. Bias: hype, precons, commander not deck. Weight 0.05. moxfield_deck_popularity: views, likes and copies per day. Bias: launch hype, creator fame. Weight 0.05.archidekt_deck_popularity: views per day. Bias: front-page spikes, views only. Weight 0.05.value_per_dollar: predicted win chance over price. Bias: prices track scarcity. Role: check only, weight 0.declared_bracket: owner-set bracket. Bias: under-declared, defined partly by card lists. Role: ordinal label with noise, weight 0.10 as input when no outcome exists.bracket_floor: minimum bracket from Game Changers and combos. Bias: a floor, not strength. Weight 0.10.edhpowerlevel_score: EDHPowerLevel.com power level. Bias: card-demand weights, unvalidated. Weight 0.10. One signal, never the answer.playgroup_power_level: Playgroup’s power level and bracket. Bias: rule-based, often owner-entered. Weight 0.05.deckwright_card_features: Deckwright’s parsed roles per card, summed. Bias: hand-set values. Weight 0.20 (docs/research/sp3-deterministic/note.md).goldfish_speed: win-turn distribution alone. Bias: overrates fragile combo. Weight 0.05.goldfish_disrupted: the same under bracket-calibrated disruption. Bias: untargeted interaction. Weight 0.10.pod_race: table win chance from the race. Bias: independent seats. Weight 0.15 (docs/research/sp4-decomposed-sim/note.md).forge_referee: Forge four-player AI games. Bias: weak AI, clock cut-offs. Role: referee when signals disagree, 265 games an hour ceiling (docs/research/sp7-engines-ai/forge-speed.md).
Popularity thesis
Jaime’s thesis: people copy and like decks they think will win, so deck popularity, adjusted for how
long the deck has existed, may track strength or strength per dollar (run notes.md, 2026-11-03).
Result: tested, n = 97 commanders, rho = 0.12 (thesis.json). Popularity is EDHREC decks per
commander over the past two years (https://json.edhrec.com/pages/commanders/year.json). The outcome is
the Playgroup.gg win-rate excess over 0.25 for the same commander, kept at 200+ games
(https://playgroup.gg/api/public/v1/commanders/by_name/). Playgroup’s own deck count gives rho 0.06.
The 95% range for rho is about -0.08 to 0.32, so the link is weak and not significant at this size.
Caveats: this is commander-level, not deck-level; the top 100 commanders are all popular, which
narrows the range; Playgroup win rates average 0.32, so pods are often under four seats.
Lifetime adjustment. A deck’s counts grow with its age, and new decks get launch hype.
popularity_rate divides weighted counts (copies 1, likes 1, views 0.01) by the deck’s age in days,
with a 30-day floor so a week-old deck is not inflated. Copies and likes cost the user an action, so
they weigh 100 times a passive view. The EDHREC count uses a fixed two-year window instead.
Deck versus card popularity. Deckwright excludes card popularity because a card’s play rate mostly measures how many decks can use it (colours, staples), not how strong it is. Deck popularity is a choice among whole decks for the same commander, so it carries a judgment that the deck works. Next test: deck-level rates from Archidekt and Moxfield against Playgroup decks with a decklist URL.
Calculators
Full survey in calculators.yaml, each checked 2026-11-03 from public pages only.
edhpowerlevel: card-demand weights to a 1-10 power level. Use as input; beat it by fitting on outcomes (https://edhpowerlevel.com).playgroup_bracket: Game Changers, banned cards, land denial, extra turns and Spellbook combos to a bracket. Borrow its rule list (https://playgroup.gg/commander/bracket-calculator).playgroup_elo: Elo and win rates from logged pods. Input and label source.wizards_game_changers: the official bracket rules. Borrow them as the bracket floor.moxfield_bracket: owner bracket plus Game Changer flags and deck popularity. Input.archidekt_bracket: owner bracket, views and dates in public JSON. Input.edhrec_salt: salt votes, commander deck counts, average decks. Input for popularity; salt is not strength.commander_spellbook: combo finder and bracket estimate API. Input for combos and the floor (https://backend.commanderspellbook.com/estimate-bracket).brackcheck: unpublished weights on Game Changers, fast mana, tutors, combos. Beat it.playscore: scores games, not decks. Ignore.deckorator_nn: a neural rater trained on power claims. Beat it, since claims are not wins (https://github.com/Noles55/DeckoratorNN).better_bracket_calc: early rule-based bracket estimate. Ignore. None publishes a check against game results. What to borrow: the Wizards rules and Spellbook combos. What to beat: every calculator, by scoring better log loss on held-out pods.
Combining
Model family: an ordered multinomial-logit (Plackett-Luce) model over the seats at a table, as SP2
and SP9 propose: each deck gets a strength score from a linear model of its features, and a seat’s win
chance is its exp(score) share (docs/research/sp2-statistical/sp2-statistical-methods.md). Add a small
per-deck random effect so heavily played decks earn their own rating, as in SP8. Gradient-boosted trees
are the later upgrade when there are 10,000+ tables.
Missing signals. Most decks lack most signals. Each feature gets a “present” flag and is set to its
bracket mean when missing, so the model learns how much a missing value costs. Before fitting,
combine() renormalises weights over present signals and reports coverage; low coverage widens the
uncertainty shown to users.
Weak or noisy labels. Winner-only results enter Plackett-Luce as a first place only. Full finish order (own logs) counts about 2.6 times more per game (SP9). The self-declared bracket is an ordinal label: fit it with an ordered-logit side task, and treat it as under-declared (owners pick lower brackets), so a declared 3 means “3 or higher” more often than “3 or lower”.
Per-bracket context. Strength is relative to the table: fit bracket as a context feature and interaction strength q per bracket, counted from interaction cards in public decklists at each bracket. A deck is scored against a typical table of its bracket, not against all decks (general knowledge).
Validation
-
Outcomes are checked against held-out tables split by date and event; this is the main test. Metric: log loss and skill against a coin flip, from SP1
sp1-ground-truthscore_games(docs/research/sp1-ground-truth/validation.md). -
Popularity, calculators and declared bracket are each checked by Spearman rho with win-rate excess, as in
thesis_check. - Simulation signals are checked against Forge four-player games, then against real pods.
- Card features are checked against hand-judged pairs (SP3) and, after phase 2, against outcomes.
-
Minimum data: about 1,000 tables to fit, 300 held out (SP8); 610 games to tell one 30% deck from 25%, so single-deck claims pool games by commander (
docs/research/sp2-statistical/sp2-statistical-methods.md). -
A signal stays in the model only if dropping it worsens held-out log loss by more than run-to-run noise (general knowledge).
Pipeline
Sources and access follow SP9 (docs/research/sp9-game-data/README.md):
- TopDeck.gg tables with decklists, by free key (https://topdeck.gg/docs/tournaments-v2).
- Playgroup.gg public decks and commanders, bulk allowed; member games after Jaime’s account (SP9
playgroup.md). - Our own pods, logged with the SP9 capture CSV (
docs/research/sp9-game-data/capture.md). - Popularity from EDHREC, Archidekt and single Moxfield deck pages, polite rates, credited.
- Calculators: Commander Spellbook API for combos and bracket; edhpowerlevel by manual paste only.
- Decklists matched to Deckwright card ids; features, goldfish, disruption and pod race computed per deck. Each pull stores the raw response with date, so every number can be rebuilt.
Build plan
-
Baseline and test set. Load 1,000 TopDeck tables and Playgroup decks with lists; compute every signal; score
combine()and each single signal. Exit: held-out log loss and skill reported for the baseline and each signal. -
Fitted model. Fit Plackett-Luce on features plus bracket context, with missing flags. Exit: beats the baseline and plain Elo on held-out decks by more than noise.
-
Simulation layer. Add goldfish, disruption and pod race features with q per bracket. Exit: held-out log loss improves, or the layer is dropped.
-
Casual check. Run on our own logged games (brackets 1-3). Exit: skill above 0 on 200+ logged games. See
docs/research/sp8-synthesis/answer.mdfor the cEDH-only version this extends; data per https://topdeck.gg/docs/tournaments-v2 and https://playgroup.gg/.well-known/openapi.yaml.
Risks
- Popularity is circular: decks rated strong get copied more. Fit popularity only from before the outcome window (general knowledge).
- Playgroup selection: loggers are not typical players; public decks are a minority (SP9).
- Pilot skill is mixed into every outcome; nothing public separates it.
- Independent seats: the pod race ignores politics, targeted removal and kingmaking (D7).
- Calculator drift: edhpowerlevel and others change weights without notice; store the date.
- Terms change: Moxfield has no API; any site may close access.
- Small casual data: about 2,600 qualifying Playgroup decks (
docs/research/sp9-game-data/playgroup.md).
First changes
What Deckwright changes first, in order (general knowledge):
- Show a deck’s bracket floor from Game Changers and Spellbook combos next to its score.
- Add
combine()as the deck score, with coverage shown, and edhpowerlevel as one input. - Add the disruption and pod race step to Deckwright’s goldfish, with q per bracket.
- Start the TopDeck and Playgroup pulls for phase 1, and log our own games with the capture CSV.
Sources
- SP1 ground truth:
docs/research/sp1-ground-truth/README.md,docs/research/sp1-ground-truth/validation.md - SP2 statistics:
docs/research/sp2-statistical/sp2-statistical-methods.md - SP3 Deckwright scorer:
docs/research/sp3-deterministic/note.md - SP4 piece simulation:
docs/research/sp4-decomposed-sim/note.md - SP5 self-play:
docs/research/sp5-self-play-rl/note.md - SP6 card encoding:
docs/research/sp6-card-encoding/sp6-card-encoding.md - SP7 engines:
docs/research/sp7-engines-ai/README.md,docs/research/sp7-engines-ai/forge-speed.md - SP8 answer:
docs/research/sp8-synthesis/answer.md - SP9 game data:
docs/research/sp9-game-data/README.md,docs/research/sp9-game-data/playgroup.md - EDHREC commanders: https://json.edhrec.com/pages/commanders/year.json
- Playgroup.gg API: https://playgroup.gg/.well-known/openapi.yaml
- Wizards brackets: https://magic.wizards.com/en/news/announcements/introducing-commander-brackets-beta