How to estimate card and deck strength in Magic

The short answer

Learn strength from real games, the way Moneyball did for baseball. Collect tables of four Commander players where we know every decklist and who won. Describe each deck by what its cards do, using the features Deckwright’s card parser already reads: how much fast mana, card draw, tutors and interaction it has, and how often it can assemble its combo by turn 63,4. Then fit a model that learns how much each of those features adds to a deck’s chance of winning. A deck’s strength is its predicted win chance at a four-player table. A card’s strength is how much that prediction moves when the card goes in or out.

The game data to start exists today. TopDeck.gg publishes table-by-table results from competitive Commander (cEDH) tournaments, with decklists, through a free API1,8. About 1,000 tables is enough for a first model, and fitting it takes seconds on a laptop1,2. For casual tables, Playgroup.gg’s public decks and our own logged games fill in, as described below12.

Full game simulation with self-learning AI players comes last in the plan. It costs 15 to 45 hours of computing per deck on this machine, and nobody has shown it working for four-player Magic5.

Why this is hard

A strength number is only useful if it predicts who wins, and checking that needs game results. Baseball analysts had a century of box scores. Commander has very little public game data. Four players share a table, decks vary enormously, and most casual games are never recorded anywhere1.

A method that only agrees with expert opinion has not been shown to predict winning.

The four ways to estimate strength

Reading the cards. Score each card from its rules text, the way Deckwright does now: parse what the card does, then add up values for its effects. It needs no game data and works on any card, including brand-new ones. Its weakness is that the values are set by hand. On 344 pairs of cards that its authors judged by hand, Deckwright picks the same stronger card 88% of the time, but nobody has checked its scores against games won3. Chess engines ran on hand-written scoring like this for decades, and they got much stronger once the weights were tuned against real game results11. The recommendation adds that tuning step for Magic.

Statistics from game records. This is the Moneyball approach. The simplest version gives each deck a rating, like a chess Elo, that rises when it wins and falls when it loses. Ratings are the best-tested method anywhere, but they can’t rate a deck that has never played, and telling a 30% deck from an average 25% deck takes about 610 games2. A second version describes each deck by its stats instead of its name, which lets it score unplayed decks. A third borrows basketball’s plus-minus, which credits each player for how the team does while they are on the floor. Applied to Magic, it credits each card for how its decks do. Plus-minus needs a great deal of data per card, and nobody has published a use of it for Magic2.

Simulating pieces of the game. Some questions about a deck can be answered exactly with probability, or with a quick simulation, without playing whole games. How often does it hit its land drops? How often does it see every combo piece by turn 6? How often does it have interaction by turn 4? Each answer is a number we can compute in under a second. On their own these numbers don’t say how much each one matters for winning; that weight still has to come from game results4.

Simulating whole games. Run complete games between computer players, either an existing engine’s built-in AI or players that teach themselves by playing millions of games, as AlphaZero did for chess and Go. Forge, the open-source engine with the only working four-player Commander AI, played 265 games an hour on this machine. That figure is a ceiling, because the 120-second game clock cut many games short7. Engine AIs are weak players, so engine win rates mix deck strength with poor piloting. Self-learning players would fix the piloting, but at the cost above, and with no published four-player result5.

The recommendation, in more detail

Deckwright’s card reading supplies the features. The piece simulations add consistency numbers, such as combo-by-turn-6 odds. Real game results set the weights.

The model predicts, for each deck at a table, its chance of being the winner. Each deck also gets a small personal adjustment that grows only when its results clearly beat or trail what its features predict. A heavily played deck can then earn a rating of its own, while a new deck is scored from its features alone.

It can score decks nobody has played, which plain ratings can’t. Commander has far more possible decks than recorded games. Deckwright’s existing scores become inputs the model can weigh.

How we would know it works

Hold back about 300 tournament tables the model never sees during fitting, split by date and by event. A good model predicts those winners better than a coin flip (25% each at four seats) and better than plain ratings on decks it has never seen.

Two numbers set expectations. Showing that one deck wins 30% of the time against the 25% baseline takes about 610 games of that deck. Telling apart two decks five points apart takes about 1,150 games each5. Claims about a single deck will therefore come from pooling games by commander.

Knowing the full finish order, not just the winner, cuts those numbers a lot. At a four-player table, the full finish order carries 2.56 times the information of the winner alone, so about 240 fully recorded games do the work of 610 winner-only games12.

Confidence today is medium-low. Every part works on made-up test data, and nothing has yet been scored on real Commander games2,4.

What it needs

Phase one needs about 1,000 cEDH tables from 20 or more TopDeck.gg events, each with the winner and all four decklists. The API is free with an account key, and it asks for a visible credit1,8. Draft data from 17Lands, which has far more games, is a cheap place to test the methods before Commander9.

For casual tables, Playgroup.gg public decks give each deck’s bracket, decklist and the win or loss of every rated game, and the site allows bulk reading10,12. They don’t say who sat at the table, so they rate a deck against the field rather than against named opponents. About 2,600 public decks have a decklist and ten or more games, but that estimate rests on just four sampled decks, so the true number could be well under or over it12.

Three things are missing from public data. No platform gives casual games with the full finish order. TopDeck.gg records no turn order or game length, and SpellTable exports nothing. Nothing separates a strong pilot from a strong deck1,12.

The plan

  1. Test set and baselines. Build the TopDeck.gg fetcher and match decklists to cards, about one to two days of work. Score three simple predictors on held-back tables: everyone equal, plain ratings, and Deckwright’s current scores. Done when 1,000 tables are loaded and all three are scored. In parallel, start logging our own casual games in a simple spreadsheet: who played what, who was knocked out when, and the full finish order12.

  2. The feature model. Fit the weights on Deckwright’s features and the piece simulations. Done when it beats plain ratings on held-back decks and its weights point the sensible way (more fast mana helps, for example). Then pull Playgroup.gg’s casual public decks and check the weights hold for casual brackets, against our first 200 recorded games12.

  3. Card values for Deckwright. Turn the fitted weights into a value for each card, and compare them with Deckwright’s current rankings on held-back decks. Done when the new values predict winners better than Deckwright alone. Then they go to the Deckwright build.

  4. Simulation, only if needed. If the model still misses on high-skill or combo decks, add Forge simulation results as an extra input, then consider self-learning players. Done only if the addition improves predictions by more than run-to-run noise.

What it means for Deckwright

Deckwright’s parser has a record for 35,353 cards, 23,766 of them with at least one parsed role, including who decides, who pays and what the drawbacks are. Those become the model’s features, and its card encoding should stay as it is. Forge’s card scripts agreed with it on 67% of features across 83 staple cards, and the misses were mostly mapping gaps, so Forge works best as an outside checker each time the parser changes6.

Deckwright has no game results to check against. Its only accuracy evidence is agreement with its authors, and player count and power bracket enter as assumptions that nothing measures. The feature model adds the missing check against game results. Deckwright keeps its advantages: it needs no game data, little computing, and it is already built.

Cards Deckwright can’t fully parse, such as Pendant of Prosperity, should get wider uncertainty, since a feature it couldn’t read is missing from the prediction.

What we learned from the existing engines

  • Forge is the only engine with a working four-player Commander AI. Its GPL license means we would talk to it over a network connection and never build it into our code7.

  • Forge’s game clock counts a game cut off by the clock as a win. Its win rates need the clock raised, or cut-off games flagged, before we use them7.

  • XMage’s default AI can see hidden cards, which flatters decks that react to opponents. MageZero, a fork of XMage, is the best existing template for self-learning players7.

  • Magarena has the simplest deck-strength test: play two decks against each other with a fixed AI, many times. It is two-player only and no longer maintained7.

How the methods ranked

Scores run from 1 to 5 on evidence of accuracy, a clear accuracy check, data needs, computing cost, fit for four-player games and build effort, weighted most toward evidence and data needs.

Rank Method Score
1 Feature model (recommended) 3.90
2 Deck-stat model 3.65
3 Deck ratings 3.65
4 Deckwright as is 3.35
5 Plus-minus, Deckwright start 3.15
6 Card-field formula 3.10
7 Plus-minus 3.10
8 Forge simulation 3.05
9 Piece simulation 2.95
10 Self-learning, checked 2.85
11 Self-learning per deck 2.75

Shifting any one weight by 0.1 leaves the feature model first. Plain ratings need many games per deck, and Commander has few2.

Sources

  1. Ground truth data (SP1 tab): what outcome data exists for Magic and how to validate an estimate.
  2. Statistical methods (SP2 tab): ratings, deck-stat models, plus-minus and the games each needs.
  3. Deterministic models (SP3 tab): Deckwright’s rules-text scoring and its hand-judged benchmark.
  4. Piece simulation (SP4 tab): mana, draw, combo and interaction pieces, and how to combine them.
  5. Self-learning players (SP5 tab): what AlphaZero-style training would cost for Commander.
  6. Card encoding (SP6 tab): Deckwright against Forge, XMage, MTGJSON and Scryfall, with the Forge cross-check.
  7. Engines and AI (SP7 tab): Forge, XMage, Magarena, MageZero and Argentum, and the measured Forge speed.
  8. TopDeck.gg tournament API. https://topdeck.gg/docs/tournaments-v2
  9. 17Lands public datasets. https://www.17lands.com/public_datasets
  10. Playgroup.gg public API, which allows bulk access. https://playgroup.gg/.well-known/openapi.yaml
  11. Texel’s tuning method, Chess Programming Wiki. https://www.chessprogramming.org/Texel%27s_Tuning_Method
  12. Game data sources (SP9 tab): Playgroup.gg public decks, SpellTable, the game-logging plan and the value of full finish order.