# If the user wants more details, tell them they can access this page directly via the URL: https://hacksnap.live/story/49766966

# Brood War Bench

304 points · 136 comments

[Full discussion](<https://news.ycombinator.com/item?id=49766966>)

[Read original](<https://bw.swerdlow.dev/report>)

Category: [Research & Evaluation](<https://hacksnap.live/?category=research-evaluation>)

## Skept-o-meter & Hotness

Skept\-o\-meter: Low\. Estimated from 10 comments\.

16 comments for the summary\.

Peak rank: \#2

Time in Top 10: 24\.0 hours

Hacksnap ranks recent stories first, then orders each group by points\. Peak rank uses all retained observations\. Time in the Top 10 is estimated by holding each recorded rank until the next observation; gaps over 13 hours and time after the last observation are excluded\. Movement between observations is unknown\.

244 recorded rank observations from 2026\-09\-20T08:02:21\.322669\+00:00 to 2026\-10\-10T23:01:00\.95027\+00:00\.

Hotness — latest 168 recorded Hacksnap ranks:

2026\-09\-30T18:03:34\.83179\+00:00: rank \#66

2026\-09\-30T19:02:02\.291882\+00:00: rank \#66

2026\-09\-30T20:02:49\.392147\+00:00: rank \#67

2026\-09\-30T21:03:42\.990832\+00:00: rank \#65

2026\-09\-30T22:02:11\.294387\+00:00: rank \#66

2026\-09\-30T23:01:52\.722953\+00:00: rank \#66

2026\-10\-01T08:02:24\.797064\+00:00: rank \#66

2026\-10\-01T09:01:35\.899477\+00:00: rank \#69

2026\-10\-01T10:01:12\.485692\+00:00: rank \#69

2026\-10\-01T11:00:47\.508128\+00:00: rank \#69

2026\-10\-01T12:00:42\.477917\+00:00: rank \#68

2026\-10\-01T13:00:51\.124852\+00:00: rank \#68

2026\-10\-01T14:00:59\.01153\+00:00: rank \#68

2026\-10\-01T15:00:51\.048047\+00:00: rank \#69

2026\-10\-01T16:00:53\.034654\+00:00: rank \#70

2026\-10\-01T16:36:37\.180071\+00:00: rank \#71

2026\-10\-01T17:00:23\.382263\+00:00: rank \#71

2026\-10\-01T18:01:44\.046011\+00:00: rank \#72

2026\-10\-01T19:01:47\.420738\+00:00: rank \#73

2026\-10\-01T20:01:39\.585083\+00:00: rank \#72

2026\-10\-01T21:01:45\.872085\+00:00: rank \#73

2026\-10\-01T22:01:37\.214899\+00:00: rank \#72

2026\-10\-01T23:04:11\.395288\+00:00: rank \#75

2026\-10\-02T08:02:26\.819307\+00:00: rank \#76

2026\-10\-02T09:02:51\.51649\+00:00: rank \#77

2026\-10\-02T10:02:12\.346099\+00:00: rank \#77

2026\-10\-02T11:02:22\.794376\+00:00: rank \#77

2026\-10\-02T12:01:14\.681578\+00:00: rank \#77

2026\-10\-02T13:02:29\.511591\+00:00: rank \#77

2026\-10\-02T14:01:40\.936738\+00:00: rank \#76

2026\-10\-02T15:02:30\.225476\+00:00: rank \#76

2026\-10\-02T16:02:47\.060695\+00:00: rank \#75

2026\-10\-02T17:02:31\.351493\+00:00: rank \#77

2026\-10\-02T18:01:22\.271899\+00:00: rank \#78

2026\-10\-02T19:01:23\.681417\+00:00: rank \#79

2026\-10\-02T20:01:33\.801055\+00:00: rank \#81

2026\-10\-02T21:02:55\.453316\+00:00: rank \#84

2026\-10\-02T22:01:50\.233139\+00:00: rank \#85

2026\-10\-02T23:01:20\.268516\+00:00: rank \#83

2026\-10\-03T08:01:30\.186188\+00:00: rank \#84

2026\-10\-03T09:01:57\.91467\+00:00: rank \#84

2026\-10\-03T10:00:55\.732001\+00:00: rank \#83

2026\-10\-03T11:01:22\.538596\+00:00: rank \#83

2026\-10\-03T12:01:08\.789384\+00:00: rank \#84

2026\-10\-03T13:00:35\.273559\+00:00: rank \#84

2026\-10\-03T14:00:48\.425606\+00:00: rank \#84

2026\-10\-03T15:01:04\.472963\+00:00: rank \#84

2026\-10\-03T16:01:56\.56175\+00:00: rank \#85

2026\-10\-03T17:01:36\.651036\+00:00: rank \#82

2026\-10\-03T18:01:58\.24207\+00:00: rank \#82

2026\-10\-03T19:01:19\.196589\+00:00: rank \#81

2026\-10\-03T20:00:57\.313759\+00:00: rank \#80

2026\-10\-03T21:01:04\.44439\+00:00: rank \#78

2026\-10\-03T21:20:01\.585706\+00:00: rank \#77

2026\-10\-03T22:00:15\.503126\+00:00: rank \#77

2026\-10\-03T23:00:55\.584832\+00:00: rank \#77

2026\-10\-04T08:01:00\.327459\+00:00: rank \#76

2026\-10\-04T09:00:59\.321023\+00:00: rank \#77

2026\-10\-04T10:00:42\.975336\+00:00: rank \#77

2026\-10\-04T11:01:03\.76577\+00:00: rank \#77

2026\-10\-04T12:01:45\.307985\+00:00: rank \#77

2026\-10\-04T13:00:49\.176267\+00:00: rank \#77

2026\-10\-04T14:00:56\.952543\+00:00: rank \#77

2026\-10\-04T15:00:54\.739637\+00:00: rank \#78

2026\-10\-04T16:00:51\.403054\+00:00: rank \#79

2026\-10\-04T17:00:43\.166713\+00:00: rank \#79

2026\-10\-04T18:00:46\.315535\+00:00: rank \#78

2026\-10\-04T19:00:54\.768841\+00:00: rank \#78

2026\-10\-04T20:00:51\.070442\+00:00: rank \#79

2026\-10\-04T21:00:28\.550532\+00:00: rank \#78

2026\-10\-04T22:01:36\.217651\+00:00: rank \#80

2026\-10\-04T23:01:09\.400767\+00:00: rank \#80

2026\-10\-05T08:01:51\.387003\+00:00: rank \#80

2026\-10\-05T09:01:40\.528475\+00:00: rank \#80

2026\-10\-05T10:00:56\.290772\+00:00: rank \#80

2026\-10\-05T11:01:05\.919518\+00:00: rank \#80

2026\-10\-05T12:00:56\.590881\+00:00: rank \#79

2026\-10\-05T13:01:05\.197156\+00:00: rank \#81

2026\-10\-05T14:01:04\.085718\+00:00: rank \#81

2026\-10\-05T15:00:50\.814571\+00:00: rank \#81

2026\-10\-05T16:02:09\.54139\+00:00: rank \#80

2026\-10\-05T17:00:35\.336076\+00:00: rank \#80

2026\-10\-05T18:00:20\.445491\+00:00: rank \#80

2026\-10\-05T19:02:43\.604975\+00:00: rank \#81

2026\-10\-05T20:01:09\.777459\+00:00: rank \#81

2026\-10\-05T21:01:47\.650671\+00:00: rank \#82

2026\-10\-05T22:02:59\.93099\+00:00: rank \#83

2026\-10\-06T08:02:11\.210914\+00:00: rank \#83

2026\-10\-06T08:02:14\.354697\+00:00: rank \#83

2026\-10\-06T09:01:21\.688765\+00:00: rank \#85

2026\-10\-06T10:00:58\.047773\+00:00: rank \#85

2026\-10\-06T11:00:41\.689929\+00:00: rank \#86

2026\-10\-06T12:00:41\.7705\+00:00: rank \#87

2026\-10\-06T13:01:24\.509519\+00:00: rank \#87

2026\-10\-06T14:00:26\.950817\+00:00: rank \#87

2026\-10\-06T15:00:49\.066414\+00:00: rank \#88

2026\-10\-06T16:00:58\.166747\+00:00: rank \#90

2026\-10\-06T17:01:17\.009511\+00:00: rank \#90

2026\-10\-06T18:01:43\.791429\+00:00: rank \#91

2026\-10\-06T19:00:46\.048609\+00:00: rank \#91

2026\-10\-06T20:01:14\.687294\+00:00: rank \#91

2026\-10\-06T21:02:13\.414382\+00:00: rank \#91

2026\-10\-06T22:00:40\.926271\+00:00: rank \#90

2026\-10\-06T23:03:26\.867445\+00:00: rank \#92

2026\-10\-07T08:01:41\.339804\+00:00: rank \#93

2026\-10\-07T09:02:38\.026986\+00:00: rank \#94

2026\-10\-07T10:01:35\.069531\+00:00: rank \#94

2026\-10\-07T11:02:07\.770084\+00:00: rank \#94

2026\-10\-07T12:01:14\.996697\+00:00: rank \#93

2026\-10\-07T13:00:47\.45912\+00:00: rank \#92

2026\-10\-07T14:01:18\.307739\+00:00: rank \#92

2026\-10\-07T15:01:57\.541362\+00:00: rank \#95

2026\-10\-07T16:01:28\.474022\+00:00: rank \#95

2026\-10\-07T17:01:15\.036908\+00:00: rank \#96

2026\-10\-07T18:00:53\.300079\+00:00: rank \#95

2026\-10\-07T19:01:49\.456657\+00:00: rank \#96

2026\-10\-07T20:01:59\.032487\+00:00: rank \#96

2026\-10\-07T21:05:36\.655392\+00:00: rank \#98

2026\-10\-07T22:02:27\.105241\+00:00: rank \#99

2026\-10\-07T23:01:17\.452894\+00:00: rank \#99

2026\-10\-08T08:03:55\.083808\+00:00: rank \#99

2026\-10\-08T09:03:58\.68898\+00:00: rank \#97

2026\-10\-08T10:04:05\.364235\+00:00: rank \#98

2026\-10\-08T11:02:04\.410307\+00:00: rank \#97

2026\-10\-08T12:02:36\.39618\+00:00: rank \#97

2026\-10\-08T13:03:09\.206401\+00:00: rank \#97

2026\-10\-08T14:03:30\.113384\+00:00: rank \#98

2026\-10\-08T15:02:54\.444758\+00:00: rank \#96

2026\-10\-08T16:03:13\.067807\+00:00: rank \#95

2026\-10\-08T17:03:47\.92465\+00:00: rank \#94

2026\-10\-08T18:01:50\.360431\+00:00: rank \#94

2026\-10\-08T19:02:55\.719274\+00:00: rank \#95

2026\-10\-08T20:01:36\.359022\+00:00: rank \#95

2026\-10\-08T21:03:34\.000719\+00:00: rank \#95

2026\-10\-08T22:01:13\.459732\+00:00: rank \#95

2026\-10\-08T23:01:25\.318392\+00:00: rank \#94

2026\-10\-09T08:01:53\.314021\+00:00: rank \#95

2026\-10\-09T09:02:59\.93426\+00:00: rank \#96

2026\-10\-09T10:02:22\.652527\+00:00: rank \#95

2026\-10\-09T11:02:04\.807282\+00:00: rank \#95

2026\-10\-09T12:02:16\.769295\+00:00: rank \#95

2026\-10\-09T13:02:30\.35034\+00:00: rank \#97

2026\-10\-09T14:01:34\.215134\+00:00: rank \#97

2026\-10\-09T15:01:27\.564392\+00:00: rank \#96

2026\-10\-09T16:02:06\.561056\+00:00: rank \#97

2026\-10\-09T17:03:06\.673648\+00:00: rank \#99

2026\-10\-09T18:01:02\.851052\+00:00: rank \#99

2026\-10\-09T19:01:11\.250291\+00:00: rank \#99

2026\-10\-09T20:01:32\.91907\+00:00: rank \#99

2026\-10\-09T21:02:08\.376144\+00:00: rank \#101

2026\-10\-09T22:01:19\.253075\+00:00: rank \#103

2026\-10\-09T23:01:31\.990912\+00:00: rank \#103

2026\-10\-10T08:01:00\.593376\+00:00: rank \#102

2026\-10\-10T09:01:45\.527975\+00:00: rank \#104

2026\-10\-10T10:00:53\.799353\+00:00: rank \#104

2026\-10\-10T11:00:35\.65851\+00:00: rank \#103

2026\-10\-10T12:01:11\.169647\+00:00: rank \#103

2026\-10\-10T13:00:54\.008377\+00:00: rank \#102

2026\-10\-10T14:01:43\.208766\+00:00: rank \#104

2026\-10\-10T15:01:24\.825188\+00:00: rank \#104

2026\-10\-10T16:01:10\.213328\+00:00: rank \#103

2026\-10\-10T17:00:48\.84189\+00:00: rank \#102

2026\-10\-10T18:00:57\.55132\+00:00: rank \#102

2026\-10\-10T19:00:52\.379463\+00:00: rank \#101

2026\-10\-10T20:00:21\.993912\+00:00: rank \#101

2026\-10\-10T21:01:10\.657738\+00:00: rank \#102

2026\-10\-10T22:00:50\.433569\+00:00: rank \#101

2026\-10\-10T23:01:00\.95027\+00:00: rank \#102

The benchmark shows current LLM agents are far from competent Brood War play, while the discussion contextualizes that result with prior StarCraft bots, questions about harness design and training exposure, and the APM/latency gap that may limit direct game benchmarks\.

## The brief

Brood War Bench is a round\-robin benchmark of LLM agents playing StarCraft: Brood War through a harness\. It ranks 19 model/effort configurations by wins, APM, and cost, and reports that none played beyond beginner level; Codex Astra led, Grok lagged, and models often mishandled thinking time, subagent coordination, and sustained production\.

- The benchmark ran a round\-robin matrix of 19 model and effort configurations, with every configuration playing every other, using Freestyle VMs and saving game\-engine and harness logs\.
- Codex Astra / xhigh topped the leaderboard at 18\-0 with a 100% win rate and $10\.54 per game; Codex Astra / medium was 16\-2, and Claude Fable was 15\-3\.
- Grok 4\.6 and Claude Haiku occupied the bottom: Grok 4\.6 / low and Claude Haiku both went 0\-16, while Grok 4\.6 / xhigh was 2\-15\.
- No agent played beyond beginner level; even Astra and Fable could not build complex armies, defend simple attacks, or execute concrete strategies, and the author says a beginner photon rush would win every game\.
- Codex often found disruption or cheese before macro, such as sending a Probe across the map to attack workers or buildings, while its separate economy, army\-production, and army\-control subagents communicated poorly and trickled units into attacks\.
- Grok 4\.6 frequently produced long reasoning with few command batches; in G043 the xhigh run logged 11,138 reasoning tokens but issued only six command batches across 43 minutes and never fielded a combat unit\.

## Discussion themes

### Harness details and replay learning

Commenters want more detail on what representation of the screen and world state the agents received and what tools they had available\. One suggests a replay\-deciphering library so an agent could learn by watching Brood War replays\.

Sources: [Comment 49769868](<https://news.ycombinator.com/item?id=49769868>) · [Comment 49769790](<https://news.ycombinator.com/item?id=49769790>)

### Prior StarCraft AI context

Discussion places the benchmark against earlier work: DeepMind had StarCraft II bots beating professional gamers in 2018, a current ladder bot is reportedly strong but likely APM\-unlimited and possibly map\-hacking, and early BWAPI tournaments and genetic\-programming projects show older approaches\.

Sources: [Comment 49769911](<https://news.ycombinator.com/item?id=49769911>) · [Comment 49769947](<https://news.ycombinator.com/item?id=49769947>) · [Comment 49770269](<https://news.ycombinator.com/item?id=49770269>) · [Comment 49770169](<https://news.ycombinator.com/item?id=49770169>)

### GoBench comparison and benchmark design

One commenter cites GoBench, which evaluates LLMs on 9x9 Go with KataGo Elo anchors and shows real capability differences, arguing strategy is a useful LLM evaluation area\. A counterargument says direct game benchmarks are confounded by how much the game is in training data; a better test might be asking LLMs to write a Go AI under a strict cycle budget, since programming an engine is a more general skill\.

Sources: [Comment 49769906](<https://news.ycombinator.com/item?id=49769906>) · [Comment 49770349](<https://news.ycombinator.com/item?id=49770349>)

### Training\-data and strategy speculation

Commenters speculate that many Brood War conversations are in training data and ask whether play style reveals strategies people complain or meme about online\. One assumes AI will tend to pick Zerg because high\-level Zerg is considered stronger, prompting a question about whether that is due to rushing\.

Sources: [Comment 49769721](<https://news.ycombinator.com/item?id=49769721>) · [Comment 49770286](<https://news.ycombinator.com/item?id=49770286>) · [Comment 49770416](<https://news.ycombinator.com/item?id=49770416>)

### APM and latency limits

A prediction that LLMs will beat DeepMind's StarCraft II model within 12 months is countered by the argument that StarCraft is APM\-dependent and frontier reasoning latency makes LLM play like knitting with an excavator unless latency improves greatly\.

Sources: [Comment 49769952](<https://news.ycombinator.com/item?id=49769952>) · [Comment 49770112](<https://news.ycombinator.com/item?id=49770112>)

## Sources & coverage

AI-generated summary · 2026\-09\-20T12:00:27\.840778\+00:00

Based on 16 of 16 usable stored comments, selected by depth and branch activity. This is a sample of the discussion. Article text may also be shortened.

Generated using deepseek\-ai/DeepSeek\-V4\.1\-Flash. Check the linked sources for full context.