The StarSkirmish benchmark v0.2 evaluated large language models (LLMs) on their ability to write Protoss bots for StarCraft: Brood War within one hour. GPT-6 Astra achieved the highest score, outperforming GPT-6.1 Sol and other leading models like Gemini 4 Argon and Claude Opus 5.5, which scored closely behind, according to starskirmish.com.
Each LLM was given one hour of wall clock time to write a C++ Protoss bot, which was then tested against a roster of competitive human-written bots and demo bots. The score reflects the average win rate over five runs. Gemini 4 Argon ran at its highest reasoning effort, while other models ran at their maximum settings. GPT-6 Astra’s top score marked a clear lead over GPT-6.1 Sol in second place.
StarSkirmish tests long-horizon reasoning and agentic coding by requiring LLMs to incorporate learnings from practice games to improve their bots. This benchmark provides a unique measure of LLMs’ capabilities in strategic game AI development. The close scores among Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 highlight competitive performance in this emerging evaluation domain.
The benchmark also compares model performance against the average API cost of a one-hour run, plotted on a log scale. This cost-performance analysis offers insight into the efficiency of different LLMs in producing competitive StarCraft bots, as detailed on starskirmish.com.