A recent study by Mercor Research found that approximately 90% of working hours in the US are spent on tasks that are not well covered by existing AI benchmarks. The analysis, based on an initial list of 29 economic benchmarks, highlights significant gaps in how current AI performance metrics align with real-world job tasks, particularly outside highly benchmarked fields like software engineering.
Mercor Research selected benchmarks to maximize coverage across different job families while avoiding oversaturation, which can inflate AI performance estimates. The study acknowledges limitations due to the provisional nature of the benchmark list and the challenges in measuring economic realism. By capping benchmarks per job family, the researchers aimed to provide a more accurate economy-wide estimate, despite potential underrepresentation in some sectors.
This finding underscores challenges in applying large language models (LLMs) and AI tools to the broader labor market, as many tasks remain underrepresented in benchmarks. The study suggests that current benchmarks may overstate AI capabilities in practical deployments, especially in tasks that require nuanced understanding or complex human judgment. This gap could affect how businesses and policymakers evaluate AI’s impact on productivity and workforce transformation.
Mercor Research plans to update its benchmark list to improve coverage and accuracy. The study’s results serve as a baseline for understanding AI’s economic reach, emphasizing the need for more comprehensive benchmarks to better reflect the diversity of US labor tasks and inform AI development strategies.