Research from Mercor suggests a substantial misalignment between the scope of American labor and the metrics used to evaluate artificial intelligence. According to the study, approximately 90% of US working hours are spent on tasks that have little to no coverage in existing AI benchmarks. This discrepancy suggests that while AI models are being tested on specific capabilities, those tests may not represent the actual day-to-day work performed by the majority of the workforce. The study notes that even when controlling for AI exposure, tasks from high-wage occupations are significantly overrepresented in benchmark prompts, potentially indicating a focus on automating high-value roles rather than general labor.
Several factors contribute to this gap, including the nature of professional work and the limitations of current testing methodologies. Some observers point out that many professional roles involve complex human interactions—such as social work or psychiatry—that are difficult to quantify or replicate with a set of prompts. Others suggest that AI benchmarks often focus on the ability to 'solve' a problem, whereas a critical component of real-world work is the ability to detect and correct errors. Furthermore, the data relies on O*NET, a database of occupational classifications that some critics argue is outdated and fails to accurately capture modern white-collar roles in fields like tech, finance, or consulting.
The practical implications of this gap involve how productivity and automation are perceived in the workplace. Some argue that the 'bottleneck' in many industries is not the execution of a task, but the subsequent review, feedback, and sign-off processes, which remain human-centric. Others observe a 'cost disease' effect where, as specific tasks like coding become more efficient through AI, workers may spend a disproportionate amount of time on non-automated administrative overhead, such as meetings and project management. These perspectives suggest that the impact of AI may be more nuanced than a simple replacement of tasks, as the remaining human work may shift toward managing the outputs and complexities created by automated systems.