Mercor scaled from fewer than a dozen active client projects to managing hundreds of projects while growing rapidly in headcount. The company had no data team and lacked a central analytics platform, collaborative dashboards, or reliable access to key operational metrics. Teams pulled raw data via VPN into AWS and relied on spreadsheets and a few technical people for custom reports. For a business operating on hour-to-hour timelines, these delays risked millions in lost revenue. Mercor made a single analytics platform the foundation for Ops, Finance, Sourcing, and Sales, connecting data from its warehouse and operational sources like Google Sheets, Airtable, and the Mercor platform. The company rolled out self-serve reporting so non-technical users could build dashboards without needing SQL or Python. Notebook-based AI assistance removed the reporting bottleneck and enabled teams to iterate on metrics and views in real time. Operations used dashboards to monitor project health across hundreds of customer engagements. Decision cycles were compressed from days to hours, enabling faster action on throughput, efficiency, quality, and revenue metrics. Over the past year, improved execution and velocity expanded capacity to take on more projects, which unlocked over $100M in revenue. Dashboards were created in hours rather than days, and the operations team tracked 60+ metrics per project across hundreds of active projects. Mercor also reported zero enterprise customer churn.

Project Details

Mercor needed to prove that a small amount of expert-labeled data could materially improve real-world agent performance on long-horizon, professional tasks. The goal was to drive measurable gains on the APEX-Agents benchmark, which tested day-to-day work across investment banking, management consulting, and corporate law. A key risk in this low-data setting was wasting scarce expert effort on data that would not transfer to the hardest benchmark tasks. Mercor partnered with Applied Compute to post-train an open-source model using an expert-labeled dev set. Mercor supplied a dev set of 874 tasks split across 50 unique “worlds,” and none of the tasks or worlds appeared in the APEX-Agents benchmark. Applied Compute deployed its proprietary long-horizon RL stack and ran single-epoch training with no SFT warmup, no filtering, and no task or rubric modifications. The team evaluated performance on the full APEX-Agents benchmark (n=480) using Pass@1, Pass@3, and mean criteria passed, starting from a GLM 4.6 baseline. The post-trained model outperformed the baseline across all metrics using just 874 expert-labeled tasks, with the largest gains in corporate law. With fewer than 1,000 high-quality data points, Pass@1 and mean score nearly doubled on APEX-Agents. On the corporate law evaluations, Pass@1 tripled. The baseline GLM 4.6 model scored 3.8% Pass@1 and 12.1% mean score prior to post-training, and the training trendline remained near-linear, indicating additional data would likely continue yielding gains.

Project Details

The customer needed to identify domain experts capable of generating problems that stumped current AI models like GPT-4 for next-generation AI training. Their prior sourcing process took weeks per search. It often returned candidates who appeared qualified but lacked actual expertise. They partnered with Sixtyfour AI to improve expert discovery and validation. Recursive enrichment agents traversed academic publications, co-authors, conference presentations, and specialized forums. This built comprehensive expertise profiles to surface qualified domain experts for AI labs. The customer reduced the time required to deliver qualified domain experts from weeks to hours. The process supported sourcing across multiple specializations, including rare genetic dermatology, investment banking, and competitive programming. It improved confidence that sourced experts had the depth required to produce AI-training problems that challenged GPT-4-level models.

Skills

Project Details