An AI research organization needed to improve LLM response accuracy while rapidly expanding its training workforce. The existing approach could not scale fast enough to meet growing task volume. They also needed consistent alignment across a larger group as task instructions changed frequently. Maintaining quality standards during rapid onboarding was a critical concern. A two-pronged strategy was implemented to increase response accuracy and scale the training team quickly. A bespoke vetting process was used to source and integrate LLM trainers through rigorous trial runs. A top-down communication approach kept the team aligned with frequent task instruction updates. A dedicated quality team provided continuous feedback and coaching to maintain standards. The effort onboarded 130+ LLM trainers in under two months. The team scaled 2X in 2 months while completing 12,000+ training tasks. Response quality and accuracy improved as quality standards were continuously reinforced. The program established an internal benchmark for future LLM initiatives.

Skills

Project Details

A global technology company needed a systematic way to understand the strengths and weaknesses of its custom LLM for coding tasks. Existing assessments did not consistently capture performance across different task types and difficulty levels. The team also needed a structured approach to surface failure modes and prioritize model refinements. Six targeted evaluation projects were implemented over a two-week sprint to assess the LLM comprehensively. The work included Guided API Evaluation, Freestyle API Evaluation, Prompt Breaking, LLM and Human Benchmark Analyses, Community Findings Aggregation, and RLHF & Calibration. Four assessment levels were used to test tasks ranging from principal engineer-level complexity to rudimentary-level tasks. A defined data split was applied to balance targeted cases, known weaknesses, and baseline scenarios. The effort delivered structured findings that clarified where the model performed well and where it broke down across task difficulty tiers. The results produced a difficulty rubric that supported iterative prompt assessment. The evaluation produced actionable insights that informed subsequent model refinement decisions.

Skills

Project Details

An enterprise software company needed high-quality multimodal training data grounded in real-world web experiences. The team required supervision that spanned code changes, visual understanding, and layout structure. They also needed consistent quality across tasks derived from diverse websites. Ensuring accuracy and reliability at scale was a central challenge. A multimodal dataset was delivered combining real-world code edits, visual question-answering (VQA), and structural sketches derived from website screenshots. The annotation pipeline included code edit tasks with HTML/CSS/JS modifications across multiple difficulty levels, web sketches with standardized component tagging, and VQA that produced five questions per screenshot. Tasks were created from a large set of real web screenshots across many domains. Each task underwent two-step human validation for quality assurance. The effort produced thousands of multimodal supervision tasks across three distinct output types. The dataset covered more than twenty real-world web domains to improve breadth and generalization. Layout sketches used over ten standardized tags to support clarity and consistency. Quality was reinforced through a two-step human validation process applied throughout.

Skills

Project Details

The customer needed a large-scale, real-world software engineering benchmark built from a complex open-source codebase. Existing approaches that relied only on unit tests did not reflect complete user workflows. The customer also needed grading that accepted any valid solution while rejecting invalid ones. They required a process that could handle large volumes of real issue reports without compromising benchmark quality. A large-scale benchmark was designed using prompts derived from real issue reports in the open-source project. Each task included a self-contained prompt and a solution-agnostic end-to-end UI test grader to validate correctness. Resolved issues were reviewed to source and curate tasks, and quality checks were applied to exclude weak candidates. Tasks were retained based on quality requirements and prepared for use as a real-world evaluation set. The effort resulted in a retained set of 1,500+ benchmark tasks built from reviewed resolved issues. The benchmark used 100% end-to-end UI test graders to evaluate complete user workflows rather than relying solely on unit tests. Approximately 500 candidate tasks were excluded for quality to improve the final dataset. The final benchmark supported solution-agnostic grading by accepting any valid solution and rejecting invalid ones.

Skills

Project Details

An AI research enterprise needed a large-scale, real-world software engineering benchmark built from a complex open-source codebase. The customer required tasks grounded in real issue reports rather than synthetic prompts. They also needed grading that could validate end-to-end user workflows and resist being gamed. Unit-test-only approaches did not meet their accuracy and robustness needs. A large-scale benchmark was designed from a complex open-source codebase by reviewing resolved issues and converting them into self-contained tasks. Each task included a prompt derived from a real issue report and an end-to-end UI test oracle. The E2E UI test graders were implemented to accept any valid solution and reject invalid ones while remaining solution-agnostic. Expert engineers also tagged task difficulty to support evaluation across skill levels. The effort resulted in a retained set of 1,500+ benchmark tasks constructed from a broad review of 2,000+ resolved issues. All tasks were paired with E2E UI test graders, enabling evaluation of complete user workflows rather than isolated unit behavior. Approximately 500 candidates were excluded for quality, improving the benchmark’s reliability. The final benchmark design supported solution-agnostic grading and harder-to-game evaluation.

Skills

Project Details