Ramp SWE-Bench is a private, production-grounded coding benchmark created from engineering work in Ramp’s backend. Public benchmarks saturate quickly and can leak into training data, with none quite resembling the work our engineers do every day. Building our own has given us and the model providers we work with a contamination-free read on how different models handle engineering at Ramp.
Modeled after SWE-Bench, we use this benchmark as a behavioral instrument for studying how coding agents handle the work Ramp engineers already delegate to them, answering questions like:
- How do they navigate massive codebases?
- Do they produce correct code?
- When do they decide to ship a change?
- How long do they run?
- What do they cost?
- Where do they break?
Tasks derive from changes that Inspect, our in-house background coding agent, shipped to production after engineer review. The benchmark's one-shot nature effectively models the background agent lifecycle where an engineer provides a prompt, the agent works asynchronously, and the expected output is a review-ready pull request.
Work spans Ramp product and platform domains including:
Tasks
Every one of the 80 tasks in the final set derives from a pull request that includes business logic and tests, carries an engineer-to-Inspect conversation, and was merged into production after code review. Each has an explicit anchor in the engineer's original intent: prompts are synthesized from what the engineer actually asked the agent to do, not from issue or PR descriptions alone.
To assemble a task, we reconstruct the repository at the PR's base commit, hold out the merged solution patch and associated test patch as gold artifacts, and synthesize a task prompt from the engineer's messages in the original Inspect conversation.
Curation
Task curation is automated except for final human approval, which acts as the primary bottleneck and quality-control step. LLMs excel at surfacing issues, ambiguity, and fairness risks, but human engineering judgement is best for ensuring benchmark quality. The pipeline extracts as many signals as possible before review, so reviewers can make fast, informed decisions.
Our task-auditing framework is inspired by BenchGuard's view of execution-based benchmarks as coupled artifacts (prompt, gold patch, tests, and environment) that must be checked jointly for fairness and validity. LLM judges cross-examine each task and sort every defect two ways: whether it lives in the prompt or the tests, and whether it says too much (over-specified) or too little (under-specified). The judges run on different model providers to prevent same-provider bias.
test_overconstrained Tests assert details like a helper name or exact string
test_missing_assertion Tests don't cover enough of the task's requirements
prompt_solution_leakage Leaks implementation details, handing the agent the answer
prompt_missing_context Doesn't include critical information for completing the task
Static review only catches so much. After a clean audit, tasks are run against a ladder of models, from small to frontier, reading where each lands on the success curve. When no model solves the task, it could mean a brittle test or broken environment over real difficulty. When every model solves it, the task is discarded for carrying no signal. Tasks worth keeping sit in between, where capability separates models and the failing traces show that the miss is clean.
Harness
All models run against the same curated tasks using the mini-swe-agent harness. Agents have bash access in a sandboxed local development environment, and every model gets the exact same environment for a task. While leaner than the scaffolds engineers use in practice, it lets us isolate model behavior from harness heuristics. Each result is a single pass@1 attempt, mirroring the bar engineers hold background agents to: a correct, review-ready change on the first try.
Scoring
A run is graded as a success if the agent's diff flips the task's failing tests to passing without breaking the others. Runs that reach a context window limit are counted as failures.
A potentially valid patch can still fail tests if they are unfair or overly specified, for example referencing a specific function name. Despite our curation pipeline greatly minimizing this failure mode, it is inherent to the SWE-Bench evaluation mechanism. In some tasks, tests are intentionally rigid, as the correct solution is to closely follow existing repository patterns and conventions.
Privacy & contamination
Because tasks derive from private Ramp production code, we cannot release prompts, patches, tests, or repository states. Our public dashboard reports aggregate metrics, pairwise comparisons, and redacted examples. Ramp SWE-Bench data is never used for training, enforced through agreements with model providers.
Future work
Ramp SWE-Bench v1 is an early look at a living internal eval. We plan to grow the task set, evaluate richer failure taxonomies, add repeated-run reliability measures, compute judge-based scoring of output quality, build further automation around trace analysis and pairwise model comparisons, and experiment with other harnesses.
References
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. R. (2024). SWE-bench: Can Language Models Resolve Real-world GitHub Issues? In The Twelfth International Conference on Learning Representations.
Tu, X., Wang, T., Lu, Y. (Minta), Huang, K., Qu, Y., & Mostafavi, S. (2026). BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks. arXiv preprint arXiv:2604.24955.
SWE-agent. (2026). mini-swe-agent: The minimal AI software engineering agent. GitHub repository.