PilotX: a field study from Robosoft Technologies
The question most engineering leaders are trying to answer right now is which AI coding tool to standardize on. It is a reasonable question. It is also a secondary one.
The finding from six weeks of controlled evaluation that changed how we think about delivery was not which tool performed best. It was this: the tool is not the primary variable. The operating model around it is.
Why we ran this study
The market for AI coding tools is loud and moving fast. Every vendor has benchmarks. Every benchmark tells a different story. None of them answers the question that matters to a team building software under real conditions: what changes about delivery when AI is in the loop?
How we ran the experiment
PilotX ran for six weeks. The starting point was a real OTT mobile application we had already delivered, built natively on Android and iOS by six developers over three months using conventional methods. We then took the same product, the same backlog of 86 user stories drawn from that live engagement, and asked a different question: what happens when agentic coding replaces traditional development from the ground up?
The same team worked across three consecutive two-week sprints, one per tool, against the same fixed backlog. Every set of stories was duplicated across tools in Jira, with identical requirements, designs, and API specifications available to each. A fourth sprint was dedicated to assessment: architect-led code reviews, unit test coverage analysis, app profiling, and a structured QA pass.
We selected the three most mature AI coding platforms available at the time. Codex and Anti-Gravity had only just launched and were excluded. Kiro had shown promise in an earlier internal trial. Cursor was gaining fast adoption, including among clients such as Warner Bros. Discovery. All three entered PilotX. Not all three finished.
Kiro exhausted its entire credit allocation within 1.5 days, largely because the frontend team was working with Figma MCP integrations that burned through tokens at high speed. We discontinued that track. Any organization evaluating AI tools at scale needs to account for that kind of constraint before committing to a decision.
Claude Code and Cursor completed the full evaluation.
What the data showed
The results were unambiguous.
Claude Code outperforms Cursor on every measured dimension. The mobile unit test coverage gap tells the clearest story: a 76% to 25% differential means more defects caught before production, fewer late-stage failures, and meaningfully greater confidence in every release. Across a sustained delivery program, that difference compounds in ways that show up in cost, timeline, and client trust.
The logical conclusion from that data: standardize on Claude Code.
We chose Cursor as our primary team tool.
Claude Code is a terminal-based coding agent that takes natural language instructions and works autonomously across multiple files and systems. It is also a fundamentally different way of working, and for engineers who have spent their careers inside an IDE, that transition takes time. Productivity climbs, but not immediately. There is a period of adjustment before teams find their footing on live delivery programs with committed timelines, that adjustment carries real cost.
Cursor sits inside VS Code. For most engineers, the switch is nearly invisible. Teams are at full capacity from sprint one.
What PilotX revealed is the gap between what a tool can achieve at its peak and what a full delivery team can sustain across months of real project work. Most evaluations only measure the first. The second is where the business value lives. Not because the tools are wrong. Because the adoption never reaches the depth the benchmarks assumed.
The answer PilotX produced is not a single tool. It is a two-tool model with clearly defined purposes.
Cursor is the delivery baseline: every engineer, every project, every day. Day-to-day feature work, backend development, and iterative refactoring. Zero onboarding friction, consistent output from sprint one. It raises the floor of daily delivery across the entire team.
Claude Code is the precision instrument: reserved for work where its depth changes what is achievable. Large-scale refactors, architecture-level changes, complex multi-file scaffolding, and accessibility passes where correctness is non-negotiable. Directed by engineers with the context and judgment to deploy it at its ceiling.
Output quality was almost entirely determined by what was put in front of the agent. Four inputs separated strong output from output that had to be rebuilt.
Requirements precision. Stories written with explicit acceptance criteria, specific inputs, expected outputs, and clearly defined edge cases produced implementations that needed one or two review cycles. Loose or assumed criteria produced code that was plausible but wrong, often in ways invisible until integration or QA. The AI does not flag ambiguity. It resolves it, silently, in whatever direction the available context suggests.
Design specificity. UI work that carried a direct Figma reference to the approved design, with assets exported and available to the agent rather than left for visual interpretation, produced output that matched intent. When Figma was handed to an agent without that constraint, it generated assets independently from visual inspection of the prototype, producing output that was structurally correct but visually wrong. That is a more expensive kind of rework than a logic error.
API contract integrity. Backend integration stories that included a stable API spec produced clean model generation, correct error handling, and well-structured integration code. When the contract was still in motion, the agent invented one. That invented code had to be found and removed.
Standardized repo context. The most significant single factor in output consistency across all three tools was whether the repository carried clear, stable engineering standards: architecture patterns, tooling choices, naming conventions, and quality expectations. When that context was absent or inconsistent, output drifted across stories in ways that accumulated cost across the sprint. We also observed agents autonomously removing unit test cases and, in some instances, deleting entire architecture layers. The agent fills the vacuum. If you do not define what must not change, it will change it.
None of these are AI-specific problems. They are delivery fundamentals that matter in any project. What AI does is make their absence immediately and expensively visible. A vague ticket in a conventional delivery model produces a slow developer. A vague ticket in an AI-native delivery model produces fast, confident, wrong code.
How delivery changes when AI is in the loop
Most teams assume AI-native delivery compresses development time. It does. What it also does is relocate effort in ways that benchmarks never capture.
Well-run AI-native teams invest more upstream. Requirements need to be written to a higher standard of precision than most teams are used to. Design references need to be in the ticket before development starts, with the agent's relationship to those references defined precisely, down to whether it reads from Figma or exports from it. API contracts need to be stable before integration stories begin. The engineering standards that live in the codebase need to be explicit, current, and consistently applied.
Strip any one of those conditions and the apparent speed becomes a liability. Volume gets generated quickly, but a sizeable portion must be unwound.
Development compresses. The thinking that precedes it does not.
What this means for your organization
For organizations running an AI engineering pilot, planning one, or trying to understand why an existing program is not delivering what it promised, the findings from PilotX point to three questions that matter more than the choice of tool.
Are your engineering foundations ready?
Vague requirements, undocumented design systems, and unstable API contracts do not slow AI-native development down. They corrupt its output in ways that are costly and non-obvious to fix. The first investment in any serious AI engineering program should be in the conditions that make AI output reliable, not in the licenses. Robosoft collaborates with clients to assess that readiness and put the right foundations in place before AI tooling reaches the engineering team.
What is your delivery partner's operating model?
The right question for any partner is not which AI tools they use. It is how they structure requirements for AI, how they govern output quality sprint by sprint, and how they manage the shift from a controlled pilot to sustained delivery at scale. Those answers tell you whether AI will compound the quality of a partner's work or simply accelerate its existing patterns.
At Robosoft, the operating model PilotX produced now travels into every client engagement: requirements and design formats built for AI, standardized engineering context across every repo, and architect review in every sprint.
Are you sequencing adoption in a way that builds lasting capability?
AI-native development delivers its clearest returns on well-scoped, well-specified work. The organizations that extract lasting value start with the right work, build the operating model around it, and expand from demonstrated results. Rushing scale before the foundations are in place is the most common reason AI programs deliver less than they promised.
What PilotX is part of
At Robosoft, we have been building ARIA around a conviction this study only sharpened: there is a meaningful difference between adding AI to the way you already work and rebuilding the way you work around AI. The first produces gains that are real but uneven. The second produces something that compounds over time, in quality, in speed, in what a team is genuinely capable of shipping.
PilotX is that conviction evaluated against evidence. The findings are now embedded in how we deliver. Every engagement benefits from a two-tool architecture matched to the nature of the work, engineering standards built for AI from the ground up, and an approach to adoption that goes well beyond installing software. The next phase of PilotX is already underway, focused on closing the remaining reliability gap and pushing agent output toward near-zero rework on the first prompt, at scale.
If you are working through what AI-native engineering should look like for your organization, we would like to talk. Not to present a conclusion, but because the organizations that get this right are the ones willing to treat it as a serious question.
PilotX was conducted by Robosoft Technologies over six weeks across mobile (Android, iOS), backend (Node.js, Strapi), and frontend (React) engineering. The study evaluated Cursor, Kiro, and Claude Code across productivity, code quality, app performance, QA and defects, developer experience, and unit test coverage. PilotX is part of ARIA by Robosoft.
