In the first post I made six predictions about what would happen if I let an AI agent implement a backlog, and set up an experiment designed to prove me wrong where I deserved it. This is the results post.

If you’re starting here, the short version. The project is jira-gateway: a REST API wrapper over the Jira API, built as an Azure Function App in TypeScript. I wrote one endpoint by hand as a seed, plus a CLAUDE.md file — the instruction file that tells the agent how the codebase is put together and what the rules are. Everything after that came from Jira tickets, implemented by Claude Code, reviewed by me, and merged through a pull request. I haven’t written a line of implementation code since the seed.

Six tickets. Four specified properly. One written deliberately badly. One written to be genuinely hard. And one control run — the same ticket run again with the instruction file deleted, to find out whether it was actually doing anything. All six shipped.

Where We Left Off

	Hypothesis
H1	Agentic development will be significantly faster for repetitive implementation work
H2	Ticket quality has a direct impact on code quality
H3	Claude Code will be surprisingly effective at tests and boilerplate
H4	Most failures come from requirement ambiguity, not coding mistakes
H5	The bottleneck shifts from implementation to review
H6	CLAUDE.md measurably improves output quality

I locked those down in the first post, before any tickets ran, in a document I couldn’t quietly edit afterwards. That matters more than it sounds. The temptation to bend a hypothesis to fit a result you like is real, and I wanted the scorecard to be able to embarrass me.

The working loop was the same for every ticket: the agent reads the ticket, proposes a plan, I approve or push back, it writes the code and the tests, opens a pull request, and I review that like I’d review anyone else’s work. The plan and the PR are the two places a human gets to say no.

What Actually Got Built

Seven endpoints, all live on Azure:

Method	Route	Written by
GET	/api/ticket/{ticketId}	Human (seed)
GET	/api/projects	Claude Code
POST	/api/ticket	Claude Code
GET	/api/boards	Claude Code
GET	/api/tickets?jql=…	Claude Code
POST	/api/ticket/{ticketId}/comment	Claude Code
POST	/api/ticket/{ticketId}/transition	Claude Code

69 tests across 7 suites by the end. Every endpoint was verified against live Jira, not just against its own mocks, and the whole backlog went from first prompt to last deploy inside a day.

How It Measured Up

	Hypothesis	Verdict
H1	Faster for repetitive work	Confirmed — but not evenly
H2	Ticket quality drives code quality	Confirmed — on a different axis than I expected
H3	Strong at tests and boilerplate	Mixed
H4	Failures come from ambiguity, not coding	Confirmed, strongly
H5	Bottleneck shifts to review	Confirmed — and that’s a problem
H6	CLAUDE.md improves output	Split: no for code, yes for workflow

H1 — Faster, But Not Evenly

The ticket-creation endpoint went from start of work to an open pull request with thirteen tests in about a hundred seconds. I went back and checked the timestamps twice.

The very next ticket took fifteen minutes. Same agent, same instructions, roughly the same amount of code. The difference was that it was the first endpoint to step off the established pattern — it needed a different Jira API, a different base URL, and a shared piece of client code had to be generalised before anything else could happen.

That’s a nine-times spread between two tickets that look identical on a board. So the prediction held, but the useful version of it is more specific: speed tracks pattern familiarity. The first endpoint of a new shape costs real time. The tenth of the same shape is nearly free. If you’re estimating agent work, “is there an existing example of this in the codebase?” turns out to be the question that matters.

H2 — Right Prediction, Wrong Axis

I expected vague tickets to produce worse code. That’s not what happened.

What actually predicted trouble was whether a ticket was correct, not how detailed it was. You can see it in how often the agent had to stop and ask me something:

Ticket	Character	Questions asked
GET /boards	Detailed, but contained a factually wrong list of valid values	3
comments	One vague, self-contradictory sentence	2
transition	The hardest ticket in the backlog, fully specified	0

The hardest ticket — a status-transition endpoint that needs two chained Jira API calls and error handling with no precedent anywhere in the codebase — asked nothing at all, because it was the most completely specified. Complexity didn’t create uncertainty. Unspecified decisions did.

The clearest illustration was the same technical detail turning up in two different tickets. Jira’s API wants rich text in a JSON structure called Atlassian Document Format, not a plain string. My detailed ticket for creating tickets confidently specified a plain string — which is wrong — and the agent had to spot that and work around it. My deliberately vague one-line ticket for comments said nothing about the format at all, and the agent simply did it correctly, because nothing in the ticket was fighting what it already knew.

A confidently wrong ticket turned out to be worse than a vague one. That surprised me, and it changes what I’d tell people to focus on when writing tickets for an agent: getting the facts right beats writing more of them.

H3 — It Writes Excellent Tests, Which Isn’t the Same as Good Coverage

Every ticket asked for one or two tests. The agent wrote eleven to thirteen each time — authentication failures, malformed request bodies, missing server configuration, error mapping in both directions. Every handler in the codebase finished at 100% coverage.

(Coverage, if you haven’t met it: the percentage of your code that your tests actually execute. 100% on a file means every line of it ran under a test at least once.)

Then there’s the part I didn’t see coming. jiraService.ts — the layer that actually talks to Jira — fell from 15.78% coverage to a low of 10.90% over the run, and only ever crawled back to 13%, while the project’s overall number sat around 74% and barely moved.

The cause was my own instruction file. CLAUDE.md tells the agent to mock the Jira service in handler tests — that is, replace it with a fake so the tests run fast and don’t need real credentials. That’s sensible advice for any single ticket. Applied over six tickets, it meant every pull request added a perfectly tested handler and a completely untested service function. The average stayed flat while the least-tested part of the codebase kept growing.

The agent did exactly what it was told. The instruction was wrong at scale, and the top-line metric hid the consequence. If I’d been watching coverage as a single number — which is how most teams watch it — I’d have seen a flat line and concluded everything was fine.

For what it’s worth, the agent noticed before I did. Closing out the comments ticket, unprompted, it wrote: “CI only exercises the mocked handler, never the actual Jira call path.”

H4 — Ambiguity, Not Coding Mistakes

Across six tickets, no reviewer — me or the automated one — found a single coding defect. Every question the agent raised was about a requirement that was missing or wrong.

The best example was the boards endpoint. My ticket stated that board types were scrum or kanban. The agent proposed validating input against that list, recommended it, and I approved it. Reasonable plan, reviewed and signed off.

Then, part-way through implementation, it called the real Jira API to check — and found that Jira also accepts a third type, simple. Which happened to be the type of the only board on my site. The endpoint as I specified it, and as I approved it, would have rejected the only board that existed.

Worth pausing on who missed that: the person who wrote the ticket, the plan that encoded it, the human reviewer who approved the plan, and the automated reviewer. All of us were reading the same wrong document. What caught it was contact with the running system.

If you take one practical thing from this post, make it this one: an agent that verifies its assumptions against the real thing is worth more than another reviewer reading the same wrong spec.

H5 — The Bottleneck Moved to Review, and Then Review Did Nothing

Across five agent-authored pull requests: zero review comments, zero rework rounds, zero human-edited lines. I verified the last one with git diff against every branch — empty, every time.

The automated review workflow ran on all five PRs, taking between 45 seconds and just under six minutes, so it was genuinely doing work. It never once left a comment. Including on a pull request that deliberately diverged from its own ticket. Including on one whose description said, in plain text, that nobody had run the build or the tests. My own review found nothing across the same five either.

The honest reading is uncomfortable: this experiment produced no evidence that either review gate is capable of returning a negative result. That’s not the same as saying the code was bad — by every check available to me, it was good. But a gate that never fires hasn’t been shown to work. It’s only been shown not to fire.

There was a smaller version of the same problem in my own behaviour. The agent asked me five clarifying questions across the run. Each one arrived with the options laid out, the code implications worked through, and a recommendation marked. I took the recommendation all five times. The choices were good ones — one of them corrected a mistake of mine using evidence from the live API. But what I was doing was ratifying, not analysing, and it felt exactly like review while I was doing it.

H6 — The Instruction File Expires, Except Where It Can’t

The control run is the cleanest result in the experiment. (A control, if the term is new: you run the same task with one thing changed, so you can see what that one thing was actually contributing.)

I rebuilt the repository at the state it was in before the comments ticket, deleted CLAUDE.md, removed every trace of it from the git history, and ran the same ticket again. Then I diffed the two implementations.

Same route. Same three-layer structure. Authentication checked first. Jira calls going through the service layer. Types in the right file. The rich-text format handled correctly. Eleven tests versus eleven. The only differences were a type name, two extra fields in a response, and one error mapping — about the variation you’d get running the same agent twice. Nothing that CLAUDE.md actually specifies about code structure changed at all.

Where the run did change was everything you can’t see in source code. It stopped and asked whether it should create a feature branch, update the README, or move the Jira ticket — and noted that these are exactly the things an instruction file would normally tell it. The run with CLAUDE.md had done all three without being asked.

So the codebase had already taught the agent everything structural. Five consistent endpoints are a better style guide than any document. What the instruction file was still carrying were the rules that leave no trace in code: branch naming, what a PR description has to contain, keeping the tracker current, the ban on adding dependencies. Those never become redundant, because no amount of reading the source reveals them.

Two caveats. The control ran seventh, against five consistent worked examples — run first, against a single seed endpoint, it would very likely have gone differently. And the agent found a comment on the Jira ticket left by the original run, which contaminated a couple of its scoping decisions. It flagged that itself, which I appreciated, but it means I can’t claim those two decisions were reached independently.

What It Got Wrong

The coverage drift above is the headline defect, and it belongs to my instructions rather than the agent’s work.

Beyond that, the defect list from the whole experiment is mostly a list of my own mistakes: my boards ticket contained a factually wrong list of valid values. My ticket-creation ticket specified the wrong type for a field. My hand-written seed code declared a field required that Jira frequently omits. My README advertised two endpoints that didn’t exist yet. All four were found by the agent. None were found by me.

The one genuine code defect anyone found in six tickets is small: when Jira rejects a request for a permissions reason, the gateway reports it as a bad ticket key or a malformed body, because it maps every Jira 400 to the same message. It only surfaced by running the endpoint against a real permission scheme — no test caught it, and no review did either.

How CLAUDE.md Evolved

The instruction file didn’t stay still during the experiment, and pretending it did would misrepresent the H6 result.

The change that mattered came from a gap I noticed a day late. The first endpoint had been implemented, reviewed, merged and deployed — and its Jira ticket was still sitting in To Do. The agent had read the ticket, commented the PR link on it, and never moved it.

That has two consequences, one worse than the other. The board showed no progress on something already running in production. And because I trigger the agent by asking for the highest-priority open ticket, the next run would have picked up the same ticket and built the same endpoint a second time.

So I added a section to CLAUDE.md: move the ticket to In Progress before writing code, In Review when the PR opens, Done only on merge. The next run picked the rule up unprompted and followed it exactly, and every run after that did too.

The uncomfortable general lesson, for anyone hoping the instruction file is a one-off setup cost: the agent followed every code convention from the first ticket, because those were written down. The tracker workflow wasn’t written down, so it didn’t happen. An agent doesn’t fill the gaps in your instructions with common sense. It fills them with nothing.

What It Actually Cost

Simpler than I predicted, and I was wrong about the shape of it.

In the first post I described two spending channels: a Claude Pro subscription for the terminal sessions, and metered API billing for the runs triggered from GitHub. That model was wrong. Both GitHub workflows turned out to authenticate with a subscription token rather than an API key, so those runs drew on the same Pro subscription as the terminal work — same budget, same usage limits. The API key I’d carefully set up as a repository secret was never used by anything.

So the total cost of the build was one month of Claude Pro: twenty US dollars. Six endpoints implemented in the terminal, one implemented from a GitHub issue, five automated code reviews, and a control run — all inside a single subscription, with no metered spend at any point. I’d also bought a ten dollar API credit before starting, on the assumption I’d need it. It’s still sitting there, untouched.

I’ll take being wrong in that direction. But it’s worth understanding why the prediction was wrong, because it wasn’t a rounding error — I’d assumed the two channels were billed separately and built a cost model on it. They aren’t. Which also kills what looks like an obvious optimisation: “move the work into CI to save the subscription” doesn’t save anything, because it’s all the same pool.

Did the Work Move Up a Level?

I ended the first post asking whether the promise of agentic development holds — does the work actually move up a level, or does it just move the typing around?

It moves up. Implementation stopped being the slow part almost immediately. What’s left is specification, review and verification — and on the evidence here, the agent was better at specification-checking than I was, and the two review gates never demonstrated they could catch anything.

That sounds like a clean win, and mostly it is. But the mistakes moved up a level too, and the level they moved to turned out to be the one I’d prepared least. That’s a longer story than fits here, and it’s the next post: what the agent’s work looked like on the systems it touched — who Jira thought was doing the work, who GitHub thought was doing it, and why that ended up being the most uncomfortable finding of the whole experiment.

What Comes Next

The practical next steps for the project itself: give the agent its own identity on every system it touches; close the loop from Jira, so a ticket transition on the board triggers the implementation run and the agent comments, assigns and transitions from there — no terminal involved; put an API management layer in front of the gateway; and try the same approach against a backlog that matters.

The repo is public if you want to read the PRs — including the one from the deliberately terrible ticket: github.com/jamienicholls/jira-gateway.

Disclosure

The first post set out where the line was: Claude helped scaffold the repository, transcribe notes and load the tickets, and from the first ticket onward there would be no human-written implementation code and no agent help that wasn’t visible in the repository history. That held. I haven’t written a line of implementation code since the seed endpoint, and every change is in the public history.

The line moved somewhere else, though, and it would be dishonest to publish this without saying so. I keep a separate notes repository for this experiment, and I work through it with Claude — that much was disclosed up front. What I hadn’t anticipated is how much of this post came out of those sessions. Two of the governance problems in the next post were found there, by Claude reading my repository configuration, not by me reviewing it. One pull request was reviewed a second time by that session, at my request, after my own review had found nothing. And one of the same problems was found independently by the agent under test, unprompted, while it was working on something else.

None of that changes the results. It does change who gets the credit for noticing them, and the credit is not mine.