Jamie Nicholls

This is the last of three posts about handing a backlog to an AI agent: the first set up the experiment, the second scored the results.

I spent a fair bit of effort deciding how I’d review an AI agent’s work. That was the part that worried me: the confident nonsense, the invented library that doesn’t exist, the test that asserts nothing and passes beautifully. So I built the experiment with review as the control point. Every change through a pull request, branch protection on, nothing merged without a human looking at it. Then I ran the tickets, and the agent wrote good code.

While I was watching the agent, several things were wrong behind me. None of them were the agent’s fault. All of them were mine, and every one was in place before it ran a single ticket. Individually, each component was working exactly as configured. Arranged together, they verified nothing, and the whole stack displayed green from end to end.

The Thing I Was Watching For

The setup was deliberately conservative. A public repository. Branch protection on the main branch: the setting that stops anyone committing straight to main and forces changes through a pull request instead, including mine. An automated code review that fires whenever a pull request opens. A deploy pipeline that builds, tests, and ships to Azure. And a CLAUDE.md file telling the agent how the codebase is structured and what it must do before opening a pull request.

For the first ticket, the agent did all of it. Read the ticket from Jira, created the branch, wrote the handler, the service method, the types and the tests, ran the build, ran the suite, pushed, opened a pull request, and commented the link back on the ticket. The automated review passed clean in forty-five seconds. I read the diff, found nothing to change, and merged.

I remember feeling quite pleased with the process.

Ready To Merge, Nothing Checked

The second ticket was the first one that writes to Jira rather than reading from it, so I was paying closer attention. The pull request arrived with four files changed and thirteen tests, which is eleven more than the ticket asked for.

At the top of the page, GitHub said Ready to merge. Next to it, in the tab strip: Checks 0.

Not “checks failed”, not “checks pending”. Zero. There were no checks at all, and GitHub was perfectly happy to call the pull request ready to merge, because by the rules I’d configured, it was.

The ruleset required a pull request. It blocked force-pushes and deletion of the main branch. That was it: zero required approvals, and no status-check rule of any kind. Nothing had to pass. Nobody had to approve. The pull request was a speed bump with a green light on it.

I’d genuinely believed I’d turned that on. If you’d asked me over a coffee whether my repo required checks to pass before merging, I’d have said yes without hesitating, and I’d have been wrong.

I wasn’t reviewing my branch protection settings when this surfaced. I was in a working session with Claude, going over what to do next, and it read the ruleset off the GitHub API and told me what was actually in it. The number had been on my screen for days. I hadn’t looked at it.

That’s a settings problem, and settings problems are the easy kind. It took about a minute to fix. Then I looked at what would have run, if anything had run.

Nobody Was Running the Tests

The workflow layout I’d built would have sounded completely standard if I’d described it to you.

A deploy workflow that installs dependencies, builds the TypeScript, runs the full Jest suite, and deploys to Azure Functions if everything passes. Sensible. It triggers on push to the main branch.

An automated code review workflow that triggers when a pull request opens.

Read those two sentences again, because I read them a hundred times without seeing it. The build and the tests only ran after a merge. Never before one. The suite that would tell me the agent had broken something ran at the point where the change was already on main and on its way to production. A pull request with a completely broken test suite could have sailed through, and I’d have found out from
a red deploy afterwards.

(If continuous integration is new to you: the idea is that every proposed change gets built and tested automatically before anyone merges it, so problems are caught while they’re still someone’s branch rather than everyone’s main branch. I had the automation. I’d pointed it at the wrong moment.)

This had been true since the first commit. Across every ticket, not one pull request was verified by anything before I merged it.

And it never once bit me. Every pull request arrived green: build passing, full suite passing, stated plainly in the description. Because CLAUDE.md says to run the build and the tests before opening a pull request, and the agent did, every single time, without being reminded.

What stood between a broken test suite and my production Function App was an instruction file, and an agent that followed it conscientiously. I’d accidentally outsourced my quality gate to the very thing the quality gate existed to check.

There’s a CI workflow now that builds and tests every pull request, about twenty-five lines of YAML. And before publishing this post I had the ruleset read again, the same way it was read the first time, because this experiment taught me not to describe a setting from memory. It lists exactly one required status check: the automated review. The build and the tests now run on every pull request, but nothing requires them to pass. So the one gate the platform enforces is the gate that has never once returned a negative result, and the gate that can genuinely fail is optional. It should all have been there before I let anything near the repository, agent or not, and apparently the ruleset still
isn’t finished.

There’s a postscript to this one I didn’t enjoy. A few tickets later the agent finished its work, opened its pull request, and ended the summary with a short section titled “Two things I noticed but did not act on”. The first was that my local settings file holds a live API token in plaintext, correctly ignored by Git, but worth treating as
exposed if that file ever leaves the machine. The second was this:

there’s no PR-validation CI workflow — deploy.yml only
runs build and tests on push to main, so the PR’s checks
won’t prove the build passes

It had found the same gap. On its own, from reading the repository, while working on something else entirely, and it correctly judged that fixing it wasn’t its job, so it left it alone and told me instead.

The tally for that one defect: the engineer who built the pipeline missed it, a Claude session in my notes repo caught it, and the agent under test caught it independently a few hours later. I’m the only participant in that list who’d been looking at the repository for weeks.

The Reviewer That Never Says No

There’s a second gate on every pull request here: an automated code review that runs when the PR opens, reads the diff, and comments if it finds something.

Across the experiment it reviewed five pull requests, taking between forty-five seconds and five and a half minutes each, so it was doing real work rather than short-circuiting.

It never left a comment. Not one, on any of them.

It had opportunities. One pull request deliberately departed from its own ticket, widening a validation rule because the ticket’s list of allowed values was factually wrong. One changed a shared piece of client code used by three existing call sites. One altered a type I’d written by hand. And one, implemented through a different channel where the agent had no permission to run the test suite, said so in its own description: the author could not run the build or the tests, the branch name broke the convention, and the tracker steps were skipped.

That pull request went green.

My own review found nothing across the same five either. Zero comments, zero changes requested, zero lines edited by hand, which I verified with git diff against every branch rather than trusting my memory.

There is one exception, and it’s the most useful data point in the section. On one pull request I asked a second Claude session, the one I use for my project notes rather than the agent under test, to review it after I’d already reviewed it and found nothing. It found a substantive point: the tests covered the handler thoroughly but never exercised the
code that actually calls Jira, so the suite could stay green while every real call failed. That became one of the main findings of the previous post. Nothing in the repository changed as a result, but three reviewers looked at that pull request, and the only one that found anything was the third, brought in after the other two had signed off.

So, the honest way to state it: this experiment produced no
evidence that either of my two review gates is capable of returning a negative result.

That’s not the same as saying the code was bad. By every check available to me (clean build, sixty-nine passing tests, seven endpoints verified against live Jira) it was good. But a gate that never fires hasn’t been shown to work. It’s only been shown not to fire. Those are different claims, and the difference is the entire point of having a
gate.

If you set this up, submit a deliberately broken pull request once, early. Confirm the thing can go red before you start relying on it being green.

You Cannot Review Your Own Pull Request

So I went back to review a pull request properly, the way I’d promised in the first post I would. Files changed, line comments, request changes if needed.

GitHub wouldn’t let me.

Not a permissions error. Something more basic. The agent runs on my machine, with my credentials, and pushes using my Git identity. So as far as GitHub is concerned, I opened that pull request. And GitHub doesn’t let you approve your own pull request; the Approve and Request Changes options simply aren’t there.

The human control point, the one thing the entire experiment was built around, was structurally unavailable, because the platform believed I’d written the code myself.

I can still comment on the diff, and I can still merge. What I can’t do is the formal part: approve or block, and have it recorded as a review by a second party. It goes further. I can’t require approvals either. If I set the required approval count to one, the repository deadlocks immediately, because the only person available to approve is the person GitHub thinks authored every pull request in it.

There’s a version of this that gets found in an audit rather than in a side project, and I’d rather have found it here.

Worth noting the exception, because it points at the fix. One ticket ran through a different channel, a GitHub issue rather than my terminal, and that run posted under its own identity: claude, marked as a bot, clearly distinct from my account. Same agent, same instructions, completely different audit trail. That difference came from how I’d wired mine up.

Who Changed This Ticket?

The same problem, one system over, and this one leaves a permanent record.

Partway through the experiment I added a rule to
CLAUDE.md: move the Jira ticket to In Progress before
writing code, In Review when the pull request opens, and leave Done for merges. The agent picked it up on the next run without being asked and executed it perfectly. Ticket to In Progress, work happened, pull request opened, ticket to In Review with the link commented on it. Exactly right.

Then I opened the ticket’s History tab.

Six entries. Every one says my name. Four of those transitions were made by the agent and two by me, and there’s nothing (not the author, not the avatar, not a flag, not a note) that distinguishes them. The agent signs in to Jira as me, so Jira has no concept that an agent was ever involved.

If someone asks who moved that ticket, the honest answer is “I don’t know, and neither does Jira”. The system will tell you it was me, confidently, and it’ll be wrong two-thirds of the time.

Compare that with the Git history, where every commit the agent makes carries a co-author trailer and the pull request body says it was generated by Claude Code. The same piece of work is fully auditable in one system and completely unauditable in the other, and I’d set both of them up myself in the same afternoon.

Being Asked Is Not the Same as Deciding

The agent stopped and asked me five questions across the six tickets. Each was a genuine gap: an undefined edge case, a body format the ticket never mentioned, a scope boundary in a ticket that contradicted itself.

Every question arrived the same way. The options laid out. The exact code it would write for each. And a recommendation, marked.

I took the recommendation five times out of five, and never once used the notes field.

The choices were good. One was better than good: it corrected a factual error in my own ticket, with evidence from a live API call to prove the point. I’m not claiming I was led astray.

I’m claiming that what I was doing was ratification, and that it feels precisely like review while you’re doing it. The agent found the gap, framed the options, wrote the implementation, and expressed a preference. I agreed. If you’d asked me at the time whether a human was steering the work, I’d have said yes without hesitating, and pointed at
those five questions as proof.

There’s a version of this that gets worse as the agent gets better. A recommendation you’d have disagreed with is easy to catch. A recommendation that’s right nine times out of ten trains you to stop reading the tenth.

Put the last three sections together and the shape is clear. My control point consisted of an automated reviewer that has never objected to anything, an approval mechanism the platform wouldn’t let me use, and five decisions I agreed with. That is the appearance of a review process, and I’d have described it to you as a real one.

When the Automation Layer Goes Down

One more, and this one wasn’t my doing. GitHub Actions had a major outage while I was mid-ticket, so the deploy triggered by a merge to main didn’t run, and neither did the automated review on a pull request opening. Git operations and pull requests stayed up the whole time, so work kept landing in a repository where nothing was verifying it and nothing was deploying it, and every screen looked normal.

The timing contrast is the bit I keep coming back to. The agent took about a hundred seconds to go from starting a ticket to an open pull request with thirteen tests in it. The platform it depends on was unavailable for over an hour. Whatever the bottleneck in software delivery is now, it stopped being the typing a while ago.

Most Places Already Have This Shape

None of this is exotic, and none of it needed an agent to exist. Most organisations already have the same arrangement somewhere: branch protection set up years ago by someone who has since left, a review gate nobody has tested against a genuinely bad change, shared service accounts with no per-agent identity behind them. Settings like those get inherited rather than audited, and no interface anywhere reports that a gate has never once fired. The agent doesn’t create the gaps. It finds them, at volume, in its first week. My repository had one engineer and six tickets in it, which is the smallest version of this anyone is going to get.

Give the Agent Its Own Identity

Most of the problems above have one fix, and it’s unglamorous: give the agent its own account. A dedicated Jira user and a dedicated Git identity, not a borrowed human one.

What that buys you:

• An audit trail that isn’t lying. “Who changed this ticket?” gets a true answer.
• Reviewable pull requests. The agent opens them, you review and approve them, the way the platform was designed to work.
• Approvals you can actually enforce. Required human approval becomes a real gate instead of a deadlock.
• A better way to hand over work. I trigger the agent by asking for the highest-priority open ticket, a sorting concept doing a queue’s job, and it has already failed once. A ticket that shipped without being moved out of To Do would have been picked up and built a second time. Assigning a ticket to the agent is unambiguous, and it’s how teams already hand work to each other.

Check three things before committing to it. A service account may consume a billable seat, and that’s the first objection you’ll get. Atlassian’s MCP server authenticates as whoever completes the sign-in flow in the browser, so running as a service account means actually signing in as one, not just pointing at a different address. And an assignee that transitions and comments on every ticket generates a lot of watcher email, so mute it early.

What I’d Set Up Before Doing This Again

• A dedicated account for the agent, on every system it touches
• Build and tests on pull requests, not just on merge
• Those checks marked required, so the platform blocks the merge instead of trusting you to look
• Required human approval, once the agent has an identity that makes it possible
• An instruction file that covers workflow, not just code structure: the tracker won’t update itself, and the agent won’t invent the convention you forgot to write down
• GitHub linked to Jira, so a ticket can be picked up where it lives instead of being copied and pasted between two systems
• A manual trigger on anything you’d need during an outage
• One deliberately broken pull request, early, to prove your review gate can go red
• Per-layer test coverage, not a single headline number. Mine sat at a comfortable seventy-four percent all experiment while the layer that actually talks to Jira sank to eleven at its worst
• The same tooling in every channel you trigger the agent from. The same instruction file produced different behaviour depending on whether I ran it from a terminal or from a GitHub issue, because the second environment had no Jira access and no permission to run the tests. Nothing warned me. The agent noticed and said so, which is the only reason I know

The Agent Wasn’t the Risk

I went in worried about the code. The code was fine.

What I’d actually built was a pipeline where nothing verified a change before it merged, no second party could approve anything, and the record of who did what was wrong by default. Every one of those was a decision I made before the agent ran a single ticket, and I’d have told you confidently that none of them were true.

I also didn’t find most of them. The missing checks and the missing CI gate both came out of a working session with Claude over my notes, the agent under test found the CI gate as well while working on something unrelated, and the one substantive review comment anyone made on any pull request came from a Claude session I asked to look after I’d already signed the change off.

What makes that more than an embarrassing anecdote is that everything looked correct the entire time. Green ticks, “Ready to merge”, pull requests opening and closing, tickets moving across a board. There’s no screen anywhere in that stack that says nothing here is actually being checked. A pipeline made of theatre looks identical to one that works, right up until you count the checks.

It’s worth being precise about the shape of what failed, because it isn’t “the tools let me down”. Every gate I had worked exactly as configured. The automated reviewer ran on every pull request and reported what it found, which was nothing. The branch protection enforced exactly the rules I’d given it, which required a pull request and nothing else. The deploy pipeline ran the tests faithfully, at the moment I’d told it to, which was after the merge. Nothing malfunctioned. Every component did its job, and the sum of them checked nothing.

That’s the general result, and it’s the part that travels beyond my repository. A set of individually correct components can be arranged so that nothing verifies anything, and the stack will display green either way. Testing the components won’t find it. The question that finds it is which part of the arrangement is capable of saying no.

Agents make that dangerous, and the reason is speed rather than carelessness. A gap in your process that produced one unverified pull request a fortnight will now produce several before lunch. Agentic development raises the bar for engineering discipline, because the volume arrives faster than the attention available to review it, and the only thing keeping my repository honest was an instruction file and an agent that followed it.

In the first post I asked whether the work really moves up a level. It does. It moves up to verifying, and to being able to prove what happened. That turned out to be the level I’d prepared least, and it’s the one I’d get right first if I were starting again.

Disclosure

The first post drew a line: from the first ticket onward, no
human-written implementation code, and no agent help that isn’t visible in the repository history. That held for the code, and the public history shows it. The same line has to extend to the writing, because I keep a separate notes repository for this project and work through it with Claude, and the sections above say where that session, or the agent
itself, got there before I did. In a post arguing that the record should say who actually did what, letting you assume it was me would be a strange way to end.

This is Part 3 of three. Part 1, Learning Agentic Software Development: An Engineering Experiment, sets up the six hypotheses and the method. Part 2, What I Predicted, And What Happened, scores them against the measurement log.