A Pipeline Bottleneck You Already Have
The AI coding agent generates a PR in 90 seconds. The CI test suite takes 45 minutes. The reviewer waits for the test suite before they look at the PR. The agent's 90-second cycle becomes a 47-minute cycle, gated by tests.
For teams introducing AI coding, this is one of the most under-discussed friction points. Your test suite duration was a tolerable annoyance when humans wrote PRs at the speed of an hour each. It becomes a hard bottleneck when the AI writes PRs at the speed of minutes.
This post is about the test infrastructure changes that clear that bottleneck.
The Math
Suppose your team has:
- 10 engineers, each opening 2 PRs per day (20 PRs/day).
- Test suite: 45 minutes per run, 4 parallel runners.
- Effective throughput: ~32 PR-tests per hour, comfortable headroom.
Add an AI coding agent producing an additional 30 PRs per day. Now you have 50 PR-tests per day, peaking in business hours. The test queue starts backing up. Median PR-to-test-completion time stretches to 90 minutes. Engineers context-switch. Reviewers stop being able to predict when a PR will be ready.
This is not a hypothetical. It is the first scaling failure most teams hit when AI throughput exceeds CI capacity.
The Patterns That Fix It
Five interventions in rough order of payoff:
1. Test sharding by file path. Run only the test files affected by the diff, plus a small fixed integration baseline. Implementation difficulty: medium (requires test-to-source mapping). Time savings: typically 60-80%. The biggest single win.
2. Pre-flight smoke gate. Before the full test suite, run a 90-second smoke test that catches the obviously-broken PRs. Failing PRs never queue for the full suite. Time savings: removes 10-15% of total test load.
3. Test parallelism scaled to PR rate. Most teams have CI capacity sized for humans. With AI throughput, the right runner count is 2-3x what you had. If your runners are cheap (cloud, ephemeral), this is the easiest fix. If they are expensive (on-prem GPU integration tests), it is harder.
4. Test result caching. Cache test results keyed by content hashes of the source files they cover. A re-run on the same code reuses the cached pass. AI-generated PRs that rebase against a fast-moving main branch benefit disproportionately.
5. Flaky test quarantine. A flaky test costs 2x test runtime: the run that flaked plus the retry. With AI throughput, the cost compounds. Aggressive flaky test quarantine (automatically pulling out tests that fail and pass on the same SHA), is no longer optional. See flaky tests hidden cost.
The Tradeoff With Coverage
Test sharding gives back time but creates a coverage gap: a change to a shared utility might affect tests in modules you did not run. Three mitigations:
- Symbol-aware sharding. Use a static call graph to expand the shard set to include tests that touch any symbol modified.
- Periodic full runs. Even with sharding, the full suite runs on every main-branch commit. Issues missed by sharding get caught at merge time.
- Confidence-gated sharding. Use sharding only for high-confidence PRs. Low-confidence PRs run the full suite.
These layers preserve correctness while keeping the median PR fast.
What The AI Agent Should Do With Test Results
Three behaviors that materially improve outcomes:
Read failing test output, do not just retry. A naive agent retries on failure with the same prompt. A useful agent reads the assertion message, traces it back to the diff, and proposes a targeted fix. The cost is one extra LLM call. The benefit is removing 80% of retry-then-fail cycles.
Generate tests for the code it just wrote. The AI's own change should include tests covering the new behavior. This is table stakes for any agent producing more than dependency bumps. The tests run as part of the PR's own suite.
Refuse to ship on flaky failures. If a test fails and passes on retry, the AI should flag it for human review rather than mark the PR green. Flakiness is a property of the test, not the code, and the AI should know the difference.
A Specific Number
For a team we tracked across a 12-week rollout:
- Pre-AI median PR-to-merge time: 4.5 hours.
- AI generation time per PR: 90 seconds (median).
- Test suite duration on full repo: 38 minutes.
- After sharding, smoke gate, and capacity scaling: 6 minutes median test time.
- Post-AI median PR-to-merge time: 22 minutes.
The headline number (4.5 hours to 22 minutes), was about a third agent throughput and two thirds test infrastructure. Most teams give the AI credit for the speedup. The truthful credit is shared.
What To Do First
If you are about to roll out AI coding and your test suite takes over 15 minutes:
- Measure: median test suite time, p95 test suite time, flaky test rate.
- Decide whether to invest in test sharding or simply scale runner capacity. For most teams, sharding pays back faster.
- Set up the smoke gate.
- Quarantine flaky tests aggressively. Do not let them stay.
- Then turn on the AI.
Tests are the rate-limiting step for AI-generated throughput. Most teams discover this after they have rolled out the agent and the bottleneck has already metastasized. Get ahead of it.
For the cycle time story end-to-end, see reducing development cycle time with AI. For how this plays into MTTR, see reducing MTTR with AI.
Frequently asked questions
Why does my CI test suite slow down AI code generation?
A test suite duration that was a tolerable annoyance at human PR speed becomes a hard bottleneck when an AI writes PRs in minutes. If the agent produces a diff in 90 seconds but the reviewer waits 45 minutes for tests before looking, the effective cycle collapses to the test suite's length. As AI throughput exceeds CI capacity, the test queue backs up and median PR-to-test time can stretch to 90 minutes.
How do I speed up test suites for AI-generated pull requests?
Apply five interventions in order of payoff: shard tests by file path to run only what the diff touches, add a 90-second smoke gate that filters obviously-broken PRs, scale runner capacity to 2-3x what humans needed, cache test results keyed by source content hashes, and quarantine flaky tests aggressively. Sharding alone typically saves 60-80% of test time and usually pays back faster than simply buying more runners.
What is test sharding and does it hurt coverage?
Test sharding runs only the test files affected by a diff plus a fixed integration baseline, instead of the whole suite. The coverage risk is that a change to a shared utility may affect tests you did not run, so mitigate it with symbol-aware sharding using a static call graph, periodic full runs on every main-branch commit, and confidence-gated sharding that sends low-confidence PRs through the full suite. These layers preserve correctness while keeping the median PR fast.
Should AI coding agents write their own tests?
Yes, generating tests that cover the new behavior is table stakes for any agent producing more than dependency bumps, and those tests run as part of the PR's own suite. A good agent also reads assertion messages to propose targeted fixes rather than retrying with the same prompt, which removes roughly 80% of retry-then-fail cycles. It should also refuse to mark a PR green on a flaky failure, since flakiness is a property of the test, not the code. See AI code generation for test coverage.
How much CI runner capacity do I need for AI coding agents?
Most teams have CI sized for humans; with an AI agent adding PRs the right runner count is typically 2-3x what you had. If your runners are cheap, cloud, and ephemeral, scaling them is the easiest fix; if they are expensive on-prem GPU integration tests, sharding and smoke gating matter more. Measure median and p95 test time plus flaky rate before you turn the agent on, because the bottleneck otherwise metastasizes after rollout. See the hidden cost of flaky tests.
How much of an AI cycle-time improvement actually comes from the agent?
Less than most teams assume. In a tracked rollout that cut PR-to-merge from 4.5 hours to 22 minutes, about two-thirds of the gain came from test infrastructure changes (sharding, the smoke gate, and capacity scaling), and only one-third from the agent's 90-second generation. The truthful credit for the speedup is shared. For the end-to-end story, see reducing development cycle time with AI.
EnsureFix DevOps Team
The EnsureFix DevOps team runs the distributed worker pool and CI integrations behind the pipeline, and writes about cycle time, MTTR, and delivery metrics.
