Debug a Failing Test With an AI Agent

Debugging a Failing Test With an AI Agent

An AI coding agent can investigate a test failure, edit code, and run checks. The challenge is keeping it focused on the actual defect rather than persuading the test suite to pass. This tutorial uses a Python pagination bug to demonstrate a repeatable workflow: reproduce, investigate, patch, and verify.

1. Prepare a Safe Working Environment

Start in a dedicated Git branch and inspect your working tree with git status. Save unrelated changes before giving an agent permission to edit files. Use your project's existing virtual environment and dependency lockfile rather than asking the agent to upgrade packages.

Set approval boundaries: the agent may read project files and run local tests, but dependency changes, destructive commands, network access, and commits require permission. Keep credentials and production data out of prompts and logs.

If you need model access, ApiCabin provides one pay-as-you-go key for indie developers and small teams. It works with Claude Code, Codex, Gemini CLI, and OpenAI-compatible SDKs. Its API address is https://api.apicabin.ai; follow the documented configuration for your chosen tool. A model such as Claude Haiku 4.5 is one example. Sign up for ApiCabin if that setup fits your workflow.

2. Reproduce the Smallest Failure

Run the failing test yourself before asking for a fix. For example: python -m pytest tests/test_pagination.py::test_first_page -q. Record the exact command, assertion output, Python version, and any relevant environment variables, with secrets removed.

Suppose the test requests page 1 with a page size of 2. It expects ['alpha', 'beta'], but receives ['gamma', 'delta']. That is useful evidence: the function returns a correctly sized slice, but starts two positions too late.

If the test passes locally, stop and investigate environment differences or nondeterminism. A patch based on an unreproduced failure is difficult to validate.

3. Give the Agent a Bounded Investigation

Provide the failing command and output, then use a specific prompt:

Investigate this failing test. First reproduce it, read the implementation and nearby tests, and explain the likely root cause with file references. Do not edit files yet. Preserve the documented behavior and existing assertions. Ask before changing dependencies or running destructive commands.

This separates diagnosis from implementation. Ask the agent to identify evidence that would disprove its hypothesis, not merely explain why its first guess sounds plausible.

4. Confirm the Contract Before Editing

In this example, suppose the implementation calculates start = page * page_size. Nearby documentation and tests define page numbers as starting at 1. The agent should connect those facts: page 1 currently starts at index 2 instead of index 0.

The proposed correction is start = (page - 1) * page_size. Confirm that callers follow the same contract. If documentation and callers disagree, resolve that ambiguity before accepting a patch; do not let the agent silently redefine the API.

5. Request the Smallest Defensible Patch

Now authorize the change:

Apply the minimal implementation fix. Keep existing assertions intact. Add focused regression coverage for the first page, a middle page, a partially filled final page, and a page beyond the available items. Avoid unrelated refactoring.

Include empty input if the existing contract supports it. For invalid page numbers or page sizes, preserve established validation behavior rather than inventing new rules. Review the diff with git diff, checking that tests exercise public behavior rather than implementation details.

6. Verify and Document the Result

Run the original failing command, then python -m pytest tests/test_pagination.py -q, followed by the project's broader test suite and configured checks. Where practical, confirm the new regression test fails against the old implementation and passes with the fix.

Ask the agent for a final summary containing:

• The root cause and changed files.
• The regression cases added.
• The exact verification commands and their outcomes.
• Any checks it could not run.

Only commit after reviewing those results. A trustworthy agent-assisted fix is not just a green test: it is a small, explainable change backed by evidence.