← Back

The Autonomous Software Factory: Engineering with AI Agents

·Bryan Lai

If your coding agent cannot run what it writes, you are not automating engineering.

You are generating review work for yourself.


The Trust Curve

  • Most teams use AI as autocomplete.
  • Human review remains the scaling bottleneck.

To scale agentic engineering, ascend the Trust Curve:

LevelModeAgent RoleVerification Gate
Level 1CopilotSuggests inline syntaxHuman writes code, tests manually
Level 2Pair ProgrammerImplements functionsHuman reads diffs, runs unit tests
Level 3Junior EngineerBuilds multi-file featuresHeadless runtime checks, human reviews PR
Level 4Software FactoryIssues reproduction test, builds, proves fixGreen mutation tests, auto-mergeable PR

Two hard rules:

  1. You cannot reach Level 4 with prompt engineering.
  2. You reach Level 4 by closing the runtime verification loop.

The Inverse Pyramid

When agents fail:

  • Beginners edit the prompt.
  • Experienced operators fix the harness.
       [   10% Prompts & Task Scope     ]
     [     20% Verification Skills        ]
   [       70% Codebase Constraints         ]

1. Codebase Constraints (70%)

  • Type Systems & Strict Schemas: Pydantic, TypeScript, Rust. If the compiler rejects bad states, the agent cannot hallucinate them.
  • AST Linters (Ban Nuance): Do not argue with agents in PR reviews about deprecated patterns or missing boundaries. Write an AST linter rule (ESLint/Ruff) that fails the build. If an agent cannot make the mistake, it won't.
  • The Weeded Garden: Agents mimic surrounding repository patterns. Dead code, loose types, and sloppy error handling are bug multipliers. Delete unused code before extending it.

2. Verification Skills (20%)

  • Headless Runtime Drivers: Drive web via headless CDP (Playwright). Drive desktop via offscreen Qt (QT_QPA_PLATFORM="offscreen") and OS process boundaries.
  • Sidecar Observers: Run memory-leak checks, network assertions (zero unauthorized outbound TCP sockets), and layout overflow guards.
  • Self-Healing Loop: If the agent sees runtime errors directly, it fixes them before a human ever looks at the PR.

3. Prompts & Task Scope (10%)

  • Narrow task definitions.
  • Explicit definition of done.
  • Prompts state intent; code and tests enforce truth.

How Agents Determine Necessary Tests

Before touching production code, the agent runs a four-step interrogation:

1. Observable Proof

What exact artifact proves completion?

  • UI: QML object property change or DOM render.
  • Data: Exact database row or filesystem write.
  • Math: Strict zero balance delta (Diff: RM 0.00).

2. Boundary Gates

What input conflicts halt execution?

  • Missing headers.
  • Inverted signs.
  • Date-range gaps.
  • Currency mismatches.

Every boundary condition becomes an explicit negative test.

3. Blast Radius

What breaks down the dependency tree?

  • Check adjacent modules.
  • Compare against working twin modules to preserve shared contracts.
  • Flag shared state modifications.

4. Mutation Testing (Prove the Test)

A test that cannot fail is worse than no test.

  1. Mutate: Invert or break the guarded source logic.
  2. Verify RED: Run the test suite →\rightarrow assert failure (exit code != 0).
  3. Revert: Restore correct logic →\rightarrow assert pass (exit code 0).

If a test stays green while the logic is broken, the test is fake.


Bottom Line

  • Writing code is no longer the hard part of software engineering.
  • Verification is.

The engineers who win in the agentic era won't be typing prompts into chat boxes.

They will be the architects who design the AST constraints and headless verification harnesses that let agents ship autonomously.