Lee Simpson

Your Test Strategy Assumes Humans Write Code

Thirteen months building a custom state government full-stack application with zero code written by hand.

7,137 tests, 5,631 of them unit tests. It was a textbook testing pyramid but still bugs stacked up.

The project moved fast - tight deadlines, whole feature sets scoped, implemented and changed in short timeframes. Any change touching existing implementation meant churning through existing unit tests. 53% of the code are tests, two-thirds of the test files change in a given month. This gets expensive fast especially when bugs are still being raised. It left us asking the same question. Should we even have unit tests?

Unit tests are anchored to code structure. When an agent refactors freely the suite stops being an independent check and becomes a mirror of the implementation. This talk looks at this problem. How scenario and end-to-end tests anchored to business outcomes provide a potentially better solution. Flipping the testing pyramid. The pyramid was designed when code structure was stable and test execution was expensive. Under agentic delivery both assumptions invert.

This talk examines what happened, what we changed, and the trade-offs. End-to-end tests are slower, costlier and can be flaky; unit tests remain valuable in the right places. The lesson is not to abandon the test pyramid blindly. It is to redesign verification for a world where code is cheap to regenerate, implementation structure is unstable, and independence between builder and checker matters more than test count.