How a Missing Plugin Silenced Our Pitest Gate for Months

When we first configured Pitest in Sprint 7 for the Lima app, we had high hopes for its impact on code quality. Our goal was to integrate mutation testing into our CI pipeline, running weekly (cron ‘0 4 * * 1’) with an aspirational 60% mutation-score gate. We used the info.solidsoft.pitest:1.15.0 Gradle plugin, believing we had a solid setup to catch regressions and drive better test coverage. For weeks, the CI job dutifully reported a mutation score of 0, which we interpreted as “mutation testing is hard” and a sign that we had work to do. But the reality was far more embarrassing: our quality gate wasn’t hard to pass, it was completely broken.

The Problem: A Silent Gate

From its creation until July 6, 2026, the weekly Pitest job never measured anything real. It consistently reported ‘Mutation score of 0’. This wasn’t because our tests were bad; it was because Pitest wasn’t finding any tests to run. The configuration was missing a single, crucial line: junit5PluginVersion=1.2.1. Without the pitest-junit5 plugin, Pitest cannot discover JUnit Jupiter tests. Consequently, its internal line coverage calculation was 0/N, and it never generated or killed a single mutant. Our aspirational 60% gate was never actually met, nor was it ever truly failed. It was just a silent observer, reporting a misleading zero.

Nobody noticed because a failing score of 0 looked like a valid outcome, rather than an indication of a fundamental setup error. It was a classic case of a quality gate that had never failed for the right reason, leading to a false sense of… well, not security, but certainly a false sense of measurement.

The Fix and The Real Score

Once we added junit5PluginVersion=1.2.1 to our Gradle configuration, the change was immediate and stark. The first real run, on 2026-07-06, finally generated meaningful results: 3,931 mutants were created. Out of these, only 1,583 were killed, resulting in a 40% mutation score. Furthermore, 1,398 mutations were completely uncovered, indicating significant gaps in our test suite. This 40% was a far cry from our aspirational 60%, but at least it was a real number.

pitest {
    junit5PluginVersion = '1.2.1' // This was the missing piece!
    // ... other pitest config
}

With a real baseline established, we embarked on four focused hardening waves. We adopted a disciplined approach, tackling one agent per class in isolated Git worktrees. We prioritized critical security classes first: DeactivatedUserFilter, AdminAccessInterceptor, and RateLimitFilter. This targeted effort systematically improved our test coverage and mutation killing ability. Through these waves, we incrementally raised the mutation score from 40% to 81% (3,212 out of 3,987 mutants killed). Line coverage on mutated classes also climbed to an impressive 98%, leaving only 41 uncovered mutations, most of which were equivalent mutants or in trivial code.

During this process, we also encountered some local environment quirks. A local run showed approximately 58 minion MEMORY_ERROR/RUN_ERROR failures. This was due to the Lima app running in IntelliJ consuming significant RAM, starving the Pitest minions. On a clean CI environment, the actual measured score was 80% (3,198 out of 3,987 mutants killed), which became our new benchmark.

Recalibrating the Gate and Takeaways

With a stable, measured score of 80%, we recalibrated our weekly gate. We set mutationThreshold=76, deliberately below the measured 80%. This buffer accounts for Pitest’s non-determinism, ensuring that the weekly job only fails on genuine regressions rather than minor fluctuations. This provides a more reliable and less noisy signal for the team. You can check out the app we’re building, Lima, at https://getlima.app.

The most important lesson from this experience is simple: a quality gate that has never failed for the right reason is not a gate at all. Always verify that your measurement tooling actually measures something before trusting its numbers. A silent pass can be far more dangerous than an immediate failure.