An AI-written test let a poker program call a straight a high card

A faulty poker program called an ace-low straight a high card, yet an AI-written test let it pass. The TestPrism study illustrates a problem one step beyond coding errors: the tests meant to catch them can also fail.

Nanjing University researchers evaluated 14 configurations. The strongest scored 59.67% against a single reference, but 28% when required to distinguish multiple valid and invalid implementations. The preprint was submitted October 8 and revised November 8.

The released framework runs the same test suite against ten candidates in clean environments. A strict pass requires detecting the initial defect, accepting valid alternatives and rejecting invalid ones. Answers and labels stay hidden from the agent. The framework uses Apache 2.0, with task and patch data supplied separately. Developers can use this approach to ask whether their tests check the requirements or merely recognize one implementation.

These are test-quality scores, not universal AI code pass rates. For people building websites or apps with AI, green check marks still need scrutiny for missed edge cases.

Source times: Paper first submitted, October 9, 2026, 00:44:39 Beijing / October 8, 12:44:39 PM EDT; revised November 9, 2026, 02:06:02 Beijing / November 8, 1:06:02 PM EST; code committed October 10, 2026, 15:01:49 Beijing / 3:01:49 AM EDT; Xinzhiyuan / 36Kr, November 12, 2026, 16:11 Beijing / 3:11 AM EST.