Why AI-Generated Tests Are Not Security Proof

Direct answer: AI-generated tests are useful regression checks, but they are not independent security proof. They can repeat the implementation’s assumptions, omit abuse cases, or pass against unrealistic mocks. Add human-defined security invariants, negative authorization and tenant tests, controlled mutation checks, and a fresh review whose evidence is separate from the generation context.

When one assistant writes both a function and its tests, the pair can agree on the same wrong behavior. The generated assertion passes because it restates what the implementation returns, not because it checks the authorization, isolation, validation, or integrity property the system must protect.

This distinction matters in how tests fit into release assurance. Keep four records separate: whether the suite ran, which code it exercised, whether a controlled security mutation made it fail, and whether evidence outside the generation context supports the claim. A green check answers only the first question unless the test record says more.

A green test can preserve the authorization bug

Incomplete test pattern. Use only with synthetic accounts.

async function getProject(sessionUserId: string, projectId: string) {
  return db.project.findUnique({ where: { id: projectId } })
}

it("returns a project", async () => {
  db.project.findUnique.mockResolvedValue({ id: "project-a", ownerId: "user-a" })

  await expect(getProject("user-a", "project-a")).resolves.toMatchObject({
    id: "project-a",
  })
})

The function accepts a session user but never uses it in the query. The test supplies the owner and asserts only that a project returns. The mock cannot reveal whether a real database query allows User A to read User B’s project.

The request “add a test for getProject” gives the assistant a function signature and an expected return shape, but no ownership rule. The resulting test follows the implementation and rewards the same omission. A security test needs an independently stated denial condition before it needs another example response.

Write invariants before examples

A security invariant states what must remain true across inputs and code changes. For this function:

An authenticated user may receive a project only when the server establishes
an allowed ownership, membership, or sharing relationship for that project.

Turn the invariant into positive and negative cases with two synthetic users and two projects. Test owner access, cross-user denial, missing membership, revoked membership, malformed identifiers, and any deliberate share path. Exercise the real query layer in a disposable database rather than mocking the authorization predicate away.

Use OWASP ASVS or another reviewed requirement set to derive invariants outside the generated patch. Select requirements that match the application architecture. ASVS is a verification framework, not proof that every listed control applies or passes.

Make a safe mutation and expect failure

Mutation testing asks whether the suite notices when behavior weakens. In a local branch, change a security condition to a fixed false or true value, run the relevant tests, and discard the mutation afterward.

For example, if the corrected query contains both id and ownerId, temporarily remove ownerId in the isolated test branch. The cross-user negative test should fail. If every test stays green, the suite does not enforce the ownership invariant.

Do not mutate a deployed environment or real customer data. Use a disposable database and synthetic fixtures. Keep the mutation small enough that its expected effect is clear, and record which tests detected it.

Mutation survival is a detected test weakness, not evidence that the production app is exploitable. A killed mutation shows that the suite noticed one controlled change. It does not prove equivalent vulnerabilities are impossible.

Reduce shared assumptions

Independence is not binary. A human can review AI-generated tests while still inheriting the same mistaken requirement. A different model can repeat common training patterns. A static analyzer can find a missing authorization predicate without knowing the business sharing rules.

Increase independence by varying the source of the claim and evidence:

• derive invariants from a threat model or reviewed requirement
• have a reviewer who did not author the implementation inspect the boundary
• run integration tests against the actual authorization and persistence layers
• use a separate analysis method for classes the test suite may miss
• compare the deployed behavior with a safe negative request in an authorized environment

The companion article on human review and threat modeling explains how to define those requirements before reviewing generated code.

Separate generation from acceptance

Ask the assistant to label which tests it generated, which requirements they cover, and which dependencies are mocked. Do not let that summary decide acceptance. A reviewer should compare the suite with the threat model, API contract, authorization rules, and known failure modes.

Review test deletions and weakened assertions as carefully as implementation changes. A generated patch may make CI green by removing a failing negative case, changing an expected denial to success, or broadening a mock. Protect security test paths with code ownership where the repository risk justifies it.

Run the accepted suite from a clean checkout using the committed lockfile and a controlled test environment. Preserve the command and result instead of relying on the assistant’s statement that tests passed. If a required external service was unavailable or skipped, record the limitation.

For each sensitive boundary, have the reviewer add at least one case that was not proposed in the generation context. Name the invariant, the synthetic identities or resources, the expected denial, the state that must remain unchanged, and the layer the test actually crosses. The case may still miss a flaw, but the acceptance record will no longer be a paraphrase of the generated implementation.

Inspect mocks and false success paths

Mocks can make a unit test fast and precise. They can also bypass database policies, middleware, token validation, queues, retries, provider signatures, and deployment configuration. Keep unit tests, then add the smallest integration test that crosses the real security boundary.

Assert durable state and side effects instead of stopping at an HTTP status. A handler can return 200 while failing to update ownership, enqueue a job, or persist an audit event. A mocked provider can accept a signature that the real SDK rejects.

Check that tests fail for the intended reason. A cross-tenant request returning 500 is not an authorization pass merely because it did not return data. Assert the documented denial status and confirm no protected record or side effect appears.

Record coverage and evidence state precisely

When a generated suite passes, record the commit, environment, command, selected tests, assertion purpose, real and mocked dependencies, result, and limitations. If only unit tests ran with mocked storage, say so. Do not label the security claim “verified” without an independent verification receipt.

A fresh deterministic retest can become retest-confirmed within its documented coverage. If the original signal came from an engine-only or incomplete path, absence on a later run may remain inconclusive. Confidence and code coverage are triage data, not proof.

The 2026 paper The Illusion of Safety compared passing tests, static findings, and confirmed runtime violations in its selected C++ task set. Use that study as scoped evidence that verification layers can disagree. It does not establish a universal failure rate for AI-generated tests or other languages.

NIST SSDF 1.1 calls for code review, analysis, and testing as parts of secure development. OWASP’s Secure Coding with AI Cheat Sheet likewise recommends reviewing and testing generated output rather than accepting it on trust.

What automation can and cannot establish

Test generators can propose boundary values and regression cases. Coverage tools can show executed lines and branches. Mutation tools can measure whether a selected set of changes is detected. Static and dynamic scanners can find patterns or behaviors within their scope.

None of these tools knows every business invariant, valid delegation, operational dependency, or untested environment. Combining them reduces blind spots, but the result remains scoped to the reviewed requirements, tools, inputs, and runtime.

The browser-local AI app security checklist can document whether negative tests, independent review, and retest evidence exist. It cannot run the suite or decide whether its assertions prove the intended boundary.

Sources

• NIST SP 800-218: Secure Software Development Framework
• OWASP Secure Coding with AI Cheat Sheet
• OWASP Application Security Verification Standard
• The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code