Why does code review miss the security defects AI writes?

Well structured code review catches most of what ordinary use would have surfaced anyway. It misses the defects that nothing surfaces until somebody tries to break in, and running more of it doesn't reach them.

AI writes more of the defects that show up when somebody attacks the software even as it writes fewer defects of other kinds. That happens because ordinary development does not have a mechanism to produce feedback about being attacked. Negative testing comes closest. It checks the wrong inputs somebody thought to write down, things like a wrong type, a value that runs too long, or a step taken before its prerequisite. An attacker sends well formed input as the wrong caller, things like somebody else's record number in a field built for record numbers, a valid session token replayed from a different account, or a request to a URL nothing in the interface links to. None of that is malformed, so a missing authorization check works fine.

The Numbers On AI Written Code

Veracode tested more than a hundred models across eighty coding tasks and found security problems in about 45 percent of them. Across model generations that measurably improved on functional scores, the security pass rate did not move.

An academic study went through 733 real snippets from three coding assistants and found weaknesses in 29.5 percent of the Python and 24.2 percent of the JavaScript, spread across 43 distinct weakness types. A separate comparison of 320 pull requests written with AI against 150 written by people counted security issues about 1.5 times as often in the first group. That same comparison counted 1.7 times as many issues of every kind, so security was not the category that stood out. At the volume end, one analysis of Fortune 50 repositories reported monthly findings going from roughly a thousand to over ten thousand in six months. That figure is directional, and it tracks volume rather than a defect rate.

There's also a set of widely quoted figures from applications built on vibe coding platforms, and those read better as illustration than as evidence. The counterfactual to a badly built vibe coded app is usually no app at all, so the sample says more about who is building than about what the tools do. It's useful for seeing that these bugs reach production. It isn't useful for much past that.

Missing Controls And Correctness Bugs

The findings from those studies sort into missing controls and correctness bugs. On correctness the sources disagree. The pull request comparison counted 75 percent more logic and correctness problems in AI code, and the study across half a million samples found model written code produced fewer defective Python samples than people did and far more defective Java ones. On missing controls they agree, in both languages.

Missing controls are protections the software never got. Cross site request forgery protection is absent, the check that stops another site from acting on your application using a logged in customer's session. Security headers and scoped access policies go the same way. An unscoped database key makes development smoother, because nothing throws a permission error. Server side request forgery shows up, where an attacker gets your server to fetch something on their behalf that it shouldn't reach. Changing an identifier returns somebody else's record, and passwords get stored badly. A deserializer runs whatever the stored bytes tell it to, and random number generation isn't random. Credentials get written into source at about twice the rate people manage. That same study found defects and vulnerabilities moving in opposite directions. One of the three models wrote fewer defective Java samples than people did and twice as many vulnerable ones.

Correctness bugs are code that runs and gets the answer wrong. The pull request comparison put its own categories as business logic errors, misconfigurations and unsafe control flow. On the enterprise systems I've worked on in insurance and healthcare, correctness bugs are the bulk of what gets found, because the business rules are that complicated.

An Edge Case Is Late, Not Invisible

My first answer to what separates missing controls from correctness bugs was that AI fails wherever the defect produces no symptom while the application is running normally.

A rounding error in a currency few customers use produces no symptom today, and produces one the first time somebody invoices in it. A race condition that needs two requests arriving in the wrong order shows up the first time real load delivers them that way. A missing authorization check has no such moment. Nothing in ordinary use is built to make the request that would reveal it, so it surfaces when somebody makes it on purpose, and otherwise by an accident nobody can plan on.

Seeing The Flaw Takes Somebody Trying To Break In

Somebody using the product as intended doesn't find the authentication bypass or reach another customer's record. Ordinary use exercises the range of input values over time but misses the attack paths.

Negative testing is the part of ordinary practice aimed at the software misbehaving. It works from the specification, so it covers what happens on a wrong type or an overlong value. Those are the wrong inputs somebody anticipated, wrote down, and turned into cases.

Before AI, on the teams I worked with, the positive path was usually the most executed due to time constraints. A sprint gives a defined amount of time to find, fix and validate an item. When one turns out harder than expected, the hours go into getting the logic right, then into defensive programming, guarding against null values, try and catch blocks, logging. Then all of that gets tested against varying levels of correct data, a large job on its own. Negative testing happens after you have resolved the issues with the happy path. Some cases get written, but time determines how much of the list gets executed. Though agent assisted development can change the time needed to run extensive negative testing.

An attacker starts from a goal instead of a list. The paths that matter are the ones cases miss because of capacity rather than diligence. A person writes cases for what they can imagine.

A model works from the published record instead, the accumulated disclosures and write ups of how other people got in. It runs that against the specific set of packages, versions and services a deployment happens to run, in a single pass. That's a scale that wasn't practical before AI, which makes "we should have thought of that" the wrong conclusion of what happened.

Researchers handed a GPT-4 agent fifteen disclosed vulnerabilities and a working environment. With the disclosure text it exploited 87 percent of them. Without the text, 7 percent. Two open source scanners run against the same fifteen got none, and the runs averaged about three and a half dollars each. That was a 2024 model on a small benchmark, so it reads as a lower bound rather than a current measurement.

The model used the disclosure text, not reasoning about the code. It's also why the scanners got none, because a rule list only covers what somebody already wrote into it, and the published record is larger than any rule list. The same capability is available to the defending side, and the difference is that the attacker is aiming it at getting in.

That puts a negative test suite and a scanner's rule list in the same category, useful and bounded by what their authors already knew.

Detectors trained on the published vulnerability datasets run into the same boundary. The one that scored best on its own test data reached 0.70 accuracy, meaning it labelled that share of cases correctly, and dropped to 0.49 against real world instances of the 25 weakness types MITRE ranks as most common. On a balanced two way call, 0.49 is a coin flip.

A Model Needs Direction

There's a version of this that blames the model for not knowing better. Ask a current model what forgery protection a form needs and the answer comes back correct. Hand it the code and ask it to check, and it usually finds the gap. The knowledge doesn't get applied unless something in the request names it.

A pipeline can't depend on knowledge that has to be asked for.

Public code skews hard toward the version without the control. Tutorials, quickstarts, StackOverflow answers and sample apps strip out what is not being demonstrated, so the version a model has seen most often is the one missing the check. Nothing in the scoring counts against that, because a version with the check and a version without it both compile, both pass the tests, and the person accepts either. That is most likely why the security pass rate stayed flat while functional scores improved. Changing it takes direction supplied when the code is written.

Your Gates Catch The Detectable Half

Static analysis, which reads source looking for known bad constructions, catches injection and cross site scripting reliably. Secret scanning finds credentials written into files, at least the ones matching a known format. Dependency checking flags libraries with published vulnerabilities. Rails and Django turn forgery protection on by default, so an application built on one of them starts with that boundary. A service built on a lighter framework starts without one.

None of them catch authorization logic, because a general purpose scanner has no way to know which records this customer is entitled to see. Reading another customer's record gets through for the same reason, since that request is indistinguishable from a legitimate one at the code level. A privilege escalation path, where an attacker turns ordinary access into more than they were given, is further out of reach, because it's a property of how components connect rather than of any file.

Of the four categories the pull request comparison separated out, three are things static analysis already catches. Cross site scripting ran highest at 2.74 times the human rate, improper password handling at 1.88, and unsafe deserialization at 1.82. The one the gates can't see is reading another customer's record by changing an identifier, at 1.91.

So the gates don't reduce the risk proportionally. They remove the detectable defects and leave what they can't see. Adding review capacity adds more of the view that already missed it.

Adversarial Review First, Substitutes After

• Adversarial review. A penetration test, a red team, anybody whose goal is breaking rather than building. It supplies the missing observer directly, and the other three approximate it. Most organizations run it last, because it costs the most and takes the most scheduling.
• Written invariants. Rules that name what correct means, so the model applies knowledge it already has. They catch what somebody already met and wrote down, and nothing past that.
• A view of the whole system. Component connections, trust boundaries, which credential reaches which service. It reaches privilege escalation and authorization, because those answers aren't in any single file. It goes stale on every deploy unless somebody owns it.
• Reading outside research. New categories arrive from incidents and published work rather than from introspection. On its own it applies nothing. It pays off by feeding the invariants.

Having a different vendor's model review the first one's code does help, though not because the two vendors have different blind spots. My position is that most of the benefit comes from the reviewing session not having already argued itself into the choices. A fresh session of the same model is also an option. Real differences between vendors add a little on top. Neither supplies attack intent, and an independent reviewer who isn't trying to break anything still doesn't take those paths.

Independence Has To Be Checked

Adversarial review, written invariants and outside research all assume the reviewer is genuinely separate from whatever wrote the code, and none of the three is guaranteed to be. A model reviewing the code was trained on much the same material as the model that wrote it. Written rules pulled from a scanner vendor come from the same public vulnerability lists the models absorbed. The research telling you which categories to watch is increasingly written the same way the code was.

I read a research note on this topic while drafting this piece. Page two says it was generated with AI assistance and never went through the publishing organization's review process.

You can add every one of these and still have one source behind them. Consider where each one got what it knows, and whether any two got it from the same place.

The whole system view is the one input that cannot come from the same place the code did. It is drawn from your own deployment, not from published material. A model cannot infer it from the files in front of it either, so it has to be built and handed over.

Measure The Disagreement Between Two Controls

You find the problems your rules describe. You can't check for something you don't know you need. So your coverage looks complete to you.

Surveyors hit the same problem. They measure their way around a loop and back to the point they started from. The position they calculate at the end should match the point they started at, and the difference between the two is how far off their measurements were. That gives them a number for their own error without needing a better instrument.

A surveyor who walks back along the same line with the same instrument does learn something. A repeat catches a misread or a transposed number. What it misses is an instrument that is off in the same direction every time, because the second pass repeats that error exactly and the two agree.

Run your own adversarial capability and an outside red team against the same release. Count what each found alone and what both found. What one found and the other did not is the size of what a single pass misses. If they produce nearly the same list, check whether they are running the same tooling against the same published record of vulnerabilities before you read that as coverage.

The Control You Buy Was Tested On Somebody Else's System

An enterprise application most likely runs dozens to hundreds of components across differing operating systems, network layouts, firewall configurations, identity setups and version combinations. The number of possible combinations is large enough that yours is unlikely to be one anything was tested against.

Which means the catch rates you've been quoted for your review tooling are averages measured on other systems. They still tell you something, the way any average does, but they are not a measurement of your stack. The detector that scored 0.70 on its own test data and 0.49 on real world code is what that gap looks like with a number on it.

Where To Spend, And What Not To Buy

Fund an adversarial capability first, not an engagement twice a year, it belongs in the deployment pipeline. A biannual engagement gives you a snapshot of your system that changes with the next deployment.

Build it AI assisted. The enumeration is the part that got cheap, and a model reading the published record of vulnerabilities against your specific packages and versions reaches combinations a person would not, for a few dollars a run. What it cannot supply is what to try when the record has nothing about your combination.

Running it safely is a separate build and this piece doesn't cover it. The isolation the attempts run inside, and the point where a person approves the next step, are what keep the capability from becoming its own exposure. The benchmark that measured these agents also recorded them going after vulnerabilities other than the ones they were pointed at. That's the argument against setting a goal and letting the run continue unattended, and a deviation from the plan should reach a person.

The second purchase is an instrument built against your system in particular, because that is the one kind the population problem doesn't reach. An evaluation set is one, meaning a fixed collection of cases with known right answers that you run your own system against and score. It's why governance discussions end up being about measurement.

Picking a cheaper version of either is fine. Believing you're covered because you picked one is the problem.

The adversarial capability needs an owner, somebody who keeps it in the deployment pipeline, runs it against real releases, and grows the rule set that tells the model what to investigate.

Standing the capability up is the first step. New attack examples and published research close more of the gap over time. You will not close all of it.

When findings went from a thousand a month to ten thousand, the reflex is buying more review capacity, because review is the control the volume appears to have outgrown. It's the purchase I'd expect to return least here, since it's more of the view you already have. Tooling is built out of what has been disclosed, and your combination has not been, so the part worth paying for is the person deciding what to try.

Speed Is The Wrong Variable

The articles quoting these numbers tend to close the same way. Your security team or an attacker, whoever gets there first.

That assumes one track and one finish line with speed as the variable. The attacker is trying to break in and your review isn't, so they aren't running the same course. Going faster on yours doesn't put you on theirs.

Is anything you run trying to break in?