Observable ≠ Exploitable: The Problem With OAuth Security Measurements
A security scanner reports: 3,148 authorization requests are missing PKCE.
…and suddenly that sounds like 3,148 vulnerabilities. Or is it?
Most large-scale security measurements, OAuth or not, share the same shape. You point a crawler at a list of popular sites, parse the requests and responses, and count what's missing. The counts come out enormous, so the headline writes itself: X% of OAuth implementations are vulnerable. What those measurements quietly skip is the distinction between a vulnerability and an exploit. In a recent study at APIsec Labs, we set out to measure exactly how large the gap between the two actually is.
A passive crawl of the Tranco top-10,000[1] (put simply, the most popular websites) captured 4,436 authorization requests from 1,786 sites, via SSO-Monitor[2]. 3,148 of them contained no PKCE parameter. If you paused right there, you'd think thousands of applications are exposed to authorization-code attacks.
Then we tried to actually exploit what we'd measured, in a lab, within ethical boundaries, and the answer to "how many of those are real" turned out to be far smaller, and far stranger, than the count suggests. It points to a broader problem in how we think about security measurement, and OAuth just happens to show it more cleanly than most.
A scanner knows something is missing. Does it know why?
Most vulnerability scanners can point at what's wrong or absent without understanding the context behind it, and anyone who's spent their Tuesday afternoon triaging reports knows that context is the whole difference between an informational note and a critical finding.
A passive observer can inspect an OAuth authorization request and see whether code_challenge is present. If it isn't, the observation is unambiguous and objectively true: code_challenge is absent.
The more important question is whether an attacker could use that against you. That depends on a number of factors. Is this a public client or a confidential one? Which library generated the request? Which version? Is there another mechanism already binding the authorization code to the session?
PKCE's security significance depends on every one of those answers. Current OAuth guidance requires public clients to use PKCE and recommends it for confidential clients, while explicitly recognizing that other mechanisms cover the same ground in particular circumstances. The request tells you the parameter is gone. It doesn't tell you if that matters.
Even so, the OAuch study[3] of 100 real-world OAuth providers found that supporting a countermeasure and enforcing it are different things. An authorization server can advertise PKCE yet still accept a request that omits it, leaving it open to a PKCE downgrade attack that a request-level scan would score as "PKCE present, all good."
The evidence boundary
Every security tool has a field of vision, and that field of vision sets a hard limit on the kind of evidence it can produce. For a passive OAuth crawler, the boundary is the authorization request.
Inside the boundary, you can observe:
response_type=token- no
code_challenge - no
nonce - a particular
redirect_uri - a particular combination of parameters
Outside it are the questions the request simply cannot answer:
- Is the behavior exploitable?
- Does the server enforce a corresponding control?
- Is the client public or confidential?
- Does the framework override the observed behavior?
- Can an attacker actually satisfy the preconditions?
- Does the attack produce any real impact?
The further a claim sits from the artifact that produced it, the more evidence it needs to stand up. In practice, security tooling crosses that boundary constantly, and rarely tells you it has.
The cost of crossing the gap
Our crawl produced thousands of genuine observations. What we couldn't always justify were the security conclusions people habitually attach to them.
Turning an observation into a finding meant making a chain of claims: this parameter is missing, and it was generated by this library, and that library should have set it, and its absence is actually exploitable here. We built 5,017 evidence chains from every observation we could attribute, and after checking each link, only 25 held up. The rest broke somewhere along the chain. Here's how:
Not applicable. OmniAuth produced an apparent nonce_missing_oidc condition. But the base omniauth-oauth2 strategy is OAuth 2.0, not OpenID Connect, so it never consumes an ID token whose integrity a nonce would protect. The parameter was absent, but given the context, the vulnerability was not there.
Wrong component. Cognito and Okta requests carried no PKCE because the request we observed was the inbound federation leg, headed to an upstream identity provider. The SDK responsible for the application-facing flow didn't own that request at all. The observation was correct; the attribution wasn't.
Configuration, not default. ASP.NET Core and msal-browser ship their protections on by default. Seeing them absent in a live deployment tells you something about that deployment, someone turned them off, but it says nothing about whether the library ships insecurely. A crawler sees what a deployment did, not why it did it.
Vulnerable vs. exploitable?
So we stopped observing and started exploiting.
We stood up all 13 libraries in our matrix, each in its documented default configuration, and ran the real attack against each.
Of the thousands of missing-PKCE observations that started this whole exercise, the number that survived to a reproduced, boundary-crossing vulnerability against a real library default was less than we can count on our fingers.
One of them was an OAuth proxy that ships tons of preconfigured providers. Not one of those enabled a CSRF countermeasure by default: no state, no PKCE, no nonce, and a callback that silently skips state validation when none was configured. We reproduced the full attack end to end. The attacker starts a login, authenticates as themselves, and hands the resulting callback to a victim whose session is mid-login. The victim lands inside the attacker's account. Flipping the same library into its hardened profile stops the attack; we're coordinating disclosure with the maintainers on this case, so we can't name it here.
passport-oauth2 ships the identical insecure default (no CSRF protection out of the box), and we reproduced the same attack against it. Our first instinct was to disclose it, but we can't, because the maintainer has publicly declined the issue as documented behavior. So the two libraries where the attack genuinely works are the two the passive study can say the least about: one it couldn't see, one the ecosystem has already argued about and moved on from.
And when we coded 105 real, disclosed OAuth vulnerabilities from the wild, 77 of them had no observable counterpart at all, nothing a passive crawl could have flagged, because the vulnerabilities lived in server behavior that was out of our crawler's field of vision.
How far does the evidence take us?
None of this means the crawl was worthless. It means every claim it produced sits at a specific level of evidence, and the levels are not interchangeable. OAuth gives a very clean and prominent ladder:
- Observation:
code_challengeis absent. - Exposure: the implementation isn't visibly using PKCE.
- Applicability: PKCE is actually the relevant control for this flow and client type.
- Exploitability: an attacker can satisfy the preconditions and the path is reachable.
- Reproduction: the attack succeeds against the measured configuration.
These five claims carry five different burdens of proof, and no level should be silently promoted to the one above it without the evidence that rung demands. Silent promotion is exactly what buries triagers and developers, who then can't tell a level-1 observation from a level-5 exploit because both arrived wearing the same CVSS score.
We observe an artifact, infer a property, and too often report the inference as though we measured the property directly. The more assumptions stacked between the observation and the claim, the more the validation matters.
This box gives you one way to reason about any security measurement in terms of exploitability:
| Exploitable | Not exploitable | |
|---|---|---|
| Observable | True positive | Informational / False positive |
| Not observable | Blind spot | Irrelevant |
Most conversations about security testing quality live in the top-right box: how many false positives does the tool produce? But the bottom-left box matters just as much, and gets far less attention: what real attack paths are fundamentally invisible to the measurement? A tool that flags everything has terrible precision; a tool that flags nothing has terrible recall.
Ask better questions while security testing
Stop asking whether a particular scanner is right. Ask instead: what is the strongest claim this evidence actually supports?
That reframes what automated tooling is for. A scanner doesn't have to prove every vulnerability; its job is to generate high-quality hypotheses. But a hypothesis has to stay visibly a hypothesis until further evidence upgrades it. The moment code_challenge absent gets reported as "this is exploitable," you've skipped four levels of proof and handed someone false certainty.
Our study's answer to "how big is the gap between observable and exploitable" turned out to be its own best example. There's more to application security than what shows up in a scan. Explore APIsec to learn more.