The Proxy Problem: Applying Goodhart's Law to AI-Generated Vulnerability Reports
Over the past year, bug bounty platforms have reported a dramatic rise in AI-assisted, "slop" vulnerability submissions. Much of the discussion has centered on whether large language models are capable of discovering genuine vulnerabilities or whether they merely generate convincing-looking but ultimately meaningless reports.
That debate, while important from other angles, misses a more fundamental shift in the way we've been seeing vulnerability disclosure as a whole. AI has fundamentally altered the economics of vulnerability reporting by making it almost costless to produce signals that bug bounty programs historically relied upon as indicators of genuine research. Viewed through this lens, the recent wave of AI-generated "slop" is not merely a quality problem but rather an illustration of Goodhart's Law in action.
When a proxy loses its power
Goodhart's Law[1] is commonly summarized as: when a measure becomes a target, it ceases to be a good measure. More intuitively, when a measure that corresponds to a target is optimised directly, agents find ways to move the measure without moving the underlying target. The familiar factory example captures the problem: if you reward workers for the number of nails produced, they make tiny nails; reward them by total weight and they make oversized ones. The metric improves while the actual objective is ultimately lost. In the taxonomy proposed by Manheim and Garrabrant[2], this is closest to adversarial Goodhart: an agent learns how to optimise the evaluator's proxy rather than the underlying goal.
As for bug bounty programs particularly, they have never directly measured "research quality" at the point of submission. Instead, triagers usually relied on a collection of inexpensive heuristics that historically correlated with genuine vulnerability discovery and report plausibility. These include:
- technically accurate reproduction steps,
- detailed exploit narratives,
- correct CWE mappings,
- structured vulnerability reports,
- references to specific functions or implementation details.
This heuristic correlation is psychologically wired in our brains. Plausibility is an inference a human makes quickly, under volume pressure, using surface cues as a stand-in for the expensive step of full verification. That's the mechanism documented in the processing-fluency literature: confident, structurally well-formed text gets processed as more credible independent of whether it's true. The reason that cognitive shortcut used to be trustworthy is a separate, economic point in itself. Producing convincing technical detail was expensive to fake before AI, so the heuristic tracked something real by construction, the same costly-signaling logic that makes a peacock's tail or an expensive degree an honest signal.
But these characteristics were never the objective themselves. The objective was simple: Does this report identify a real, exploitable security vulnerability?
Historically, these surface features worked remarkably well because producing them required substantial effort. A polished report usually implied that someone had invested time in understanding the application and validating their findings. I mean, nobody really hand-crafted a convincing buffer-overflow report without understanding buffer overflows.
The cost of producing convincing-looking reports has collapsed, while the cost of discovering genuine vulnerabilities remains largely unchanged. The proxy has become dramatically cheaper to optimize without improving the underlying signal. That is precisely the situation Goodhart's Law predicts.
What the numbers say
Stenberg's own published figures trace the shape of this collapse. Before 2025, curl's confirmed-vulnerability rate ran north of 15% of submissions[3]. Through 2025, as AI-generated reports climbed to roughly a fifth of total volume, that rate fell below 5%[4], effectively not even one in twenty submissions was impactful.
Confirmed-vulnerability rate and relative submission volume, reconstructed from Stenberg's dated blog posts (Jul 2025–May 2026). These are phase-level snapshots, not continuous monthly data.
Flooding a cheap channel with cheap noise is a familiar spam-economics story on its own, no Goodhart framing required. But the interesting part is what happened after the incentive was removed. In February 2026, curl removed the monetary reward entirely and moved reporting off HackerOne. By March, they'd returned to HackerOne, still with no bounty attached[5]. The confirmed rate recovered to 15–16% (back to, even slightly past the pre-2025 baseline) while submission volume kept climbing, eventually running four to five times 2024 levels. Stenberg has offered a different interpretation of this recovery. In a June interview, he attributed at least part of the improvement to newer models producing technically accurate reports that he had not expected AI systems to generate. That explanation cannot be dismissed: model capability and the incentive structure both changed between the low-validity and recovery periods.
LLMs don't know money, but report quality clearly tracked changes in the incentive structure.
So if degraded report quality were simply a function of LLMs being unreliable at finding real bugs, removing a cash reward shouldn't have changed the confirmed rate at all, the model's hallucination rate doesn't respond to money. It is also worth noting that solely model improvement over time is unlikely to cause a difference this steep, but the available data does not allow the contribution of model improvement to be separated cleanly from the contribution of the incentive change.
My interpretation is that some submitters were optimising for reward eligibility, using AI as the cheapest available way to satisfy the surface-plausibility proxy that historically gated payment. Removing the bounty weakened that target, but it did not remove every incentive to submit. Accepted reports, platform reputation, leaderboard position, public attribution, and resume value remained available, which helps explain why submission volume stayed four to five times above 2024 levels even after the cash reward disappeared.
HackerOne reported roughly 76% year-over-year submission growth through March 2026[6], with the share of submissions flagging genuine vulnerabilities holding around 25%. Bugcrowd saw its triage queue grow by more than 334% over three weeks in March, even after excluding legitimate submissions produced through traditional researcher workflows[7]. Trey Ford described the phenomenon as "sloptimism": high-volume, low-validation submission driven more by unwarranted confidence in AI output than by deliberate fraud. Nextcloud paused its own program in April, citing the same dynamic.
Bugcrowd and Nextcloud both attribute their growth explicitly to low-validation AI submissions, so the pattern isn't an artifact of curl's specific setup. HackerOne's platform-wide 76% is murkier, the company's own data suggests a fairly steady signal rate, meaning that growth looks more like AI raising real discovery volume than flooding the channel with noise. Either way, none of these platforms ran curl's reward-removal intervention, so this data speaks to scope, not mechanism.
The layered Goodhart dynamic
There's a related but distinct Goodhart story on the researcher's side. Accepted-report count, acceptance rate, and platform leaderboard rank have long functioned as a portfolio signal. These metrics are inexpensive ways for employers, conference organizers, and other researchers to infer "this person finds real, impactful bugs" without auditing the work directly. That signal held for the same reason the triage proxy held: sustaining a high count used to require actually doing the work.
Bad actors flooding a common signal devalue it for everyone drawing on it, honest researchers included, which kind of pushes it closer to Akerlof-style adverse selection[8] than to single-agent Goodhart. So, when low-quality participation becomes difficult to distinguish from high-quality participation, the shared signal loses value for everyone, including legitimate researchers. Uncertainty about quality allows lower-quality participation to devalue the market signal produced by higher-quality work. I personally believe platform responses are consistent with an attempt to rebuild a harder-to-fake version signal; Bugcrowd now requires identity verification after ten invalid reports from an account, and several programs have adopted minimum HackerOne Signal-score gates before new submitters can participate at all.
A move toward AI-assisted or fully automated triage is a natural response to volume but if that system is trained or calibrated on historical human triage decisions, it inherits the fluency heuristic those decisions were made with. It learns "what acceptance looks like" from labels that were themselves fluency-biased, which automates the vulnerability rather than closing it. That reopens the same evaluator-gaming dynamic one layer up so a generator that has learned what an automated triager's proxy responds to has an easier target than one that had to fool a human.
The bigger picture
A proxy's validity is conditional on the cost regime it was calibrated under. Report-surface-plausibility was a decent proxy for validity as long as producing that surface required real work. That regime shifted because the cost of faking the proxy collapsed on the input-generation side. The same structural claim shows up anywhere a cheap-to-produce signal has been standing in for an expensive-to-verify property: citation count standing in for research quality, or even a confident tone standing in for correctness.
The practical response follows directly from the diagnosis. Programs can either make the proxy expensive again, through identity verification, rate limits, reputation thresholds, deposits, or other forms of staking, or they can verify the target more directly by requiring executable proofs of concept, reproducible test cases, runtime evidence, or automated validation. The two explanations advanced here also produce different predictions. If incentives are the dominant mechanism, bountied programs with low submission costs should remain more degraded than comparable non-bountied programs. If improving model capability is the dominant mechanism, confirmed-vulnerability rates should rise across both rewarded and unrewarded programs. The likely outcome is some combination of both.