Generative AI tools have become superhuman in their capacity to discover vulnerabilities in computer code. Describing the Claude Mythos Preview, Anthropic recently wrote:
During our testing, we found that Mythos Preview is capable of identifying and then exploiting zero-day vulnerabilities in every major operating system and every major web browser when directed by a user to do so. The vulnerabilities it finds are often subtle or difficult to detect. Many of them are ten or twenty years old, with the oldest we have found so far being a now-patched 27-year-old bug in OpenBSD—an operating system known primarily for its security.
Similarly, Mozilla reports that Claude Mythos Preview identified 271 vulnerabilities in Firefox, which is another mature and very thoroughly reviewed piece of software. &c.
Linus’s Corollary
Once upon a time, the primary consequence of open sourcing your code was enabling collaboration with your users, making the software more helpful for them and improving its security. Eric Raymond called the security implications of open source Linus’s Law:
Given enough eyeballs, all bugs are shallow.
One way of understanding Linus’s Law in the context of generative AI is that a model as powerful as Claude Mythos Preview provides essentially infinite eyes. Perhaps we now need Linus’s Corollary:
Given powerful enough AI, all bugs will be found and exploited.
I haven’t heard enough detail yet about the Canvas breach to know how the system was compromised, but when I heard about it my first thought was: I bet this is because Canvas is open source. Someone used powerful AI to review the source code, found vulnerabilities, and exploited one. Then my second thought was: is the Canvas breach the beginning of the end for open source?
The Asymmetry of AI for Defense
Mozilla points out that AI tools empower developers, too, giving defenders superpowers as they review and secure their own code. And according to the report, Mozilla used it very effectively for this purpose. But not every open source project is run by a major organization like Mozilla.
The majority of open source projects are passion projects with a single, part-time contributor. Do we now expect each of these hobbyists to pay for access to frontier AI models and consistently integrate them into their release process? If they can’t or for any reason don’t want to, what should they do? Is sharing code that is all but guaranteed to be exploited an appropriate choice to make?
What Now?
Perhaps in the future there will be sufficiently powerful open weights AI models, and these can be run locally (for free) and integrated directly into whatever future development tools and workflows look like. But until then, what now?