I was thinking a bit about the bugs I found in the Piefed codebase yesterday. And these led to an emergency fix by the dev that’s now been implemented. https://codeberg.org/rimu/pyfedi/commit/093a466935849f27b3ecf2eab159129186320417
And what the real takeaway for me here is that the whole dynamic of how we approach security has now changed in ways most people don’t appreciate.
It used to take a lot of effort to find exploits in software projects because you’d have to spend a long time to familiarize yourself with the codebase, then comb through the code looking for mistakes that could be exploited. And to even do that, you’d need a good understanding of the protocols and specifications used by the application.
You basically had to be a domain expert with a deep understanding of how the application works. A random person looking at the source code would have little chance of finding any non trivial problems or figuring out how to actually exploit them.
And in that world, doing a private disclosure made a lot of sense because you did a lot of hard work to find it, and it wasn’t easy for somebody to replicate. This was valuable and dangerous knowledge that had to be communicated in a responsible fashion.
But now, anybody can throw an LLM at the code and it’ll sniff out vulnerabilities and even explain step by step how to exploit these security holes. So, the information itself isn’t really that valuable anymore. If I can throw an LLM at the code and find these problems in a few minutes, anybody else can do the same thing too.
I’m not a Python developer, I don’t have any deep knowledge of the Python stack used in Piefed, and on my own, I’d have zero chance of finding these exploits. But once the LLM identifies them, it’s very easy for me to verify that they are indeed real exploits, and to realize how they can be used maliciously.
The attacker doesn’t even need to have any deep knowledge of programming because the LLM can guide them through the exploit step by step.
Open source projects are particularly vulnerable here since anybody can just grab the source and throw an LLM at it to see if it can find exploits.
I’d argue that raising awareness that this is now the state of things is really important, and I would suggest that running an LLM against the code is minimal due diligence at this point.
Obviously, the LLM vulnerability check is not exhaustive, and if it doesn’t find anything that doesn’t mean there aren’t exploits in the code. But anything it does find should absolutely be checked by the developers.
People should be aware that we’re now living in the world where the bar for finding vulnerabilities is far lower than it used to be. And that means security must be taken far more seriously.
Private disclosure still makes sense to me when you add LLMs into the mix. It is possible that an LLM outputs some plausible-sounding story that over-estimates the actual risk and impact of the exploit. If this story is publicly announced to people who use the software but are not capable of assessing these risks themselves, this can easily have a negative unnecessary consequence - for example, people may bring their server down until an expert or developer provides an assessment or fix.
This is a source of noise, and I don’t agree that this is better than private disclosure. Via public disclosure one is applying a lot of pressure to the developer(s) to prioritize whatever is being disclosed, which may not always be the nicest thing to do, especially if the impact is not as significant as the LLM suggests. This may not have been what happened in your case (I don’t know the details), but I am thinking about the idea of the average person disclosing publicly LLM-discovered vulnerabilities.
I’m not talking about a scenario of a hypothetical exploit here. I’m talking about a concrete scenario where somebody finds an exploit and verifies it. In that case, people operating the software need to be aware of the vulnerability in the application they are running. Since the exploit is very easy to find, it should be assumed that malicious people would have found it as well.
You’re arguing against a case where you have an unverified exploit that LLM might’ve hallucinated. This is not the case I’m describing. And this provably did not happen in my case as is clearly evidenced by the fix the dev had to make in their server.
Definitely, disclosing (either private or publicly) a vulnerability that has been verified is significantly better than passing on the LLM output without verifying it.
It isn’t my intention to argue one specific case. What I think is that normalizing public disclosure of LLM-inspired vulnerabilities would lead to a wide distribution of cases. We would have some successful cases like yours, and also some cases of the type that I have mentioned. Increase in disclosures will raise the noise floor, and the fact that it is done publicly adds the additional pressure that I mentioned.
I see your point, but I don’t agree that the benefit of public awareness offsets the increase in noise. This disagreement isn’t rooted in aspects that we can objectively quantify though - we just have a difference of opinion here.
It would all depend on the rate of false positives, and as you say, we’d have to wait and see how this plays out. At the very least, what I’d want people to take away from this is that project maintainers absolutely should be using these tools themselves. They’re the people who are in the best position to decide whether something is a real issue, and it’s better to be safe than sorry here.