AI tools were apparently used for locating the bugs but the reports were real and legit.
Yes, but the FFMPEG developers do not know this until after they triage all the bug reports they are getting swamped with. If Google really wants a fix for their 6.0 CVE immediately (because again, part of the problem here was google’s security team was breathing down the necks of the maintainers), then google can submit a fix. Until then, fffmpeg devs have to work on figuring out if any more criticial looking issues they receive, are actually critical.
It’s nuts to suggest continuing to ship something with known vulnerabilities without, at minimum,
Again, the problem is false positive vulnerabilities. “9.0 CVE’s” (that are potentially real) must be triaged before Google’s 6.0 CVE.
It would be great if Google could fix it, but ffmpeg is very hard to work in, not just because of the code organization but because of the very specialized knowledge needed to mess around inside a codec. It would be simpler and probably better for Google to contribute development funding since they depend on the software so heavily.
Except google does fix these issues and contribute funding. Summer of code, bug bounties, and other programs piloted by Google contribute both funding and fixes to these programs. We are mad because Google has paid for more critical issues in the past, but all of a sudden they are demanding free labor for medium severity security issues from swamped volunteers.
Being able to find bugs (say by fuzzing
Fuzzing is great! But Google’s Big Sleep project is GenAI based. Fuzzing is in the process, but the inputs and outputs are not significantly distinct from the other GenAI reports that ffmpeg receives.
Those approaches would be ridiculous bloat, the idea is just supply some kind of wrapper that runs the codec in a chrooted separate process communicating through pipes under ptrace control or however that’s done these days.
Chroot only works on Linux/Unix and requires root to use, making it not work in rootless environments. Every single sandboxing software comes with some form of tradeoff, and it’s not ffmpeg’s responsibilities to make those decisions for you or your organization.
Anyway, sandboxing on Linux is basically broken when it comes to high value targets like google. I don’t want to go into detail, but but I would recommend reading maidaden’s insecurities (I mentioned gvisor earlier because gvisor is google’s own solution to flaws in existing linux sandboxing solutions). Another problem is that ffmpeg people care about performance a lot more than security, probably. They made the tradeoff, and if you want to undo the tradeoff, it’s not really their job to make that decision for you. It’s not such a binary, but more like a sliding scale, and “secure enough for google” is not the same as “secure enough for average desktop user”.
I saw earlier you mentioned google keeping vulnerabilities secret, and using them against people or something like that, but it just doesn’t work that way lmao. Google is such a large and high value organization, that they essentially have to treat every employee as a potential threat, so “keeping vulns internal” doesn’t really work. Trying to keep a vulnerability internal will 100% result in it getting leaked and then used against them.It would be great if Google could fix it, but ffmpeg is very hard to work in, not just because of the code organization but because of the very specialized knowledge needed to mess around inside a codec. It would be simpler and probably better for Google to contribute development funding since they depend on the software so heavily.
It’s nuts to suggest continuing to ship something with known vulnerabilities without, at minimum, removing it from the default build and labelling it as having known issues. If you don’t have the resources to fix the bug that’s understandable, but own up to it and tell people to be careful with that module.
You have no fucking clue how modern software development and deployment works. Getting rid of all CVE’s is actually insanely hard, something that only orgs like Google can reasonably do, and even Google regularly falls short. The vast majority of organizations and institutions have given up on elimination of CVE’s from the products they use. “Don’t ship software with vulnerabilities” sounds good in a vacuum, but the reality is that most people simply settle for something secure enough for their risk level. I bet you if you go through any piece of software on your system right now you can find CVE’s in it.
You don’t need to outrun a hungry bear, you just need to outrun the person next to you Cybersecurity is about risk management, not risk elimination. You can’t afford risk elimination.
Yes, but the FFMPEG developers do not know this until after they triage all the bug reports they are getting swamped with.
With a concrete bug report like “using codec xyz and input file f3 10 4d 26 f5 0a a1 7e cd 3a 41 6c 36 66 21 d8… ffmpeg crashes with an oob memory error”, it’s pretty simple to confirm that such a crash happens. The hard part is finding the cause and fixing it. I had understood the bug search to be fuzzing controlled by AI so I referred to it as fuzzing. Apparently though the AI is also writing the bug report now, so yeah ok, maybe there is potential slop there.
“Don’t ship software with vulnerabilities” sounds good in a vacuum,
I said KNOWN vulnerabilities. Make it known vulnerabilities without known mitigations if you prefer.
I wrote a few of those GNU coreutils that the Rusties are now rewriting. I don’t remember hearing of any CVE’s connected with any of them, though that is mostly because they are uncomplicated.
Here’s all the Debian security advisories for the past year or so. There aren’t THAT many, and they are mostly in complicated network programs, the Linux kernel, etc. Also a lot aren’t actual vulns: https://www.debian.org/security/
This was the first search hit about ffmpeg cve’s, from June 2024 so not about the current incident. It lists four CVE’s, three of them memory errors (buffer overflow, use-after-free), and one off-by-one error. The class of errors in the first three is supposedly completely eliminated by Rust. No idea about the fourth. Not claiming that a Rust reimplementation of ffmpeg is anywhere near feasible. Dunno if the current set of CVE’s are comparable but it’s a likely guess. Anyway, as SPJ likes to say about Haskell’s type system, the idea is to stop fixing bugs one by one, and instead eliminate entire classes of bugs. We can’t fix everything but we can certainly do better than we are doing now.
I saw earlier you mentioned google keeping vulnerabilities secret, and using them against people or something like that,
That was listed as an example of what not to do, not a proposal of an approach to take.
With a concrete bug report like “using codec xyz and input file f3 10 4d 26 f5 0a a1 7e cd 3a 41 6c 36 66 21 d8… ffmpeg crashes with an oob memory error”, it’s pretty simple to confirm that such a crash happens
Google’s big sleep was pretty good, it gave a python program that generated an invalid file. It looked plausible, and it was a real issue. The problem is that literally every other generative AI bug report also looks equally as plausible. As I mentioned before, curl is having a similar issue.
Stenberg said the amount of time it takes project maintainers to triage each AI-assisted vulnerability report made via HackerOne, only for them to be deemed invalid, is tantamount to a DDoS attack on the project.
So you can claim testing may be simple, but it looks like that isn’t the case. I would say one of the problems is that all these people are volunteers, so they probably have a very, very limited set of time to spend on these projects.
This was the first search hit about ffmpeg cve’s, from June 2024 so not about the current incident. It lists four CVE’s, three of them memory errors (buffer overflow, use-after-free), and one off-by-one error. The class of errors in the first three is supposedly completely eliminated by Rust.
FFMpeg is not just C code, but also large portions of handwritten, ultra optimized assembly code (per architecture, too…). You are free to rewrite it in rust if you so desire, but I stated it above and will state it again: ffmpeg made the tradeoff of performance for security. Rust currently isn’t as performant as optimized C code, and I highly doubt that even unsafe rust can beat hand optimized assembly — C can’t, anyways.
(Google and many big tech companies like ultra performant projects because performance equals power savings equals costs savings at scale. But this means weaker security when it comes to projects like ffmpeg…)
Have any of the google-submitted vulnerability reports turned out to be invalid? Project Zero was pretty well regarded.
Yes I know about the asm code in ffmpeg though IDK if it’s doing anything that could lead to a use after free error. I can understand if an OOB reference happens in the asm code since codecs are full of lookup tables and have to jump around inside video frames for motion estimation, but I’d hope no dynamic memory allocation is happening there. Instead it would be more like a GPU kernel. But, I haven’t examined any of it.
Anyway there’s a big difference between submitting concrete input data that causes an observable crash, and sending a pile of useless spew from a static analyzer and saying “here, have fun”. The Curl guy was getting a lot of absolute crap submitted as reports.
From the GCC manual “bug criteria” section:
If the compiler gets a fatal signal, for any input whatever, that is a compiler bug. Reliable compilers never crash.
Yes, but the FFMPEG developers do not know this until after they triage all the bug reports they are getting swamped with. If Google really wants a fix for their 6.0 CVE immediately (because again, part of the problem here was google’s security team was breathing down the necks of the maintainers), then google can submit a fix. Until then, fffmpeg devs have to work on figuring out if any more criticial looking issues they receive, are actually critical.
Again, the problem is false positive vulnerabilities. “9.0 CVE’s” (that are potentially real) must be triaged before Google’s 6.0 CVE.
Except google does fix these issues and contribute funding. Summer of code, bug bounties, and other programs piloted by Google contribute both funding and fixes to these programs. We are mad because Google has paid for more critical issues in the past, but all of a sudden they are demanding free labor for medium severity security issues from swamped volunteers.
Fuzzing is great! But Google’s Big Sleep project is GenAI based. Fuzzing is in the process, but the inputs and outputs are not significantly distinct from the other GenAI reports that ffmpeg receives.
Chroot only works on Linux/Unix and requires root to use, making it not work in rootless environments. Every single sandboxing software comes with some form of tradeoff, and it’s not ffmpeg’s responsibilities to make those decisions for you or your organization.
Anyway, sandboxing on Linux is basically broken when it comes to high value targets like google. I don’t want to go into detail, but but I would recommend reading maidaden’s insecurities (I mentioned gvisor earlier because gvisor is google’s own solution to flaws in existing linux sandboxing solutions). Another problem is that ffmpeg people care about performance a lot more than security, probably. They made the tradeoff, and if you want to undo the tradeoff, it’s not really their job to make that decision for you. It’s not such a binary, but more like a sliding scale, and “secure enough for google” is not the same as “secure enough for average desktop user”.
I saw earlier you mentioned google keeping vulnerabilities secret, and using them against people or something like that, but it just doesn’t work that way lmao. Google is such a large and high value organization, that they essentially have to treat every employee as a potential threat, so “keeping vulns internal” doesn’t really work. Trying to keep a vulnerability internal will 100% result in it getting leaked and then used against them.It would be great if Google could fix it, but ffmpeg is very hard to work in, not just because of the code organization but because of the very specialized knowledge needed to mess around inside a codec. It would be simpler and probably better for Google to contribute development funding since they depend on the software so heavily.
You have no fucking clue how modern software development and deployment works. Getting rid of all CVE’s is actually insanely hard, something that only orgs like Google can reasonably do, and even Google regularly falls short. The vast majority of organizations and institutions have given up on elimination of CVE’s from the products they use. “Don’t ship software with vulnerabilities” sounds good in a vacuum, but the reality is that most people simply settle for something secure enough for their risk level. I bet you if you go through any piece of software on your system right now you can find CVE’s in it.
You don’t need to outrun a hungry bear, you just need to outrun the person next to youCybersecurity is about risk management, not risk elimination. You can’t afford risk elimination.With a concrete bug report like “using codec xyz and input file f3 10 4d 26 f5 0a a1 7e cd 3a 41 6c 36 66 21 d8… ffmpeg crashes with an oob memory error”, it’s pretty simple to confirm that such a crash happens. The hard part is finding the cause and fixing it. I had understood the bug search to be fuzzing controlled by AI so I referred to it as fuzzing. Apparently though the AI is also writing the bug report now, so yeah ok, maybe there is potential slop there.
I said KNOWN vulnerabilities. Make it known vulnerabilities without known mitigations if you prefer.
I wrote a few of those GNU coreutils that the Rusties are now rewriting. I don’t remember hearing of any CVE’s connected with any of them, though that is mostly because they are uncomplicated.
Here’s all the Debian security advisories for the past year or so. There aren’t THAT many, and they are mostly in complicated network programs, the Linux kernel, etc. Also a lot aren’t actual vulns: https://www.debian.org/security/
This was the first search hit about ffmpeg cve’s, from June 2024 so not about the current incident. It lists four CVE’s, three of them memory errors (buffer overflow, use-after-free), and one off-by-one error. The class of errors in the first three is supposedly completely eliminated by Rust. No idea about the fourth. Not claiming that a Rust reimplementation of ffmpeg is anywhere near feasible. Dunno if the current set of CVE’s are comparable but it’s a likely guess. Anyway, as SPJ likes to say about Haskell’s type system, the idea is to stop fixing bugs one by one, and instead eliminate entire classes of bugs. We can’t fix everything but we can certainly do better than we are doing now.
That was listed as an example of what not to do, not a proposal of an approach to take.
Google’s big sleep was pretty good, it gave a python program that generated an invalid file. It looked plausible, and it was a real issue. The problem is that literally every other generative AI bug report also looks equally as plausible. As I mentioned before, curl is having a similar issue.
And here’s what the lead maintainer of curl has to say:
So you can claim testing may be simple, but it looks like that isn’t the case. I would say one of the problems is that all these people are volunteers, so they probably have a very, very limited set of time to spend on these projects.
FFMpeg is not just C code, but also large portions of handwritten, ultra optimized assembly code (per architecture, too…). You are free to rewrite it in rust if you so desire, but I stated it above and will state it again: ffmpeg made the tradeoff of performance for security. Rust currently isn’t as performant as optimized C code, and I highly doubt that even unsafe rust can beat hand optimized assembly — C can’t, anyways.
(Google and many big tech companies like ultra performant projects because performance equals power savings equals costs savings at scale. But this means weaker security when it comes to projects like ffmpeg…)
Have any of the google-submitted vulnerability reports turned out to be invalid? Project Zero was pretty well regarded.
Yes I know about the asm code in ffmpeg though IDK if it’s doing anything that could lead to a use after free error. I can understand if an OOB reference happens in the asm code since codecs are full of lookup tables and have to jump around inside video frames for motion estimation, but I’d hope no dynamic memory allocation is happening there. Instead it would be more like a GPU kernel. But, I haven’t examined any of it.
Anyway there’s a big difference between submitting concrete input data that causes an observable crash, and sending a pile of useless spew from a static analyzer and saying “here, have fun”. The Curl guy was getting a lot of absolute crap submitted as reports.
From the GCC manual “bug criteria” section:
That sounds like timelessly good advice to me.