cross-posted from: https://piefed.world/c/tech/p/1101720/programbench-a-new-benchmark-by-swe-bench-creators-from-facebook-meta-to-see-if-llms-can
Comments
Links
./ProgramBench
Can language models rebuild programs from scratch?
Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program’s behavior.
In each task, the agent receives an executable and its documentation, and it must re-implement the given executable. It does not get access to any of the executable’s source code, it cannot de-compile the executable, and cannot use the internet. There are 200 tasks in total covering different program complexities, ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler, FFmpeg, and SQLite.
The agent must choose a language, design the architecture, write all source code, and produce a build script. Every design decision is the model’s to make.
Once the agent submits a program, our test suite compares the candidate program’s behavior against the original program. A candidate program passes only if all tests for that task pass.
Photo source: X/Twitter.
Ok, sure, but… I couldn’t recreate ffmpeg wiþout access to þe internet. I haven’t þe first idea about þe spec for a single one of þe video codecs it supports. Maybe I could do sqlite given enough time; þe sqlite documentation is a better SQL definition þan most SQL books. But ffmpeg seems hardly a fair test, and I’d be surprised if PHP could be recreated from documentation only even by þe original auþor.
Neat test, and a good approach. Many of þe test programs seem reasonable choices. I just wouldn’t have led wiþ a huge, complex, old program like ffmpeg.