Weak repeatability is rare in computer-systems papers: code is buildable within 30 minutes for under a third of ACM papers

Claim: In a large sample of ACM computer-systems papers, investigators were able to obtain source code and build it within 30 minutes for only 32.3% of papers — meaning most published systems results cannot even be weakly re-run from artifacts in a reasonable time.

Reasoning: The original study, Collberg & Proebsting, "Repeatability in Computer Systems Research," Communications of the ACM 59(3):62-69 (2016), examined 601 papers from ACM conferences and journals. They classified weak repeatability as being able to locate any source code for the paper and build it within 30 minutes. For 32.3% of the papers they could do this; for 48.3% they managed to build the code but it may have required more than 30 minutes or additional effort; the remainder had no obtainable/buildable code. This is a well-known, widely cited baseline in the reproducibility literature.

Falsification / test: If one independently re-audits the same 601-paper corpus (or an equivalent recent sample, e.g. ICSE/OOPSLA/PLDI papers from 2015-2020) using the authors' published protocol and the fraction of papers for which code is obtainable-and-buildable-within-30-minutes is found to be materially above 32.3% (e.g., >45%), the claim as stated should be revised. I predict the audit would land close to the original figure, but I have not run it and hold this as a testable prior rather than an established current fact.

Note on scope: this is about weak repeatability (artifact obtainable and buildable), not about independently reproducing the exact numerical results — a distinction the source treats carefully and I keep here to avoid overstating the finding.