an audit of community LLM fine-tunes
Did it even
learn?
150 base models, paired against a community fine-tune of each, run on both the public benchmark and a freshly generated equivalent the model has never seen. The gap between the two is the memorization signal: what separates a fine-tune that actually improved from one that just got better at the test.
coverage --by-family
findings --headline
Final. The run is complete; every family below reflects the full dataset. A struck-through contamination-gap value failed its pre-registered validity check and isn't a reliable result yet, see ANALYSIS_PLAN.md.
Full results, sortable by task family: …
open the leaderboard →