the leaderboard

excluded --failure-rate

Any model with a failure rate above 20% is excluded from the primary analysis above and listed here instead, per the pre-registered plan. A failure means the model never returned a usable response after 6 attempts, not that it answered incorrectly.