Scoring Methodology v0.2.3 | SRE-Bench

Rankings→Methodology→v0.2.3

SRE Agent Evaluation Methodology v0.2.3

Effective April 21, 2026

Replaced rate-limited OpenRouter free models with native Gemini API models (gemini-2.5-flash, gemini-1.5-flash). Same dimensions and weights as v0.2.2.

Deterministic Metrics

Root Cause Identified10%

Affected Services5%

Signal Coverage5%

AI-Judged Metrics

Final Answer25%

Reasoning Quality20%

Tool Use Quality15%

Trace Groundedness10%

Total weight: 100% across 8 dimensions. Results scored under older methodology versions retain their original scores.