Skip to main content

Handling 100% scores in the Bayesian ECI

A simple fix for the ECI’s new sensitivity to Epoch’s 99.9% clip

Epoch’s switch to a Bayesian ECI makes some results of interest depend on how benchmark scores of 0 or 1 are clipped (to 0.001 or 0.999), but extending the beta distribution to handle scores of exactly 0 or 1 provides similar results without the clip’s arbitrariness.
Published

November 12, 2026

Updated

November 16, 2026

The Epoch Capabilities Index (ECI) has become the canonical measure of progress in AI capabilities.1 It stitches together benchmarks of different difficulties, allowing the seamless addition of the new, harder benchmarks required by rapidly advancing capabilities. Its transparent, elegant math has helped make it a trusted measure. When Epoch recently announced that Claude Fable 5.2’s ECI was “above what the pre-Astra trend for reasoning models would have predicted,” the ensuing conversation (rightly) took for granted that the underlying methodology is sound.

Scatter plot of models' ECI against release date, with a trend line through the frontier reasoning models and a shaded band around it. Claude Fable 5.2, at 174, is above the line, at the top edge of the band.
Figure 1: Epoch AI’s chart (source, CC BY). Fable 5.2 sits at the upper edge of the trend’s shaded 90% prediction interval.

There is, however, one ad hoc aspect of Epoch’s methodology that has seemed to me to detract from its elegance (though I could not come up with a better alternative myself). The underlying math can’t handle benchmark scores of exactly 100% (or 0%), so Epoch “clips” such scores to 99.9% (or 0.1%). Though the arbitrariness of this choice bothered me, I came to understand that the choice of which particular number close to 100% they used had no meaningful effect on the resulting ECI estimates—under the original least-squares methodology, that is.

Epoch’s November 4 shift to a Bayesian methodology,2 however, deliberately made small score differences near 100% (or 0%) matter more than they had under the original methodology.3 The way the original ECI seemed to underweight scores near 100% seemed a genuine shortcoming (which I had also tried but struggled to address myself), but fixing it has introduced an unintended side effect. The inevitably arbitrary choice of which number near 100% to clip perfect scores to now meaningfully impacts certain results of interest.

How 100% scores are clipped moves Fable 5.2’s ECI

In particular, the choice of clip impacts how far above the prior trend Fable 5.2 lands. Its 4.0 point jump with Epoch’s 0.1% clip would instead be 3.4 points with a 1% clip, or up to 6.8 points with a 10−12 clip, 2 times the 1% clip’s result.4 The jump would be even higher for clips still closer to the actual scores of exactly 100% (or 0%).5

Whether the jump lands outside the trend’s 90% prediction interval also depends on the clip. With Epoch’s 0.1% clip, Fable 5.2 sits just inside the interval’s upper edge, as in Epoch’s chart. With a 0.01% clip or tighter, it lands above the interval. That said, the size of the jump is fairly stable for clips near Epoch’s choice of 0.1%.

Figure 2: Claude Fable 5.2’s ECI from a full refit using the given clip, minus that fit’s own trend prediction at Fable 5.2’s release date. Each line is one fit’s reasoning-era frontier trend (best reasoning models on release, o1-mini through Claude Fable 5.1, as in Epoch’s chart), with its 90% prediction interval shaded. The extended-support beta (XB, described below) gap rounds to the 1% clip’s.

The reason Fable 5.2’s score shifts relative to others is that it boasts a perfect score on both FrontierMath Tier 4 and FrontierMath Tiers 1–3,6 out of only 10 benchmark scores currently available for Fable 5.2. Smaller clips therefore pull Fable 5.2’s ECI higher relative to models without these perfect scores, as explained further in the following sections.

Using the original least-squares methodology (my refit of Epoch’s earlier code on the same data), however, the impact of different clip values is barely perceptible: Fable 5.2 lands 5.72 ECI points above the trend at a 1% clip, 5.63 at 0.1%, and 5.62 at every clip from 0.01% to 10−12.

The clip choice also shifts Sol v. Astra

GPT-6.1 Sol similarly benefits relative to other models from smaller clips due to its rare perfect score on FrontierMath Tier 4. In the official ECI, GPT-6.1 Sol narrowly trails GPT-6 Astra by 0.29 ECI points, with probability 39% that Sol is actually ahead.7 Moving the clip to 10⁻⁵ is enough to give GPT-6.1 Sol the edge, and clips still closer to the actual scores of 0% and 100% widen its lead.8

Clip 6.1 Sol − Astra P(6.1 Sol ahead of Astra)
1% −0.61 0.27
0.1% −0.29 0.39
0.01% −0.06 0.48
10−5 +0.13 0.54
10−6 +0.27 0.60
10−8 +0.53 0.69
10−10 +0.69 0.74
10−12 +0.85 0.79

The probability that 6.1 Sol is ahead of Astra rises from 0.27 at a 1% clip to 0.79 at 10−12. A tighter clip treats Sol’s perfect score on FrontierMath Tier 4 as ever stronger evidence.

Figure 3: The probability that GPT-6.1 Sol is ahead of GPT-6 Astra in full refits of Epoch’s model, by clip. The labels give Sol’s lead in ECI points. The diamond shows the extended-support beta result (XB, described below), placed at the clip it resembles at the boundary, δ / (1 + 2δ).

Tighter clips result in a higher ECI for GPT-6.1 Sol than GPT-6 Astra’s even though Astra scores at least as high as 6.1 Sol on every shared benchmark except FrontierMath Tier 4. Figure 4 below illustrates all benchmark scores that go into determining each model’s ECI.9

Figure 4: GPT-6 Astra’s and GPT-6.1 Sol’s scores on each benchmark (as Epoch rescales them; the higher in bold), each placed at the ECI that score implies under Epoch’s current method (the capability at which it would be most likely). Each line runs from 0.1% to 99.9% (so Epoch’s clip assigns a 100% score to its right end), with ticks at 10%, 20%, …, 90%. Dashed lines show the two models’ ECIs.

How the clip impacts ECI estimates

What the clip choice directly changes is the placement of a perfect score on the ECI scale. Figure 5 below shows the row for FrontierMath Tier 4 from the benchmark comparison of Astra and Sol above, redrawn for four different clips:10 Astra’s 40/41 implies an ECI around 171 regardless. Sol’s perfect 41/41 has no place on the scale unless it is clipped, so where it lands depends on the clip: 176 with Epoch’s 99.9%, 178 with 99.99%, up to 183 with twelve nines.

Nothing about the models differs between the rows. Sol solved the same 41 problems in each case. What differs is how much better than 40/41 the ECI methodology is told a perfect score is. Under the original least-squares method the answer hardly mattered. Under the new Bayesian method, there is no natural limit to how much it matters: each tightening of the clip pushes the end of the scale further out.

Figure 5: What a perfect score is worth depends on where it is clipped. Each line is FrontierMath Tier 4’s scale from Figure 4, marked at the ECI each score implies under Epoch’s model. Astra’s dot does not depend on the clip; GPT-6.1 Sol’s 100% sits wherever the clip puts the end of the scale. Hover for details.

A more indirect way the clip affects results is that it raises the estimated scatter σb of benchmarks with 0% (or 100%) scores, reducing their influence overall.11 A benchmark’s influence in determining ECIs falls roughly as 1/σb2. In particular, for Mystery Game Puzzles, with 14 scores of 0%, σb is 0.150 with Epoch’s 0.1% clip, 0.171 with 0.01%, and 0.243 at 10−12.12 So a 0.01% clip would give it about 22% less weight than Epoch’s 0.1% does, whether intended or not.

Still, it is generally true that most ECI results are not sensitive to the clip choice. For instance, models at ECI 150+ move by only 0.06 on average between the 0.1% and 0.01% clips (at most 0.68).

A simple fix: the extended-support beta (XB)

Fortunately, a small change to Epoch’s new Bayesian methodology would allow for direct handling of 0% and 100% scores, eliminating the need for a clip. The idea is to extend the beta distribution (used to model the score on a given benchmark, given an ECI and other relevant parameters) to a slightly wider interval, [−δ, 1 + δ], and use the probability mass outside the [0, 1] interval to represent the probability of exact 0% or 100% scores.13 Kosmidis and Zeileis (2026)14 call this the extended-support beta (XB),15 available in R’s betareg since version 3.2-0.

The δ is estimated, not chosen like Epoch’s current clip. For the current ECI data, it is estimated as 0.37% (90% interval 0.25%–0.53%).16

The results are broadly similar to the current official ECI, but models at ECI 150+ do shift by 0.30 points on average, somewhat more than under any of the alternative clips considered above (at most 0.24). The largest change is for Fable 5.2 itself, whose ECI falls from 174.2 to 172.9. But the trend line shifts down as well, so its gap above the trend falls by less (from 4.0 to 3.4). XB’s Astra v. 6.1 Sol result is in between that for the official ECI and a 1% clip, as shown in Figure 3 above.

XB measures of fit

In terms of fit, the extended beta has overall log predictive density (PSIS-LOO) higher by 34.7 (SE 12.1).17 Broken down, it’s +73.6 on 0% and 100% scores, but −38.9 on the rest. So the gain comes entirely from the 0% and 100% scores, whereas the scores in between are predicted somewhat worse.

The other measures below follow Epoch’s report (main text and Appendix B).18

Measure (PSIS-LOO) Epoch (0.1% clip) XB
RMSE 0.0925 0.0928
MAE 0.0681 0.0679
median abs error 0.0503 0.0501
beta-scaled RMSE 0.1094 0.1136
CRPS 0.0479 0.0480
mean bias −0.0013 −0.0011
coverage 90% 89.6% 89.5%
coverage 90%, scores <= 0.05 81.0% 85.5%
coverage 90%, scores >= 0.95 92.0% 92.8%
coverage 50% 50.9% 51.7%
coverage 99% 98.2% 98.6%
PIT KS 0.025 0.025
predicted share < 0.1% 1.4% 2.5%

The extended-beta is similar on accuracy and 90% coverage, closer on 99% coverage, and better on coverage near 0% (85.5% against 81.0%), a weak spot Epoch’s report flags. Near 100%, both over-cover, though XB somewhat more, and its 50% coverage is a bit further off (51.7% against 50.9%). The beta-scaled RMSE is a weak point for XB, as is the posterior predictive check on the variance.

The relative weaknesses in XB’s predictive fit are somewhat improved by a further extension (with one parameter more), tying δ to each benchmark’s scatter, δb = δ (σb / median σ)γ.19 More broadly, the point here is not that XB is the ideal model but rather that at least one reasonable alternative to the ad hoc clip is available.

Conclusions

In the end, Epoch’s current ECI with a 0.1% clip yields results similar to my proposed extended beta, both in terms of Fable 5.2’s jump above trend and the relative ratings of GPT-6 Astra and GPT-6.1 Sol. These similar results provide reassurance about the robustness of the current ECI’s results. Alternatively, Epoch could adopt the more conceptually coherent XB approach (or a better alternative addressing XB’s relative weaknesses in terms of predictive fit) and leave behind the need to choose a clip.

Data: Epoch AI Benchmarking Hub (CC BY 4.0), as of November 10, 2026.

AI use: Opus 5.5 developed and ran all code. Fable 5.2 checked it, and I provide it for inspection. Opus 5.5 also, in conversation, suggested the extended beta alternative. The writing is mine.

Back to top

Footnotes

  1. For example, Ethan Mollick’s “30 Months on the Exponential (in Charts)” (October 20) uses the ECI for its two initial charts showing the pace of frontier model progress. Likewise, Anthropic’s internal version of the ECI is central to METR’s recent report on AI R&D acceleration there (November 5).↩︎

  2. The primary motivation for changing methodologies seems to be the benefits a Bayesian model provides in terms of better grounded error bounds and improved model comparisons, which I agree are alone well worth the change.↩︎

  3. “The previous frequentist methodology, which minimized the squared prediction error, is comparable to a Bayesian model with a normal distribution that has the same noise for every predicted score. When using least-squares minimization, predicting 50% against an observed 55% is penalized just as much as predicting 90% against 95%. However, it seems reasonable to expect the noise to shrink as the predicted score gets close to 0 or 100%.”↩︎

  4. Under the extended-support beta (XB) proposed below, Fable is 3.4 ECI points above trend, also inside the trend’s 90% prediction interval.↩︎

  5. One might think the clip should be as small as possible, since 99.9999% is closer to a true 100% than 99.9% is. But the results do not settle down as the clip shrinks: at 10−12, Fable 5.2’s gap is still growing by about 0.3 points with each tenfold tightening. Alternatively, one could argue for a larger clip proportionate to benchmarks’ resolutions based on their question counts. (FrontierMath Tier 4 has 41 questions.)↩︎

  6. Epoch now considers its score of 95% on the latter the maximum possible.↩︎

  7. These numbers are from my rerun of Epoch’s code (needed to vary the clip). Epoch publishes Astra 169.16 and Sol 168.87.↩︎

  8. Using the original least-squares methodology (my refit of Epoch’s earlier code on the current data), I confirmed that the clip barely matters: 6.1 Sol trails Astra by 0.90 ECI points at a 1% clip, 0.88 at 0.1%, and 0.87 at every clip from 0.01% to 10−12.↩︎

  9. In a future post, I plan to provide a tool for comparing any two models benchmark by benchmark, also showing how much each score affects the models’ ECIs.↩︎

  10. The benchmark’s difficulty and slope are held fixed for illustrative purposes here, though in a full refit they adjust in response to the clip changes.↩︎

  11. A note in Epoch’s code acknowledges this channel, explaining that the clip “is not a free numerical tolerance” because shrinking it “hands the likelihood an unbounded reward” for pushing 0% rows down, “and the model pays for it by lowering the capability of the models involved and raising the sigma of the benchmarks carrying the zeros.” (eci-public, bayesian.py)↩︎

  12. Its σb under XB, described below, is 0.147.↩︎

  13. This approach is conceptually similar to the Censored Normal alternative likelihood Epoch tried, as reported in their post’s Appendix A.↩︎

  14. Kosmidis and Zeileis (2026) Journal of the Royal Statistical Society Series C 75: 139–157 (open access).↩︎

  15. The model their paper actually recommends, XBX, is a continuous mixture of XB distributions in which the exceedance, δ, follows an exponential distribution with unknown mean ν. They say this shrinks XB towards the beta to avoid certain identifiability problems that do not arise here because δ is small and tightly estimated (and its estimate barely moves under tighter or flatter priors). As a check, I also fit XBX (estimating a mean exceedance ν of 0.50%). It gives broadly similar answers (Fable 5.2’s gap above the trend 3.1 against XB’s 3.4; 6.1 Sol − Astra −0.56 against −0.41) and fits slightly better (log predictive density +5.6, SE 2.5)—but not enough to justify the additional conceptual complexity.↩︎

  16. One implication is that the latent mean now reaches 100% at a finite capability: 48 ECI points above the median benchmark’s 50% point.↩︎

  17. The two models’ likelihoods cannot be compared directly because they treat 0% and 100% scores differently: Epoch’s model assigns the clipped score a probability density, while XB assigns the exact 0% or 100% a probability. So I score both models on the same set of outcomes. A score of 0.1% or below counts as the event “bottom” and one of 99.9% or above as the event “top,” and each model is credited with the probability it gives that event (for Epoch’s model, its fitted beta’s probability below 0.1% or above 99.9%). Every other score is credited with its density. (No observed benchmark score lies strictly between 0% and 0.1% or between 99.9% and 100%.) The measures in the table below apply the same convention to Epoch’s model: a simulated score beyond its clip counts as an exact 0% or 100%.↩︎

  18. Note that the observed share of scores below 0.1%, the target of the final row, is 2.1%.↩︎

  19. γ is estimated to be 4.2 (90% interval 2.6–5.8), so δb runs from 0.01% (METR Time Horizons) to 2.0% (Furniture Assembly): noisier benchmarks hit 0% and 100% more often than one shared δ allows. It then predicts better (log predictive density +14.4, SE 4.5, over XB: +9.2 on 0% and 100% scores, +5.2 on the rest). It has beta-scaled RMSE 0.1113 against 0.1136 (Epoch 0.1094); predictive p-value for the variance 0.77 against 0.83 (Epoch 0.57).↩︎