---
title: "Decomposing the Epoch Capabilities Index"
subtitle: "ECI visualized as a weighted average of benchmarks"
date: 2026-11-17
draft: false
execute:
echo: false
warning: false
resources:
- eci_model_panels.html
date-modified: 2026-11-18
---
```{python}
#| include: false
import json
import pandas as pd
# Everything this post reads sits in its folder, written by ../eci_assets.py (run it, or
# ../../scripts/update_eci_posts.py, before rendering). Output is frozen after rendering (see _metadata.yml).
w = pd.read_csv("weights.csv")
eci = pd.read_csv("eci_scores.csv")
adj = pd.read_csv("rescaled_benchmarks.csv").set_index("benchmark")
pages = pd.read_csv("benchmark_pages.csv").set_index("benchmark")
dom = pd.read_csv("benchmark_domains.csv").set_index("benchmark").domains.fillna("")
st = pd.read_csv("post_stats.csv").iloc[0]
fitb = pd.read_csv("benchmark_fit.csv").set_index("benchmark") # sigma_b from our refit of Epoch's model
info = json.load(open("data_info.json"))
data_date = pd.Timestamp(info["data_date"]).strftime("%B %-d, %Y")
top = eci.sort_values("eci", ascending=False).iloc[0]
n_models, n_bench = w.model.nunique(), w.benchmark.nunique()
nm = lambda b: pages.display.get(b, b)
url = lambda b: f"https://epoch.ai/benchmarks/{pages.slug[b]}" if b in pages.index else "https://epoch.ai/benchmarks/search"
def neff(model):
g = w[w.model == model]
return 1 / (g.share ** 2).sum()
def p(model, bench): # Epoch's (rescaled) score
return w[(w.model == model) & (w.benchmark == bench)].performance.iloc[0]
def sc(model, bench):
return f"{p(model, bench):.0%}"
def E(model):
return w[w.model == model].ECI.iloc[0]
def nb(model):
return int((w.model == model).sum())
def bset(model):
return set(w[w.model == model].benchmark)
def shared(a, b):
return len(bset(a) & bset(b))
def contrib(a, b): # each benchmark's contribution to E(a) - E(b), as in the chart's right-hand column
A, B = (w[w.model == m].set_index("benchmark") for m in (a, b))
mid, idx = (E(a) + E(b)) / 2, A.index.union(B.index)
term = lambda X: (X.share * (X.implied_ECI - mid)).reindex(idx).fillna(0)
return term(A) - term(B)
def top_k(model, k=2):
return list(w[w.model == model].sort_values("share", ascending=False).head(k).benchmark)
floors = adj[adj.random_baseline >= 0.01].random_baseline.sort_index()
ceils = adj[adj.score_ceiling < 1].score_ceiling.sort_index()
ceil_list = ", ".join(f"{nm(b)} {c:.0%}" for b, c in sorted(ceils.items(), key=lambda kv: nm(kv[0])))
gq_rb = adj.random_baseline.get("GPQA diamond", 0.0)
gq_c = adj.score_ceiling.get("GPQA diamond", 1.0)
gq_94 = (0.94 - gq_rb) / (gq_c - gq_rb)
opus = "Claude Opus 5.5"
opus_top4 = w[w.model == opus].sort_values("share", ascending=False).head(4)
HK, G55, FL, KK = "Claude Haiku 5.5", "GPT-5.5", "Gemini 3.8 Flash", "Kimi K3"
FM4, PB, SQA, A2 = "FrontierMath-Tier-4-v2-Private", "ProofBench", "SimpleQA Verified", "ARC-AGI-2"
fk = contrib(FL, KK)
fk_two, fk_rest = fk[[PB, FM4]].sum(), fk.drop([PB, FM4]).sum()
fk_best = fk.drop([PB, FM4]).idxmax()
extra = bset(G55) - bset(HK)
# How far the weighted average of readings sits from the published ECI, models at 150 and above
gm = w.groupby("model").apply(lambda g: pd.Series({"ECI": g.ECI.iloc[0], "avg": (g.share * g.implied_ECI).sum()}), include_groups=False)
gm = gm[gm.ECI >= 150]
gap150 = (gm.avg - gm.ECI).abs()
sig_lo, sig_hi = fitb.sigma.idxmin(), fitb.sigma.idxmax()
fk_pull = list(fk.nsmallest(2).index) # the two benchmarks pulling hardest toward KK
# The prose below makes claims about these numbers. If new data breaks one, stop here so a person
# rewrites the sentence instead of publishing a false one.
checks = {
"GPQA Diamond has a guessing rate (the footnote's example)": gq_rb >= 0.01,
f"{HK} and {G55} within 1 ECI point": abs(E(HK) - E(G55)) < 1,
f"{G55}'s extra benchmarks mostly agentic or coding": sum(("Agent" in dom.get(b, "")) or
("Software engineering" in dom.get(b, "")) for b in extra) > len(extra) / 2,
f"{HK} far ahead on ProofBench": p(HK, PB) - p(G55, PB) > 0.2,
f"{HK} well behind on FrontierMath Tier 4 and SimpleQA Verified": min(p(G55, FM4) - p(HK, FM4), p(G55, SQA) - p(HK, SQA)) > 0.15,
f"{FL} and {KK} within 1 ECI point": abs(E(FL) - E(KK)) < 1,
f"{FL} leads on ARC-AGI-2 and SimpleQA, {KK} on ProofBench and FrontierMath Tier 4":
p(FL, A2) > p(KK, A2) and p(FL, SQA) > p(KK, SQA) and p(KK, PB) > p(FL, PB) and p(KK, FM4) > p(FL, FM4),
f"ProofBench and FrontierMath Tier 4 pull hardest toward {KK}": set(fk_pull) == {PB, FM4},
f"FrontierMath Tier 4 among both models' two heaviest": FM4 in top_k(FL) and FM4 in top_k(KK),
f"a few benchmarks carry much of {opus}'s weight": opus_top4.share.sum() >= 0.4,
"weighted averages within half a point of published ECI at 150+": gap150.max() < 0.5,
f"those two favor {KK}, the rest {FL}": fk_two < 0 < fk_rest,
}
failed = [k for k, ok in checks.items() if not ok]
assert not failed, "Prose needs revisiting: " + "; ".join(failed)
```
The [Epoch Capabilities Index](https://epoch.ai/benchmarks/eci) (ECI) puts `{python} n_models` AI models on a single scale. What differences in that number ultimately mean boil down to scores on `{python} n_bench` different benchmarks.
@fig-panels below shows, for any pair of frontier or near-frontier models, how benchmark scores determine their ECIs.
## The math underlying the ECI {#sec-model}
Very briefly, an AI model's score on a given benchmark is [modeled](https://epoch.ai/data/eci-documentation/methodology) as a function of its capability $C_m$ (also referred as its ECI):
$$
y_{mb} = S\big(\alpha_b \, [C_m - D_b]\big)
$$ {#eq-eci}
where $D_b$ is the benchmark difficulty, $\alpha_b$ the benchmark slope, and $S(x) = 1 / (1 + e^{-x})$ is the S-shaped logistic curve rising from 0 to 1. The scatter of the "noise" in each benchmark's scores is denoted $\sigma_b$.
Epoch fits all the capabilities, difficulties, slopes, and $\sigma_b$'s at once as a [Bayesian model](https://epoch.ai/publications/eci-bayesian-methodology#what-changes-in-the-bayesian-model).
## Interpreting a model's ECI via its scores {#sec-panels}
Because all parameters are fit jointly, each benchmark score influences the difficulty, slope, and scatter parameters at the same time as it influences a model's ECI, $C_m$. As an approximation, though, we can use the estimated values of each benchmark's difficulty, slope, and $\sigma_b$ to assess the influence of a given benchmark score on $C_m$. Each individual benchmark score can then be thought of as a separate measurement of capability: a score $y_{mb}$ implies $\hat{C}_{mb}$, the capability at which that score is most likely.^[For mid-range scores this is close to inverting @eq-eci, $D_b + S^{-1}(y_{mb}) / \alpha_b$ with $S^{-1}(s) = \ln[s / (1 - s)]$. Near 0% or 100% it can differ by a few ECI points, because the assumed noise there is lopsided.] A model's ECI can be thought of as a weighted average of these measurements, $C_m \approx \sum_b w_{mb} \hat{C}_{mb} / \sum_b w_{mb}$, with weights that are, approximately,[^exact]
$$
w_{mb} \propto \frac{\alpha_b^2 \, \hat{p}_{mb} (1 - \hat{p}_{mb})}{\sigma_b^2}, \qquad \hat{p}_{mb} = S\big(\alpha_b \, [C_m - D_b]\big)
$$ {#eq-weight}
A benchmark score counts most when the corresponding part of that benchmark's curve is steep and there is little noise. A score near the top or bottom ends of a benchmark, or one anywhere on a benchmark whose estimated curve is flat, counts for little. Otherwise, the scatter $\sigma_b$ of benchmark noise runs from `{python} f"{fitb.sigma.min():.3f}"` (`{python} nm(sig_lo)`) to `{python} f"{fitb.sigma.max():.3f}"` (`{python} nm(sig_hi)`), so, other things equal, a score on the former counts about `{python} f"{(fitb.sigma.max() / fitb.sigma.min())**2:.0f}"` times as much as one on the latter.
[^exact]: @eq-weight approximates the information a score carries about $C_m$ under Epoch's likelihood, $[\alpha_b \hat{p}_{mb} (1 - \hat{p}_{mb})]^2 \kappa_b^2 [\psi'(\hat{p}_{mb} \kappa_b) + \psi'((1 - \hat{p}_{mb}) \kappa_b)]$ with $\kappa_b = 1 / (4 \sigma_b^2) - 1$ and $\psi'$ the [trigamma function](https://en.wikipedia.org/wiki/Trigamma_function). The weight shares shown in @fig-panels use these more exact weights, which make the weighted average the capability that best fits the model's scores given every benchmark's fitted curve and scatter. Not included in the calculations depicted in @fig-panels is the way Epoch's ECI also averages over uncertainty in the benchmark curves, so the two differ slightly: by `{python} f"{gap150.median():.2f}"` points for the median model at ECI 150 and above, and at most `{python} f"{gap150.max():.2f}"`. (Epoch does not publish $\sigma_b$ updates, so the values here come from a refit of Epoch's [published model code](https://github.com/epoch-research/eci-public), which reproduces its ECI to within sampling error.)
(@fig-panels can optionally display analogous measures for an alternative ECI fit, XCI v0.1 [<i class="bi bi-github"></i>](https://github.com/hugetim/xci){title="XCI v0.1 code" aria-label="XCI v0.1 code on GitHub"}. It is the [XB+ variant](https://hugetim.github.io/posts/eci-clip/#xb-median-centered-beta-with-capability-scale-noise-also) with the addition of benchmark-specific capability noise, $\tau_b$.)
::: {#fig-panels}
```{=html}
<iframe class="eci-frame" src="eci_model_panels.html" style="width:100%;height:900px;border:0" scrolling="no" allow="clipboard-write"></iframe>
```
ECIs as weighted averages of benchmarks. Dots are the ECIs individual benchmark scores imply, the bars are the weights, and a dashed line shows each model's ECI. Hover over or tap a dot for the score behind it.
:::
The effective number of benchmarks behind a score can be summarized as $1 / \sum_b (\text{share}_{mb})^2$: about `{python} f"{neff('Claude 3.5 Sonnet'):.0f}"` for Claude 3.5 Sonnet and `{python} f"{neff(opus):.0f}"` for `{python} opus`, whose top four benchmarks (`{python} ", ".join(nm(b) for b in opus_top4.benchmark)`) carry `{python} f"{opus_top4.share.sum():.0%}"` of the weight.
*Data: [Epoch AI Benchmarking Hub](https://epoch.ai/benchmarks) ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)), as of `{python} data_date`.*
```{=html}
<script>
// size each embedded chart to the height it reports
window.addEventListener('message', e => {
if (!e.data || !e.data.eciFrameHeight) return;
document.querySelectorAll('iframe.eci-frame').forEach(f => {
if (f.contentWindow === e.source) f.style.height = (e.data.eciFrameHeight + 4) + 'px';
});
});
</script>
```