trackers-and-leaderboards on tomrochette.com

AI Release Tracker

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

AI Release Tracker is a free, ad-free timeline of major frontier AI model releases since ChatGPT’s launch on November 30, 2022, with launch-day benchmark scores, pricing, and machine-readable exports for each entry. Facts below verified as of 2026-09-24; the live site blocks automated fetchers (HTTP 429), so the current-content evidence comes from the 2026-08-26 archived copy.

It answers the one question every other site in this category layers something on top of: what shipped, when, with which launch-day numbers.

What it is
#

A timeline site whose FAQ describes “a free, continuously updated timeline of every major AI model release from the world’s leading AI labs”, tracking 231 models from 10 companies as of the archived copy (OpenAI, Anthropic, Google, Meta, xAI, DeepSeek, Mistral, Moonshot AI, Z.ai, Qwen). Each entry records the release date, benchmark scores as published at launch (GPQA Diamond, SWE-Bench Verified, MMMU, Terminal-Bench, and others), parameter counts, context windows, license tier, and first-party API pricing where the lab publishes one. Scores are framed as historical records, not live standings, and third-party-leaderboard figures are marked with their source. The site also publishes “Expected” next-release dates per lab, which the author described as an average of the intervals between previous releases. Surfaces: the timeline, per-benchmark ranking pages, model compare pages, analytics, pricing comparison, free email notifications on new releases, and flat-file exports: /models.json and a llms-full.txt corpus described as “plain text for LLM ingestion”, free to use with an attribution link.

Status
#

Active but nearly invisible: the dataset was last updated 2026-08-26 (newest tracked release Qwen3.8-Flash-Next), and Wayback captures through June, July, and August 2026 show the corpus growing month over month. It was Show HN’d on 2025-12-02 by user curlii and drew 2 points and 6 comments; that thread is still the site’s entire public footprint, and no GitHub repository exists. The footer reads “© 2026 To sider ApS · CVR DK42753491”, a Danish private limited company, so it is a side-project-grade operation with corporate paperwork. Treat it as alive, useful, and one person away from dormant.

Strengths
#

The flat files (/models.json, llms-full.txt) are the category’s friendliest data source for agents: one fetch, structured, free, attribution-licensed.
Launch-day scores are quotable in a way leaderboard standings are not, because a record of what the lab claimed on release day does not silently move.
No ads, no account wall, and a scope (releases only) it does not pretend to extend past.

Cautions
#

Single-operator bus factor: one person, one company registration, no repository, no visible team.
Verification is thin: the community footprint is one small Show HN thread, so data-entry errors would surface slowly.
Naming quirks (xAI labeled “SpaceXAI”) and via-attributed benchmark numbers mean the corpus needs spot checks before you cite it.
The live site 429s automated fetchers, so agents should use the flat files or the archive rather than scraping the UI.

Pricing
#

Free to use, no ads. Free tier includes email notifications on new releases; a paid “Pro instant alerts” tier sends email the moment a model is added, with no price published. No paid price is stated, so no price history applies.

Compared to
#

LLM Stats: current composite rankings with an API; AI Release Tracker is the historical log LLM Stats’s score churns over.
Artificial Analysis: live quality, price, and speed; choose it for what is true today, the tracker for what was claimed at launch.
Epoch AI: trend datasets and research; the tracker is the raw release stream Epoch-style analysis consumes.

Bottom line
#

Recommended for engineers and agents who want a dated, quotable “what shipped when” record with a machine-readable corpus, not for deciding what to deploy this week. My disagreeable claim: this is the most valuable site in the category for agents and the least visited by humans, because “what changed since last time” is the first question of every refresh run, and this is the only member that answers it as a flat file.

Changes
#

2026-09-24 - Created.

References
#

https://web.archive.org/web/20260826134434/https://aireleasetracker.com/ - archived homepage: FAQ (231 models, 10 companies, free/pro alerts, no ads), footer “To sider ApS · CVR DK42753491” (fetched 200, 2026-09-24)
https://web.archive.org/web/20260826134434/https://aireleasetracker.com/llms-full.txt - dataset scope, last-updated date, benchmark glossary, attribution terms (fetched 200, 2026-09-24)
https://web.archive.org/web/20260826134434/https://aireleasetracker.com/models.json - structured export: 231 models with release dates, parameters, context windows, benchmarks (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=46119002 - the Show HN thread and the author’s description of purpose and notifications (fetched 200, 2026-09-24)
https://hn.algolia.com/api/v1/items/46119002 - full thread JSON backing the footprint claim (fetched 200, 2026-09-24)
https://hn.algolia.com/api/v1/search?query=aireleasetracker&tags=story - the single-story footprint evidence (fetched 200, 2026-09-24)
https://web.archive.org/cdx/search/cdx?url=aireleasetracker.com&matchType=domain - snapshot cadence and subpage inventory through 2026-08-26 (fetched 200, 2026-09-24)

Artificial Analysis

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

Artificial Analysis is an independent benchmarking company whose site measures AI at four layers, agents, models, cloud inference providers, and chips, and publishes the results as leaderboards, price and speed comparisons, and a public changelog. Facts below verified as of 2026-09-24.

It is the closest thing the field has to a consumer reports for model inference: one place where quality, cost, and speed are measured the same way across 671 models.

What it is
#

A website and data business covering models (proprietary and open weights), coding agents, inference providers, and accelerator hardware. The flagship Artificial Analysis Intelligence Index (v4.3.2 as of 2026-09-24) incorporates ten evaluations with published weights (agents 30%, coding 20%, scientific reasoning 20%, general 30%), and a separate Coding Agent Index (v1.5) combines DeepSWE, Terminal-Bench, and SWE-Atlas-QnA. The homepage compares 671 models on price per token, output speed, and latency, and its Endpoint Accuracy Index re-runs the same evals against 16 third-party providers of one model to measure how much accuracy each endpoint loses to quantization or configuration. Products around the data include Optima (build-your-own benchmarks), MicroEvals, a Model Recommender, and a Data Playground. Scale claims from the about page: 500+ models benchmarked, 100+ inference providers, 1,000+ endpoints, 1T+ evaluation tokens.

Status
#

Very active and heavily cited: the changelog had entries dated 23 September 2026, one day before verification, and the Intelligence Index itself iterates weekly (v4.2 on September 5, v4.3 two days later, v4.3.2 by September 24). Its numbers are market-moving enough that HN threads are titled by its rankings (“GLM-5.2 is the new leading open weights model on Artificial Analysis”, 916 points in June 2026), and providers market against its measurements (Baseten’s “fastest Kimi K2.5” post). Founded by Micah Hill-Smith (CEO, ex-McKinsey) and George Cameron (CPO); started as a side project in 2023, launched January 2024, went viral after a Swyx retweet, and raised a seed from Nat Friedman and Daniel Gross’s AI Grant with angels including Andrew Ng, Adam D’Angelo, Clem Delangue, Guillermo Rauch, and swyx.

Strengths
#

Breadth no rival matches: quality, price, speed, providers, and now chips, in one methodology.
The methodology hub publishes evaluation lists, weights, and scoring detail, and the Endpoint Accuracy Index measures an error source (endpoint drift) nobody else prices in.
A dated public changelog makes its own update cadence auditable.

Cautions
#

Its own methodology concedes prompts are partly written in-house and grading partly uses LLM judges, so the indexes measure the measurer as much as the models.
HN criticism is persistent: “the benchmarks are bunk”, a “lost all trustworthiness with the astra blunder” comment on the v4.2 thread, and a pay-to-play analogy for the benchmark-firm business model; a counterpoint in the same thread calls it “the best option currently available”.
It sells private custom benchmarking and an enterprise insights subscription to AI companies, the same companies it ranks; the founders state “no one pays to be on the public leaderboard” and describe a mystery-shopper policy, which you can accept or audit.
Index version churn means any citation needs the version number and date or it goes stale within weeks.

Pricing
#

The public site is free. Revenue comes from an enterprise benchmarking-insights subscription and private custom benchmarking; a paid data platform exists (its terms PDF is published) with no public prices. No public price is stated, so no price history applies.

Compared to
#

LMArena: blind human votes instead of controlled evals; choose the arena for preference, Artificial Analysis for price and speed.
Epoch AI: trends and open datasets rather than leaderboard-style comparisons.
OpenRouter Rankings: revealed preference (actual spend) rather than constructed measurement.

Bottom line
#

Recommended as the default first stop when a decision needs a specific model’s quality against its price and speed across providers, not for trend or adoption questions. My disagreeable claim: the Endpoint Accuracy Index is the most underrated page on the site, because quantization and endpoint defaults are the silent error source teams accept when they pick the cheap provider, and this is the only ranking in the category that makes that tradeoff visible.

Changes
#

2026-09-24 - Created.

References
#

https://artificialanalysis.ai/ - homepage: 671 models, Intelligence Index v4.3.2, Coding Agent Index v1.5, Endpoint Accuracy Index across 16 providers (fetched 200, 2026-09-24)
https://artificialanalysis.ai/about - founders, backers, scale claims, four-layer scope (fetched 200, 2026-09-24)
https://artificialanalysis.ai/methodology - methodology hub: scope, blended-price definition, benchmark inventory (fetched 200, 2026-09-24)
https://artificialanalysis.ai/methodology/intelligence-benchmarking - the ten evaluations, weights, and scoring detail (fetched 200, 2026-09-24)
https://artificialanalysis.ai/changelog - update cadence, entries dated 23 September 2026 (fetched 200, 2026-09-24)
https://www.latent.space/p/artificialanalysis - founding story, seed round, revenue model, mystery-shopper policy (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=48567759 - 916-point HN thread showing citation scale (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=49586403 - the “astra blunder” criticism on the v4.2 thread (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=45706969 - the “benchmarks are bunk” skeptical take (fetched 200, 2026-09-24)
https://artificialanalysiscdn.com/legal/ProDataPlatformTerms.pdf - existence of the paid data platform terms (fetched 200, 2026-09-24)

Epoch AI

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

Epoch AI is a 501(c)(3) research nonprofit that maintains open datasets on AI models, compute, data centers, chips, and companies, runs its own benchmarks, and publishes research, a newsletter, and a podcast, all under a transparency policy that itemizes its funders and paid consultations. Facts below verified as of 2026-09-24.

Where the other sites in this category tell you what is winning this week, Epoch tells you how fast the whole field is moving, and lets you download the series behind the claim.

What it is
#

A research organization, roughly 50 people, whose public products are data explorers (AI Models with 3,200+ entries from 1950 to today, AI Data Centers, Chip owners, Companies, Polling), benchmarks (the Epoch Capabilities Index, FrontierMath, MirrorCode, EBR-bench), papers and reports, short Data Insights, the Gradient Updates newsletter, and the Epoch After Hours podcast. Data is free to use with attribution under Creative Commons (CC BY), and a Python client library and data repositories live on its GitHub org. Its numbers are the citation of record for AI trend claims, used by Our World in Data, government AI-safety reports, and the financial press.

Status
#

Actively maintained at near-daily granularity: the data page was stamped “Updated Sep. 24, 2026”, the day of verification, and its GitHub repositories show pushes the same day. HN traction is substantial and recurring: the FrontierMath launch drew 185 points in 2024, “FrontierMath was funded by OpenAI” drew 483 points in January 2025, and “Epoch confirms GPT5.4 Pro solved a frontier math open problem” drew 480 points in March 2026. Founded by Jaime Sevilla and collaborators; funding is donations (Coefficient Giving grants of $8.5M in 2025, $4.13M in 2024, and more, plus the Survival and Flourishing Fund, Jaan Tallinn, and Schmidt Sciences), with paid consultations disclosed on the transparency page.

Strengths
#

The most checkable actor in the category: funders, grants, and paid consultations (OpenAI, Google DeepMind, the EU AI Office) are named on one page.
CC BY data plus a Python client means their series can go straight into your own analysis instead of a screenshot.
Coverage spans hardware, data centers, and companies, not just model leaderboards, which is where the multi-year questions live.

Cautions
#

The FrontierMath funding controversy is the load test it had to pass: TechCrunch reported in January 2025 that Epoch “didn’t disclose that it had received funding from OpenAI” while developing a benchmark OpenAI’s models were scored on.
Its response was structural (full disclosure plus a 2026 FrontierMath data audit), but the structural tension remains: the labs it benchmarks are also its consulting clients and neighbors in the funding ecosystem.
Trend series are backward-looking; nothing here tells you which provider to buy this week.

Pricing
#

Free: a donation-funded nonprofit whose data is CC BY licensed. It sells consultations and custom research to organizations, with no public price list. No reader-facing price is stated, so no price history applies.

Compared to
#

Artificial Analysis: live model-versus-model comparisons; Epoch answers the multi-year questions those snapshots sit inside.
AI Release Tracker: the raw release stream; Epoch’s model dataset is the curated, parameterized version of the same history.
Stanford HAI’s AI Index: a single annual curated report; Epoch is continuous and downloadable.

Bottom line
#

Recommended for anyone who needs to cite how fast AI capability, compute, or cost is moving, with the data behind the number, not for weekly model selection. My disagreeable claim: the FrontierMath episode made Epoch more trustworthy, not less, because it is the only member of this category that responded to a conflict-of-interest scandal by publishing an itemized funder and client list, and none of its commercial peers has matched that.

Changes
#

2026-09-24 - Created.

References
#

https://epoch.ai/ - homepage: tagline, 3,200+ model dataset, product map, update stamps (fetched 200, 2026-09-24)
https://epoch.ai/about - “data-first research nonprofit” self-description and mission (fetched 200, 2026-09-24)
https://epoch.ai/about/transparency - 501(c)(3) status, itemized funders, disclosed OpenAI and DeepMind consultations, the 2026 FrontierMath data audit (fetched 200, 2026-09-24)
https://epoch.ai/data - data explorers, CC BY licensing, “Updated Sep. 24, 2026” stamp (fetched 200, 2026-09-24)
https://epoch.ai/team - team size and leadership including Jaime Sevilla (fetched 200, 2026-09-24)
https://epoch.ai/frontiermath - the FrontierMath benchmark surface (fetched 200, 2026-09-24)
https://techcrunch.com/2025/01/19/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai/ - the funding-disclosure controversy (fetched 200, 2026-09-24)
https://github.com/epoch-research - active data and benchmark repositories, pushes on 2026-09-24 (fetched 200, 2026-09-24)
https://epochai.substack.com/ - the Gradient Updates newsletter (fetched 200, 2026-09-24)

LLM Stats

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

LLM Stats is a model-comparison platform that ranks 398 canonical models on a composite “LLM Stats Score”, publishes task-level leaderboards, pricing, and comparison pages, and exposes the whole dataset through a REST API and an MCP server aimed at agents. Facts below verified as of 2026-09-24.

It is the only member of this category that treats an agent, not a human, as the primary consumer of the leaderboard.

What it is
#

A website and data service: a composite leaderboard (“Rankings for 300+ Top AI Models by Intelligence, Speed & Price”), category boards for reasoning, coding, agents, math, long context, vision, and more, model compare and pricing pages, an AI news feed, a weekly newsletter, and arena products hosted under its sibling brand huggle.ai. The methodology (score v3.1, updated 2026-09-02) is more careful than the aggregator genre demands: scores normalize within each benchmark, missing results are treated as missing rather than as failures, models with less evidence carry wider uncertainty, and lab-reported numbers are labeled separately from independently verified ones. The developer surface lists endpoints for models, benchmarks, scores, rankings, and updates, plus “11 MCP tools” with one-click OAuth for Cursor, Claude Code, and Windsurf. It is operated by ZeroEval Inc., which describes itself as building “the independent measurement layer for AI” and names LLM Stats as its first product; the founder is Jonathan Chavez, who Show HN’d earlier versions in November 2024 and January 2025.

Status
#

Actively maintained: the methodology page was modified 2026-09-02, the docs repository was pushed 2026-09-08, and the news page is titled by the current month. Traction is modest but growing: two small Show HNs (3 and 7 points), 20 HN comments referencing the site, and a small GitHub org, with self-reported reach (people at OpenAI, Anthropic, Google, Meta, “400,000+ more”) that cannot be verified. The corporate backing is self-displayed: a “Backed by” strip on zeroeval.com lists Y Combinator, Hugging Face, Harvard Medical, Google, and Datadog.

Strengths
#

The agent surface is the category’s best: a free API tier plus MCP tools means your next session can query leaderboard data directly.
The uncertainty-aware, missing-is-missing scoring policy is more statistically defensible than the genre’s usual single-number bravado.
Breadth of coverage (398 models, 50+ benchmarks claimed) with a compare tool and per-task boards for quick narrowing.

Cautions
#

The site is dressed for search engines: dozens of “best AI for X” landing pages (lawyers, architects, YouTube thumbnails) whose purpose is search traffic rather than measurement.
Methodology details are partly gated: “We share additional methodology details with researchers, customers, and partners when appropriate”, and pages sit behind a Cloudflare Turnstile for non-interactive clients.
Early HN feedback flagged coverage gaps (Cerebras missing, “makes me wonder if other evaluations are missing”) and the author’s own admission that “some labs cherry pick the benchmarks they want to report”.
Paid evaluation services (custom benchmarking, data labeling) put revenue on the same side of the table as the companies being scored.

Pricing
#

Free to browse, with a free Community API tier (250 data responses/day, 50 req/min, rolling 6-month history). Builder: $99/month with 5,000 responses/day, 300 req/min, 12-month history, bulk snapshots, and an incremental-updates API. Commercial: contract pricing with redistribution licensing and signed webhooks.

Price history
#

Date	Plan	Change	Source
2026-09-24	Community	Baseline: free tier at 250 data responses/day, 50 req/min, 6-month history	https://llm-stats.com/developer
2026-09-24	Builder	Baseline: $99/month at 5,000 responses/day, 300 req/min, 12-month history	https://llm-stats.com/developer

Compared to
#

Artificial Analysis: first-party controlled evals with published weights; LLM Stats aggregates public evidence and labels it.
AI Release Tracker: the dated release log; LLM Stats is the current-rankings view of the same population.
LMArena: human preference rather than benchmark aggregation.

Bottom line
#

Recommended for wiring leaderboard data into agents and scripts on the free tier, and for a quick conservative read across many models; not as the deciding source for a deployment commitment. My disagreeable claim: the MCP server is the most consequential feature introduced in this category in years, because the reader of these leaderboards is becoming an agent that cannot click through a chart-dense SPA, and LLM Stats is the only member that noticed.

Changes
#

2026-09-24 - Created.

References
#

https://llm-stats.com/ - homepage: 398 canonical models, composite score, task boards, newsletter (fetched 200, 2026-09-24)
https://llm-stats.com/methodology/llm-stats-score - score v3.1 construction, evidence policy, limitations, 2026-09-02 modification date (fetched 200, 2026-09-24)
https://llm-stats.com/developer - API and MCP endpoints, plan tiers and quotas, “updated within hours” claim (fetched 200, 2026-09-24)
https://llm-stats.com/about-us - founder Jonathan Chavez and the zeroeval relationship (fetched 200, 2026-09-24)
https://llm-stats.com/ai-news - the news feed and self-reported reach claim (fetched 200, 2026-09-24)
https://zeroeval.com - ZeroEval Inc. description, product list, self-displayed “Backed by” strip (fetched 200, 2026-09-24)
https://api.github.com/orgs/zeroeval - org metadata and docs-repo activity (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=42590841 - the January 2025 Show HN and authorship evidence (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=42231372 - the November 2024 launch thread with the Cerebras coverage-gap criticism (fetched 200, 2026-09-24)

LMArena

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

LMArena ranks AI models by blind human preference votes: two anonymous models answer the same prompt, you pick the winner, and Bradley-Terry-style statistics turn millions of those picks into leaderboards spanning text, image, video, vision, search, web development, and agents. Facts below verified as of 2026-09-24; lmarena.ai is a client-rendered app, so vote and model counts beyond the founding paper’s figures could not be read from its HTML.

It is the field’s mood ring: the most cited signal of which model people prefer, and the easiest leaderboard in existence to game.

What it is
#

A website (lmarena.ai), a family of leaderboards (Agent Overall, Text, WebDev, Image, Video, Vision, Document, Search), a WebDev arena (web.lmarena.ai), a blog, and open methodology repositories, run by Arena Intelligence Inc., the company that grew out of the UC Berkeley and LMSYS Chatbot Arena project. The founding paper (arXiv:2403.04132, March 2024) describes the pairwise crowdsourcing method and 240K+ votes at the time; the leaderboard methodology source is published as the arena-rank repository, pushed August 2026. The company raised $100M at a $600M valuation in May 2025, led by Andreessen Horowitz and UC Investments, and labs including OpenAI, Google, and Anthropic partner with it to put flagship models in front of voters. Recent product motion includes AutoEval scores added to the leaderboards (to complement slowly collected human votes), agent leaderboard categories with task costs (August 2026), and a HarnessTax research post (September 2026).

Status
#

The most-cited leaderboard in the field: a critical paper describing Chatbot Arena as “the go-to leaderboard for ranking the most capable AI systems” is itself the best evidence of that status. The blog posts within days of verification, the arenas run continuously, and HN threads routinely open with its rankings as the premise. The company is well capitalized and has converted the academic project into a venture-scale business, which is also the source of its hardest questions.

Strengths
#

Preference at scale is a measurement no benchmark suite replaces: it captures whatever makes people pick one answer over another, including style, format, and thoroughness.
The methodology code and the voting procedure are public, so the statistics are checkable even when the data pipelines are not.
The arena expansion (WebDev, agents, image, video) follows usage: it measures the surfaces people actually use models on.

Cautions
#

The gaming record is documented: Meta’s Llama 4 Maverick episode (an “experimental chat version” tested on the arena that differed from the shipped model) forced a policy update, and LMArena’s own statement conceded “Meta’s interpretation of our policy did not match what we expect from model providers”.
The Leaderboard Illusion paper (arXiv:2504.20879) documents “undisclosed private testing practices” that “benefit a handful of providers”, counting 27 private Meta variants tested before the Llama 4 release and sampling-rate asymmetries favoring closed models.
The sharpest criticism (Surge AI’s “LMArena is a cancer on AI”, 246 points on HN in January 2026) argues the format “rewards superficiality over accuracy” because “the easiest way to climb the leaderboard isn’t to be smarter; it’s to hack human attention span”.
A preference rank is not a capability claim: verbosity and sycophancy win votes that lose tasks, so the number is routinely over-read by headlines.

Pricing
#

Free to use and to vote. No public pricing page exists; the company is venture funded, and its “Try Arena” product surfaces are free at the time of verification. No reader-facing price is stated, so no price history applies.

Compared to
#

Artificial Analysis: controlled first-party evals with published weights; choose it when you need price and speed, the arena when you need preference.
OpenRouter Rankings: revealed preference (spend) versus stated preference (votes).
LLM Stats: benchmark aggregation, which at least labels what it cannot verify.

Bottom line
#

Recommended as the fastest read on which models feel better to people, and as a required filter on any headline of the form “X tops the arena”; not as the number you commit money against. My disagreeable claim: the leaderboard is the least valuable thing the arenas produce, because the vote stream is quietly one of the largest human-feedback datasets ever assembled, and whoever holds it holds a training asset, not just a ranking.

Changes
#

2026-09-24 - Created.

References
#

https://lmarena.ai/ - homepage meta: blind comparison, vote-driven leaderboards across text, image, and code (fetched 200, 2026-09-24)
https://blog.lmarena.ai/ - Arena Intelligence Inc. identity, leaderboard families, 2026 post dates (fetched 200, 2026-09-24)
https://blog.lmarena.ai/how-it-works/ - the vote flow and identity reveal procedure (fetched 200, 2026-09-24)
https://arxiv.org/abs/2403.04132 - the founding paper: method and 240K+ votes (fetched 200, 2026-09-24)
https://arxiv.org/abs/2504.20879 - The Leaderboard Illusion: private testing, 27 Meta variants, sampling asymmetries (fetched 200, 2026-09-24)
https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/ - $100M seed, $600M valuation, investors, Berkeley origin (fetched 200, 2026-09-24)
https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks-gaming - the Maverick gaming episode and LMArena’s policy response (fetched 200, 2026-09-24)
https://surgehq.ai/blog/lmarena-is-a-plague-on-ai - the strongest critical essay on the format (fetched 200, 2026-09-24)
https://github.com/lmarena - methodology repositories including arena-rank, pushed 2026-08-04 (fetched 200, 2026-09-24)
https://web.lmarena.ai/ - the WebDev arena surface (fetched 200, 2026-09-24)

OpenRouter Rankings

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

OpenRouter Rankings ranks models by the tokens actually processed through the OpenRouter gateway, published with CC BY 4.0 licensing, a public Data API, and views by task, cost per session, market share, and app, and it is the only ranking in its category built on revealed spend rather than votes or benchmarks. Facts below verified as of 2026-09-24.

Every other leaderboard in this category measures opinion; this one measures invoices.

What it is
#

A rankings surface inside openrouter.ai, the LLM gateway founded in early 2023 that claims 500T+ monthly tokens, 10M+ users, 80+ providers, and 500+ models. The page states plainly what it does not measure: “They do not rank models by accuracy, reasoning ability, or benchmark performance”, and it excludes private requests while bucketing usage into daily UTC aggregates per model variant. Views include top models (today, week, month), top models by task share of spend, cost per session for coding agents, market share by model author, fastest models, and top apps (Hermes Agent, Claude Code, Kilo Code, and Cline led in tokens as of 2026-09-24). Data access is unusually open: rankings are CC BY 4.0 with a required citation format, and the Data API serves rankings-daily (top 50 public models per day, history back to 2025-01-01, 30 requests/minute, live-updating) to any inference key.

Status
#

Fresh and commercially central: the page showed “Usage data through Sep 23, 2026” on the day of verification, a one-day lag, and updates flow live as traffic arrives. The underlying gateway is a top-of-category business: $113M Series B led by CapitalG in May 2026 at a reported $1.3B valuation, then an announced acquisition by Stripe on August 19, 2026 at a reported price above $7B, with the company claiming 10T+ tokens per day. Its numbers are routinely quoted as market signal in press and on HN, and OpenRouter itself turned the dataset into research (a 100T-token empirical study that drew 207 points on HN).

Strengths
#

Revealed preference: tokens and spend through a paid gateway are harder to game with a demo day prompt than a vote or a benchmark run.
Openness is a policy, not a favor: CC BY 4.0, a documented citation format, and a free JSON API with a year-plus of history.
The by-task and cost-per-session views answer questions (what does a coding-agent session actually cost) that no benchmark board even frames.

Cautions
#

The sample is one gateway, and the page says so: “They do not include usage on a model provider’s own API or across the whole market”, and “token volume is not a count of requests, users, or spend” since verbosity and tokenization differ.
Free-tier promotions distort the ranks: Max Woolf’s analysis of the Tencent Hy3 model topping the leaderboard documents a free period on a single provider driving the rank, and the Kilo Code free-Grok episode did the same in 2025.
Independence is now a clock: the rankings belong to a gateway being absorbed by Stripe, so treat “market share” as “OpenRouter’s customers’ share” and re-verify who publishes the numbers after the deal closes.
Figures restate as snapshots refresh, so any number you cite needs its as-of date recorded alongside it.

Pricing
#

Free to view and to query, CC BY 4.0 licensed with attribution. OpenRouter monetizes the gateway (inference and enterprise controls), not the rankings. No reader-facing price is stated, so no price history applies.

Compared to
#

Artificial Analysis: constructed measurement of quality, price, and speed; the rankings measure adoption, and the two disagree for good reasons.
LMArena: stated preference versus revealed preference.
LLM Stats: benchmark aggregates with an agent-facing API; the rankings’ API is the usage-side mirror.

Bottom line
#

Recommended for answering “what are developers actually paying for this week” with numbers you can legally republish, not for any claim about which model is more capable. My disagreeable claim: this is the most misused ranking in the category, cited as market truth from a sample that excludes every provider’s first-party API traffic, and the Stripe acquisition makes reading its provenance, not its numbers, the skill that matters going forward.

Changes
#

2026-09-24 - Created.

References
#

https://openrouter.ai/rankings - methodology, caveats, windows, top apps, CC BY 4.0 note, data-through date (fetched 200, 2026-09-24)
https://openrouter.ai/docs/cookbook/administration/data-api - Data API endpoints, limits, citation format, dataset history from 2025-01-01 (fetched 200, 2026-09-24)
https://openrouter.ai/about - company scale claims and founding date (fetched 200, 2026-09-24)
https://openrouter.ai/blog/announcements/series-b/ - $113M Series B and investors, May 2026 (fetched 200, 2026-09-24)
https://openrouter.ai/blog/announcements/openrouter-is-joining-stripe/ - the Stripe acquisition announcement, August 2026 (fetched 200, 2026-09-24)
https://techcrunch.com/2026/08/16/stripe-will-reportedly-acquire-ai-gateway-startup-openrouter-for-7b/ - the reported price and valuation (fetched 200, 2026-09-24)
https://minimaxir.com/2026/05/openrouter-hy3/ - the free-tier distortion analysis behind the Hy3 rank (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=48317294 - the HN thread questioning free-driven ranks (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=48330499 - the “market signal” counterpoint (fetched 200, 2026-09-24)
https://news.ycombinator.com/item?id=46154022 - the 100T-token study showing the dataset’s research use (fetched 200, 2026-09-24)

Trackers and leaderboards

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

Sites that watch the AI field itself and publish a continuously refreshed number from that watching: release timelines, independent benchmarks, open datasets, composite rankings, preference arenas, and usage statistics. The category’s dividing line is what the number measures, since each member answers a different question about the same field.

AI Release Tracker - a free, ad-free timeline of frontier releases since ChatGPT, with launch-day scores and flat-file exports for agents.
Artificial Analysis - independent quality, price, and speed measurement across 671 models, 100+ providers, and the chips underneath.
Epoch AI - the research nonprofit’s open datasets and trend series on models, compute, data centers, and AI’s trajectory.
LLM Stats - a conservative composite leaderboard over 398 models with the category’s only MCP server for agents.
LMArena - blind human-vote arenas ranking models by preference, the field’s most cited and most gamed signal.
OpenRouter Rankings - real token spend through the OpenRouter gateway, CC BY licensed, the category’s only revealed-preference number.

Its members are compared on shared rows in the Trackers and Leaderboards Feature Matrix.

Changes
#

2026-09-24 - Added AI Release Tracker.
2026-09-24 - Added Artificial Analysis.
2026-09-24 - Added Epoch AI.
2026-09-24 - Added LLM Stats.
2026-09-24 - Added LMArena.
2026-09-24 - Added OpenRouter Rankings.

Trackers and Leaderboards Feature Matrix

tom@tomrochette.com (Tom Rochette) — Thu, 24 Sep 2026 00:00:00 +0000

This matrix compares the six members of the Trackers and leaderboards category: sites whose product is a continuously refreshed number about the AI field itself. The members split cleanly on what their number measures: what shipped (AI Release Tracker), what the operator measured (Artificial Analysis), how the field moves (Epoch AI), what public evidence aggregates to (LLM Stats), what people prefer (LMArena), and what people pay for (OpenRouter Rankings).

No row of this matrix crowns a winner, because the rows are different questions; the failure mode is citing a site for a question it does not answer, usually LMArena ranks quoted as capability or OpenRouter tokens quoted as market share.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell traces to a source cited there or in the references.

The matrix
#

	AI Release Tracker	Artificial Analysis	Epoch AI	LLM Stats	LMArena	OpenRouter Rankings
Kind	release timeline plus flat-file corpus	independent benchmarking site and data business	research nonprofit with open datasets	composite aggregator with agent-facing API	blind preference arena	gateway usage rankings
The number measures	launch-day facts: what shipped, when, with which claimed scores	the operator’s own controlled evals, prices, and speed runs	long-run trends: compute, cost, capability over time	public benchmark evidence, normalized with uncertainty	blind human preference votes	tokens processed through one gateway
Run by	To sider ApS, a Danish side project (one visible operator)	venture-backed independent company (AI Grant seed)	501(c)(3) nonprofit, itemized donors, about 50 people	ZeroEval Inc. (self-displayed YC backing)	Arena Intelligence Inc. ($100M seed at $600M)	OpenRouter, a gateway acquired-by-Stripe (announced 2026-08-19)
Coverage	231 releases, 10 labs (as of 2026-08-26)	671 models, 100+ providers, 1,000+ endpoints, chips	3,200+ models since 1950, data centers, chips, companies	398 canonical models, 50+ benchmarks claimed	text, image, video, vision, search, webdev, agent arenas	500+ models via the gateway, 80+ providers
Update cadence	on each release; dataset last updated 2026-08-26	daily changelog; index versions iterate weekly	near-daily data updates, page-stamped	continuously; “within hours of release” claimed	continuous votes; product posts within days	daily UTC buckets, about one day of lag
Methodology published	~ FAQ and data notes, no formal methodology	✓ methodology hub with evaluation lists and weights	✓ transparency page, papers, and data documentation	~ score construction published, details gated	✓ methodology repo (arena-rank) plus founding paper	✓ on-page caveats plus Data API docs
Data access	✓ /models.json and llms-full.txt, free with attribution	~ web charts free, data platform paid, no public API documented	✓ CC BY datasets and a Python client	✓ REST plus 11 MCP tools, free tier	~ arenas open, methodology open, raw votes not re-offered	✓ CC BY 4.0 JSON via Data API, history to 2025-01-01
Reader pricing	free, no ads; Pro instant alerts, price unpublished	free site; enterprise and data-platform tiers, prices unpublished	free, donation funded	free tier; Builder $99/month; Commercial contract	free	free, attribution required
Independence caveat	one operator, one company registration, no second pair of eyes	sells private benchmarking to the labs it ranks (disclosed policy)	consulted for OpenAI and DeepMind, disclosed and audited	paid eval services and a consumer sibling in the same company	lab partnerships and a documented private-testing history	owned by the gateway it measures, being absorbed by Stripe
Verification hooks	✓ full corpus re-downloadable and diffable	~ changelog and methodology public, raw runs private	✓ data, notebooks, and code public	~ API re-queryable, scoring internals partly gated	✓ methodology code public, vote data not re-offered	✓ daily snapshots re-fetchable under CC BY

Reading the matrix
#

The row that sorts the category is “the number measures”, and it doubles as a citation guide: release questions go to the tracker, present-tense quality to Artificial Analysis, trend claims to Epoch, quick composites to LLM Stats, preference to LMArena, spend to OpenRouter. The two most-quoted members are also the two most misquoted: LMArena ranks get read as capability, and gateway tokens get read as market share, when the matrix’s own rows show both measure something narrower.

Verification strength tracks institutional type, not popularity. The nonprofit (Epoch) publishes its funders, its consultations, and its data; the gateway (OpenRouter) publishes its raw daily numbers under CC BY; the side project (AI Release Tracker) lets you diff its whole corpus; while the three venture-scale companies publish methodology and products but keep raw runs, votes, or scoring internals private. If you need to check the work rather than read the work, the checkable columns are the nonprofit’s, the gateway’s, and the side project’s, which is the opposite of what citation frequency would predict.

Every column carries a conflict row, and the conflicts are structural rather than scandals: the benchmark firm sells benchmarking to the benchmarked, the arena partners with the labs it ranks, the aggregator sells evaluation services, the gateway’s data is its own marketing, and the nonprofit consults for the labs. Epoch’s FrontierMath episode and LMArena’s Maverick episode are the two documented failures, and both produced their institution’s strongest disclosure artifacts, which is the pattern worth watching on every refresh of this page.

Choosing from the matrix
#

Needing a dated record of what shipped and what the lab claimed that day: AI Release Tracker, via its flat files.
Picking a model or provider this week and needing quality against price and speed: Artificial Analysis, with the Endpoint Accuracy Index for the cheap-provider question.
Citing how fast the field moves, with downloadable data: Epoch AI.
Wiring leaderboard data into an agent or a script: LLM Stats’s free API and MCP tools, accepting the gated internals.
Sensing which model people prefer right now: LMArena, read as a mood ring and never as a capability claim.
Asking what developers actually spend on: OpenRouter Rankings, quoted with its as-of date and its single-gateway caveat.

Changes
#

2026-09-24 - Created with six columns (AI Release Tracker, Artificial Analysis, Epoch AI, LLM Stats, LMArena, OpenRouter Rankings) when the category was seeded at the owner’s request.

References
#

https://web.archive.org/web/20260826134434/https://aireleasetracker.com/ - AI Release Tracker column: scope, footprint, alerts, operator (fetched 200, 2026-09-24)
https://artificialanalysis.ai/ - Artificial Analysis column: model count, indexes, provider coverage (fetched 200, 2026-09-24)
https://artificialanalysis.ai/methodology/intelligence-benchmarking - Artificial Analysis column: published evaluation weights (fetched 200, 2026-09-24)
https://epoch.ai/about/transparency - Epoch AI column: nonprofit status, funders, consultations (fetched 200, 2026-09-24)
https://llm-stats.com/developer - LLM Stats column: API tiers, MCP tools, quotas (fetched 200, 2026-09-24)
https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/ - LMArena column: funding and company identity (fetched 200, 2026-09-24)
https://arxiv.org/abs/2504.20879 - LMArena column: the private-testing findings (fetched 200, 2026-09-24)
https://openrouter.ai/rankings - OpenRouter Rankings column: methodology, caveats, licensing (fetched 200, 2026-09-24)
https://openrouter.ai/blog/announcements/openrouter-is-joining-stripe/ - OpenRouter Rankings column: the ownership question (fetched 200, 2026-09-24)
https://minimaxir.com/2026/05/openrouter-hy3/ - OpenRouter Rankings column: the free-tier distortion evidence (fetched 200, 2026-09-24)

trackers-and-leaderboards on tomrochette.com

AI Release Tracker

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

Artificial Analysis

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

Epoch AI

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

LLM Stats

What it is #

Status #

Strengths #

Cautions #

Pricing #

Price history #

Compared to #

Bottom line #

Changes #

See also #

References #

LMArena

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

OpenRouter Rankings

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

Trackers and leaderboards

Changes #

Trackers and Leaderboards Feature Matrix

The matrix #

Reading the matrix #

Choosing from the matrix #

Changes #

See also #

References #

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Price history
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

Changes
#

The matrix
#

Reading the matrix
#

Choosing from the matrix
#

Changes
#

See also
#

References
#