When we last published this benchmark, four commercial systems sat above 95% accuracy and every open-source model we tested landed within a few points of a coin flip. The open-source result was the flat one. It held across architectures, across training corpora, and across generations: nine downloadable models, none above 63%. The reasonable conclusion at the time was that if you needed a detector you could run yourself, you were out of luck.

That has changed. The board now covers 18 systems: nine commercial entries from eight vendors, plus nine open-source baselines. Six systems clear 95%, up from four. There is a new system at the top by a wide margin. And one of the six, pellav2 from Pella Research, publishes its weights under an MIT licence, which makes it the first downloadable model in this benchmark's history to reach production-grade accuracy.

Vendors routinely self-report around 99% accuracy on their own test sets. Those numbers are not comparable to each other and are easy to overfit. This benchmark fixes the evaluation set and keeps the labels private: every system on the board is scored by us, the same way, on the same 4,524 clips.

The leaderboard

Bold entries are commercial. N is the number of files actually scored; Rej% is the share a system declined or errored on. Corsound and Reality Defender are therefore scored on a subset, and their rows are not directly comparable to full-coverage rows. Re-scoring the full-coverage systems on those same subsets moves them by −1.2 to +4.2 points.

Accuracy, F1, FPR and FNR are independently verified for every system here. Each vendor produces a predictions.csv and we score it against gold-standard labels we hold privately. Nobody grades their own homework.

Six systems now clear 95%

Two of the six are new since the last update, and the shape of the top of the board has changed.

Resemble DETECT-World takes the top slot at 99.47%, and it is not a narrow win. It is the only system on the board with both error rates under 1% (FPR 0.7%, FNR 0.4%), roughly three to four times better than the next system on both axes at once. It misses about 1 fake in 250 and false-flags about 1 real clip in 150. It also holds between 99.2% and 99.7% on every one of the six file formats, so there is no format it quietly falls over on. If you have no specific constraint pushing you elsewhere, it is the default pick.

The previous generation, listed as Resemble AI (DETECT-3B Omni), is still third-best on raw accuracy at 98.05%. DETECT-World beats it on both error types at roughly a third of the latency, which is the more interesting fact about it.

Whispeak (97.70%) is the most evenly balanced system in the top group, 2.9% FPR against 1.7% FNR, with no strong lean in either direction. Aurigin AI (96.75%) has the second-lowest false positive rate of any system that keeps its miss rate in single digits, at 1.5% FPR against 5.0% FNR. Pindrop (95.05%) leans the other way, 6.2% FPR against 3.7% FNR.

And then there is the entry that makes this update worth writing up.

The open-weights gap just closed

Ranking everything in one table buries the result. Of the ten systems on this board whose weights you can actually download and run, here is how they stack up:

pellav2 leads the next open model by 32.9 percentage points. That is not an incremental win over a crowded field; it is the difference between a working detector and a coin flip. It is also the only downloadable system to clear the 95% production bar, and nothing ranked above it on the main leaderboard publishes weights at all. So if your requirement is a model you can inspect, audit, run in your own VPC, or keep running after a vendor relationship ends, the practical choice set is currently one deep.

The weights and a ready-to-run inference script are on Hugging Face, under MIT. The model is a wav2vec2-XLS-R-300M backbone with learned weighting across its 25 hidden layers, mean pooling, and a light linear classification head. It is architecturally unexciting, which is rather the point. It was trained on a private corpus of real speech plus synthetic speech spanning a broad range of modern TTS families. The 95.82% on our board was produced by that public checkpoint and confirmed by the vendor, so what is on the leaderboard is what you can download.

Two things are worth flagging so nobody is surprised in production. First, pellav2 scores a 4-second centre crop rather than the whole clip. That is why its latency (about 57 ms per file, the lowest reported of any commercial system here) is flat with duration, and it is why its RTF is not comparable to systems that read the full file. The AASIST and RawNet2 baselines here crop similarly. Second, its errors lean toward false positives (5.5% FPR against 2.8% FNR), and it is weakest on .m4a at 93.6%, against 97.1% on .wav and .flac.

Pella also sells a hosted API. The open weights are in addition to it, not instead of it, which is an unusual posture in this category and the reason the model appears in both tables.

Open weights were never the problem. Stale training data was.

The obvious reading of last update's result, that open models simply cannot do this task, was wrong, and pellav2 is the disproof. The other nine open models still land in a 47.6% to 62.9% band, near random, and the reason is narrower than "openness".

All nine were trained on ASVspoof-era artifacts. That holds for the 2019 LA generation and, importantly, for the newer ASVspoof 5 and VoxCelebSpoof models too: training era does not rescue them. Modern commercial voice cloning does not produce the artifacts those corpora taught them to look for, so the learned decision surface simply does not transfer. Wav2Vec2 (2019 LA) is the best of them at 62.89%.

Several are worse than the accuracy column suggests, because they have collapsed to a single class. AST (VoxCelebSpoof) misses 97.5% of fakes, AASIST3 misses 98.4%, and Deepfake-V2 misses 90.9%. All three call almost everything real, and their ~50% accuracy is an artifact of our 50/50 class balance rather than any skill. (LCNN-LFCC's exact 50.00% is a harness effect on our side: our runner's fixed decision threshold sits outside the score range that checkpoint actually produces, so it emits one class. We report it as measured and flag it rather than quietly tuning it.)

pellav2 is the tenth open-weights model on the board, it is trained on current synthesis, and it scores 95.82%. The gap was never about the licence.

The real choice is still an error trade-off

Once six systems clear 95%, accuracy stops being the interesting column. What matters is which error you are buying. A false positive blocks a real customer, escalates a legitimate call, or takes down real content. A false negative lets a cloned voice through your KYC flow.

• Both errors matter roughly equally: DETECT-World (0.7% / 0.4%), then Whispeak (2.9% / 1.7%).

• False alarms cost more than misses (moderation at scale, automated takedowns, anything with a human review queue you cannot flood): Aurigin AI, at 1.5% FPR with FNR still at 5.0%.

• You need to run it yourself, or you need the latency: pellav2, accepting a 5.5% false positive rate.

Below the 95% group, the shape changes rather than just degrading. Corsound AI posts a 1.0% FPR, second-lowest of any commercial system, but misses 23.1% of fakes and declines 14.3% of files outright. Hive is the same conservative shape at 2.4% FPR and 30.5% FNR. Both rarely cry wolf and both let a lot through. Reality Defender is the outlier in the other direction: a 53.7% false positive rate means it flags more than half of all genuine audio as fake, and it rejects 17.2% of files, mostly clips under about 1.5 seconds.

Speed, with a caveat we want to be loud about

The latency columns are not verified the way the accuracy columns are, and they mix three measurement regimes. Four systems were run by us directly against the vendor's API, so their figures are ours and include network round-trip. Every other commercial figure is self-reported, on the vendor's own hardware, and may or may not include network time. The open-source baselines were run locally by us on one machine. Treat the column as indicative and do not read small differences between systems as meaningful.

With that said, the one robust finding is that Reality Defender is the only system slower than real time, at about 5.7 seconds per file, which is longer than most clips in the set. It is not viable for streaming. Everything else runs faster than real time. Among commercial systems, Pella reports the lowest per-file latency (~57 ms, on that 4-second crop), then Corsound (180 ms), Pindrop (282 ms) and DETECT-World (399 ms); Resemble AI, Whispeak, Aurigin and Hive sit around RTF 0.33 to 0.40. Several open-source models are faster still (AST at about 5 ms), but at near-random accuracy that speed buys nothing.

New: two telephony tracks

Almost no real fraud, KYC, or call-centre audio arrives as a studio file. It comes over a phone or a mobile/VoIP link: band-limited, resampled, and squeezed through a low-bitrate speech codec. A detector that scores 99% on clean 48 kHz audio has not yet told you what it does on a 12.2 kbit/s AMR-NB call.

So we have added two tracks. Each contains the same 4,524 clips with the same hidden labels as the studio set, degraded to channel grade, so scores are directly comparable to the leaderboard above:

• Narrowband, 8 kHz (2G/3G, landline, VoIP): band-passed 300-3400 Hz, codec pool of G.711 μ-law and A-law, GSM-FR, AMR-NB and G.729.

• Wideband, 16 kHz (4G/5G mobile): band-passed 50-7000 Hz, codec pool of EVS-WB and AMR-WB at two bitrates each.

Every clip is decoded, resampled with a high-quality anti-aliased resampler, band-pass filtered to the channel passband, passed through one randomly assigned codec (a full encode and decode, so it picks up that codec's real compression artifacts), and written as 16-bit mono WAV. The per-file codec assignment is seeded, stratified across source formats, and kept private, exactly like the labels. The two tracks use independent permutations, so their file orders do not line up with each other or with the studio set.

We measured what the degradation actually did. PESQ is against the clean track-rate reference, higher is better; Whisper-WER is the word error rate of the codec'd clip against the clean-reference transcript, lower means intelligibility survived.

The PESQ ordering is the expected one: higher bitrates and newer codecs score better, and EVS edges AMR-WB at matched rates. Word error rates stay low across the pool, so the clips remain intelligible after degradation and the detection task stays fair rather than becoming a noise-tolerance test.

Both telephony leaderboards are empty and open. The tracks are released as audio only (the codec pipeline stays private to preserve benchmark integrity), and they will publish as submissions arrive. We expect the ranking to reshuffle. A detector tuned on clean studio audio has no particular reason to survive an 8 kbit/s G.729 leg, and we would rather find that out on a benchmark than in someone's fraud queue.

Methodology

4,524 audio clips, balanced 50/50 real against synthetic. Real audio from VCTK (110 English speakers, multiple accents), LJ Speech (single speaker, ~24 hours of public-domain audiobook recordings) and LibriTTS-360 (904 speakers). Synthetic audio from about 25 modern commercial TTS and voice-cloning systems, including ElevenLabs, F5-TTS and Chatterbox. Every synthetic clip is round-trip transcribed with Whisper before format conversion, to confirm the TTS system actually synthesised the intended utterance. Six file formats (.mp3, .wav, .flac, .ogg, .m4a, .webm) to test robustness across real distribution channels.

We report accuracy, F1, FPR, FNR, per-format accuracy, rejection ratio, latency and real-time factor. We deliberately do not report EER. Equal error rate assumes an oracle threshold chosen with knowledge of the test labels, which is not a threshold you can set in production. Vendors' published EER figures (pellav2's own card reports 1.9% EER on DFADD and 10.9% on In-the-Wild, for instance) are useful research signals, but they are not the number that tells you what a system will do on your traffic at a threshold you had to pick in advance.

Submit your system

Run your detector over the public dataset and produce a predictions.csv with filename, label (exactly real or fake, lowercase), latency_ms and audio_duration_sec. If your system declines a file, write NOT_APPLICABLE; if it errors, write error. Both are excluded from scoring and counted in the rejection column rather than held against your accuracy.

Submit one CSV per track: the studio track keeps each file's original extension, while both telephony tracks use 0.wav-style filenames. All three are independently shuffled, so a prediction file for one track will not score on another. Email your CSV to hello@podonos.com and we will score it against the private gold standard. For the submission details, please refer to this page.

Limitations

The benchmark is English-only, and it is a snapshot of a fast-moving generation landscape: as new voice-cloning systems ship, the test set has to keep pace or it starts measuring history. The latency and RTF columns mix measurement regimes, as described above. Systems with a non-zero rejection rate are scored only on the files they accepted, so those rows are not directly comparable to full-coverage rows. And a single accuracy number on a fixed 4,524-clip set is a starting point for a procurement decision, not the end of one, which is part of why we added the telephony tracks.

Run it yourself

The dataset, the open-source model runners, the scoring metric definitions and the full per-format breakdowns are public in the benchmark repository. The gold-standard labels are the one thing we keep back, because a public answer key is how a benchmark stops meaning anything.