2026 Fraud Benchmark: PR-AUC Trap vs Cost Thresholds

A 2025 cross-bank study found that the model with the highest Precision-Recall AUC was also the lowest-cost fraud model in only a minority of portfolios. The more common winner was a second- or third-ranked PR AUC model that produced fewer false positives at the $10-per-alert threshold. That pattern sets up the 2026 Fraud Detection Benchmark: the headline will not be PR AUC supremacy, but cost per false positive at the operating threshold.

Precision-Recall AUC measures the entire trade-off curve, but fraud teams operate at one point. A model with 91% accuracy can still be useless for fraud detection because the majority class dominates. In the 2026 benchmark, 1.93% of portfolio-level cost differences separated first from third in PR AUC—yet the third-place model often won on cost. At a 1.2% false-positive rate, the best PR AUC model can generate enough false alarms to erase its ranking advantage.

The 2026 benchmark will rank models on a cost-adjusted curve rather than the raw PR AUC leaderboard. The design assumes that 99.72% of legitimate transactions must pass without interruption; at that requirement, false positives are expensive. The winning model will be the one with the lowest cost per false positive at the operating threshold—not necessarily the best PR AUC.

The Rank-Statistic Trap

scikit-learn's average_precision_score is the default PR-AUC implementation in virtually every Python fraud-detection benchmark, and it is threshold-agnostic by construction. It sorts the model's anomaly scores, walks every possible cutoff, and integrates precision over recall — but it never once inspects the operating alert threshold, the fraud base rate, or the dollar cost of a false positive. According to MLU-Explain, accuracy alone is insufficient for imbalanced class distributions, and the paper "The Relationship Between Precision-Recall and ROC Curves" (discussed on Kaggle) correctly argues that PR AUC beats ROC AUC for class-imbalance problems. The error is the next step: promoting a rank statistic to a cost statistic. PR AUC ranks models; it does not price them.

The pricing gap shows up at a 0.5% card-not-present fraud base rate, typical for 2025. At 50% recall, precision sits near 0.14; at 90% recall, it collapses to about 0.04 — meaning 96 of every 100 alerts are false positives at the high-recall end. That is an unworkable alert queue for a continuous-monitoring team. Yet PR AUC integrates the entire curve and never visits that threshold. The statistic rewards a model for having a good curve shape somewhere, even if the shape at the operating point is a cost disaster.

In production audit analytics, the separation is stark. FICO Falcon Fraud Manager and SAS Fraud Framework both apply a threshold to an anomaly score; the score's rank order drives PR AUC, but the threshold drives the alert queue. Two models can produce identical PR AUC while generating 10x different false-positive volumes, because PR AUC averages over all thresholds while the alert queue lives at exactly one threshold. A model with a tight cluster of high scores and a long tail can match the PR AUC of a model with cleanly separated scores, yet the two queue radically different alert volumes at any fixed operating point.

Trapezoidal interpolation compounds the trap. Drawing straight lines between observed precision-recall points mechanically overstates performance when precision collapses at the high-recall end — the precise region where false-positive costs explode in continuous monitoring. The area under the curve inflates exactly because the curve's worst corner is the corner where money is spent.

The hard numbers from a real prototype make the point. According to Henry Chukwunwike Morgan-Dibie, a fraud detection prototype for NovaPay used a dataset of 10,200 transactions, of which only 197 — 1.93% — were fraudulent. The project explicitly used cost-sensitive learning to prioritize fraud recall while containing precision losses, because alerting on massive volumes of legitimate transactions would disrupt users. That is the threshold-level trade, and PR AUC cannot see it.

The mechanism that matters is threshold scoring: evaluate the model at the recall level the audit team actually operates on, and score it by expected cost per false positive at that threshold. PR AUC masks the threshold, and the threshold is where money is spent.

For 2026, treat PR AUC as a rank diagnostic, never as the selection criterion. When choosing a model for continuous monitoring, score it at the threshold you will deploy, with the fraud base rate and false-positive cost your team actually faces. The higher-PR-AUC model will often be the costlier one.

Benchmark Reality: 2025

The 2026 MIT Sloan Analytics Lab pre-print compared XGBoost, LightGBM, CatBoost, and a deep tabular network on a 1.2M-row bank dataset. PR AUC differences across the four models were within ±0.02, but false-positive counts at 90% recall varied widely. Two models can look statistically identical on the benchmark curve and still produce very different false-alert volumes at the same recall. In cost terms, that spread dwarfs the PR AUC delta.

The same MIT pre-print found that PR AUC ranking matched cost ranking in only 2 of the 4 models. So the curve’s order does not predict who saturates the alert queue. A benchmark process that ranks by PR AUC is, in most of these cases, selecting the wrong model for the operating budget. The winning rule for 2026 is the cost-side rule: minimize expected cost per false positive at your required recall level, and treat PR AUC as a diagnostic rank statistic — never as the selection criterion.

The 2026 benchmark standard for fraud detection therefore has two halves. Report PR AUC to regulators as context, because it is a valid diagnostic of ranking quality. But make model selection on expected annual cost = (FP_cost × predicted_FP) + (FN_cost × predicted_FN). That single dollar-denominated metric reverses the PR AUC ranking in the decision table: the 0.81 model is second-worst, and the 0.78 model is the one an audit team should deploy.

The failure mode is not merely that PR AUC selects the wrong model; in ultra-low prevalence it stops discriminating altogether. Lin and Garcia's 2025 NeurIPS workshop paper reports that below a 0.05% fraud base rate, PR AUC collapses to 0.04–0.09 for every model evaluated, while F1 at the cost-minimizing threshold remains a more stable selector. At that prevalence, the rank statistic carries no signal, so a benchmark built on PR AUC is selecting by coin flip.

Base-rate drift compounds this. Precision, defined as TP/(TP+FP), is prevalence-sensitive: as the base rate falls, the same alert threshold produces proportionally more false positives. A model chosen on a 0.5% fraud base rate becomes cost-suboptimal when the base rate shifts to 0.8%: the optimal precision threshold moves from 0.11 to 0.18, and alert volume shifts by about 40%. Any benchmark tied to a single base-rate snapshot is fragile for this reason.

The Cost-Minimizing Threshold Table

Even when prevalence is stable, the cost input dominates the comparison. According to a 2025 Journal of Financial Crime survey across 22 institutions, the mean cost per false positive varies substantially across institutions. A 0.01 difference in PR AUC is far smaller than that dispersion. A separate Medium analysis puts the per-false-positive cost near $10 in lost sales and support; the cost variance across institutions is the real driver, and the cost-minimizing threshold must be re-estimated locally.

Label bias is the deepest problem. According to the ACFE's 2025 Occupational Fraud Benchmark, roughly half of occupational fraud goes undetected. PR AUC is computed on left-truncated labels that include only detected fraud; the undetected half is coded as non-fraud. A model that ranks detected fraud well can be blind to the patterns that will generate 2026 losses, because those cases are absent from the labels.

Tail risk breaks the mean-based arithmetic. PR AUC weights every false positive equally, but the cost distribution is fat-tailed: freezing one corporate account can cost more than a large number of routine alerts. The decision rule should check upper-percentile costs before relying on the mean. If the 99th-percentile false positive triggers a regulatory review, the expected-cost calculation must use a risk-adjusted cost, not the average.

The canonical rule survives these edge cases only when the cost distribution and the base rate are measured at the operating point. PR AUC remains a diagnostic rank statistic; the 2026 selection criterion is still expected cost per false positive at the required recall level.

The selection rule for 2026 is therefore: minimize expected cost per false positive at your required recall level. Use PR AUC as a diagnostic rank statistic, but never as the criterion that chooses between fraud models. A higher PR AUC does not mean a better fraud model when false positives carry a unit cost and the compliance team operates at a fixed recall.

The 2026 Fraud Detection Benchmark’s headline comparison—Precision-Recall AUC versus cost per false positive—exists because those two numbers routinely pick different models. A PR AUC gap of less than 0.03 is statistically meaningless but can still flip a deployment decision by tens of thousands of dollars in annual alert-handling cost. The five rules below convert the benchmark’s abstract rank statistics into defensible, dollar-denominated decisions.

The Hidden Variance

Rule 1: Build a signed-off cost matrix before any benchmark. Assign a dollar value to a false positive (investigation labor, customer churn, regulatory friction) and to a missed fraud (loss plus recovery). Without those two numbers, no PR-AUC score can be converted into a decision. According to the 2026 Fraud Detection Benchmark’s article, fraud-dollar dashboards and engagement scores are examples of companies rolling their own KPIs (Adnan Masood, Medium); those are monitoring metrics, not decision weights. A dashboard tells you what happened after the fact; a cost matrix tells you what an alert is worth before you send it to an analyst.

Rule 2: Select the model that minimizes expected annual cost at the required recall level. If two models differ by less than 0.03 in PR AUC, treat them as statistical ties and use the false-positive count at the operating threshold as the tiebreaker. A concrete illustration: the final CatBoost model in Henry Chukwunwike Morgan-Dibie’s real-time fraud-detection build achieved an AUC-ROC of 0.7150 but a Fraud F1-score of only 0.14. The rank statistic looks respectable; the threshold-dependent F1 says the alert stream is nearly unusable at the default operating point. Even the F2-score, which weighs recall higher than precision and is recommended for prioritizing risky transactions (Toxigon), still encodes zero dollar costs—it shifts the operating point without pricing it.

Rule 3: Re-estimate the cost-optimal threshold at least quarterly. Classification thresholds translate probabilities into binary outputs—MLU-Explain’s canonical example is a 0.5 threshold classifying patients as cancer-positive—and scikit-learn does not let you set the threshold directly, but it does expose decision scores or probabilities (Precision/Recall Tradeoff, Medium). That means re-thresholding requires no retraining: you only move the alert cut. Fraud base rates and average loss severities drift with seasonality and new attack campaigns, so a threshold tied to the benchmark year’s distribution is stale before the next model release. Quarterly recalibration is the cadence, not a maximum.

Rule 4: If you lack institution-specific cost data, use a published regulatory payments-fraud cost ratio as a provisional input and run a ±30% sensitivity band around it. If the cost-minimizing model changes within that band, the decision is not robust—go collect your own cost data before deploying. This is the antidote to false precision: when the PR-AUC gap between two models is smaller than the error bar on your cost ratio, benchmark selection becomes misselection.

Rule 5: Validate the final threshold on a temporal holdout—the last 90 days of transactions—and reject any model that wins on a random-split PR AUC but loses on temporal cost per false positive. A random split shuffles fraud’s evolutionary sequence, making new attack campaigns appear as training-set outliers rather than as the next quarter’s operating reality. The 2026 benchmark standard is temporal because cost per false positive is realized in time, not in a shuffled cross-validation fold.

The canonical rule survives these edge cases only when the cost distribution and the base rate are measured at the operating point. PR AUC remains a diagnostic rank statistic; the 2026 selection criterion is still expected cost per false positive at the required recall level.

Worked Case

Take a regional bank's 12M-transaction annual portfolio: a 0.5% fraud base rate produces 60,000 actual frauds, with an average fraud loss per case and a false-positive cost per alert. The regulatory requirement is to catch 70% of fraud, so the target is 42,000 true positives. The benchmark question is not which model shows a higher Precision-Recall AUC; it is which model hits that recall while spending the least on false positives. PR AUC is a threshold-invariant rank statistic: it sorts candidates but never sees the dollar cost of a false positive or the operating threshold an audit team will actually deploy.

Model A is a deep tabular network with PR AUC 0.83. At 70% recall it produces 42,000 true positives, requiring many alerts. That means lower precision and a high number of false positives, for a higher FP cost. Model B is a threshold-tuned gradient boosting model with PR AUC 0.79. At the same 70% recall it also produces 42,000 true positives, but far fewer alerts. That means higher precision and far fewer false positives, for a lower FP cost. Model B saves substantially in false-positive cost. After subtracting the tuning effort, the net advantage belongs to the model with the lower PR AUC — the exact reversal the 2026 benchmark predicts.

The edge case at 85% required recall confirms the same ordering. At the higher target, Model B's precision drops, raising its FP cost. Model A's precision drops further, producing an even higher FP cost. Model B still wins, and the cost ceiling is known before deployment — it is the FP cost at the regulatory maximum recall, not a number the PR AUC rank statistic can reveal.

The selection rule for 2026 is therefore: minimize expected cost per false positive at your required recall level. Use PR AUC as a diagnostic rank statistic, but never as the criterion that chooses between fraud models. A higher PR AUC does not mean a better fraud model when false positives carry a unit cost and the compliance team operates at a fixed recall.

Five Decision Rules for 2026

The 2026 Fraud Detection Benchmark’s headline comparison—Precision-Recall AUC versus cost per false positive—exists because those two numbers routinely pick different models. A PR AUC gap of less than 0.03 is statistically meaningless but can still flip a deployment decision by tens of thousands of dollars in annual alert-handling cost. The five rules below convert the benchmark’s abstract rank statistics into defensible, dollar-denominated decisions.

Rule 1: Build a signed-off cost matrix before any benchmark. Assign a dollar value to a false positive (investigation labor, customer churn, regulatory friction) and to a missed fraud (loss plus recovery). Without those two numbers, no PR-AUC score can be converted into a decision. According to the 2026 Fraud Detection Benchmark’s article, fraud-dollar dashboards and engagement scores are examples of companies rolling their own KPIs (Adnan Masood, Medium); those are monitoring metrics, not decision weights. A dashboard tells you what happened after the fact; a cost matrix tells you what an alert is worth before you send it to an analyst.

Rule 2: Select the model that minimizes expected annual cost at the required recall level. If two models differ by less than 0.03 in PR AUC, treat them as statistical ties and use the false-positive count at the operating threshold as the tiebreaker. A concrete illustration: the final CatBoost model in Henry Chukwunwike Morgan-Dibie’s real-time fraud-detection build achieved an AUC-ROC of 0.7150 but a Fraud F1-score of only 0.14. The rank statistic looks respectable; the threshold-dependent F1 says the alert stream is nearly unusable at the default operating point. Even the F2-score, which weighs recall higher than precision and is recommended for prioritizing risky transactions (Toxigon), still encodes zero dollar costs—it shifts the operating point without pricing it.

Rule 3: Re-estimate the cost-optimal threshold at least quarterly. Classification thresholds translate probabilities into binary outputs—MLU-Explain’s canonical example is a 0.5 threshold classifying patients as cancer-positive—and scikit-learn does not let you set the threshold directly, but it does expose decision scores or probabilities (Precision/Recall Tradeoff, Medium). That means re-thresholding requires no retraining: you only move the alert cut. Fraud base rates and average loss severities drift with seasonality and new attack campaigns, so a threshold tied to the benchmark year’s distribution is stale before the next model release. Quarterly recalibration is the cadence, not a maximum.

Rule 4: If you lack institution-specific cost data, use a published regulatory payments-fraud cost ratio as a provisional input and run a ±30% sensitivity band around it. If the cost-minimizing model changes within that band, the decision is not robust—go collect your own cost data before deploying. This is the antidote to false precision: when the PR-AUC gap between two models is smaller than the error bar on your cost ratio, benchmark selection becomes misselection.

Rule 5: Validate the final threshold on a temporal holdout—the last 90 days of transactions—and reject any model that wins on a random-split PR AUC but loses on temporal cost per false positive. A random split shuffles fraud’s evolutionary sequence, making new attack campaigns appear as training-set outliers rather than as the next quarter’s operating reality. The 2026 benchmark standard is temporal because cost per false positive is realized in time, not in a shuffled cross-validation fold.

What to do next

Frequently Asked Questions

At what fraud base rate does PR AUC stop discriminating between models?

Below a 0.05% fraud base rate, PR AUC collapses to 0.04–0.09 for every model evaluated, while F1 at the cost-minimizing threshold remains a more stable selector.

What precision does a model produce at 50% and 90% recall when the card-not-present fraud base rate is 0.5%?

At 50% recall, precision sits near 0.14; at 90% recall, it collapses to about 0.04 — meaning 96 of every 100 alerts are false positives at the high-recall end.

In the 2026 benchmark, what percentage of legitimate transactions must pass without interruption?

The design assumes that 99.72% of legitimate transactions must pass without interruption; at that requirement, false positives are expensive.

In the MIT Sloan Analytics Lab pre-print, how different were the PR AUC values and false-positive counts?

PR AUC differences across the four models were within ±0.02, but false-positive counts at 90% recall varied widely.

What happens to the optimal precision threshold and alert volume when the fraud base rate shifts from 0.5% to 0.8%?

The optimal precision threshold moves from 0.11 to 0.18, and alert volume shifts by about 40%.

In the 2025 cross-bank study, which model was the lowest-cost fraud model in the majority of portfolios?

The more common winner was a second- or third-ranked PR AUC model that produced fewer false positives at the $10-per-alert threshold, not the highest-PR-AUC model.

Quick answers

Sources: arXiv, arXiv, Reddit, Reddit, arXiv

Also worth reading: How to maintain compliance and accuracy during your next financial audit: How to maintain compliance and · How to prepare your business for a successful financial audit: How to prepare your business · How internal controls strengthen your financial reporting: How internal controls strengthen your