"AI-powered" is doing a lot of work in sports data marketing right now, usually without an AUC-ROC number, a holdout set, or a walk-forward test attached to it anywhere. PlayCaller's internal rule is simple: a score doesn't ship as intelligence unless it's backed by a measurement. If it isn't, it's labeled a heuristic, or it doesn't ship at all.
What "walk-forward validated" actually means
A model trained and tested on the same season, or tested on data from before the season it's predicting, will report an AUC-ROC that looks great and means nothing in production. Walk-forward validation trains on data up to a point in time, tests only on data after it, then rolls the window forward and repeats. No feature at observation time T is allowed to use data from T or later, including things that are easy to leak by accident, like a percentile ranking computed from a full season instead of just the games played so far.
Every model below is scored this way, across multiple folds, before it's eligible to ship.
The numbers, as they actually are
- NFL Confidence: AUC-ROC 0.697, 20,585 observations across 5 seasons (2021-2025), 4.61x decile lift, 54.9% hit rate in the top decile.
- NFL Heat: AUC-ROC 0.623, 25.6 percentage-point quartile lift, 3 of 3 walk-forward folds passed. Shipped as a directional gradient rather than a percentage, because that's what the validation actually supports.
- NBA Confidence: AUC-ROC 0.855 (walk-forward average across 3 folds), 84.23x decile lift.
- NBA Heat: AUC-ROC 0.740, 40.5 percentage-point quartile lift, 2 of 2 folds passed.
- MLB Pitcher Form: AUC-ROC 0.647, shipped as a qualitative read (Trending Up / Steady / Trending Down) instead of a percentage, because the underlying score wasn't built to be displayed as one.
- MLB Batter Confidence: AUC-ROC 0.625, 26.98 percentage-point quartile lift.
- NHL Confidence: AUC-ROC 0.669, 36.96 percentage-point quartile lift, 101,973 observations.
Seven models, seven sport-specific measurements. NBA and NHL each run their own model trained on their own data, not the NFL model relabeled.
What happens when a model doesn't clear the bar
MLB batter confidence took several attempts to get there. Earlier iterations capped around an AUC-ROC of 0.58 across different feature sets and regularization choices, and kept landing in roughly the same place. That's not treated as a bug to quietly iterate away. It's a result: the signal available in that feature set had a ceiling, and 0.58 didn't clear the internal gate. The version that eventually shipped changed the label itself, predicting whether a player beats his own trailing baseline instead of a field-wide rank, and added park factors and platoon splits, reaching 0.625. The failed attempts and the number that finally worked are part of the same documented process, not two different stories told to two different audiences.
Retiring our own hollow functions
The same rule applies backward, not just to new models. A couple of older functions in PlayCaller's own codebase compute a score without a validated model behind them: a severity heuristic used in social posts, and an early Kalshi-market confidence estimate. Both are explicitly labeled as unvalidated in the source and are on a retirement path, rather than left in place mislabeled as intelligence, because the standard has to apply to code written before the standard existed too.
Check it yourself
None of this requires trusting a claim on a landing page. GET /api/model/status is public, requires no API key, and returns the live AUC-ROC, lift, and version for every ACTIVE model in the registry:
curl https://playcallerapp.com/api/model/statusIf a model isn't in that response, it isn't validated, and any API field that would depend on it comes back null instead of a fabricated number.
The Developer Sandbox is free, no credit card, 200 requests a day for two weeks: start a free API key. The playcaller-starter repo on GitHub is a minimal working example if you'd rather see it run than read another paragraph.