How Pikola is tested About three times the titles of a cheat-sheet drafter

A good cheat sheet is the bar to beat

Most people walk into an auction with a cheat sheet: a ranked list of players with a price beside each. It's a sensible plan, and it's what Pikola has to beat to be worth paying for.

Why a cheat sheet isn't enough

An auction doesn't stand still. By the middle rounds, the money left in the room, the players left and the holes in your roster have all changed, and a printed list can't see any of it. The same $40 can be a bargain for a team that needs that player's categories and a mistake for a team that already has three players just like them. Pikola reprices every player after every sale, for your roster and the money left.

Does that actually win more, and by how much? One draft can't tell you, because luck swamps everything in a single season. So we ran thousands.

How we tested it

Picture a 14-team Yahoo auction. One seat is Pikola. Another is the cheat-sheet manager. They rank every player by projected stats across the nine categories, then pay the 10th-best player what the 10th most expensive player cost in this league last year, and so on down the list. They never panic, never chase a player and never change plan. That makes them a disciplined opponent, not a pushover.

Both use the same projections, sit in the same auction and face the same prices and the same luck. The only difference is how they draft. When the auction ends, we play the season out 32 times, playoffs included. Then we run it again: 40 auctions against each kind of league.

• 3kinds of league to draft against
• 40auctions against each
• 1,280simulated seasons against each

Who else is bidding matters, because the rest of the league sets the prices. So we ran it against three kinds of league:

• Last year's prices. Managers who rank players mostly the way Yahoo does, then pay what that rank cost in this league last year: whoever they rank 10th goes for last year's 10th-highest price. Each has favourite categories, and a couple commit to a punt.
• Yahoo's numbers. Managers who pay what Yahoo shows beside every player, somewhere between its projected value and the average price in Yahoo auctions.
• A mix of both.

Pikola won by about the same margin in all three.

Pikola won about three times as many titles

Share of seasons that ended in a title

Pikola against the cheat-sheet manager in the same auctions, 1,280 simulated seasons for each kind of league. The figure at the right is how many times as many titles Pikola won.

Against all three kinds of league, Pikola won 26% to 27% of seasons and the cheat-sheet manager won 9% to 10%. That's 2.8 times as many titles against last year's prices, 3.0 against Yahoo's numbers and 2.7 against the mix, each give or take about 0.2 from luck. That's where “three times the championships” on our homepage comes from.

Small weekly edges become big title edges

Week to week, the gap is modest. Pikola's team won about five or six more of every 100 category battles. But five or six extra category wins per 100 became 16 to 18 extra titles per 100 seasons, because each stage of the season builds on the last.

A slightly better team makes the playoffs more often: Pikola's team got in 93% to 94% of the time, the cheat sheet's 71% to 74%. It gets a better seed. Then it has to win three straight one-week rounds, and a small edge in each round compounds. For scale, an average team in a 14-team league wins the title one season in 14 (7.1%) and makes the playoffs eight seasons in 14 (57%). Both of our drafters beat that easily, because the computer managers price players off last year rather than this year's projections. So don't read the totals as what you'd win in your league. Read the gap.

It holds up when the projections are wrong

Projections are never right. So we replayed the mixed league's auctions and played the seasons out on numbers that were 15% off from the projections both drafters used. Pikola's title rate fell from 25.7% to 21.6%, and the cheat sheet's from 9.6% to 8.4%. Pikola still won about 2.6 times as many titles. Worse projections cost it some titles, not its edge.

Where it breaks: a league that knows more

Now the opposite test. Suppose the people in your league know something the projections don't. Say a player is about to get more minutes, and the league bids that player up because of it. We replayed the mixed league with every player's season pulled halfway toward what the league paid, so the league's prices were half right.

In that world, anyone who trusts projections is the one overpaying for the wrong players. Pikola won the title in only 3.4% of seasons and made the playoffs 42.8% of the time, below par (57%). The cheat sheet did worse still, at 0.4%. Pikola can't know what the projections don't. If your league is sharper than the projections, no projection-based tool will save you.

The full numbers and method

Titles, playoffs and weekly wins, room by room

Championship, playoff and win-rate margins by room

League	Pikola titles	Cheat-sheet titles	Title gap	Playoff gap	Weekly win gap
Last year's pricesrank like Yahoo, pay last year's price for that rank	26.4%	9.4%	17.0 ± 2.0	19.3 ± 2.1	4.9 ± 0.4
A mix of bothhalf last year's prices, half Yahoo's numbers	25.7%	9.6%	16.1 ± 1.6	21.6 ± 2.2	5.6 ± 0.4
Yahoo's numberspay between Yahoo's value and average price	26.6%	9.0%	17.6 ± 2.2	22.4 ± 2.6	6.1 ± 0.5
mixed, 15% projection noisestress test: the mixed room's auctions, seasons played on projections wrong by 15%	21.6%	8.4%	13.1 ± 2.7	22.4 ± 4.9	5.4 ± 0.9
mixed, the room half right about pricesstress test: the mixed room's auctions, production pulled halfway toward the room's prices	3.4%	0.4%	3.0 ± 0.7	27.7 ± 3.8	4.1 ± 0.6

Titles are the share of 1,280 seasons won. Gaps are Pikola minus the cheat sheet in percentage points, paired by auction, ± one standard error across the 40 auctions. Pikola made the playoffs in 93.0% of seasons against last year's prices, 94.0% against the mix and 93.0% against Yahoo's numbers; 88.8% with 15% projection noise and 42.8% in the room half right about prices. In those two stress tests the playoff and weekly win gaps held: 22.4 ± 4.9 and 5.4 ± 0.9 with noise, 27.7 ± 3.8 and 4.1 ± 0.6 with the room half right.

How the test works. Every auction has 14 teams drafting 13-man rosters from the same player pool and projections. Pikola and the cheat sheet sit in fixed seats, so neither gets a better position or nomination order. The cheat sheet ranks players by the sum of their z-scores across the nine categories and pays last year's price for that rank. Each room is tuned so that the second-keenest bidder, the one who sets an auction price, pays last year's price for a player of that rank. Ties in the auction and the standings are broken by seeded coin flips; a tied playoff week goes to the higher seed, as in Yahoo. Each finished auction is played over 32 seasons of head-to-head category scoring with playoffs. Each ratio's error, about ±0.2, is the title gap's standard error divided by the cheat sheet's title rate. Across the three rooms built like real leagues, a five-to-six point edge in weekly category wins became a sixteen-to-eighteen point edge in titles.

Pikola is a Chrome extension that advises live in Yahoo fantasy basketball drafts. Try it free in a mock draft · Should you punt from pick one? · More Draft Science