Deconstructing a Model (APM)

A great breakdown of the flaws of adjusted plus/minus.

about 1 year ago MrPants 15 comments 3 recs |

Story-email Email Printer Print

Comments

Well that's a lot to digest

It’s good analysis, but I don’t think I agree with the conclusion.

Pat Riley is the devil.

by Poloplaya14 on Mar 5, 2011 10:53 PM CST reply actions

I couldn't digest it.

I’ve read Tolkien books less sophisticated.

Road Warriors East Coast Takeover Assassination Missions:

Wizards: Mission Accomplished. MIssed the game, but looks like a good all-around effort... except from Bogans.
Hawks: Mission Scratched. This game never happened. Don't try and tell me it did, you bastard!
Magic: Mission Accomplished. Omer=Tseng, Hedo=Cissnei.
Heat:

by Dr. Handsome, D.D.S. on Mar 5, 2011 11:41 PM CST up reply actions

He basically destroys the argument for adjusted-plus minus. It made me rethink a lot of what I have thought about APM numbers.

by fundamentallysound on Mar 5, 2011 11:55 PM CST up reply actions

I felt the same way iniitally

But then I thought, is it necessarily a bad thing that the model involves tweaking the coefficients for statistical and pure +/-? The author paints it as cheating to achieve a higher R-2 value, but I think it might be justified.

Here’s the way I think of it. First off, using box score statistics is an effective tool in evaluating players, but because there is so much to the game of basketball that isn’t captured by the box score, such analysis falls well short of completely describing a player’s worth. However, for some players, those who primarily contribute in ways accurately measured by the box score (namely scoring and rebounding) it’s a fairly close measure. For other players, those who contribute in ways that aren’t accurately measured by the box score (i.e. good perimeter defenders who don’t rack up many steals/blocks), stats like PER fall short.

When calculating APM, statistical +/- is based on box score statistics while pure +/- is based simply on whether or not a player’s team does well when he’s out there. Going back to what I said before, some players are accurately represented by box score statistics while others aren’t. So, doesn’t it make sense to weigh statistical +/- (the box score side of things) and pure +/- (the “everything that isn’t in the box score” side of things) differently for each player? And if we’re going to do that, shouldn’t we do so in a way that maximizes the correlation between the model and the observed results?

So that would be my best attempt at defending the way APM is currently calculated. I’m gonna reread this article again tomorrow though and rethink my position.

Pat Riley is the devil.

by Poloplaya14 on Mar 6, 2011 12:33 AM CST up reply actions

Ew, no.At least ... I don't think so.

If you don’t know whether the box score stats or general plus-minus stats are better, just cherrypicking which stats are used so that it correlates better to a predictive model is not a great idea. With that rationalization, you could make up any statistical model you wanted, add an adjustment to make it accurate, and then claim that the model was relevant.

I only post cotton candy. Because it's delicious.

by Prevenge on Mar 6, 2011 4:00 AM CST up reply actions

Your understanding of the process is off

There’s no cherrypicking involved. The process for assigning weights to the stats isn’t done arbitrarily. The whole basis behind regression analsyis involves finding a model that minimizes the difference between the predicted and observed values. In the case of APM, the model is mathematically determined in a way such that the individual players’ +/- values add up and line up with the teams’ total +/- as best as possible.

Pat Riley is the devil.

by Poloplaya14 on Mar 6, 2011 4:13 AM CST up reply actions

So it's not done arbitrarily, it's just done so that the stats look like they're accurate?

That’s arbitrary enough to me … especially here, where the weights change from year to year and player to player. There appears to be no rational explanation for the weight except that it makes the data hew closer to expected values, whereas a good model would already produce statistics close to expected values.
But I’m extremely skeptical of regression analysis as a whole, especially in a field where samples are fairly small and can be affected by any number of ameliorating factors [where the adjustments for these factors are basically guesses]. So … eh.

I only post cotton candy. Because it's delicious.

by Prevenge on Mar 6, 2011 4:42 AM CST up reply actions 1 recs

Well the weighting of stats part is a little sketchy

It does make for a more comprehensive model, but at the cost of a great deal of consistency. It’s not dine arbitrarily though. It’s done using a mathematical algorithm to fit the data as best as possible.

And everything done up until that point (which is where the actual regression analysis is done) is completely valid.

Pat Riley is the devil.

by Poloplaya14 on Mar 6, 2011 6:16 AM CST up reply actions

Maybe, but probably not

When calculating APM, statistical +/- is based on box score statistics while pure +/- is based simply on whether or not a player’s team does well when he’s out there. Going back to what I said before, some players are accurately represented by box score statistics while others aren’t.

Remember that the whole point of calculating APM is to get at the value box score stats don’t get at. Now, as you say, box score stats are better predictors for some players but not others. They will generate a predicted value for some guys that will be close to their value, and others will be far away. The technical term for this is heteroscedasticity. The variables have differing variance (dispersion from the mean) against different predictors.

Adding these box score vars in, because they accurately work for some players, raises the overall predictive power of the model (it’s r2). However, it comes at the cost of increasing the standard error (due to this increased variance) around the predictions. This is exactly what we see with most APM results. You typically get a number that’s extremely noisy, even for many obviously good players.

Hence, the end result isn’t all that great. You don’t get a statistically significant, stable result for most players, and you especially don’t get one for the players that were “hard to measure” using box score stats. Adding the stats doesn’t let you explain much more than you could have explained without appealing to APM.

Think of it like this. Suppose you’ve got 10 players you’re trying to figure out. 5 of them you’ve got a pretty good idea about via the box score stats (maybe 2 of them are really terrible and 3 are obviously great). 5 of them are kind of a mystery. If you do +/- on the 10 as a whole, you get a generic, confusing response for almost everyone. If you then add in the box score results, you it will maybe provide you with more clarity about the 5 that the box score already told you about, but it will leave the mysterious 5 just as mysterious as they ever were.

BullsTwo > Back up and running!

by Sports2 on Mar 6, 2011 11:13 PM CST up reply actions 4 recs

Another problem with plus minus- the data is inaccurate.

To give a non-controversial example, look at the rookie-soph game play by play and the box score. Notice that Stephen Curry is -3. If you read the box score though, by any logical interpretation, he should be plus -2 (hint: look at the sequence at 5:28 left in the first half). Curry goes -1 by coming into the game in the midst of a foul shot situation. He wasn’t on court when the foul was committed, but gets dinged for the point.

This is a problem for two reasons. First, fouls and substitution sequences are not random events. They’re highly correlated and a systematic calculation error in the plus-minus aggregation routine will create inaccurate data.

Second, a point or two a game makes a big difference, and the way this data is usually presented makes it that much worse. Notice in that game, Curry player just under 28 of the 40 minutes. So his on court (as calculated by the NBA) was -3 and off court was -5. On a per 48 minute basis (the sort they do on 82games), that’s -5.2 on court and -19.9 off, for a net of 14.7.

If you correct for the mis-allocated point, he was -2 on court, and his off court is -6. Per48 minutes, that’s -3.4 and -23.8, for a net of 20.4.

In other words, a single freaking point (which are regularly miscalculated by the NBA) leads to nearly a 6 point difference in the raw +/- value you get out of the situation. What’s one measly point? Well, it’s a 33% error in that -3 data point we’re starting with. Not exactly a solid foundation for analysis, is it?

BullsTwo > Back up and running!

by Sports2 on Mar 6, 2011 11:43 PM CST reply actions 6 recs

Wow great point

I never thought of that.

Pat Riley is the devil.

by Poloplaya14 on Mar 7, 2011 12:52 PM CST up reply actions

very good insight

so why wouldn’t the NBA make the simple adjustment of assessing all foul shot points to the players who were on the floor when the foul was committed? Seems like an obvious fix.

by EuroBullsFan on Mar 8, 2011 10:36 AM CST up reply actions

My guess would be that the NBA doesn't actually spend much time poking around databases

The teams themselves might, but probably any team that notices it might prefer to re-do the data collection themselves rather than alert everyone else to the problem.

I dunno whether many of the APBR folks who use the data have thought much about it. Arturo (the guy writing the original post) actually alludes to concerns about the data, but he’s also asking for a much broader array of info to be collected as well. I think lots of other folks have just been so happy to get their hands on numbers like this that they’ve not gone and looked at it in great depth.

BullsTwo > Back up and running!

by Sports2 on Mar 8, 2011 11:02 AM CST up reply actions

Another related issue = offense/defense substitutions

I wonder how they might skew the results for plus/minus. If a coach actively substitutes late in games, then the better defensive player would seem to be badly shortchanged. Plus/minus presupposes a 50/50 split in offensive and defensive possessions for each player in order to be objectively fair. A player’s raw +/- cannot improve when the other team has the ball, and it cannot decline when his own team has the ball.

If a coach actively substitutes based on game situation, the better defender might play 55% of his total possessions when the other team has the ball, giving him a superficially low +/-. The better offensive player’s +/- would then be superficially high. I am not sure which adjusted +/- measures, if any, account for this, but raw +/- certainly would not.

by from the window to luol on Mar 8, 2011 11:47 PM CST up reply actions

Another excellent point

I noticed Arturo Galletti’s aproach, if I read it correctly, would partially solve this problem because he throws out any stints of less than 2 minutes

BullsTwo > Back up and running!

by Sports2 on Mar 9, 2011 8:17 AM CST up reply actions