Fans, CPM, and Academic Rigor

A while back, I got pretty interested in something called “Player Efficiency Rating” (PER). Being a fan of the NBA – and in particular the Lakers – I’m interested in the recent shift across some clubs to a more stats-based approach. It turns out that the teams that use “advanced statistics” to make personnel decisions and to inform their players before each game do significantly better than those that do not. “Advanced Statistics” are things like PER that use an algorithm to bring together multiple individual metrics into a single number. There are plenty of criticisms to this approach, many of which focus on the way the algorithm is built.

Just a couple of weeks ago, I looked at teams that have stats people integrated into the decision process. (Boston, Cleveland, Dallas, Denver, Houston, Oklahoma City, Portland and I may have included Orlando — I’m not certain what they do exactly.) It was seven or eight teams. They had won 60% of their games, and that’s counting Houston, which has only won half their games because they’re missing Yao Ming and Tracy McGrady wasn’t playing.

The teams that don’t have quants won 40-some percent. And it was pretty linear … the more or less they had someone integrated into their decision making, the more or less they were at the extremes of winning and losing. [Emphasis mine]
TrueHoop – The State of Basketball Analysis

I’ll reiterate the main point for emphasis: NBA teams that use a stats-based approach win 60% of their games, while those that don’t win 40%.

But I guess *my* point is not that stats-based approaches are always the way to go, but rather that if you’re going to take a stats-based approach, it’s important to really think hard about how (and why) you’re using data.

Which leads me to a quick dissection of the state of nerdery in basketball today:

That, friends, is the formula for PER. It takes all the valuable individual metrics for a player (which have been recorded since around 1950) and rolls them up into one number that represents the total contribution of a single player per minute of each game. There are plenty of other ways to judge players using advanced metrics, including Adjusted +/- (how the team does when a player is on the floor versus off), Rebound Rate (just what it sounds like) and Wins Added (also what it sounds like).

And while I won’t bore you with an explanation of every portion of the algorithm, you can see that it takes into account good things (points scored and how they’re scored, blocks, steals, rebounds, assists) and bad things (turnovers, shots missed, fouls), and includes the team’s pace-of-play in comparison to the overall league. Each of the “contributions” is weighted to essentially create an index where the “average” player in the NBA has a PER of 15.

When you anecdotally compare players’ performances to their PER numbers, things start to make sense:

Only 14 times has a player posted a season efficiency rating over 30.0. All of them are between 30.23 and 31.84. Michael Jordan leads with four 30+ seasons, with Shaquille O’Neal and Wilt Chamberlain having accomplished three each, and LeBron James, David Robinson, Dwyane Wade and Tracy McGrady having accomplished one each. The 2008-2009 season was unique in that it was the only season in which more than one player (LeBron James and Dwyane Wade, with Chris Paul just missing the cut with a PER of 29.96) posted efficiency ratings of over 30.0.
Wikipedia – PER

I bring this up now because there’s a lot of talk lately of the value of a “fan” on Facebook (or rather, the value of a “liker”), and the analysis seems pretty shallow. Compare PER (judging the value of an NBA player per minute of play) to the formula proposed by Vitrue and Edelman for the value of a “fan”/”liker”:

Don’t get me wrong: there is absolutely value in a fan relationship that can be measured by looking at the free impressions that relationship creates. And while it’s nice to have a low-level algorithm to determine that component of a fan’s “value”, I think we can do a whole heck of a lot better.

To me, judging fan value by how many free impressions they can create seems like judging an NBA player only by their free-throw shooting ability. Or even by how fast they can run end-to-end on a court. Or in a pretty good case by their PER. In all cases, there’s a much bigger picture to take into account.

Take into account this bit from a fascinating article in the Times Magazine from last year, about analyzing the performance of players like Shane Battier, the “No Stats All-Star”:

There are other things Morey has noticed too, but declines to discuss as there is right now in pro basketball real value to new information, and the Rockets feel they have some. What he will say, however, is that the big challenge on any basketball court is to measure the right things. The five players on any basketball team are far more than the sum of their parts; the Rockets devote a lot of energy to untangling subtle interactions among the team’s elements. To get at this they need something that basketball hasn’t historically supplied: meaningful statistics. For most of its history basketball has measured not so much what is important as what is easy to measure — points, rebounds, assists, steals, blocked shots — and these measurements have warped perceptions of the game. (“Someone created the box score,” Morey says, “and he should be shot.”) How many points a player scores, for example, is no true indication of how much he has helped his team. Another example: if you want to know a player’s value as a rebounder, you need to know not whether he got a rebound but the likelihood of the team getting the rebound when a missed shot enters that player’s zone. [Emphasis mine]
Times Magazine – The No-Stats All-Star

I’m not arguing that the creators of the “Fan Value” metric should be shot, but none of us should feel good with an impressions-based valuation of fan relationships. Instead, I’d argue that each business needs to figure out (as basketball nerds already have):

  • What constitutes a “win”
  • Who/what contributes to those wins
  • How those contributions are made
  • and Who else competes for those contributions

Only then can you figure out the value of a fan.

Or not. Thanks to Mike and Alex for looking at this pre-post.

Comments

10 Comments so far. Leave a comment below.
  1. I’m wondering if comparing the NFL with Facebook Fan Values makes sense? The NFL has been struggling to find metrics to measure and evaluate players (and still hasn’t found any to my mind) and I feel that we’ll struggle to find some clear metrics in the “Friend/Fan” space?

    A quick article about the NFL and player evaluation: http://scienceblogs.com/cortex/2010/04/the_nfl_draft.php?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+scienceblogs%2FwDAM+%28The+Frontal+Cortex%29&utm_content=Google+Reader

    Just a thought to add to the discussion

  2. Clay,

    Excellent post and I appreciate your analysis. Just one clarification though; the “fan value” formula was calculated by Virtue. The post I wrote was just my observations and point of view on the formula.

    Oh, and as an FYI, I am NOT a metrics guy and I suck in math.
    In reading your post though, I find it difficult using the “PER” as an example (or illustration) on how to possibly calculate fan value for a variety of reasons. I do believe and agree that it needs to be more than just impressions but it’s a decent start.

    • There is a finite time the game is played, 60 minutes w/ a ton of historical data. There are no outliers or behavioral issues (unless Artest gets kicked out of a game; and even then, one game of not playing won’t change his metrics)
    • There are no assumptions made. Through observation I know Nash’s APG, Kobe’s PPG and Howards RPG so I can safely assume using the data of how many assists, rebounds, points each will make in future games based on averages
    • It’s a controlled environment and the coach is telling them which plays to run
    • I had a few more reasons but I forgot them…whatever

    Assigning monetary value to a consumer is difficult, if not impossible. In some cases, like the NBA Facebook fan page, they are posting Bit.lys (tracking) directly to their e-commerce store so their fan value is based on purchase, not impressions. Also, consumers’ attitudes, behaviors change all the time. They like brands, they hate brands, they try new brands … they buy online, offline, sometimes repeat purchases, sometimes not. It’s impossible to get an exact “value” on any consumer because they are all over the place. I am sure this will change once Facebook builds in their e-commerce capabilities.

    Oh and one last note …. Despite what any data/metrics/stats /rings say, I believe in my heart that Magic Johnson is a greater player than Jordan. I am sure my Edelman colleagues in Chicago won’t appreciate that, but whatever….and, there is nothing you or they can say or do to change my mind on that one either!

    : )

  3. @ Josh Premuda

    A note on the word ‘clear’ in marketing metrics:

    I’m not sure why the lack of a ‘clear’ metric would stop anyone from trying to create a scoring system. Or, on that note, why we’d assume that clear equates to correct. What PER provides is a basket of metrics to evaluate performance in a nuanced way. The scoring system is fallible and only works in concert with an evaluator’s qualitative judgement.

    We can’t rely on data to make decisions for us – especially without the quantitative chops to create predictive models and the limited access to information about our fans. (We can thank Facebook’s shitty insights for that.) We’d only struggle to find something ‘clear’ if our goal was to make causal relationships between the data and consumer value. But we can’t rely on figures for that. Not in isolation.

  4. First, thanks all for commenting.

    Starting from the top:

    The struggle that the NFL faces is perhaps similar to the struggle marketers face; in both cases, it’s hard to look at surface-level numbers and judge effectiveness. Case studies include a whole truckload of quarterbacks and and a few others. In the NBA, there’s some new data that suggest playing time (early in a career) may be an important determinant for future success. Most of the players in any sports league are within a few percentiles of each other in terms of ability, smarts and raw talent, and the opportunity to get reps on a *huge* stage would seem to be a huge factor.

    Moving on…

    None of these equations make any sense for marketers/digital strategists to use as a 1-to-1 foundation for judging the value of a fan.

    However…

    I’m interested in (potentially) broad valuations that are tied to valuable outcomes.

    For basketball, it’s clear which contributions result in a win for one team. Score lots of points, don’t miss shots, don’t foul, and don’t turn the ball over. PER weights all the possible, measurable contributions and turns those contributions into a single, digestible number.

    It’s important to note: this may not be possible for fan value. And that’s okay.

    What I’m calling for, though, is a more nuanced look at fan value. Acquired fans are “worth” some free impressions, but they also may be worth *other* things that are valuable for a business: higher lifetime value, higher likelihood to share positive experiences with a brand, etc. And my contention is that it’s important for each brand to (1) figure out what’s valuable, (b) figure out how fans contribute and (3) figure out how to evaluate what those fans contribute.

    It is absolutely more complex. It’s definitely harder. But it’s a lot better than just looking at impressions.

  5. @ Alexander Chung

    I completely agree with you. My choice of the word “clear” may not have been the best. I was simply trying to make a comparison that the NFL (and marketers) needs to get more rigorous in how they measure and evaluate talent (and likers).

    Great post, Clay.

  6. First off, this post was fantastic. I guess I’m not a big enough basketball fan, because I had never heard of PER until today. So thanks for that.

    In a basketball game, the ultimate goal, or metric, is a win. It is a closed system. Players play the same game, in similar environments, in a world completely controlled by rules. The PER statistic is extremely valuable because the statistic is ecologically valid.

    Each Facebook Fan page is a unique beast. Each brand is unique. Hell, every Facebook user is influenced by numerous factors that make determining any sort of a causal relationship nearly impossible. Maybe you “un-liked” the Clorox fan page because your friend Becky told you that bleach gives you cancer. Maybe you saw a billboard for a cookie company and then later in the day saw one of your friends “liked” their Facebook fan page, inspiring you to join.

    Maybe basketball statistics are more valuable because we better understand the behaviors that end in a “win”.

    If the ultimate goal of a Facebook fan page, for a business, is to generate revenue we need to better understand what drives behavior before we can place a set value on a fan.

    So what can we measure in terms of Facebook fans, before we better understand the behaviors that create this outcome? Cost per fan? Qualified leads driven from the Facebook fan page over time? Maybe we could even go so far as to analyze the activity of each fan and place a higher value on those who engage more often and in more productive ways?

    These are just some (unoriginal) ideas. Thanks for the awesome post, excuse the rambling.

  7. I like this. I am not an NBA fan, never watched a single game in my life and this post is the first time I’ve heard the term PER – but it makes sense to me.

    Over the past few weeks, I have had a tough time trying to understand why so many people are trying to figure out a way to assign a CPM based number for the facebook fan. The direction you steer that metric makes more sense. Multiple variables, each variable has a certain weight (the different weights are relative of course). But I think your way of defining the value of a fan can go beyond being a number. Done right, it can do wonders for a brand’s social strategy. If I’m a brand and I have more than a million fans on a social network, it is next to impossible to converse with or even listen to all of them – leave alone putting it all in the right context. I think what a well defined “fan value” metric might help in is figuring out who are the most valuable fans. Add some text mining in there, and you’ve got context too.

    Although, I think truly quantifying fans will need more variables – from multiple disciplines – from technographics to web tracking. But this is definitely a place to start and the four basic criteria you’ve listed make sense.

    Great post.

  8. Eric,

    The devil’s in the details; the word “value” is too subjective in this case. For Vitrue, “fan value” is very much that straightforward, literal valuation – which is perfectly fine (I think). Media impressions make sense as a valuation if you put value in media impressions.

    But if you put value beyond simple fan-ship and the impressions it garners, than something like PER makes perfect sense. To Clay’s comment-point (“None of these equations make any sense for marketers/digital strategists to use as a 1-to-1 foundation for judging the value of a fan.”), I agree – to an extent. Vitrue has put together a model that fits their needs (it seems), and that’s fine. If we can come up with something more rigorous and refined – digital’s version of PER (i.e. time of day of wall posts) – I think it would immediately provide more *value* to the broader marketer/digital strategist world. (Bill James has convinced me we could quantify and valuate something like wall post color scheme.)

  9. Sam Chotiner,

    Clay- it should come as little surprise that I agree completely with what you have said here but I did want to add one point on top of Eric’s comment.

    The reason that it is so much harder to truly value fandom is that we are not even sure what it is that we are attempting to measure/value. In my mind the big difference between vFan and PER is that PER (and sports in general) rely on numbers and statistics however with fandom we seem to be trying to quantify sentiment. Though possible, I am not sure if one can quantify sentiment solely from facebook fandom alone. The motives for why I am a fan of ‘target’ vary incredibly (I love their products, I want to stay informed with their updates, I want potential discounts, I did it as an inside joke with a friend) and matter so much more than the motives for why a Lamar scored 31 points (his father died and he is playing for him, he is playing in his home city for the first time, Kobe was hurt and he thought he would have to pick up the slack). With facebook it would seem important to understand the motivations for fandom as it is those very motivations that we value. As it stands vFan seems like an incomplete (and shallow) form of measuring sentiment as it makes no attempt to account for consumer motivation.

    PER is a fantastic tool because it has a deep data pool that allows it measure value but it is equally impressive because it is able to measure relevant value. In this way, vFan is doomed by its shallow data pool AND by the fact that it doesn’t address sentiment (the very thing it aims to make a claim about).

    At least so I think…

  10. So why haven’t marketers replicated the analytic stat rigor that we find in most major sports (PER in baseketball, slugging % in baseball)?

    It’s easy in sports–everyone has the same common goal: winning. The holy grail of metrics can’t exist online because everyone has different objectives. Brand building. Converting. Awareness. Sharing. You need a different metric for each of these goals.

    And while PER maybe be comprehensive, how many coaches do you think pay attention to stats like PER? How many coaches even pay attention to +/-? Similarly, I feel like even the greatest marketing metric would be ignored by management if it’s too complicated or inactionable. Hell, most of my friends don’t even understand PER, and they are Econ majors. Creating a stronger metric for Facebook may just be academic gobbley-guck if it resembles anything like Hollinger put together.

Add Your Comments

Disclaimer
Your email is never published nor shared.
Required
Required
Tips

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <ol> <ul> <li> <strong>

Ready?