Showing posts with label Referees. Show all posts
Showing posts with label Referees. Show all posts

Tuesday, August 14, 2007

Biased Towards Bias 2: The Umpires Strike Back

As much as I wanted to name this post “Referee Bias 2: Electric Boogaloo,” I thought I’d go for the incredibly obvious wordplay instead. Substance and parody over randomness, I guess. Let’s move on.

A few months ago, the New York Times featured a “juicy” study concerning the racial biases of NBA referees. Naturally, everyone went apeshit over the idea. I mean, how could they not? After all, the halo of authenticity provided by academic study gives you a free pass to run with the sexiest of all headlines:

NBA REFS: RACIST?

Obviously, I exaggerate. Still, it’s hard to deny that talking heads love the idea of research backing up stories on topics as volatile as race. University! Academia! STATISTICS! It’s all just so sexy… a columnist’s wet dream. Problem is, people rush to conclusions without really getting to the bottom of what the research says.

Still, you might be wondering why I’m talking about the NBA Ref study in August. Well, history tends to repeat itself: It’s baseball’s turn to share the headlines.

And here we are again. Another study, another claim concerning inter-racial call biases. Unfortunately, it’s probably going to prove to be another example of exaggerated columnist fodder.

So, like I did last time, I will throw out the following disclaimer: I am not an expert. I am not a statistician. I’m a former research assistant in psychology who decided to jump ship to the much more exciting world of trading. This qualifies me for…well, dick. And man did that sentence sound better in my head. Point is, I’m just a guy with some basic stats background. Do with this information what you will.

Anyway, unlike the NYT article (which provided a link to the actual NBA ref study), I had to do a little digging to find Hamermesh’s study/results. The study’s design is actually pretty interesting, and I found their comparison of QuesTec to non-QuesTec umpire calls fascinating – I like the idea of comparing ump decision making with and without scrutiny. Regardless, I spent most of my time going over their data and conclusions. Here’s a quick recap:

“The analysis of individual pitches and game outcomes suggests that baseball umpires express racial/ethnic preferences in their decisions about players’ performances. Pitches are more likely to be called strikes when the umpire shares the race/ethnicity of the starting pitcher, an effect that becomes significantly stronger when umpire behavior is less well monitored. The evidence also suggests that this bias is strong enough to affect measured performance and games’ outcomes. As in many other fields, racial/ethnic preferences work in all directions—most people give preference to members of their own group. The difference in MLB, as in so many other fields of endeavor, is that power belongs disproportionately to members of the majority—White—group (Hamermesh, 19).”

In studies with claims like these, I try to immediately look for the common test – the comparison across comparable conditions that truly shows me a difference BETWEEN groups. This is an important point if you’re going to claim bias; showing me that white umps call more strikes for white pitchers is only bias if, say, black umps AREN’T doing the same. If both seem to call more strikes for white players, there are questions to be asked, but they aren’t questions of racial bias. Table 2 (Page 26 of the PDF) has the basic percentages of pitches called for strikes by ump/pitcher ethnicity.

Just looking at the percentages, there’s some variability between umpires by race… but then you notice the number of total pitches analyzed by each umpire ethnicity. Holy mother of discrepancy! Here’s a simplified table from page 26, including only called pitch data and some simple sums:


Um. This looks like a data problem. A quick review of the study’s first table (which I should have spent some more time on initially) reveals that our umpire sample size is 93, and a whopping 85 of the 93 umps tracked are white. 5 are black, and 3 are Hispanic. And right here, at least in my mind, you can throw out the study’s results, regardless of how elegant the rest of the paper may seem.

You cannot make sweeping statements of race and racial bias with subject groups this small. There is nothing to infer. Sure, you might have enough total pitches viewed by umps of all ethnicities to generate a statistical comparison that looks legitimate… but, at the end of the day, you’re making key assumptions about racial attitudes based on the work of 3-5 people. An experiment/study’s conditions need to be comparable in number. You wouldn’t compare the averaged IQ tests of 85 students to the average of 5 other students and expect to gain any kind of brilliant insight. It’s the same deal here.

Here’s a nice FAQ about the study as composed by the authors. They address a lot of issues, including the point that their study does NOT make any assumptions concerning the consciousness of the ump’s bias. It’s worth a read, but it doesn’t address my point concerning their umpire sample sizes. In fact, they assert the opposite:

The basic idea behind these tests is that, because randomness is completely unpredictable, its average effect will diminish as the size of the sample increases. An effect that persists despite a very large sample, as ours is, almost certainly is not random. All our results used standard methodology to account explicitly for the possible role of randomness (Hamermesh FAQ, 1).

The problem is that, while the sample of pitches is quite large, the umpire sample is not. 3 umpires can’t possibly make for a full comparison group. It’s that simple.

Look, I’m not here to try and deny that racism (or racial disparity) exists in the world of sports. If anything, the study’s presented ethnic breakdown of umpires and players emphasizes the discrepancy in baseball’s racial composition. That, however, does not mean we need to jump to conclusions about racism and its effect on the game – a study does not a reality make.

But hey, if Sportscenter needs a headline, I’ll just generate a few to make for good controversial conversation and general screaming:

Tour de France Riders: Terrorists?

NFL Refs: Baby-rapists?

NASCAR Drivers: Necrophiliac-Cult Leaders?


Talk amongst yourselves.

Are baseball Umpires Racist?

Strike Three: Umpires’ Demand for Discrimination

FAQs: Strike Three

Biased Towards Bias: What’s With Those NBA Refs?

Thursday, May 03, 2007

Biased towards Bias - What's with those NBA Refs?

The big non-playoff news in the NBA yesterday was the announcement that there may be a racial bias in how NBA officials referee games. Naturally, the minute that the words “race,” “bias,” and “sports” hit the front page of the New York Times, the sports media collectively had an orgasm at the thought of a juicy racial scandal to discuss. In short, the Times’ piece describes an upcoming study by a University of Pennsylvania professor which points to a racial disparity in fouls called during NBA games.

Oftentimes, the root of such potential controversies are sports columnists looking to stir the pot, but this story emerged - and was promptly leapt on - because of a certain authenticity associated with an academic conclusion (as opposed to some blabbering idiot’s opinion). Here’s the problem: while an academic study isn't the equivalent of a yammering columnist blindly pushing a race issue, it still seems like the media is flouting a finding for the sake of scandal-mongering. The main issue is that the media is running with the same headline and synopsis of Wolfers' study that the New York Times presented… but why isn't anyone delving in to the actual data? The immediate reaction has either been to embrace the study or point out superficial criticisms that don't apply when reading the actual study (i.e. assuming that the study is stating the obvious because there are more black players in the league when, in fact, that is accounted for).

I don't claim to be an economist/statistician – in fact, I left academia a year ago to avoid dealing with these kinds of studies in the first place - but I wanted to at least TRY and understand the findings. Why simply take a summary at face value? The first step to interpreting a finding is determining what went into the study in the first place, and it seems like most of the media threw an academic halo around the results without digging deeper.

Admittedly, I can't parse out all of the statistical data and manipulation because some of it is a bit dense and over my head, but there are some fundamental criticisms I took away from the study:

  1. This isn't a criticism of the paper itself, but it's important to remember that this has NOT been published or peer reviewed. Warrants mentioning, because the people most qualified to interpret an academic paper are peers in the field. Personally, I’d like the counterpoints to be available before an orgy of media attention.
  1. The study uses box scores to collect their referee/foul/player data. This means that individual calls are impossible to discern, so unless the officiating team was comprised of one ethnicity, it's hard to tell who's actually making the call. So while a scenario with an all-white or all-black team of refs is possible, it certainly isn’t the most prevalent combination. The box-score data compilation also removes ALL context of fouls, intentional or otherwise.
  1. Most importantly, the heart of the data (which seems to then be extended and extrapolated into more detailed observations with more complicated analysis) does not seem to support the synopsis given by the Times.

Here’s how the finding is summarized by the Times:

"A coming paper […] says that, during the 13 seasons from 1991 through 2004, white referees called fouls at a greater rate against black players than against white players."

But let’s take a look at claim a bit closer, shall we? Here’s a PDF of the study manuscript itself.

What caught my attention was the information presented in Table 3 (Page 36 of the Study). My impression is that most of the study’s farther reaching analyses are based on this initial presentation of foul rates. Simply put, the table shows the difference in rates of fouls called on white and black players by white and black referees. The term foul rate is defined as a player's fouls earned per minute times 48 (48*Fouls/Minutes Played).

Here’s what’s striking about the data in this table. First, there is NO difference in the foul rates for black players between black and white refs (4.33 vs 4.329). There IS a difference in the foul rate called against white players (4.954 called by White Refs, 5.023 by black refs). The second, more detailed part of Table 3 (which measures these changes by the percentage of white or black refs on an officiating team) shows the same finding.

This leads to one of two possible conclusions: either white refs favor white players by calling fouls less frequently, or black refs call more fouls on white players. It is IMPOSSIBLE to determine which one is actually at play.

In the authors' own words (PDF page 30):

"There are also two ways in which these own-race biases may emerge: they may reflect referees favoring players of their own race, or alternatively disfavoring those of the opposite race[…] Table 3 is instructive, showing that the rate at which fouls are earned by black players is largely invariant to the racial composition of the refereeing crew. By contrast the rate at which fouls are earned by white players responds quite strongly to referee race […] suggesting that the impact of the biases we document is on white players, who are either favored by white referees, or disfavored by black referees."

This is a far cry from what the New York Times article presented, but people ran with that interpretation for the story’s sake. In other words, what we may have on our hands is ridiculous case of rabble rousing, which is especially infuriating because it's veiled by a pretend shield of statistics. At this point several people have come out to criticize the study (PTI and ESPN.com's John Hollinger come to mind), but it almost seems like a fundamental piece of information isn't being given the light of day.

I need to reiterate that I am NOT a statistician, so my ultimate goal would be to see an explanation by someone more qualified than I. Nevertheless, I’m simply trying to point out that the very root of this “scandal” shouldn’t have to do with whether white refs call more fouls on black players than on white players; it’s how they call these fouls compared to black refs. This is the logical apples-to-apples comparison, a common test for one group of referees compared to another. If the study’s most fundamental data shows that there is an ambiguous discrepancy between officials calling fouls (i.e. are white refs under-calling fouls on white players or are black refs over-calling them?), why aren’t we discussing THAT finding instead of immediately pointing a finger of blame?

Now that question I don’t have an answer for.

read more | digg story