The mathematics of random walks: skill ratings in Overwatch

UPDATE: Added a new condition below

Overwatch is a very fun team-based shooter that has an interesting skill rating (SR) system. Every player has an SR, and a matchmaking system tries to create fair games by assembling teams with similar SR. The winners get an SR increase and losers get a decrease, but it can be different amounts for each player. A hidden algorithm picks the actual value based on both the expected outcome of a match and individual performance.

For this post, I wanted to look at the contributions of skill vs. chance in the way SR can change over time. After all, players should theoretically only be able to win if they’re more skilled than their opponents, but it’s possible for the matchmaking system to place someone with very good teammates, leading to a win even though that person’s skill is low. The opposite is possible too, where a person should win a game but doesn’t due to their teammates. These two scenarios shouldn’t have an effect in the long run, but it still leads to terms like “Elo hell” (named for the Elo rating system), which refers to the supposed impossibility of climbing out of the lower ranks to the “true” SR. But does Elo hell actually exist?

Random walks

The way in which SR climbs and falls can be described by the concept of a random walk. For a simple example, say you’re standing on a sidewalk and flip a coin. For heads, you take a step forwards, and for tails, you take a step back. Where do you end up after 100 coin flips? 1000? This concept of a random walk can be used to describe things like stock prices and the physics of diffusion.

To put an example in the context of Overwatch, say your SR is right in the middle at 2500 out of a maximum 5000. If you have a 50% chance to win every game and play 300 games over the course of the 90 day season, what is your final SR? How likely is it that you climb 500 SR, which is the amount needed for the next rank? (Overwatch has ranks at different tiers called gold, platinum, diamond, etc.). I tried to answer these questions with Monte Carlo simulations, meaning I simulated a large number of season SR paths and calculated probabilities from the distribution of results.

To demonstrate, here are some simulated SR paths where the probability of winning was a constant 50%, starting from 2500 SR and assigning +25 SR for wins and -25 SR for losses. Each path is has different color in this plot, and the maximum season SR over 300 games is marked with a dot.

You can see each of these SR paths peaked at different times during the season. Since the next highest rank (diamond) occurs at 3000 SR, then only the blue path made it to diamond in this simulation.

If I repeat this process many many times, I can take the maximum SR of each path and get a distribution of season peak SRs. Below is a histogram of this distribution for 100,000 random walks when the win probability was always 50%.

You can see from this distribution that most paths had a peak below 3000 SR, i.e. they did not climb to diamond. But a significant fraction still did (0.25 or 25% in this case). This fraction is what I use below to estimate the probability that a player can climb a complete rank through pure chance. That’s the power of Monte Carlo simulations.

Making the simulations more accurate

One thing to notice in the simulations above is the tendency for the paths to diverge, i.e. if a path shoots up after a string of wins, there’s nothing to force the SR back to 2500. In reality, if a player gets many consecutive wins, they’ll likely find themselves playing with much more skilled players, which should eventually decrease their rank back to the “true” value. The difficult thing to know is whether a climb is due to a true increase in skill or merely chance. In a simulation, though, these are easy factors to separate.

To see the contribution of chance to climbing, I compared three win probability conditions:

  1. Flat – The win probability was always 50%, i.e. a purely random walk.
  2. Shallow – The win probability decreased linearly by 5% for each 500 change in SR
  3. Steep – The win probability decreased linearly by 10% for each 500 change in SR.

So for the shallow and steep conditions, there will be a normalizing force that will tend to drive the SR paths back to 2500 once they start to diverge.

Here’s what these win probabilities look like as a function of SR. The lines are centered at 2500, since that was the assumed “true” rank for these simulations.

Other Technical Details

  • Points per game – In the simulations above I used a constant 25 SR per game, but for the full set below, I made this value another random number. Specifically, it was a normal random number with μ = 23 SR and σ = 4 SR, i.e. 95% of the values ranged from 15 to 31 SR. These parameter estimates are from a personal data set of mine that I recorded from season 2 of about 120 games. I think these estimates are still accurate for more recent seasons.
  • Games per season – Here I used another three conditions: 100, 300, and 500 games per season. Since each season is ~90 days, I think these values span the range from casual to avid Overwatch players. If anything, 500 games might be an underestimate, since I’ve seen streamers with >1000 games in a season, so >11 games per day. That’s a lot of Overwatch.
  • Draws – I decided to leave out draws for the time being, because not only are they rare, but Blizzard is continually making changes in order to make draws as rare as possible. In my season 2 data set, <8% of my games were draws, and this was before Blizzard made their changes.
  • Streaks – Blizzard has a streak system in place to allow players very far away from their true SR to climb quickly. However, the details of this system are unknown, and it was also recently changed to become less significant. Therefore, I decided not to consider streaks.

Results

To demonstrate the different win probability conditions, here is a sample of 50 paths from the Flat and Steep conditions over 300 games. It’s impossible to distinguish individual paths with so many on the same plot, but you can get a sense of the distribution. The steep paths rarely go above 3000 SR, whereas a few of the flat paths do manage to climb ranks.

To estimate the probability that a path climbs a full rank (500 SR), I generated 100,000 paths for each condition and measured the fraction of maximum season SRs above 3000 SR. The starting point of 2500 SR was arbitrary. These results can be applied just as easily for a climb from silver to gold (1500 to 2000 SR) or diamond to master (3000 to 3500 SR). The only place I wouldn’t trust these results are for the very edges, e.g. a climb from 4000 to 4500 SR, since there are so few players at these high ranks that matchmaking becomes a bit of a mess.

Anyway, here are the results. The numbers reported are fractions, e.g. a value of 0.21 means there is a 21% chance of climbing a full 500 SR through chance alone.

There are two main trends. (1) Playing more games always increases the chances of climbing. It seems very unlikely to climb by playing only 100 games. (2) As the slope of the win probability curve increases (i.e. a stronger normalizing force), it becomes more and more difficult to climb through chance alone.

I was surprised by how large some of these probabilities were. Considering that tens of millions of people have played Overwatch, it certainly seems possible that some people have managed to climb ranks without any significant increase in actual skill.

But what if skill actually increases?

I next wanted to consider the cases when the “true” SR does increase, which should theoretically happen over time with more games played. The easiest way to mimic this in my simulations was to shift the win probability curves to a higher SR, so the point at which the win rate equals 50% will increase over time.

I repeated the previous set of simulations for two more conditions.

  1. 500 SR increase – Over the course of 500 games, I linearly increased the true SR by 500 points. This means that for the 100 game season, the true SR only increased by 100 points.
  2. 1000 SR increase – I linearly increased the true SR by 1000 points over 500 games, so by the end of a 500 game season, the true SR will have climbed two complete ranks.

Just to see what these SR paths look like, here is a sample using the steep win probability curve, this time with 5 paths for each of the rank increase conditions. There’s still a significant amount of individual variation, but overall there’s a gradual increase in SR over time, with a larger increase in the 1000 SR condition.

I next recalculated the probabilities of climbing a single rank with 100,000 simulated SR paths. Here are the probabilities for the 500 true SR increase.

The flat probabilities remained the same, as they should since shifting a flat line doesn’t change anything. The probabilities for 100 games also did not change much, since 100 games was not enough time for the true SR increase to kick in. However, there are significant increases in the 500 game probabilities, e.g. an increase from 5% to 62% for the steep condition. Interestingly the steep probability actually surpassed the shallow case (62% vs. 47%). This is counter-intuitive since the steep case allows less variation, but it actually benefits with a true SR increase, since the force driving the SR to the true value is stronger.

Lastly, here are the probabilities for a 1000 SR increase. Note that the probabilities are still for climbing a single rank, even though the true SR increased by two ranks.

Again, the chances for climbing in only 100 games are practically zero, even with this relatively quick increase in skill. I think this supports the claim that it’s quite difficult to climb as a casual player. On the other hand, chances of climbing with more games are much higher, and practically certain for 500 games. I think the most extreme examples of this are the Twitch streamers that do a “Bronze to Grandmaster” stream. These players climb very quickly, but their true SR is much, much higher than their opponents, which makes winning almost certain.

Conclusions

So does “Elo hell” actually exist? It’s hard to conclude anything certain with this data since this is all essentially a glorified thought experiment. But there are a few points in which I’m fairly confident.

  • Due to the nature of random walks, it’s possible (but unlikely) to climb a full rank through pure chance.
  • Climbing through chance alone is highly sensitive to how the win probability depends on SR. Even a 10% decrease at a higher rank can make it virtually impossible to climb with pure chance.
  • If your actual skill increases over time, climbing is much easier, but still not guaranteed. It’s only certain after many games or large differences in SR, e.g. it’s easy to climb to platinum if you actually deserve diamond.
  • Playing more games always increases you chances of climbing. Said another way: it is extremely difficult to climb when only playing casually, even if you get better over time.

Now onto more speculative conclusions: does Elo hell exist? I think for some people, it has. Of the millions of people playing Overwatch, some unlucky players might have legitimately increased in skill but not climbed due to chance. However, I think the chances of this are extremely rare, to the point that nearly every person complaining of Elo hell is actually at their true SR and just venting. For those very few unlucky souls where it’s actually true, hopefully they just keep playing, because they will inevitably climb after more games.

None of these conclusions are probably surprising to those that play a lot of competitive Overwatch, but I still think it’s useful to consider how big a role chance can play in a skill-based game. I was certainly surprised at how high some of these probabilities were. For another perspective on this skill vs. chance trade-off, check out this super interesting video from Vox on professional sports. Or just play more Overwatch. After all, you can’t git gud if you don’t play.

UPDATE:

After getting some feedback, I decided to run one additional set of simulations. Specifically, what happens when you start the season with a true SR at a higher rank, i.e. your true rank is 3000 but you’re trying to climb from 2500. I kept the true SR at 3000 for the entire season and recalculated the probabilities of reaching 3000 with the actual SR.

I also added another win probability condition, called Very Steep, which has twice the slope of the Steep condition, i.e. the win probability decreased by 20% linearly for a 500 change in SR. I think the true shape of the win-probability curve is more complicated than these simple lines, but they do illustrate a spectrum of normalizing forces. In reality, the curve is likely shallow for nearby SR, steep for larger differences, and then very steep near the edges.

Here are the probabilities for climbing when the true SR starts 500 points above the actual SR and remains constant throughout the season.

Interestingly, even with the very steep condition, the chances of climbing after 100 games is still <50%. I believe this very steep condition overestimates the normalizing force (especially near one’s true SR), so keep that in mind.

The story changes quickly with more games, and the chances of climbing are much higher after 300 games and especially after 500 games. This supports the conclusion I made earlier, that even if you’re unable to climb even while getting better at the game, it just takes more time.

The Ship Wars Part II: Another look at the most popular pairings

In my post last week, I took a look at how the most popular pairings in Harry Potter fanfiction changed over time. Specifically, I looked at the rate at which fics were published. This time, I wanted to look more closely at a wider range of pairings, as well as other measures of popularity (i.e. favorites, follows, and reviews).

The data

The dataset is the same from last week, e.g. the descriptive info from all 724,315 Harry Potter fanfics I could scrape from Fanfiction.net. From these, I looked only at fics with either (a) two characters, or (b) fics with more than two characters where pairings were explicitly marked with brackets. This shortens the number of fics to 505,406 consisting of 5,414 unique pairings.

Below are the 20 most popular of these pairings (in terms of number of fics). At the top are Draco/Hermione and Draco/Harry, which have more than 50,000 fics each. To give you a sense of that number, this means that there were, on average, more than 8 fics published per day. For 17 years.

To take a closer look at other popular pairings, I decided to consider the top 100 ranked pairings. These pairing each have more than 500 fics apiece and make up a large majority of all the fics with pairings. Here are some stats for the top 100 pairings.

Therefore, the bottom 98% of pairings only account for ~15% of fics. This may be some serious inequality, but sadly I don’t think anybody’s going to be protesting for more Harry/Stan Shunpike fics. Even though Starry is an amazing portmanteau for a pairing. #OccupyDiagonAlley #MakeStarryHappen.

Pairings with the most favorites

The next thing I looked at were average number of favorites, follows, and reviews that each pairing receives. These average values were highly correlated with each other, so for the sake of simplicity I’m only showing the number of favorites. Sorting the pairings by follows or reviews does switch up some of the rankings, but there is a large degree of overlap.

Here are the top 20 pairings with the most favorites per fic, i.e. the highest average values.

I think that there are a few interesting things to point out

  • Daphne/Harry and Fleur/Harry have ~800 and ~600 fics, respectively, but still have averages of >500 favorites. This is impressive, considering that over 98% of all fics have fewer than 500 favorites.
  • Two of the top 10 most popular pairings made it to this list: Harry/Severus (#9) and Harry/Hermione (#4). This is also impressive, since their average values are from ~16,000 and ~25,000 fics, respectively.
  • Every pairing on this list has either Harry or Hermione in it. Also, many of these are with significantly older men, which is, well, interesting. Of course, fics with pairings may not necessarily be romantic in nature, but I’m guessing a fair amount still are.

Case study: Daphne/Harry vs. Bellatrix/Rodolphus

It may seem like high average values are inflated by a small number of fics with thousands of favorites, and it is true that these distributions are highly skewed, e.g. most fics have only a few favorites. To look at this most closely, below I’m showing the distribution of favorites for two pairings with very similar fic counts: Harry/Daphne (#72) and Bellatrix/Rodolphus (#71). Besides their similar fic counts, I picked Bellatrix/Rodolphus because its average is quite low: ~10 favorites per fic.

This plot is a cumulative distribution plot, i.e. for a given x-value (number of favorites), the corresponding y-value is the fraction of fics with that number of favorites or less. So for a value of 10 favorites, Daphne/Harry has a value of 0.04, meaning 4% of Daphne/Harry fics have 10 or fewer favorites. The value for Bellatrix/Rodolphus is 0.8, so a full 80% of fics have 10 or fewer favorites. Also, note the x-axis has a log scale, so the values range from 1 to 10,000.

You can see the distribution of favorites for Daphne/Harry is shifted to the right (i.e. it has more favorites). Around 99% of Bellatrix/Rodolphus fics have <100 favorites, whereas only 40% do for Daphne/Harry. In fact, ~20% of Daphne/Harry fics (~170 fics) have more favorites than highest-rated fic for Bellatrix/Rodolphus (634 favorites). So there are many highly-rated fics for Daphne/Harry that help to increase its average.

Why are certain pairings so highly rated?

There are many valid answers for why one pairing might receive more favorites/follows/reviews than another. There’s always the obvious one: readers probably enjoy some pairings more than others, at least enough to favorite those fics more often (I certainly do). But there are two other aspects I want to explore: length and timing.

Possibility 1: Length

This one is fairly self-explanatory. Across all fics, there is a significant correlation between length (in words) and the number of favorites. This also holds true when considering the average values for these top 100 pairings. So below I’m showing the top 20 pairings by average fic length. There is significant overlap between this list and the one sorted by most favorites.

As a sidenote, I think it’s interesting to point out that half of these pairings are OC pairings (original characters). I wonder if OC fics tend to be longer because they have to include more character development instead of starting with established characters.

Possibility 2: Timing

In my last post, I showed that the popularity of pairings can change over time, sometimes to the point where previously popular pairings seem to disappear. Conversely, there is something interesting when you look at pairings that have been popular only recently. Below I have 20 top pairings ranked by what I call date “center-of mass”, i.e. it’s a weighted sum of the publication times for each pairing. The higher the “center-of-mass”, the more recently popular that pairing has been.

One thing to notice here are all the next-generation Harry Potter characters, which makes sense as these characters were only introduced in the epilogue of the final book. However, there are several pairings on both this list and the highest-rated list, namely the top two of Daphne/Harry and Fleur/Harry.

Case study: Daphne/Harry vs. Cho/Harry

To see these timing differences more clearly, I have two pairings compared below with similar numbers of total fics but very different trends over time: Daphne/Harry (#72) and Cho/Harry (#73). This graph shows the number of fics published each year, so you can see how the popularity has changed over time.

The Cho/Harry pairing peaked in popularity in 2003, likely due to the release of Harry Potter and the Order of the Phoenix, which was the first (and last) book which had this pair together romantically. On the other hand, Daphne/Harry has only become popular very recently, leading to its high “center-of-mass” value.

It’s possible that there is some degree of favorite “inflation,” i.e. readers are more likely to favorite stories in recent years than in the early 2000s. On the other hand, older fics have had more time to accumulate favorites if we assume the overall rate were constant.

Overall, there isn’t much of a correlation between publication date and number of favorites (especially when considering the full population of fics), but I still found it interesting that the pairing with the most favorites (Daphne/Harry) also tended to have the newest fics.

Final Thoughts

To briefly summarize, there are a lot of popular pairings in Harry Potter fanfiction, but a popular pairing doesn’t necessarily translate to higher ratings (in terms of favorites/follows/reviews). Two factors that might partially explain higher ratings are length (more likely) and timing (less likely), but they certainly don’t explain everything. The true predictor for an individual fic’s ratings is always going to be how much readers like it, but I hope this discussion and my previous post show that there are always interesting trends to see if you look closely enough. Until next time!

The Ship Wars: How the popularity of pairings in Harry Potter Fanfiction changed over time

In a previous post, I looked at character choices in Harry Potter fanfiction. One thing I focused on were the popularity of various pairing, e.g. I noticed that non-canon pairings like Draco/Harry tended to be more popular than canon pairings like Harry/Ginny. I think this is typical for fanfiction, which loves to explore “what-if” scenarios. I decided to take another look, this time asking the specific question of whether the popularity of the various pairings seemed to change over time.

The data

In my previous post, I looked at a random sample of ~200k fics from the total of ~700k. In order to avoid the sampling issue, I decided to bite the bullet and scrape all the fic info from Fanfiction.net. So the figures below are based on analyzing a total of 724,315 Harry Potter fics.

To make the figures below I looked at all the fics which listed two characters. This doesn’t necessarily mean a romantic pairing, so keep that in mind. If more than two characters were listed, I included pairings that were specifically mentioned with brackets, a feature which was added in 2013. This isn’t used all that often, so the figures below mostly reflect the relative popularity of two-character fics.

Overall popularity

Below are the number of fics for the eight most popular pairings over time. There is one data point for each year, e.g. the 2001 dot represents the number of fics that were submitted between January 1, 2001 and December 31, 2001.

There are a few obvious things to point out, like the extremely high popularity of fics with Draco, or how only three of these eights pairings are canon (James/Lily, Hermione/Ron, and Ginny/Harry). But it’s also interesting to notice that many of the curves have a similar shape, which reflects the overall popularity of Harry Potter fanfiction on Fanfiction.net.

I think this time period can be separated (more or less) into the five following stages:

  1. Initial growth: From 2001-2004, all the pairings increased in popularity as the Harry Potter community on Fanfiction.net grew in size.
  2. “Golden age”: From 2004-2007, the overall rates were pretty high. J.K. Rowling was still writing the books, so fanfiction authors got to explore all the wonderful “what-if” scenarios as each book revealed more and more of the story.
  3. Post-book lull:  From 2007-2010, most pairings dipped as fewer people submitted fics. My guess is overall interest in Harry Potter dropped as no new books were released.
  4. Movie spike: There was a big spike in submissions to Fanfiction.net in 2010 and 2011, which I suspect is a result of increased interest as the two Deathly Hallows movies came out.
  5. Post-movie lull: From 2011 to the present, overall popularity has waned but could be stabilizing. It will be interesting to see if the Fantastic Beasts movies get made and whether these will cause a resurgence in popularity.

Popularity rankings

While the overall numbers are neat to look at, I think it’s also interesting to see how the relative rankings of pairings has fluctuated over time. So below you can see how the pairings shifted in popularity. If a particularly pairing has a dot at, say, rank 4 in 2008, then that pairing was the 4th most popular for that year. Note that some pairings that aren’t shown here (e.g. Draco/Ginny) occasionally broke into the top 8 rankings, but I’m just showing how the most popular overall pairings changed relative to each other.

This doesn’t reveal much new information for the top three pairings, which have had relatively stable rankings. But I think there are some interesting things to see in the less popular pairings.

For example, the Harry/Hermione pairing had a very high ranking in the early 2000s, but dropped to rank 6 by 2007. I think this was caused by the release of Half Blood Prince and Deathly Hallows books, which cemented the canon pairings of Hermione/Ron and Ginny/Harry. You can see these canon pairings were rank 4 or 5 from 2007 to 2010, but both have dropped in ranking from 2011 to 2016. Conversely, the non-canon pairings of Harry/Hermione and Hermione/Severus seemed to increase in rank.

It’s important to note from the raw data above that all of these pairings dropped in overall popularity from 2011 to 2016, just that the non-canon pairings like Harry/Hermione and Hermione/Severus dropped less compared to the canon pairings. Part of this might be the effect of the movies. I know Harry/Hermione fans like use the movies to support their choice (e.g. that dance scene in the tent). But it might be that these non-canon pairings are more popular over time simply because they’re not canon. Fanfiction allows authors and readers to explore the unknown with their favorite characters, so it’s only natural that non-canon pairings are very popular.

Final Thoughts

I think it’s interesting to see that these pairings do shift in popularity over time, presumably as the different books and movies were released. Fans can be very vocal about their favorite pairing, so I think this is a good way to look at the fandom as a whole.

There are other interesting trends I didn’t explore, so hopefully I’ll get to revisit this data set in the future. Draco/Ginny, for example, was actually very popular in the early 2000s (#4 in 2003), but has since dropped out of the top 10 (#16 in 2015). Or there’s Rose/Scorpius, which obviously didn’t exist until 2007 when the characters were introduced, but has stayed firmly in the top 10 since 2010. Stay tuned.

Voter Representation in the United States

In the last month, there has been a lot of discussion in the news about how Hillary Clinton lost the presidential election to Donald Trump, despite winning the popular vote. This quirk of the Electoral College has now occurred five times in U.S. history and twice in the last twenty years. Instead of rehashing other explanations for how this happened, I wanted to take a different look at the Electoral College, and by extension, the U.S. Congress. I’ll try to keep my personal politics out of this post and focus on the data, but discerning readers should keep in mind that I chose my examples carefully to illustrate a point.

Historical look: representation over time

The U.S. congress is a compromise to the different ways a legislative body can represent a collection of states. The Senate has two members per state regardless of size, while the House of Representatives assigns its members based on each state’s population. The intention is to ensure states with a small population are still fairly represented compared to bigger states. The Electoral College assigns its members in the same way, with Washington D.C. being the sole exception. It has three votes in the Electoral College, but only non-voting representation in the U.S. Congress. My graphs below focus on just congressional representation, but the same numbers apply for the Electoral College.

To begin, below I have a plot of the U.S. population over time, specifically in states with congressional representation. The counts are from the U.S. census (according to Wikipedia), which is collected every ten years.

U.S. Population in states with congressional representation, so excluding territories before they became states and Washington D.C.

U.S. Population in states with congressional representation, so excluding territories before they became states and Washington D.C. Source

The main thing to point out here is simply that the population keeps growing. While the pre-Civil War population grew at an approximately exponential rate, it has now slowed to pretty much a linear rate since World War II.

Next I have the sizes for the two houses of Congress over this same period. By law, the House of Representatives has to re-assign its members after each census, but as you can see, the size of the House has not changed in the last 100 years. Instead, the same number of members are re-apportioned according to relative population changes in each state

Size of the U.S. Congress over time, specifically voting members.

Size of the U.S. Congress over time, specifically voting members. Source

This shows how the U.S. Congress grew quickly in early years as new states were admitted. After the 1920 census, though, Congress couldn’t agree whether to to add new members, supposedly due to size constraints (and political reasons). This led to the Reappointment Act of 1929, which established a way for reappointment to occur automatically. Importantly, it did not increase the size of the House, a tradition which continues to this day.

As a result, a growing number of people are represented by a single member of congress, or put another way, representation per capita is shrinking. Below is a graph that illustrates this, obtained by dividing the values from the two graphs above. Note that this uses the total population, not the number of elligible voters.

Stuff

Representation per capita in the U.S. Congress. The values are calculated by dividing the total number of congressional members by the total population in states with voting members in Congress.

Despite the increases in the size of Congress, the representation per capita has continuously fell in the U.S., with the largest decreases occurring before the Civil War. These are fairly consistent relative changes, so when plotted on a log scale (not shown), the rate of decrease is fairly constant.

I think the most important takeaway here is simply how low these numbers have fallen in recent years. Since the year 2000, there are fewer than 2 members of Congress per million people (and by extension, members of the Electoral College). This means each member of the Senate represents, on average, ~3 million people, while each member of the House represents ~700,000 people. These values have more than tripled since the last time the House increased in size 100 years ago. And of course, as the population continues to grow, these numbers will not get any better.

Representation by state

While the decrease in representation has a negative effect on every state in the union, it certainly affects some states more than other. Specifically, states with larger population will have fewer congress members (and electoral college members) per capita. As I said earlier, this is intentional by the U.S. constitution, but I still think it’s interesting to look at how it breaks down by state. Below is a chloropleth map below using data from the 2010 census.

Stuff

Congress members per capita (House + Senate) according to the 2010 census. This is equivalent to the number of Electoral College members per capita. Note that the coloring is based on a log scale.

Smaller states like Wyoming, Vermont, North Dakota, and Alaska have the highest representation, while many of the larger states have lower representation. Thirty states have fewer than 2 members of Congress per million people. For another look, I have the same values in a bar graph below.

Stuff

Congress members per capita in each state, ordered by decreasing population size. Click on the graph to see the full size figure with legible text.

Since the states are arranged here by population, the mostly decreasing trend implies smaller states have better representation than larger states. The only thing that keeps this trend from strictly decreasing is the discrete nature of assigning House members. Rhode Island is the smallest state to receive two House members, making its representation appear higher than Montana and some others with only one member.

So what? Warning: opinions ahead

I hope these figures provide some insight into whether the current system of the U.S. Congress and Electoral College provides a “fair” representation of the American people. The logic that led to the the compromise between states and individuals certainly still applies, i.e. smaller states still have reasons to fear that they would be “ignored” without this system. However, I want to argue that most small states are already being “ignored,” along with many of the larger states.

Specifically, I want to repeat what CGP Grey pointed out in his excellent video on the electoral college: the system does not guarantee that small states receive attention during a national campaign, since it’s still possible to win a Presidential election with just the 11 largest states. But even those states don’t receive most of the attention. Instead, a huge majority of campaign resources are delegated towards the “Battleground” states, i.e. states with close races that have a chance in tipping the Electoral College vote one way or another.

To illustrate this, I made maps of the states that had scheduled events with Hillary Clinton and Donald Trump during the official campaign, meaning after their respective party conventions but before the election. The counts for Donald Trump came from this list of rallies on Wikipedia, while the counts for Hillary Clinton came from this archive of speeches. I don’t pretend either source is 100% correct or complete, but they both demonstrate my point.

Stuff

Number of campaign events attended by Donald Trump (top) and Hillary Clinton (bottom) during the presidential campaign but after their official nominations.

Both candidates spent most of their time in an extremely small number of states: Florida, Ohio, Pennsylvannia, and North Carolina. Also, it’s not as if the white states in these maps all received 1 or 2 visits. According to the sources, Donald Trump only visited 19 states, while Hillary Clinton only visited 16 states.

One could argue that this was time well spent, since states like Florida and Pennsylvania did end up having close races and results that were very different from poll predictions. However, I argue that these maps are not the intention of the Electoral College. It isn’t big states or small states that are benefiting from this system, it’s only the states with contentious and often bitter races. This system encourages polarization. And while I don’t know what would happen if the presidential election became decided by a popular vote, I think this is an opportunity to have an honest discussion about what “fair” representation really means.

Subreddit breakdown: Gentlemanboners

UPDATE: Added a new graph below

After my previous look into /r/AskScience, I wanted to do a similar analysis for another interesting reddit community: /r/Gentlemanboners. It’s a place where people submit (mostly) SFW pictures of female celebrities, usually candids from red carpet events or glamour shots from magazines. It’s a fairly popular subreddit that usually gets a post to /r/all each day.

I thought it would be interesting to analyze the submissions to /r/Gentlemanboners for a couple reasons. First, it’s very easy to tell who’s in each picture because there’s a strict rule of requiring that the full name is included in the title. Second, certain girls seem to be much more popular than others, e.g. Emma Watson. To see if this was true, I downloaded the JSON files for all submissions between April 5, 2011 and June 22, 2016 using this script for a total of 89,842 files. Below are some some simple analyses, but I might get motivated to look more closely at this dataset in the future.

Popularity rankings

The first thing I looked at was ranking the most popular girls. You can see below the number of submissions for the 15 girls with the most submissions. I ignored incorrect/alternate spellings of names, and a single submission could be counted twice for two different girls if they were both in the picture. Also, this post was my first attempt to try D3.js, so you can see the exact numbers by placing the mouse cursor over each bar.

Emma Watson is by far the most popular girl on /r/Gentlemanboners, with 40% more submissions than Taylor Swift, Anna Kendrick, or Scarlett Johanssen, who are essentially tied for 2nd place. It’s interesting to see that this 40% difference can’t be seen between any of the lower ranks, meaning that Emma Watson’s popularity surpasses the power law trend that describes the rest of the data fairly well.

Popularity vs. Rating

The number of submissions is probably not the best measure for which girl is the most well-liked, so I also wanted to look at the scores for each submission. Below I have a scatter plot of the number of submissions vs. the average score for those submissions. The plot has the 500 girls with the most submissions. You can mouse over each dot to see the name associated with it.

There’s an overall positive correlation between the average score and the number of submissions, with Emma Watson’s dot located to the far right. However, you can also see that many girls have higher average scores despite fewer submissions.

I think the most interesting thing here are the names with very high average scores but few posts. At the top is Milana Vayntrub, with only 62 submissions but an average score of 985, much higher than the average score for anyone else in this group.

So who is Reddit’s favorite?

I can’t really say who deserves to be called Reddit’s favorite, since it depends on whether you consider the score or number of submissions as more important. Besides Emma Watson and Milana Vayntrub, there’s also girls to consider like Alison Brie. She was ranked 13th in the bar graph above, but her average score is higher than the top 89 most popular girls. The next most popular girl with a higher average score is Katherine McNamara (who has over 500 fewer submissions). So who is reddit’s favorite? I’ll leave that for you to decide.

Update: Total Karma

By request, I made an additional plot of the total karma, i.e. the sum of the scores for each post. Below are the top 15 girls ranked by total karma. Emma Watson is still at the top, probably thanks to her large lead in overall submissions, but the difference between 1st and 2nd is definitely smaller. Taylor Swift was probably the hardest hit, dropping down to 11th. The stars of Black Swan (Natalie Portman and Mila Kunis) were knocked out of the top 15, replaced by Natalie Dormer and Blake Lively.