[ { "title": "Simulating Madness with bigdance", "url": "/posts/BigDance/", "categories": "Data Science, Sports", "tags": "march-madness, probability, python", "date": "2026-03-19 00:00:00 -0600", "content": "March Madness is a combinatorial playground unlike any other. The sheer number of possible outcomes leads to a plethora of bracket-picking strategies, some more logical than others. Knowing close to nothing about college basketball but still wanting to participate, I started building a python package in 2019 as an attempt to bring some numerical insight into an event that is and will always be best described as madness (and we love it for that). But this toy model of a package also became an incredibly valuable experience for my career in the context of software development and package management. In data science and open source coding, it is remarkably easy to come down with imposter syndrome. “I’m not a real software engineer…” “What if my PR looks like crap?” Fantasy sports gave me a gateway to ask questions that in other more serious contexts might feel “stupid” (even though they’re probably not). “What the hell is PEP8?” “How do I load this damn thing into PyPI?” “Unit tests are incredibly annoying, why would I ever need them… oh wait, that’s why…” The less-than-serious nature of the subject gave me a sandbox to learn new concepts, ask questions, and most importantly, not be afraid of messing up. What started as a thesis-writing distraction in grad school and evolved into a pandemic project sadly grew stale after becoming a father. But with the advent of AI coding assistants like Claude Code, I decided to dust it off and return to the sandbox one more time. It has once again taught me way more than I thought it could around what’s possible with open source software, especially in the new frontier of AI agents. After playing around in my free time for a few years, I finally feel like it’s in a developed enough state to show the rest of the world, hence the purpose of this article! Getting into the details of the package itself, bigdance pulls real-time college basketball ratings from Warren Nolan, simulates tournament brackets with adjustable upset factors, and can even scrape your ESPN Tournament Challenge pool to estimate each entry’s win probability. It also includes game importance analysis to figure out which remaining matchups matter most for your bracket’s chances. Whether you’re a fan looking to improve your bracket picks, a data scientist analyzing tournament patterns, or a researcher studying sports predictions, bigdance offers customizable tools to help you simulate and analyze the Big Dance. You can install it with a simple pip install bigdance (PyPI)), and the full source code is available on GitHub. There’s also a live web app at bigdance-bracket.streamlit.app — no installation or login required. It lets you pick your bracket, configure your pool size, and run Monte Carlo simulations to estimate your win probability, all from the browser. It supports both the men’s and women’s NCAA tournaments and includes an Upset Strategy tab with pre-computed analysis of winning bracket patterns by pool size, so you can see how your picks compare to what historically wins. For data scientists just starting their career, I highly encourage you to find a subject that you derive joy from and put it down in code. It can be as serious as climate change or cancer research, or as silly as fantasy sports or cooking shows, whatever brings you joy. It alleviates some of the frustrations in learning something new, while hopefully putting a smile on your face as you progress in your craft and your career. (Can’t think of what subject to play with? I highly recommend scrolling through the TidyTuesday datasets for inspiration!)" }, { "title": "Chasing .400", "url": "/posts/Chasing400/", "categories": "Baseball, Statistics", "tags": "baseball, probability", "date": "2020-07-12 00:00:00 -0600", "content": "With baseball starting back up in a couple of weeks, the shortened 60-game season brings up an interesting question: what are the odds that someone hits for a .400 batting average, a feat last accomplished in 1941 by Ted Williams? Pitching has greatly improved since then, so the probability in a regular 162-game season is astronomically low, but you would have to think that the shortened season improves those odds significantly. It’s much easier to believe someone could have a flukishly hot 60-game stretch compared to an entire 162-game season. I am obviously not the first person to think about this question, but I thought it would be fun/helpful to break things down in a game theory mindset. To simplify things a bit, let’s look at this in three steps: Given a player’s career batting average and assuming four at bats per game, what are the odds that player ends up with a season batting average of .400? Given the distribution of career batting averages among current MLB players, what are the odds of at least one player hitting .400? Does accounting for the variance in the number of at bats per game change these probabilities at all? Starting with point #1, let’s assume that a player gets \\(N_{ab}\\) at bats over the course of the season and that their career batting average, \\(p\\), is representative of the underlying probability of them getting a hit in each at bat. With these assumptions, we can approximate the distribution of the total number of hits over these at bats, which we’ll call \\(N\\), using a binomial distribution: [P_h(N,N_{ab},p) = \\frac{1}{Q} \\frac{N_{ab}!}{N!(N_{ab} - N)!} p^{N} (1 - p)^{(N_{ab} - N)}] [Q = \\sum_{n=0}^{N_{ab}} \\frac{N_{ab}!}{n!(N_{ab} - n)!} p^{n} (1 - p)^{(N_{ab} - n)}] This is easily converted to a distribution of season batting averages by dividing the input number of hits by the number of total at bats (240 in the case of a 60-game season). The graph below illustrates the probability of a player getting a certain season batting average given their career batting average (different color circles), and as you can see, these distributions can get pretty wide. It’s entirely realistic for a player to hit 50 points above their career batting average this year, but this means that even with the shortened season, somebody has to be in at least the low .300’s to even have an iota of a chance of hitting .400 for the season. In fact, if we sum up the probability of season averages above .400, a player with a career average of .305 (that of Mike Trout) only has a 0.11% chance of accomplishing the feat. Keep in mind that this is a significant improvement over the 0.000013% chance he would have in a full 162-game season… In case you have a particular player in mind, the interactive graph below plots the probability of hitting .400 as a function of a player’s career batting average (blue dots) and even allows you to change the number of games in the season via the slidebar at the bottom. These odds may seem negligible, but when you think about it, with ~270 players in the entire league, it seems reasonable to think that at least one of them might get hot and come close, right? This brings us to point #2: let’s try to calculate the probability of at least one MLB player hitting .400. To start, we can formalize the probability of a player hitting .400 using the same definitions laid out above: [P_{.400}(N_{ab},p) = \\sum_{n=\\lceil 0.4 N_{ab} \\rceil}^{N_{ab}} P_{h}(n,N_{ab},p)] From there, the probability of an individual player not hitting .400 would be \\(1 - P_{.400}\\), the probability of everyone not hitting .400 would be the product of those values for every single player, and finally, the probability of at least one player hitting .400 would be the complement of that product: [P_{tot}(N_{ab}) = 1 - \\prod_{p} [1 - P_{.400}(N_{ab},p)]^{N_p}] where \\(N_p\\) is the number of players with a career batting average of \\(p\\). The last piece of the puzzle is deciding what subset of players we should be looking at to produce a realistic view of this season’s collection of MLB talent. We definitely don’t want to include any young players that had a fluky cup of coffee in the majors to cover for an injured teammate. To account for this, we’re only going to consider players that started at least half of the 2019 season (according to stats pulled from Baseball Reference). For the 195 players who meet this criteria, after all of this complicated math, the probability that at least one of them hits for .400 is a whopping… 3.4%. Probably not going to happen, but again, an astronomical improvement over the 0.002% chance in a regular 162-game season (second column of the first table below). These numbers are pretty close to those of Tom Tango, so it sounds like we’re doing something right! The one final wrinkle that might affect these odds is removing the assumption of exactly four at bats per game, as brought up in point #3. Someone very well could get five at bats one day and three at bats the next, so how do we account for this variation accumulating over the course of the entire season? We can start by creating an empirical distribution of the number of at bats per game from the 195 “typical” players mentioned above. Since the distribution of the sum of two random variables is the convolution of their respective distributions, a season-long distribution would simply be the single-game distribution convolved with itself 59 times. If we call this season-long distribution \\(S(n)\\), the probability of at least one player hitting .400 is just a weighted sum over all possible values for the number of at bats: [P_{rand} = \\sum_{n=0}^{\\inf} S(n) P_{tot}(n)] Ultimately, accounting for this only increases the probability to 4.4% when applied to the same 195 players as above (third column of the first table above), but it does affect things! I think the logic is that some players will wind up with fewer than 240 at bats over the course of the season, requiring fewer hits to reach .400 and making it easier to join the ranks of Rogers Hornsby, Shoeless Joe Jackson, and Ty Cobb. The red circles in the second figure illustrate a generic player’s probability of hitting .400 when accounting for the variance in at bats over a season, and as you can see, the difference is noticeable but miniscule. Even if we look up each player’s respective at bats per game distribution and use that to produce individualized \\(S(n)\\) functions, the probability of at least one player hitting .400 still sticks around 3.8%. The second table above lists the individual probabilities for every player in the league when we account for their respective at bats per game distribution. Honestly, I hope it happens! I know some people are going to roll their eyes at it and say it doesn’t count, but look at the odds we just calculated. Even with the shortened season, even if someone barely misses the mark by a few points, it’s still a pretty tremendous feat. Maybe it will be forgotten sooner than most records, but it’ll be a fun piece of baseball ephemera, like Charles Radbourn’s 59 wins in a season, Red Barrett’s 58-pitch complete game, and Fernando Tatis’ two grand slams in the same inning. Baseball is full of weird anomalies, so I hope someone meets the mark, even if it does have an asterisk next to it. The python code and data used to generate the values and figures above are available on my GitHub page." }, { "title": "Riddler Answer:<br>Fool the Bank", "url": "/posts/RiddlerCounterfeit/", "categories": "Puzzles", "tags": "riddler, probability, optimization", "date": "2019-08-25 00:00:00 -0600", "content": "Put on your Hamburglar costumes, Riddler fans! This week’s puzzle takes a foray into racketeering by putting us in the shoes of an expert counterfeiter specializing in $100 bills. In one last hurrah, we need to optimize the number of fake bills to deposit alongside $2500 worth of real bills, knowing that the bank will randomly inspect 5% of them (rounded up) and have a 25% chance of detecting fake bill after inspecting it. Let’s do this thing… Solution To start, let’s define \\(a\\) as the portion of bills inspected, \\(R\\) as the number of real bills you deposit, and \\(F\\) as the number of fake bills you deposit. This would put the number of randomly inspected bills at [n = \\text{ceil}(a(R + F)).] From here, we need to think about the probability (and therefore combinatorics) of how many fake/real bills the bank chose to inspect. If we set \\(f\\) as the number of fake bills the bank happens to scrutinize, the number of ways the bank could choose \\(f\\) fake bills and \\(n\\) bills total can be defined as [\\Omega(f) = \\frac{F!}{f!(F - f)!} \\times \\frac{R!}{(n - f)!(R - n + f)!}.] This would put the total number of inspection combinations at [Q = \\sum_{f=0}^{n} \\Omega(f).] Let’s also define \\(p\\) as the probability of a fake’s detection. To calculate the probability of success given that \\(f\\) fake bills were analyzed, we can normalize \\(\\Omega\\) using \\(Q\\) and add a factor of \\((1 - p)^{f}\\) to account for the bank teller missing the fakes: [P(f) = \\frac{\\Omega(f)}{Q} (1 - p)^{f}.] Finally, we can sum this function over all possible values of \\(f\\) and multiply by the cash to be gained to get the expected return of our hypothetical caper: [E(R,F) = 100(R + F)\\sum_{f=0}^{n} P(f).] Using \\(a = 0.05\\), \\(p = 0.25\\), and \\(R = 25\\), we can vary \\(F\\) to produce a graph of expected return vs. number of fake bills. The maximum expected return occurs at \\(F = 55\\) with a 47.0% probability of winning $8000, suggesting an expected return of $3756.83. Note: This post originally included an interactive widget that let you adjust the inspection rate, detection probability, and number of real bills to see how the optimal strategy changed. The widget has been retired, but the math and solution above remain valid. The python code used to generate the values and figures above are available on my GitHub page." }, { "title": "Presidential Vocabulary", "url": "/posts/VocabularEntropy/", "categories": "Data Science", "tags": "nlp, presidents, entropy", "date": "2019-08-24 00:00:00 -0600", "content": "It seems immensely difficult to form a presidential voice that’s strong, reassuring, informative, and fifty other adjectives all at once. Too sophisticated: you’re an out-of-touch elitist. Too simple: you’re unqualified for the most complex job on the planet. Too soft: you’re spineless. Too strong: you’re heartless. In an attempt to better characterize presidential vocabularies, I went through every presidential transcript in UC Santa Barbara’s American Presidency Project and analyzed the relative frequencies of different words. I recognize that a president’s lexical fingerprint extends far beyond just the words they choose, but this will hopefully be a good start. Because less material available as you go further back in time, we’ll focus on presidents with more than 1,000 documents in the database, i.e. Herbert Hoover and onward. Furthermore, we’ll exclude functional words like “the”, “is”, “where”, etc. and only consider descriptive lexical words. Note: This post originally included an interactive widget that let you compare word frequencies across presidents, broken down by party or individual. The widget has been retired, but the analysis and findings below remain. Here are a few interesting examples from the data: “Democrats” vs. “Republicans” broken down by party: Apparently, politicians are obsessed with whatever party they’re not in because Republican presidents are much more likely to mention “democrats” over “republicans” and vice versa for Democratic presidents. Presidential last names broken down by president: This provides an interesting perspective on which presidents greatly influenced their successors (Roosevelt/Kennedy) and which presidents are the biggest narcissists (Hoover/Trump). “America” vs. “American” vs. “Americans” over all presidents: I might be reading a bit too far into this one, but it’s interesting that presidents tend to emphasize “America” and something being “American” a bit more than actual “Americans”. I will be the first to point out that the comparisons above don’t paint the whole picture. They depend heavily on the exact word you choose to look at and the interpretations largely depend on the observer. To eliminate this subjectivity and get a broader lexical characterization, we can look at statistics that apply across a president’s entire vocabulary. One such metric could be the average number of syllables per word for each president, loosely gauging verbal complexity. Along the same lines, we can get a little more sophisticated and measure the informational entropy of each president’s vocabulary. Explicitly, we will define it as [S = - \\sum_{i} P_i \\log_2 P_i] where \\(P_i\\) is the probability of a particular word (\\(i\\)) being said. In a nutshell, this quantity represents how much information is contained in each word on average. If I only use eight unique words equally (corresponding to \\(S=3\\)), my speech patterns are pretty predictable and I can’t be expected to provide any sort of detailed description. But if I use 3500 unique words equally (corresponding to the presidential average of \\(S=11.8\\)), I can speak with much more precision. The table below contains both statistics for every president since Hoover and an interesting trend emerges: the number of syllables per word gradually decreases with time while informational entropy varies president to president regardless of time. And before anyone jumps to conclusions, if we average over parties instead of presidents, both parties are equal at 2.068 syllables per word and \\(S=11.74\\). (By the way, is anyone surprised that Trump is at the bottom of both lists? While we all intuitively knew that 45 wasn’t the bright bulb in the box, now we have statistical proof…) Information Contained in Presidential Vocabularies President Syllables Per Word Informational Entropy (Bits) Barack Obama 2.04 11.63 Donald J. Trump 1.943 11.18 George W. Bush 2.012 11.39 William J. Clinton 2.013 11.48 George Bush 2.063 11.7 Ronald Reagan 2.081 11.76 Jimmy Carter 2.143 11.56 Gerald R. Ford 2.129 11.45 Richard Nixon 2.158 11.51 Lyndon B. Johnson 2.073 11.61 John F. Kennedy 2.207 11.49 Dwight D. Eisenhower 2.208 11.53 Harry S. Truman 2.16 11.43 Franklin D. Roosevelt 2.146 11.61 Herbert Hoover 2.237 11.53 A few questions naturally follow: How much does the language of each president’s era increase/decrease their informational entropy? What about the relative frequencies of two-word combinations? Can we categorize the major themes of each president using text classification? These are all ideas that are in the works and will come soon in a sequel to this post. As we have seen in recent weeks, words absolutely matter, especially those coming from a President of the United States. They can lift the hopes or stoke the fears of the citizens who hear them. We should all keep track of them in any way we can. All statistics presented here are based on documents curated by the American Presidency Project of UC Santa Barbara. Detailed datasets and the python code used to generate them are available on my GitHub page." }, { "title": "Centennial State<br>Political Geography", "url": "/posts/PoliticalGeographyCO/", "categories": "Politics", "tags": "colorado, elections, mapping", "date": "2019-07-18 00:00:00 -0600", "content": "Colorado has been considered purple since the 90’s, but has been steadily turning blue in recent years. So why is the upcoming Senate race between incumbent Republican Senator Cory Gardner and his hypothetical Democratic opponent still something the citizenry should pay attention to? For starters, the Senate is an unfair playing field, so we should care about every race. The way Senators are apportioned provides disproportionate power to citizens of certain states, even more so than the electoral college, so every seat is important. But perhaps a more interesting reason is that the divergent political geography of the Centennial State makes the potential outcome of this race much foggier. In order to illustrate this, I set out to map the behavior of political partisanship throughout the state of Colorado. Mapping partisanship by county is actually pretty simple considering that county outlines are readily available from the US Census Bureau. However, different parts of the county take part in different races. For instance, Douglas County votes in both the 4th and 6th congressional districts. Grouping these voters into one statistic would create an apples to oranges comparison in that the 58,000 DougCo voters that voted in the 6th congressional district faced a different choice than the 117,000 that voted in the 4th congressional district. Furthermore, counties can be rather large (Douglas County is 840 mi2) so the partisan characteristics of one end of the county could be very different from the other. On the other hand, individual voting precincts are much smaller in area and all voters in each precinct vote in the same races, providing a fair comparison across the board. However, the actual mapping of these results is rather difficult for multiple reasons. Precincts are drawn by each respective county, so you have to go to each of the 64 county clerk offices to find the outlines of the precincts they preside over. In the easiest cases, GIS shapefiles are readily available for download on their website. In the more difficult cases, hand drawn maps need to be manually converted to shapefiles using WebPlotDigitizer and multiple Python modules. In the worst cases, the county clerk doesn’t have an email address and won’t return your phone calls. As a result, I wouldn’t recommend using the following maps in any sort of official capacity to identify which precinct you vote in, but I do think these maps are pretty unique and comprehensive. To define the scope of the analysis shown here, let’s consider results from presidential and midterm elections in the state of Colorado between 2012 and 2018. As a disclaimer, the metrics I’ll be using are similar (if not identical) to those used by analysts at FiveThirtyEight. There’s no way I’m smart enough to have invented these metrics, just smart enough to know how to apply them. Starting with the basics, partisan lean is a fairly standard metric of whether an area is more Republican or Democratic, but we will define it explicitly as [L_i = \\frac{1}{N_i} \\sum_{j=1}^{N_i} (D_{i,j} - R_{i,j}).] where \\(i\\) is the precinct index, \\(N_{i}\\) is the number of races in the precinct, \\(j\\) is the election index (e.g. 2018 District 3 State House race, 2014 Senate race, etc.), and \\(D_{i,j}\\) and \\(R_{i,j}\\) are the percentage of Democratic and Republican votes, respectively, for race \\(j\\) in precinct \\(i\\). The map below illustrates this metric for each precinct throughout the state in the form of a heat map with darker shades of red representing more Republican areas and darker shades of blue representing more Democratic areas. The general trends are not surprising in that more densely populated areas (e.g. Denver, Boulder, Fort Collins) tend to be more Democratic and more sparse rural areas (e.g. the entire eastern third of the state) tend to be more Republican, but seeing where the exact lines are drawn is both interesting and informative. However, these partisan predilections do not remain static over time. Precinct leans can easily change based on both the impact of current events and the changing makeup of voting populations. For instance, the 2018 midterms in Colorado saw an 8.1% increase in voter turnout compared to 2014. Numerous things changed in those four years, but the most obvious and earth-shattering one in terms of politics was the election of our current president. To illustrate its effect on voters’ behaviors, we’ll use what I’m calling the Trump effect: it calculates the lean of a precinct using only races from the 2018 midterms and subtracts the lean of the same precinct using only the 2014 midterms. This isolates where voters are trending due to the instability and antics of the current resident of the White House. As you can see in the map below, the sea of cerulean is not good news for Cory Gardner, especially considering he votes in line with Trump 90% of the time. Another interesting and lesser-known metric is partisan elasticity, an index of how stubborn voters are with their partisanship, i.e. are they voting party lines down the ballot or will they vote for the other party if the right candidate comes along. This will be defined as the standard deviation of the partisan lean, or using the same variables from the first equation, [E_i = \\sqrt{\\frac{1}{N_i} \\sum_{j=1}^{N_i} (D_{i,j} - R_{i,j})^2 - L_i^2}.] In isolation, the relevance of a precinct’s elasticity is kind of hard to grasp, but when put in context with its associated lean, elasticity teases out valuable insight for potential candidates. Let’s say a campaign wanted to identify areas where a candidate’s effort would have the most impact. To do this, we can look at “flippable” precincts, precincts that have elasticities larger than the magnitude of their lean. To get the most bang for our buck, let’s also limit the scope to precincts with voter densities higher than 100 per square mile. Fascinatingly, the map above shows us that the majority of these flippable precincts are in the suburban neighborhoods surrounding the major cities along the Front Range. Keep in mind that winning over precincts is not the goal in a Senate race, winning over voters is, so this metric of flippability should not be the golden standard. In order to get past the primary, a Democrat needs to visit the major cities since that’s where the majority of Democrats reside. Even in the general, a smaller elasticity could still swing more votes overall if the precinct is dense enough. However, these suburban conditions provide a great opportunity for a candidate to knock on doors, shake a few hands, and have the fabled “kitchen table discussions” with everyday voters. Gaining Democratic support in these neighborhoods would kill two birds with one stone: they vote for you in the primary and persuade their undecided neighbors during the general. So which candidates might we be seeing in these flippable precincts come March? And which of them should we be paying attention to? If online presence has anything to do with it, former State Senator Mike Johnston should be at the top of the list. Of the declared candidates, Johnston has the highest relative search interest via Google Trends over the past two years, the most Facebook likes, and the third most Twitter followers as of June 23, 2019 (see table below). And since money definitely has something to do with primaries, it should be mentioned that Johnston has twice the fundraising as the rest of the field combined. Honestly, he should be more worried about individuals who have yet to join the race, e.g. if Hickenlooper decides to stop the exercise in narcissism that is his presidential campaign, or if Perlmutter decides it’s time to upgrade from the House of Representatives. Online Presence and Fundraising of Declared Democratic Senate Candidates Candidate Relative Search Interest Twitter Followers Facebook Likes Fundraising Mike Johnston 369 14,610 14,789 $1,804,923.52 Andrew Romanoff 76 4,509 7,800 $504,095.35 Diana Bray 0 195 527 $72,514.38 Trish Zornio 19 72,487 2,983 $59,048.41 Lorena Garcia 53 808 1,941 $14,266.15 Dustin Leitzel 0 420 415 $5,603.00 John Walsh 181 7,263 2,301 $0.00 Stephany Rose Spaulding 66 3,225 3,228 $0.00 Dan Baer 29 15,353 5,490 $0.00 Alice Madden 10 1,021 766 $0.00 Ellen Burnes 0 62 128 $0.00 All of these factors aside, I do think Mike Johnston would make an excellent Senator for Colorado. To paraphrase the wise (albeit fictional) Sam Seaborn, education is the silver bullet and Johnston’s record in that department is second to none. I will note that he may want to tread lightly on that subject considering the recent results of Colorado Amendment 73, which would have increased public education funding via higher income, corporate, and property taxes. In the flippable precincts mentioned above, only 44.2% of voters voted in favor of the amendment. I’m not saying don’t tout your education credentials, just don’t rely entirely on that issue, especially if it’s paired with a tax increase. Regardless of who ends up being the Democratic nominee, Cory Gardner has not acted with Coloradoans’ best interests in mind, despite his best efforts as of late. Senator Gardner has voted to repeal health coverage for over 400,000 Coloradoans, uphold President Trump’s wasteful and xenophobic border wall, oppose common-sense gun laws, and provide 60% of the state’s tax cuts to the richest 1% of its citizens. A change needs to be made, and if the Democratic candidate can deftly navigate the unique political geography of Colorado, this historically purple state may just turn blue. All statistics presented in the maps shown here were calculated using official election results from the Colorado Secretary of State website. All precinct shapes were collected from each county's Clerk and Recorder. The relevant data and python code used to calculate them are available on my GitHub page and precinct shapefiles are available on request." }, { "title": "Methods to the Madness", "url": "/posts/MarchMadness/", "categories": "Data Science, Sports", "tags": "march-madness, probability", "date": "2019-05-27 00:00:00 -0600", "content": "March Madness has come and gone and it was an exciting (albeit fairly chalk-ish) tournament! To get a little skin in the game, many people join bracket leagues where they pick a winner for each of the sixty-three match-ups in the bracket. A correct pick is worth 10 points in the round of 64, 20 in the round of 32, 40 in the Sweet Sixteen, 80 in the Elite Eight, 160 points in the Final Four, and 320 points in the championship. At the end of the tournament, the bracket with the most points wins. This year would have been my first victory, were it not for the non-existent three-pointer foul by Samir Doughty (thanks for nothing, Wahoos), so I decided to look into the different methodologies of bracket selection to hopefully improve my odds for next year. There are some pretty strange techniques out there (my personal favorite is which mascot would win in a fight), but the most logical one I could come up with uses the projections from our friends over at FiveThirtyEight: Let’s say you have no previous experience with college basketball, but you still want to play in order to have something to talk about with coworkers/old high school buddies/random strangers. To get around your lack of experience, you decide to flip a coin to pick who wins each matchup, i.e. heads means the better seed wins, tails means the worse seed wins. However, to make sure you don’t pick a 16-seed going all the way to the Final Four, you decide to use the latest innovation in sports-related coinage: a self-biasing coin whose heads/tails probabilities shift based on the two teams in each matchup and FiveThirtyEight’s Elo ratings system. For simplicity’s sake, let’s ignore FiveThirtyEight’s travel and injury adjustments (partly because they seem to be shrouded in secrecy) as well as the four initial play-in games (because those games are pointless). Let’s also assume we only know the initial team ratings. So as an example, in the first round Gonzaga-FDU matchup (Go Zags), Gonzaga has a rating of 95.02 and Fairleigh Dickinson has a rating of 70.04. This would suggest that the probability of the coin coming up heads (and Gonzaga winning) would be [P_{H} = \\frac{1}{1 + 10^{30.464*(70.04 - 95.02)/400}} = 98.8\\%] and the probability of the coin coming up tails (and FDU winning) would be [P_{T} = \\frac{1}{1 + 10^{30.464*(95.02 - 70.04)/400}} = 1.2\\%.] For both the men’s and women’s tournaments, how much better does the specially designed self-biasing coin do compared to a plain old unbiased quarter? What about compared to “chalk” where the better seed always wins? To assess how this method would perform, we can apply it to the past three tournaments using a Monte Carlo simulation. Using the code located here, we can pick 10,000 brackets for each year and see what the distributions of point values look like for each year compared to chalk or an unbiased coin flip, distributions like the ones below! March Madness Results - Chalk vs. Biased Coin Season Gender Chalk Simulation Avg. Simulation St. Dev. Unbiased Coin 314.9 134.7 2017 Men 820 680.4 221.7 2017 Women 940 930.4 176.5 2018 Men 810 662.7 261.4 2018 Women 1190 924.7 174.4 2019 Men 920 802.8 248.9 2019 Women 1470 1218.3 252.6 As you may have already guessed, the unbiased coin flip (blue traces) does not do well at all, producing far fewer points than our technically advanced currency regardless of the season. Even when we compare the biased coin results (solid lines) to chalk results (dashed lines), we can see that, on average, the sophisticated FiveThirtyEight method comes close but also falls short of a chalk bracket. However, where the benefit of this method can be seen is in the width of these distributions. If we normalize the three biased coin distributions relative to the chalk point values for the respective years and average them together, this produces the graph below. Again, for both the men’s and women’s tournaments, the average ratio is below one, but 24.7% of the men’s brackets and 21.8% of the women’s brackets earn more points than a chalk bracket. Picking chalk may give you more points on average, but if you want to win in a bracket league of ten people, you have to take risks. The real test would be to compare this method’s results to the actual brackets that fanatical sports enthusiasts submit, but I’ll leave that for next year’s analysis. Hopefully this method helps all of us in next year’s basketball-related spring psychosis! The python code and simulations used to generate the values and figures above are available on my GitHub page." }, { "title": "Riddler Submission: Loteria!", "url": "/posts/Loteria/", "categories": "Puzzles", "tags": "riddler, probability", "date": "2019-05-25 00:00:00 -0600", "content": "Loteria is a traditional Mexican game of chance akin to bingo. Each player receives a 4-by-4 grid of images. Instead of the comically large rotating bin of numbered balls, the caller randomly draws a card from a deck containing all 54 possible images. If a player has that image on their grid, they mark off the corresponding location with a corn kernel or pinto bean. Exact rules can vary, but in the version I was taught, the game ends when one of the players fills their entire card (and screams “Loteria!”). Each of the 54 possible images can only show up once on each card, but other than that restriction, let’s assume image selection and placement on each player grid is random. One beautiful day, you and your friend Christina decide to play a friendly game of Loteria. What is the probability that either of you end the game with an empty grid, i.e. none of your images were called? How does this probability change if there were more/fewer unique images? Larger/smaller player grids? Solution The inspiration for this problem actually comes from reality: while on vacation, a friend of mine ended a game with zero matching images and I seemed to be the only one flabbergasted by the odds. To illustrate that I wasn’t crazy (or perhaps confirm that I was), I set out to calculate the exact odds of such a thing happening. First, the two grids cannot have any overlap. If any of the images on the first card are also on the second, the first card cannot possibly cover all its images without covering one on the second. With \\(G = 16\\) images on each grid and \\(D = 54\\) possible images, there are \\(\\frac{D!}{G!(D - G)!} = \\frac{54!}{16!38!}\\) possible grids that either player could have. Regardless of which images are on the first player’s grid, the second grid now only has \\(D - G = 38\\) images to choose from, hence leaving \\(\\frac{(D - G)!}{G!(D - 2G)!} = \\frac{38!}{16!22!}\\) non-overlap grids. Thus, the probability of the two grids having no overlap is [P_{d} = \\frac{(D - G)!}{G!(D - 2G)!} \\times \\frac{G!(D - G)!}{D!} = \\frac{(D - G)!(D - G)!}{(D - 2G)!D!} = \\frac{38!38!}{22!54!} = 0.00105428] Second, we need the order of the deck to be such that all of one player’s images will come up before any of the other player’s images. After \\(N\\) cards are revealed from the deck, there are \\(\\frac{D!}{N!(D - N)!} = \\frac{54!}{N!(54 - N)!}\\) possible card combinations that could be revealed and \\(N!\\) possible orderings of those cards, giving us a total of [Q = \\frac{D!}{N!(D - N)!} \\times N! = \\frac{D!}{(D - N)!} = \\frac{54!}{(54 - N)!}\\text{ possible outcomes.}] How many of the possible outcomes result in the first player getting their sixteenth and final match on the \\(N^{\\text{th}}\\) card and the second player getting no matches at all? Out of the \\(N\\) cards revealed, \\(G = 16\\) of them have to match, but the remaining \\(N - G\\) cards can’t match any image on either grid. This suggests \\(\\frac{(D - 2G)!}{(N - G)!(D - G - N)!} = \\frac{22!}{(N - 16)!(38 - N)!}\\) possible combinations of non-matching cards to come up. Furthermore, the first \\(N - 1\\) cards can be in any order they want, giving us \\((N - 1)!\\) possible orderings, but the \\(N^{\\text{th}}\\) card has to be one of the G = 16 matching cards, resulting in G possible endings. This produces [W = \\frac{(D - 2G)!}{(N - G)!(D - G - N)!} \\times (N - 1)! \\times G = \\frac{22!(N - 1)!16}{(N - 16)!(38 - N)!}\\text{ possible winning outcomes.}] With all possible outcomes being equally probable, we can calculate the probability of one player winning and the other remaining at zero after revealing \\(N\\) cards as [P_{w}(N) = \\frac{W}{Q} = \\frac{(D - 2G)!(N - 1)!G}{(N - G)!(D - G - N)!} \\times \\frac{(D - N)!}{D!} = \\frac{22!(N - 1)!16}{(N - 16)!(38 - N)!} \\times \\frac{(54 - N)!}{54!}.] Next, let’s put these two probability functions together. Since either player can be the winner or loser, the probability of a game ending with either player having an empty grid after \\(N\\) turns is [P_{z}(N) = 2 \\times P_{d} \\times P_{w}(N) = 2 \\times \\frac{(D - G)!(D - G)!}{(D - 2G)!D!} \\times \\frac{(D - 2G)!(N - 1)!G}{(N - G)!(D - G - N)!} \\times \\frac{(D - N)!}{D!}] [P_{z}(N) = \\frac{2(D - G)!(D - G)!(N - 1)!G(D - N)!}{D!(N - G)!(D - G - N)!D!} = \\frac{38!38!(N - 1)!(54 - N)!32}{54!(N - 16)!(38 - N)!54!}.] Finally, we can plot this function as shown below and sum up over all possible values of \\(N\\) to calculate a total probability of [P_{tot} = \\sum_{N = G}^{D - G} P_{z}(N) = 3.508 \\times 10^{-12}.] Think about that for a second. That means that even if we played three billion times, we would still only have about a 1% chance of a zero match game occurring. And yet it happened to us in real life. I would say that we should have bought a lottery ticket, but sadly that’s not how random variables work. As for the second part of the question, we can easily calculate the zero match probability for larger/smaller grid sizes or more/fewer unique images simply by varying \\(G\\) and \\(D\\) respectively in the same equations above. As your intuition might have guessed, more unique images and smaller grids produce higher zero match probabilities, but play around with the interactive graph below to see exactly how it varies. Hope everyone has as much “divertido” with this as I did! The python code used to generate the values and figures above are available on my GitHub page. Simulations that validate the equations above are also available." }, { "title": "Uneven Democracy", "url": "/posts/UnequalRepresentation/", "categories": "Politics", "tags": "elections, democracy", "date": "2019-02-22 00:00:00 -0700", "content": "Americans pride themselves on their democracy, on the idea that everyone has an equal voice in our government, and to be honest, I’m pretty proud of it too (less so in recent years). But if anyone tells you that every citizen truly has an equal voice in our system, they possess one of two things: 1. a misunderstanding of how our government is designed and operated, or 2. a loose definition of the term “equal”. In an ideal scenario, each person’s voice would be equally valued and political representation on a macroscopic level would be proportional to the views of the American public. As a simple example, a group of 1800 Democrats and 1200 Republicans could be represented by 9 Democrats and 6 Republicans, giving each voter a representative power of 1/200 or 0.005. But let’s say that a neighboring group of 900 Republicans and 600 Democrats was represented by 9 Republicans and 6 Democrats. Locally, this is proportional and fair, but macroscopically, we now have equal representation (15-15) for unequal populations (2400-2100) because the second group has twice the representative power (1/100 or 0.01). This may seem like an unrealistic example, but this is basically how the United States Senate has operated since 1789 thanks to the Connecticut Compromise. Gerrymandering has led to similar (but less extreme) situations in the House of Representatives. To illustrate this, we can calculate the representative power of each state or congressional district by dividing the number of representatives/senators they have by the number of voters who participated. Just to clarify units, we will define a 1 representative to 1 voter ratio as a “rep” which would make 1 representative to 1,000,000 voters a “microrep” or μrep. Furthermore, since there are 435 representatives in the House and 100 in the Senate, we will define “Total Power” as 4.35*(Senate Power) + (House Power). The map below illustrates darker colors as higher representative powers and shows the exact values for a congressional district when you hover over it. Taking a look at the two extremes, we can see a pretty shocking disparity in representation. A voter in Wyoming has more representative power than a Floridian, a Pennsylvanian, a New Yorker, a Californian, an Ohioan, a North Carolinian, a Michigander, an Illinoisan, and a Texan combined. This is purely due to the fact that while each of these five states are represented by two senators, only 200,000 voters pick those two people in Wyoming while multiple millions of voters pick them in each of the other four states. Using state-by-state voter demographics, we can expand the scope of our power metric to compare the average representative power of different demographics to see if any group in particular benefits from this arrangement. Comparisons based on sex and age show fairly similar representation between different groups, but race, political affiliation, and urban/rural delineation produce very unbalanced playing fields. As seen in the tables below, white voters have about 10% more representative power than non-white voters, Republican voters have 29% more than Democratic voters, and rural voters (based on CityLab’s classification) have 39% more than urban voters. This is again due to the disparate proportioning of representatives in the Senate and highlights the inherent representative inequity in our country’s democracy. Representative Power By Race in 2016 (microreps) Race House Power Senate Power Total Power White 3.36 0.79 6.81 Black 3.40 0.66 6.26 Asian 3.63 0.72 6.77 Hispanic 3.61 0.50 5.79 Representative Power By Political Party in 2016 (microreps) Party House Power Senate Power Total Power Republican 3.85 1.00 8.19 Democratic 3.16 0.73 6.33 Other 0.00 0.43 1.88 Representative Power By Urban, Suburban, and Rural Classification in 2016 (microreps) Party House Power Senate Power Total Power Rural 3.29 1.18 8.42 Suburban 3.26 0.69 6.25 Urban 3.90 0.49 6.05 On a macroscopic scale, this leads to representation that is disproportionate to the actual populations they represent. Look at the percentages of elected congressional representatives in each party compared to the number of voters who actually voted for that party shown in the first figure below. Wins and votes in the House of Representatives have generally been proportional because the number of representatives is allocated based on population – until 2010 that is when congressional districts were conveniently redrawn by state governments. Can we talk about 2012 please? Democrats received more votes in aggregate during that election and only received 45% of the delegates. The equivalent graph applied to the Senate is even worse. For each year, we can add up the vote totals for each senator’s most recent election (e.g. for 2016, a senator elected in 2014 would contribute their 2014 vote totals) and again compare votes by party to wins by party. As you can see in the second figure, the relationship between these two values is loose at best. Since 2004, the percentage of votes cast for Republican candidates has never exceeded that of Democratic candidates, and yet, four of the seven congresses have had as many or more Republican Senators as Democratic Senators. That doesn’t exactly seem fair to me… So what does this imbalance of representative power lead to? Unpopular ideas becoming law and popular ideas getting ignored. Despite being very unpopular amongst the general public, the attempted “skinny repeal” of Obamacare in 2017 almost passed in the Senate with votes from Senators representing only 39.8% of American voters, and it would have, were it not for the dramatic thumbs down of the late Senator John McCain. What did become law was the 2017 Tax Cuts and Jobs Act, another unpopular bill which only needed the votes from Senators representing 40.2% of the electorate. The Supreme Court confirmations of both Neil Gorsuch and Brett Kavanaugh were pushed through by Senators elected by less than 42% of the electorate. Even ignoring actual politics, it’s counterintuitive that such a diverse country would elect congress after congress of old white dudes rather than the melting pot that America so eagerly describes itself as. So why does this happen and what can we do to fix it? Low voter turnout is a big factor that tends to benefit conservative candidates, which possibly explains why Republicans have received disproportionately higher representation since 2010. The 2018 midterms did show a huge surge in turnout but I can’t help but wonder if this is purely a reaction to outrage against President Trump that will quickly fade in future elections. Voter suppression continues to rear its ugly head, though often masquerading as concerns for “voter fraud”. Encouraging those around us to get involved in democracy and protecting the voting rights of every American citizen will help to align the voices of power with the voice of the people. But the more data-centric problem I want to address is gerrymandering and the restructuring of Congress. The exact geometry of congressional districts is a very complicated issue, so I won’t get into that level of detail here (though a future post might), but there might be a simpler solution. For those of you who think I’m trying to rewrite the Constitution, take a chill pill, this is just a thought experiment. Let’s imagine a scenario where no districts are drawn. In this scenario, everyone still gets two Senators and the same amount of representatives in the House as before, but rather than voting in individual races, each person votes for a particular party (obviously this is problematic since people’s views don’t always align with any single party, but bear with me). Representatives and Senators are then divvied out to each party relative to the votes for each party. As you can see in the first two figures below, this setup compensates for most of the mismatch between votes and representation in the House, but the Senate remains inequitable. At least now the number of Senators for each party is proportional to the vote share, but Republican votes still go much further than Democratic ones. In fact, after trying a few different iterations of this model, the only version that produces a Senate that somewhat accurately represents the votes of the people (as seen in the third figure below) is when Senators are reapportioned based on census populations, the number of Senators is increased to 150, and people again vote for a party instead of a candidate. Crazy how far you need to go just to produce a level playing field… I realize that the founding fathers established our government this way for a reason, that the Connecticut Compromise was necessary to get smaller states to ratify the Constitution, and that, like the Electoral College, it helps to guard against the tyranny of the majority. But right now, it feels a hell of a lot like we’re at risk of the opposite. A congressional majority elected by a minority of Americans is enabling an unpopular president (also elected by a minority of Americans) to enact legislation opposed by a majority of Americans. The framers were very intentional in their preambular word choice of forming a more perfect union, implying a good start but plenty of room to improve. Maybe it’s time to start thinking about making Congress a bit more perfect in terms of representation. All statistics presented here were calculated using official election results from the FEC. The relevant data and python code used to calculate them are available on my GitHub page." }, { "title": "Riddler Answer: Printers are from Mars, Scanners are from Venus", "url": "/posts/MartianPrinter/", "categories": "Puzzles", "tags": "riddler, probability", "date": "2019-01-14 00:00:00 -0700", "content": "This week’s edition of the Riddler from Jerry Myers takes us to our rusty planetary neighbor and asks us to calculate the odds of our survival based on three (fairly unreliable) 3D printers. With Mars being a harsh environment, a vital piece of equipment will break exactly once per day and one of the printers must print a replacement part. However, each printer can only print one piece per day and they each have a 10%, 7.5%, and 5% chance respectively of breaking each day. Based on this less than ideal scenario, what are the odds that you would survive a 1825 day long journey? Solution If I understand the description of the scenario correctly, the only way you would die is if all three printers break in the same day because all other scenarios are fixable. For instance, if printers A and B both fail, printer C can print a part to fix printer B, printer B can then print a piece to fix printer A, and finally, printer A can then print a new part for the vital equipment. By this logic, the probability of surviving the entire journey would be [P = (100\\% - 10\\% * 7.5\\% * 5\\%)^{1825} = 50.4\\%.] However, the problem gets much more interesting if we make it a bit more realistic. Specifically, we can make the vital equipment/printer failures a random process using a Gillespie algorithm like the one shown below. On average, one piece of vital equipment will still break each day and each printer will have the same odds of breaking each day, but there is now a significant chance of having multiple breakages in a single day. When anything breaks, the next available printer will print a replacement but then remain unavailable for the next 24 hours to recharge. import numpy as np printerLag = 1.0 # Number of days before a printer is ready to print again numDays = 1825 # Number of days in your martian adventure printerRates = np.array([0.05,0.075,0.1]) # Probabilities of each printer breaking vitalRate = 1.0 # Rate at which vital equipment breaks (measured in malfunctions per day) numSims = 10000 # Number of simulations to run before averaging times = [] for numSim in range(numSims): if (numSim + 1)%1000 == 0: print('Try #' + str(numSim + 1)) time = 0.0 printers = np.ones(len(printerRates)) # 1 if operational, 0 if broken lastPrint = -1*np.ones(len(printerRates)) # Time of each printer's last print job while time &lt; numDays: probs = np.append(printerRates*printers,vitalRate) overallRate = sum(probs) randNum1 = np.random.rand(1)[0] time -= np.log(randNum1)/overallRate probs = probs/overallRate randNum2 = np.random.rand(1)[0] while np.any(printers == 0) and np.sum(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0)) &gt; 0: fixInd = np.where(printers == 0)[0][-1] printInd = np.where(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0))[0][0] printers[fixInd] = 1 lastPrint[printInd] += 1 del fixInd del printInd ind = np.where([randNum2 &lt;= sum(probs[:ind + 1]) for ind in range(len(probs))])[0][0] if ind &lt; len(printers): printers[ind] = 0 if np.sum(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0)) &gt; 0: printInd = np.where(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0))[0][0] lastPrint[printInd] = time printers[ind] = 1 del printInd else: if np.any(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0)): printInd = np.where(np.all([printers == 1,time - lastPrint &gt;= printerLag],axis=0))[0][0] lastPrint[printInd] = time del printInd else: break del ind times.append(time) del numSim print('Average Survival Time = ' + str(round(np.average(times),1)) + \\ ' +/- ' + str(round(np.std(times),1)) + ' days') print('Probability of Survival = ' + str(round(100*sum(np.array(times) &gt;= numDays)/len(times),1)) + '%') This added stochasticity changes the entire game. We intrepid Riddler-nauts now have essentially a zero percent chance of surviving our martian journey with an average survival of \\(13.2 \\pm 12.6\\) days. So what needs to change??? What do our highly-skilled RASA engineers need to focus on to get us to the 1825 day finish line? For starters, let’s pretend that the Riddler Nation Senate decided to approve funding for as many 5% printers as we need to have a 50% chance of survival (man, those senators love their coin flips). If that were the case, we would need seven printers instead of three, as seen in the second figure. But with space travel costing $1250 per pound at best, each printer weighing 50 pounds, and each printer costing $1500, the senators are starting to clamor about the extra $256K. Instead, they are encouraging us to optimize the three printers we already have. In the end, lowering the breaking probability of our printers won’t help much – even if the printers never break, it’s reasonably probable for more than three pieces of vital machinery to break in a single day and the lag between printing jobs will do us in. Sure enough, simulating this perfect-printer scenario produces an average survival of \\(20.5 \\pm 20.0\\) days so we should definitely focus on reducing the rate-limiting 24 hour recharging period. In order to reach the 50% goal set by the Senate with our original three printers, we would need to lower the lag time to just under 3 hours as seen in the final figure. This problem is a perfect example of how deterministic, average-based methods can sometimes be misrepresentative of the real scenario. These simpler methods told us that we had a fighting chance, but let’s be honest, malfunctions aren’t going to happen like clockwork. Only by accounting for the randomness and noise of the stochastic processes of the real world do we realize that something needs to change or we’re going to end up as martian toast. If anyone in Riddler Nation wants to mess around with the simulation or see if a different strategy would be more effective, the underlying code is available here. Safe travels, Riddler-nauts!!!" }, { "title": "When Red Becomes Purple", "url": "/posts/DougCoRecap/", "categories": "Politics", "tags": "colorado, elections", "date": "2018-11-23 00:00:00 -0700", "content": "Canvassing in any scenario is uncomfortable. Canvassing for a Democratic candidate in a historically Republican district is panic-attack-inducing. But in any campaign, it’s a necessary component. So when my partner and I began volunteering for the Douglas County portion of Jason Crow’s congressional campaign in Colorado’s 6th district (CO-6), it seemed like a daunting but worthy challenge. Given the chaotic state of our democracy, a little more is required of average Americans. We are called to endure the occasional door slam to find the few individuals who remain undecided, who aren’t sure where to vote, who need a little extra motivation to even vote at all. We are called to tolerate the surly “leave me alone”’s to ensure that the voice of every citizen is heard in our society. These efforts were particularly necessary in the region that we were assigned by the Crow campaign. In the three previous House races in Douglas County, Democrats have won exactly zero precincts and lost by an average of more than 26 percentage points. With this level of incumbency advantage, we had our work cut out for us, but other indicators left a whiff of possibility in the air. Unprecedented individual contribution levels of Democrats across the country (including this one) provided a significant advantage and could have suggested a grass roots mobilization of previously silent Democrats across the country. Midterm elections are always difficult for the party controlling the White House, and with the considerable unpopularity of President Trump, the fact that the Republican incumbent Mike Coffman voted with Trump 96% of the time was a major selling point to independent voters. And having a moderate candidate with a military background as an Army Ranger and a progressive stance on gun violence in a region deeply effected by several tragic shootings in recent years would definitely make a difference. So in the days leading up to the election, when Jason Crow’s aides would jokingly ask “you gonna win us DougCo?”, I understood the sarcasm but confidently replied, “we’ll win a few precincts for sure.” As results came pouring in, this prediction turned out to be an understatement. Jason Crow went on to take thirteen precincts in Douglas County and only lost the county by six percentage points. Losing in any fashion may not sound like an uplifting fact, but the 22.0% jump from losing by 27.9% in 2016 to only 5.9% in 2018 is the largest gain out of all the counties in the district (Adams: 16.4%, Arapahoe: 19.8%). Recent House Races in the DougCo Portion of CO-6 Year Dem. Candidate Vote % Margin of Defeat (%) Precincts Won 2012 Joe Miklosi 35.4 24.7 0 2014 Andrew Romanoff 34.5 27.5 0 2016 Morgan Carroll 33.3 27.9 0 2018 Jason Crow 45.8 5.9 13 So what happened to Douglas County voters? Why the sudden purple hue? Races further down the ballot give us some insight into why the citizens of Douglas County voted the way they did, and we see that these results don’t quite fit the “blue wave” narrative that everyone is so fond of using. Across the board, Democrats did get a bump compared to previous years because of all the factors listed above, but some of that advantage disappears down the ballot. The higher profile races with national implications, i.e. the House and Governor’s races, only had a Republican edge of 5.9% while the race for Attorney General ballooned up to 13.5%. This could suggest that voters wanted to rebuke the White House while keeping local government conservative. Races Down the Ballot in the DougCo Portion of CO-6 Position Republican Vote % Democrat Vote % Margin (%) House 51.7 45.8 5.9 Governor 51.6 45.7 5.9 Secretary of State 53.6 44.6 9.0 State Treasurer 54.3 43.8 10.5 Attorney General 55.5 42.0 13.5 So if the suburbs were what handed Democrats the House in 2018, the Douglas County portion of CO-6 should be exhibit A. Obviously, part of Jason Crow’s success in this area was regression to the mean – it was already so red that there was really nowhere to go but up – but it even stands out when comparing apples to apples. Out of the the top ten districts most similar to CO-6 according to FiveThirtyEight’s similarity metrics, three were held by a Republican before the midterms. All three seats flipped to Democrats, but in terms of gain in vote margin, CO-6 just barely finishes second to Kansas’s 3rd district (KS-3) by only 0.3%. In fact, the comparison between Jason Crow’s victory and Sharice Davids’ victory in KS-3 is actually rather fitting. In the primaries, Davids’ defeated a Bernie-Sanders-endorsed candidate to move on to the general election, possibly suggesting a desire for moderate candidates in KS-3 similar to CO-6. And while the storyline of a gay Native American MMA-fighter candidate will generate more national attention than yet another white male running for congress, the energy levels behind both campaigns in historically red districts seem reminiscent of each other. Both benefitted from early grass roots organizing by neighborhood volunteers refusing to accept the status quo. Demographically, Geographically, and Politically Similar Districts to CO-6 District 2016 Margin of Defeat (%) 2018 Margin of Victory (%) Democratic Gain (%) CO-6 8.3 11.2 19.5 CA-49 0.5 7.4 7.9 CA-25 6.3 6.4 12.7 KS-3 10.7 9.1 19.8 Two and a half weeks of door-knocking, organizing, and getting out the vote felt far more like two and a half months. It would normally be difficult to imagine how people would have survived the entire campaign, but the energy and passion for change of the volunteers that came to make a difference had to have something to do with it. There’s no way you make it through an entire campaign without neighborhood leaders like Darien and Kathy, without expert phone-bankers like Connie, without unstoppable doorknockers like Brian and Sam, and without energy boosts from seasoned campaign veterans like my partner, Ashley (of course I was going to shamelessly brag about my wife at some point during this article). Jason Crow and his team ran a hell of a campaign, but if there’s anyone to credit with this win, it’s the citizens of Colorado’s 6th district that gave their time, sweat, and effort to the idea that change can materialize anywhere. More than anything, that is what it takes to turn red into purple. All election results presented here are unofficial as of November 23rd, 2018 and a spreadsheet containing all relevant data is available here on my GitHub page." }, { "title": "Riddler Answer: Grazed and Confused", "url": "/posts/GrazedAndConfused/", "categories": "Puzzles", "tags": "riddler, optimization", "date": "2018-10-13 00:00:00 -0600", "content": "On this week’s edition of the Riddler, Moritz Hesse introduces us to a farmer that owns a circular field of radius \\(R\\) and a particularly hungry goat. Growing tired of the kid running amok, the farmer decides to tie the goat up to a post on the fence surrounding the field, but needs to ensure that the goat doesn’t graze the entire field. The question posed is this: How long does the goat’s tether need to be to ensure that the goat only eats half of the circular field? To better visualize this agricultural quandry, see the gif shown to the right. At shorter tether lengths, the more limiting factor is the actual tether restricting the goat from running free within the circle, but as it gets longer, the fence is really the only thing limiting the goat’s appetite. Solution To find the correct tether length, we really just need a bit of clever geometry. The grazing area covered by a tether of length \\(L\\) can be split into two regions shown in the diagram to the right: the area swept out by the tether if the goat is pulling it taut, which we will call \\(A_1\\), and the pizza crust looking areas to the sides of the post that the goat is tied to, which we will call \\(A_2\\). Let’s start by focusing on \\(A_1\\) and laying things out a bit more clearly. In the second diagram, I define \\(\\theta\\) as the angle from vertical that the goat can reach with a taut leash until hitting the fence. This can be calculated in terms of \\(R\\) and \\(L\\) as [\\theta = \\cos^{-1}\\frac{L/2}{R}.] This quickly leads us to the area of \\(A_1\\) as [A_1 = \\theta L^2.] The area of \\(A_2\\) is a bit more complicated, but I compared them to pizza crust for a reason. We can calculate \\(A_2\\) by cutting out a slice of pizza out of the circular field of angle \\(\\pi - 2\\theta\\) and subtracting away the cheesy triangular portion with all the toppings. This leads us to [A_2 = (\\pi/2 - \\theta) R^2 - \\frac{L}{2} R \\sin \\theta] Finally, we can combine these areas to calculate the entire grazing area as [A_g = A_1 + 2 A_2 = \\theta L^2 + 2 \\left[ (\\pi/2 - \\theta) R^2 - \\frac{L}{2} R \\sin \\theta \\right]] Setting \\(A_g = \\pi R^2\\) and algebraically solving for \\(L\\) would prove difficult/impossible so I decided to use a Nelder-Mead simplex algorithm to numerically solve for \\(L\\) in the following Python script: import numpy as np from scipy.optimize import minimize targetPortion = 0.5 def grazing_area(tether): global targetPortion ### Assuming field radius is 1 for ease of calculation ### theta = np.arccos(tether/2) area = (theta*(tether**2) + 2*((np.pi/2 - theta) - tether/2*np.sin(theta)))/np.pi return (area - targetPortion)**2 res = minimize(grazing_area,0.5,method='nelder-mead',tol=1e-6) print(str(round(100*targetPortion,1))[:-2] + '% grazing at L = ' + str(round(res['x'][0],4)) + 'R') This code advises us that in order to restrict the goat to half of the grazing area, we must tie up the ravenous kid with a tether of length \\(L = 1.1587R\\). In fact, just in case the local veterinarian recommends increasing or decreasing the goat’s diet, we can vary the target portion of the grazing area to produce the dependency as shown in the graph to the right. Hope this explanation was reasonably clear! Stay classy, Riddler Nation!" }, { "title": "The (Un)Luckiest Season of All", "url": "/posts/TheUnLuckiestSeason/", "categories": "Baseball, Statistics", "tags": "baseball, luck, probability", "date": "2018-10-04 00:00:00 -0600", "content": "On September 19th, the Seattle Mariners were 84-68, a record that in any other season would send even the ficklest Seattle fan clamoring for their suddenly fashionable Griffey jerseys gathering dust in the back of their closets. But in this logic-defying year of baseball, their season was effectively over despite being a game or less behind four of the ten available playoff spots. With the A’s and Astros streaking into October, what would normally amount to at least a playoff chase was reduced to nothing more than consolation games. But before we dive too far into self-pity, Mariners fans should probably be thankful that they even got this far considering the team’s performance in terms of runs. The easiest stat to point to is run differential where, despite having won more games than lost, the M’s allowed 34 more runs than they scored. But the far more interesting stat is the Pythagorean Win Expectancy (PWE). According to this metric, they should be sitting at a .478 win percentage, a season-long difference of 12 games compared to the Mariners’ final .549 pace. Obviously, credit has to be given to the Herculean effort of Edwin Diaz and his 57 saves, but a little bit of luck had to have been involved, considering that only nine other teams since the deadball era have ever outperformed their PWE by more. (One notable example of these outperformers is the 1981 Reds, who actually got shafted much harder than this year’s M’s due to a strike-shortened season. Despite having the best overall record in the NL, a midseason strike reset the standings, leaving Ken Griffey Sr. out in the cold.) Just to give you an idea of how unlucky/lucky this season has been, a team with 89 wins (as the Mariners ended up with) makes the playoffs about 70% of the time according to a logistic regression run by FiveThirtyEight. But a team with just 77 wins (as the M’s PWE projects them to have) makes the playoffs less than 1% of the time. The bottom line is that while we should have a better view of the mythical land of playoff baseball, the fact that we’re within a nautical mile is a miracle in itself. But that doesn’t take away any of the sting, does it? Why is a franchise that has had notoriously terrible playoff luck in recent years getting clubbed over the head by the baseball gods once again? In 2016, Rob Arthur unsurprisingly crowned the Seattle Mariners the “Unluckiest MLB Franchise Since 1998” by comparing the number of actual and expected playoff appearances based on regular season records. If we update this to include the last two seasons, it gets even worse. Based on their records, the Mariners should have been to the playoffs an additional three times in the last two decades, almost twice as many as the nearest franchise. Even if we utilize the PWE instead of the actual win percentage to calculate expected playoff appearances, the M’s remain the unluckiest team by almost an entire playoff berth. Playoff Karma By MLB Franchise Franchise Actual Expected Difference Expected (PWE) Difference (PWE) Seattle Mariners 2 5.27 3.27 4.91 2.91 Boston Red Sox 12 13.44 1.44 13.35 1.35 Tampa Bay Rays 4 5.26 1.26 4.64 0.64 Los Angeles Angels 7 8.24 1.24 6.05 -0.95 San Francisco Giants 7 8.06 1.06 7.6 0.6 Toronto Blue Jays 2 2.72 0.72 4.02 2.02 Chicago White Sox 3 3.67 0.67 3.38 0.38 Cincinnati Reds 3 3.56 0.56 3.8 0.8 Cleveland Indians 8 8.53 0.53 8.19 0.19 Washington Nationals 4 4.5 0.5 5.73 1.73 Oakland Athletics 9 9.24 0.24 8.82 -0.18 New York Mets 5 5.21 0.21 4.38 -0.62 Kansas City Royals 2 2.09 0.09 1.4 -0.6 Miami Marlins 1 1.04 0.04 0.32 -0.68 Milwaukee Brewers 3 3.02 0.02 2.18 -0.82 Detroit Tigers 5 4.98 -0.02 3.73 -1.27 Baltimore Orioles 3 2.91 -0.09 1.59 -1.41 Philadelphia Phillies 5 4.86 -0.14 5.13 0.13 Texas Rangers 7 6.8 -0.2 4.49 -2.51 Pittsburgh Pirates 3 2.62 -0.38 2.03 -0.97 San Diego Padres 3 2.5 -0.5 2.3 -0.7 Arizona Diamondbacks 6 5.48 -0.52 5.04 -0.96 Los Angeles Dodgers 10 9.28 -0.72 8.24 -1.76 New York Yankees 17 16.14 -0.86 14.26 -2.74 Houston Astros 8 6.89 -1.11 7.83 -0.17 Atlanta Braves 12 10.83 -1.17 11.51 -0.49 St. Louis Cardinals 12 10.79 -1.21 11.34 -0.66 Colorado Rockies 4 2.71 -1.29 2.09 -1.91 Chicago Cubs 8 6.57 -1.43 6.49 -1.51 Minnesota Twins 7 5.07 -1.93 2.86 -4.14 So why can’t the M’s catch a break after seventeen long years of frustration? Well, for a couple reasons: (1) That’s not how probability works. Past outcomes do not affect future probabilities. If I flip a coin and get ten heads in a row, the odds of getting heads an eleventh time are still 50-50. A baseball doesn’t remember previous seasons and then adjust its path accordingly. (2) As many of you may have seen by now, the AL has a parity (read: tanking) problem, where bad teams appear worse and good teams appear better. The standard deviation of win percentages in the AL is the largest in either league since 1962 and the range is the largest since 1954 (thanks Baltimore). This wide spread means that just being good won’t cut it. Only absurd 97-win seasons will keep you in the running, a feat that was achieved by the second wild card Oakland Athletics. So to mentally put this season to bed, the 2018 Mariners are either extremely lucky or unlucky based on what statistics you look at. I choose to see it in a positive light as a lucky one, but all of this analysis boils down to the age-old philosophical question of whether your beer is half empty or half full. Regardless, order another one because professional sports’ longest playoff drought is lasting at least another year and the A’s and Astros are playing in October. MLB season-by-season data was provided by Retrosheets and Baseball-Reference, and the Python code used to process this data is available here on my Github page." }, { "title": "Riddler Submission", "url": "/posts/RockPaperScissorsHop/", "categories": "Puzzles", "tags": "riddler, game-theory, python", "date": "2018-09-15 00:00:00 -0600", "content": "As you will come to notice throughout the course of this blog, I am a big fan of the website FiveThirtyEight and one weekly segment called The Riddler edited by Oliver Roeder. Every Friday, it challenges its readers to two math/probability/logic puzzles, one shorter, more accessible problem and one longer, more time-consuming problem for the hardcore puzzle nerds. It’s a fun way to end the week while stretching your brain and maybe picking up a computational tool or two while you’re at it. After getting hooked and submitting answers for a few months, I started itching to submit a question for the Riddler community. It took a little while, but a problem worthy of this esteemed group came to me a couple of weeks ago, and much to my surprise, it was actually published! At the time, the following video of a unique schoolyard game had been making the rounds on Facebook and it seemed like a fun thing to model: Idealized Rules of Rock-Paper-Scissors-Hop \tKids stand at either end of N hoops. \tAt the start of the game (t = 0), one kid from each end starts hopping at a speed of v = 1 hoop per second until they run into each other, either in adjacent hoops or if they land in the same hoop. \tAt that point, they play rock-paper-scissors at a rate of 1 game per second until one of the kids wins. \tThe loser goes back to their end of the hoops, a new kid steps up at that end, and the winner and the new player hop until they run into each other. \tThis process continues until someone reaches the opposing end and that player wins! Now, imagine you are a gym teacher having a bad day and you want to make sure the kids stay occupied for the entire class. If I put down 8 hoops, how long on average will the game last? How many hoops should I put down if I want the game to last for the entire 30 minute period on average? Solution Obviously, I had to have a solution in order to submit the problem, but I was flabbergasted at the kinds of the solutions that were submitted. Multiple solvers had code that was far more elegant than mine and the analytical solutions from Laurent Lessard and Tim Black (which cleverly involved a hyper-intelligent firefly) were mind-blowing. So please be warned ahead of time that my solution is not as pretty as those ones, but it still works… #AforEffort #YesIKnowHashtagsDontWorkInBlogs #OrDoThey The easiest solution would probably be a Monte Carlo simulation. Mimicking the rules of the game, the code below advances players from either end until they meet in the middle. It then picks a random number to decide which player wins the rock-paper-scissors matchup and this repeats until either side reaches the other side. The time length of the game is recorded and then the entire game is repeated 106 times to create a distribution of game lengths. My coding language of choice is Python, but this can easily been done using pretty much any language: import numpy as np numHoops = 8 leftWinProb = 1.0/3.0 rightWinProb = 1.0/3.0 hopTime = 1 rpsTime = 1 timeVals = [] for numTry in range(1000000): \tif (numTry + 1)%1000 == 0: \t\tprint('Try #' + str(numTry + 1)) \ttimeVals.append(hopTime) \tleftPos = 0 \trightPos = numHoops - 1 \twhile rightPos - leftPos &gt; 1: \t\ttimeVals[-1] += hopTime \t\tleftPos += 1 \t\trightPos -= 1 \twhile leftPos &lt; numHoops and rightPos &gt;= 0: \t\ttimeVals[-1] += rpsTime \t\trockPaperScissors = np.random.rand() \t\twhile rockPaperScissors &gt;= leftWinProb + rightWinProb: \t\t\ttimeVals[-1] += rpsTime \t\t\trockPaperScissors = np.random.rand() \t\ttimeVals[-1] += hopTime \t\tif rockPaperScissors &lt; leftWinProb: \t\t\tleftPos += 1 \t\t\trightPos = numHoops - 1 \t\telif rockPaperScissors &lt; leftWinProb + rightWinProb: \t\t\tleftPos = 0 \t\t\trightPos -= 1 \t\twhile np.all([rightPos - leftPos &gt; 1,leftPos &lt; numHoops,rightPos &gt;= 0]): \t\t\ttimeVals[-1] += hopTime \t\t\tleftPos += 1 \t\t\trightPos -= 1 del numTry I also made an analytical solution that calculates the time and probability of every possible path of a game up to 99.9% of probability. It’s a bit of a “brute force” method and is definitely impractical for larger hoop numbers, but it gets the job done. Overlaying the distributions for an eight hoop game from simulations (orange trace) and the analytical method (blue trace) produces the figure to the right, and as you can see, things line up perfectly. These distributions suggest that an average eight hoop game would last 59.46 seconds, answering the first part of the original question. If we adjust the “numHoops” variable in the code, you can also find that it takes 57 hoops for a game to take more than thirty minutes on average, answering the second part of the original question. Just for kicks, you can also generate a sample gif of a single game (like the one at the top of this article) to demonstrate how the dynamics work! My answer may not be quite as elegant as the rest of Riddler Nation, but it was fun coming up with the question in the first place and I’m inspired by the originality of everyone’s answers. Thanks for tuning in! Hope everyone had fun!" } ]