Narratives and Moral Membership

  1. You subscribe to narratives that are closest fit to your limited scope of experience. (Hence, shifts in narrative are predated by events, not deliberation.)

  2. You are mostly exposed to others whose experiential reality matches yours. (Either by choice or proximity.)

  3. Homogeneity reinforces narratives.

  4. Language is contextual. Therefore narrative shapes meaning. (You are not reading what I’m writing; you are reading what you think I’m writing.) (To understand a narrative you must adopt it.)

  5. Narrative shapes experience.

  6. GOTO 1.

Do your beliefs closely match (or merely appear to) those of your in-group? Do you think this is a coincidence? — You give yourself too much credit for being a moral being.

Now consider a difficult exercise: make an attempt to detach your thinking from narratives. Now try to evaluate the world only in terms of causal relationships between actions. So that you say “This is a description of this event, or of this person’s action, and only what I know about it, and this is the result I observe (intended or unintended), and that is all I know; and anything else is unknown and pure conjecture”. Observe actions and their effects only, and subdue the instinct to form a story around it.

If the observed act was beneficial, does embellishing and forming a story around it, and saying “This is the sort of person that does such an act, and this is the sort of person that doesn’t”, make it more so? Or is it enough to study the act, its benefits, and act according to it. If the act was harmful, does it serve any purpose to distort its causes and effects through your preferred narrative lens, and dress it up in idioms so to signal to others your membership to this or that moral group, or is it enough to study the causes and effects dispassionately, and act according to your own reason.

Note then how moral narratives serve only the purpose of gatekeeping social memberships. Otherwise it’s enough to do actions that are beneficial, and refrain from actions that are harmful, without inventing stories and adorning yourself with vapid mannerisms that signal your adherence to this or that social group and their narrative.


I find there’s something about learning from a physical object (with volume, texture, scent) that makes things “click” better in my head; and I doubt I’m an oddity in this respect — the root of this effect may be more biological than preferential: For example, when recalling some piece of information, often I can remember where — within some textbook I read half a decade ago — the subject is explained. And I do mean, a literal, physical “where”: a page, (a thing,) which at all times can be found tucked somewhere within a physical stack of bound paper, which I can touch and feel; I might even remember where I was sitting (or standing, or lying), as I held that particular book, open at that particular page, when I studied the subject, and what it felt like to the touch, and how the lighting in the room fell on the paper; maybe I jotted some note on the margin, maybe I spilled tea on the pages before, giving that “where” its own unique irregular texture and color — all of which my brain will forever associate with that information. Digital can’t do any of that.

In this sense, there is nothing left to improve about the physical book.

On the other hand, if once I open this book I find myself confused, (maybe because the book was written by a guy named named Spivak, who does not like to explain what happens in between those little “=” symbols,) I can’t press on anything and expect it to expand magically and reveal more detail. It’s at this point that I will turn to my laptop, and search for more information, maybe even ask a stranger half a world away, or watch an interactive video. My copy of “Calculus on Manifolds” can’t do any of that.

So even though ebooks do offer some perks, in many ways the printed book remains superior. I think it’s a safe bet that books aren’t going anywhere. Radio did not replace print, TV did not replace radio. (Interactive did not replace passive; print did not replace the spoken word.) Each of these media is better suited for different tasks.

(I also still find that the best way to solve a problem (even a programming problem), is with pen and paper. Physically jotting down an idea beats pressing buttons while staring at an obnoxious, headache-inducing glowing screen any time (which I try to avoid as much as possible, even as a software developer). To me flipping pages still beats searching and clicking — what a dreadful user experience is a digital textbook!)

Sorting By Relative Popularity

Hey, looks like I’m sorting user content again! last time, I sorted user posts by “interestingness”; this time around I’ll be sorting players from a set of sports teams. Once again we’ll look at why sorting things based on popularity alone is a bad idea, we’ll get a primer on standard deviation, and finally a bit of scripting to put it all together.

The Task

To drive up user engagement at theScore, we introduced an onboarding screen that’s shown when you open the app for the first time.

First, you get a list of sports teams that are popular in your area, and the option to subscribe to some of them.


Now based on the teams you choose, it would be nice to also recommend some players for you to follow. So how would one go about choosing which players to recommend out of all those teams?

Let’s assume we have three six-player teams, where each player has the following number of subscribers:


  1. 100 000
  2. 110 000
  3. 90 000
  4. 80 500
  5. 140 000
  6. 140 500


  1. 120 000
  2. 250 000
  3. 180 000
  4. 135 000
  5. 157 000
  6. 202 000


  1. 3 000
  2. 100
  3. 234
  4. 301
  5. 250
  6. 400

Now let’s consider some properties we want from our algorithm:

  • A compute-once, static value: we don’t want to run our algorithm on every user. We want to give each player in our database a static recommendation_score; a numeric value that is cheap to index.
  • Simple: the algorithm should be simple and use elementary methods. Recommending players is a small part of the app; there should be little code maintenance involved.

  • Variety: The purpose of the onboarding process is to get you engaged, so we want to recommend a variety of players from different sports and leagues.

The Naive Approach

NOTE: I will be using the Julia programming language for all my examples. You can find the complete implementation at the bottom of this post.

The first obvious solution is to simply sort players by their subscription_count. The more popular the player, the more recommendable he is. Here is our naive sorting function:

function naive_sort(teams)
    [[team.players for team in teams]...],
    by=x -> x.subscribers,

Which yields:

5-element Array{PlayerRecommender.Player,1}:
  Big Team 2 Player | subscribers: 250000
  Big Team 2 Player | subscribers: 202000
  Big Team 2 Player | subscribers: 180000
  Big Team 2 Player | subscribers: 157000
  Big Team 1 Player | subscribers: 140500

The problem with this approach is that we only get results from the most popular teams. So if you’re a fan of both a very popular NFL team and another team that is not as popular (your local basketball team, perhaps), even the least popular player from the NFL team will be recommended to you, whereas the most popular player from your favorite local basketball team will not show up in your list at all!

A Better Approach

What are we really looking for?

Well, I think the players we want to recommend are the not necessarily the ones who are most famous, but rather, the ones who are most popular compared to the rest of their team, regardless of how popular that teams is (we already know you’re a fan of the team or you wouldn’t have subscribed to it in the first place).

In other words, we want an algorithm that answers the following question:

Which players deviate the most in popularity from the rest of their team?

Luckily, there’s a mathematical tool for figuring out just this: standard deviation.

Standard Deviation tl;dr

Standard deviation is a pretty straight-forward concept: Take a set of values, and figure out the average value for that set (also known as the arithmetic mean). The standard deviation simply tells us by how much the value of the typical element in our set deviates from the average.

The mathematical representation of this calculation is:

s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}

Where N is the population size, and \overline{x} is the arithmetic mean, which itself is represented by:

\overline{x}_N=\frac{1}{N}\sum_{i=1}^{N} a_i

For example, the following two sets have the same average value, but clearly the values are spread out differently:

close_to_average = [11,8,9,12]
spread_out = [0,1,19,20]

average(close_to_average) == 10
average(spread_out) == 10

standard_deviation(close_to_average) == 1.59
standard_deviation(spread_out) == 9.51

Thus we arrive at our simple scoring function. All we need to do is find players whose deviation from the average is substantially higher than that of their teammates:

function recommend(player, team)
  player_dev = player.subscribers - team.average
  if player_dev == 0
    player.recommendation_score = 0
    player.recommendation_score = player_dev / team.std_dev

Using this scoring function on our original teams, we get the following results:

5-element Array{PlayerRecommender.Player,1}:
 Small Team Player | subscribers: 3000 | score: 2.2276261544470644
 Big Team 2 Player | subscribers: 250000 | score: 1.749555170961297
 Big Team 1 Player | subscribers: 140500 | score: 1.3134034577576315
 Big Team 1 Player | subscribers: 140000 | score: 1.291753950212176
 Big Team 2 Player | subscribers: 202000 | score: 0.6445729577225832

Much better. This time around, our top player actually has the least number of subscribers, but this makes sense, because even though he belongs to a team that is not very popular, his subscription count is tenfold that of his teammates; clearly someone to keep an eye on! (perhaps a rising star in a college league? Certainly wouldn’t want our recommendation script to ignore that one.)


This algorithm has many limitations.

  • New players won’t be very well represented (they will by nature have low subscription counts).
  • All-star teams might result in nobody being particularly recommendable. Though this one might be less of a problem: thanks to our good ol’ friend the bell curve, even among rare anomalies, there are rare anomalies.

While there are ways to address these limitations and improve the accuracy of the algorithm (for example, taking into account the rate of change in subscription_count), one has to remember the purpose of this feature: to drive up user engagement during onboarding. Is the added complexity of such changes worth the minimal improvement in the recommendations?

Point is, it’s Friday night and I should go out for a beer now. I’m also looking forward to testing out the enormous Chinese fermentation jug I bought yesterday. It looks something like this, but a LOT bigger:


And it was only $30. What a bargain.

Here is the code used in these examples (working as of Julia 0.4.1). Our actual code at theScore is in Ruby.

Continue reading

The Diminishing Returns of Focusing on the Right Thing

There was a recent thread on Hacker News about the search for the “cure for aging”:

The thought has recently occurred to me that we should all be working on the “aging problem”, until it’s solved, and then spending all our extra time on other pursuits. — b_emery

I’m sure you’ve heard (or expressed) a similar sentiment about other pressing Big Problems.

After all, why should we spend millions in tax dollars funding “useless”, or at the very least, non-vital research, when there are real problems to be solved? Why do we pay pure mathematicians to dream up infinite-dimensional spheres and Hilbert spaces, when that money could be better spent training engineers to build next-generation sustainable transportation?

This line of thinking may be noble in principle (“Let’s prioritize research that focuses on finding solutions to our most urgent real-world problems, instead of wasting resources on useless/unprofitable endeavours”), but it tends to lead to stagnation, not innovation.

Focusing on big problems is important, but there are diminishing returns to focusing on the right thing, possibly even negative returns. That is, focusing too much on solutions makes us less likely to find them.

While, improvements and advancements come from focused research, paradigm-shifting discoveries tend to come from unexpected places; often based on information that is initially discarded as not useful, even by experts.

By taking away funding from “useless” pursuits (like pure maths, arts, theoretical research) and focussing solely on the bottom line (“real-world problems”, what is profitable, what is immediately useful), we make great advancements less likely, not more.

This is why we can’t simply pour infinite money into “Finding the cure for x” and expect results; we can’t pour infinite money on engineers, while defunding basic research, and expect innovation.

Reminds me of the Louie bit where David Lynch instructs Louie to “Be funny: 3… 2… 1… Go”. — Well, that’s not how funny work. Achieving that level of mastery of his craft is 90% non-funny related activities (life experiences, personal growth, self-reflection etc.), and 10% actually focusing on “being funny” in itself.

What if the key to unlocking “disease x” comes from unrelated research in metabolism, what if the techniques needed to sort through the data come from an applied mathematician investigating economic trends? What if the technology required to model the problem in order to even ask the right question in the first place is developed by computer scientists modelling data for a social network? There’s just no way to know but to explore.


I was at the lake last Summer. I picked a mossy old bench to sit, when a group of adolescents gathered to throw stones into the water. For an hour they busied around, making loud gestures and arguing over whose stone made the bigger splash. They would try every angle, and then bicker about which one spattered out the widest, or which one sent the water flying the highest, and how could they agree on what they’d seen if in a moment it was gone? Finally they approached me hoping I would settle their dispute, but I’d not seen the water splash– they scoffed, frustrated, and went on their way, and I was free to continue watching the ripples that now adorned the surface of the lake.