Integrity work is hard because of core company metrics

This is a quick and dirty little post — I tried to explain a theory of integrity to a friend via a series of texts. Wonder what you think of it:

Everyone is asking “how do I understand feeds and algorithms?”. Well, luckily we don’t have to start from scratch. How do the companies themselves understand these systems that they created?

They do it through metrics. Every time a change is teed up, it’s tested in a randomized controlled trial. By comparing the changed metrics from control vs the new feature, they get a sense of how well the feature does.

Those changed metrics are the skeleton key to understanding these companies. Each team has their own particular metrics, but the entire company shares a set of top metrics — every experiment in every team is evaluated in regards to those company metrics. Those core metrics matter. The top metrics generally measure two things — growth, and engagement. Let’s simplify for the moment and shorthand it to “growth” for now.

We can think of the news feed (or twitter feed, or whatever) as being shaped by a *search*. To simplify just a bit, engineers are turning knobs of settings slightly, then checking the output — did growth go up? It’s hill-climbing. Just a slower process of what machine learning is — finding local optima in n-dimensional space. We can think of the entire platform as being shaped by that same search — not just the ranking algorithms but the design choices of features themselves!

The job of an integrity team is to *not* optimize on that metric. In a heavily optimized platform, that means that to do their work well, they’ll almost always have to erode growth somewhat. (Again, it’s not *necessarily* true, but in a world that is heavily optimized, that means that every setting is tuned perfectly to only growth). Imagine that they’re able to successfully fight the internal battles to be able to make the change that moves the company off the top of that hill. Now, every other team is heavily incentivized to roll back those changes and move back up that hill.

It doesn’t have to be conscious — often it isn’t! It’s just that there’s a juicy ability to get lots of growth impact by moving the settings back. They won’t necessarily even know they’re doing it — but it’ll probably happen.

This is why integrity work is so hard — and why organizational design needs to be part of the discussion.