Boosting: Can One Algorithm Save the World?

Mad World

We've all been at that low spot, the dark corner where we feel like our efforts aren't good enough. Whatever it is that the real stars of our world have, we feel we don't have it. Imposter syndrome is an insidious leech. It sucks our confidence and diseases our ability to see problems and ourselves as they are.

When problems become group problems, imposter syndrome may get lost in the tragedy of the commons. Many (most?) of the problems we face are much bigger than us. We are a part of a system, an enormous, globalized, complex system. And our influence seems to be far below what we think can make a difference.

So it is understandable that we can sometimes feel a measure of impotence. I want to describe an algorithm that gives me hope. It provides a model for how I believe we as individuals, societies, businesses, and humanity as a whole, can address and solve very complex problems.

One Algorithm to Rule them All

(Spoiler, It’s Not Deep Neural Nets)

Image Credit: New Line Cinema

Boosting is an algorithm elegant in its logic and powerful in its execution. (For those not inclined towards algorithms, just replace the word algorithm with ‘recipe’. They really mean the same thing.) The beauty of the boosting algorithm is that its main decision unit is a very simple classifier called a decision tree. Typically, these split decision trees also known as “stubs”. How these stubs make the split isn’t so important here, but know that it’s a pretty straightforward approach.

Boosting Algorithm Explained

Here's a primer on how boosting works. Boosting takes each observation (or row) of data and sequentially applies the decision trees. The decision trees decide whether the observation is in one class or another. Its output is A or B, yes or no. At first, we weight each row of data equally.

However, after the first run, the algorithm assigns higher weights to observations that are more difficult to classify. We then run the reweighted data through again. This continues for hundreds or thousands of cycles. In the end, the majority vote of all the classifiers for each row determines how they are classified.

Each generation knows observations' difficulty based on prior generations' attempts to classify them. The current generation takes that difficulty weighting and then makes the decision based on their simple approach. We pass on how this generation performed to the next generation through the reweighting of the observations.

So Many Metaphors

The boosting algorithm is powerful in a couple ways. Firstly, it outperforms many other algorithms in terms of predictive accuracy. Many of those it beats out are more mathematically sophisticated. Second, it has some great metaphorical power on the personal, business, and societal levels.

Anthropomorphizing the Algorithm

Here's where I start to anthropomorphize the algorithm. The stub is a symbol for the decision maker. To help broaden the scope of metaphors you can think of the decision maker as an individual, a single generation of parents, a single generation of society, a team of leaders at a company, a set of governmental leaders, a council of church or community leaders, the board of a university.

Each decision-making generation has only so much information and capability. The problems they face are huge and ultimately they must make a decision, so they do. They then observe their mistakes and success and this wisdom is passed along to the next generation. The next generation then focuses on the hardest problems in the set.

Practices for “Boosting” Decisions

The metaphor makes some assumptions that may highlight important practices for “boosting” decisions. These lessons help us continue to improve decision making in our organizations and personal lives.

1. We must measure our success. If we don't measure our success then the algorithm (or "recipe for success" to use a well-worn-out phrase) breaks down because you can't know how well you've done and thus can't improve.
2. We must measure our success accurately and correctly. If we measure success but inaccurately or with arbitrary or wrong metrics for our goals, then we'll head in the wrong direction.
3. We must focus on the hardest problems. If we focus on the problems our forebears have already solved, then we won't make progress on the problems that haven't been solved.
4. We must keep a record of decisions made and solutions found, including the surrounding circumstances. If we don't pass down a record of the solutions and unsolved problems, then the process starts all over again. This is knowledge transfer.
5. The next generation must actually consult that record and focus on the hardest problems. If the next generation ignores the progress made on the problems faced, then it is like starting all over again and progress is stopped or lost.

It's highly likely that you've heard and followed at least some of these practices before. The difference here is that the model motivating them comes from an unrelated field that corroborates their effectiveness.

Each generation must learn from those who came before.

Principles for Decision Makers

So what does the boosting algorithm imply for the decision makers themselves? How does this help the individual who needs to know they contribute and make a difference? How does this impact business, societal, and family decision making? Where is the hope, where is the advice, where is the progress?

1. A group of simple yet organized and systematic decision makers can in many instances make better decisions than an extremely sophisticated (genius) decision maker.
2. Related to 1: It doesn’t necessarily require genius to overcome difficult problems, sometimes it just requires persistence.
3. Measuring progress at key juncture points and making very clear and clean knowledge transfer is key to making wise decisions.
4. Focus on those issues that most greatly detract from your success.
5. Doing our best, though simple we may be, meaningfully improves the world for the next generation. This is true even if our best is only identifying or emphasizing those problems that are hardest.
6. Companies, societies, families, and organizations are likely to be more successful when A) they pass along the collected wisdom of previous generations as embodied in their knowledge, beliefs, culture, practices, and traditions. And B) the next generation takes advantage of that collected wisdom. (I feel the need to caveat this. We define wisdom, according to our algorithm, as a correct and accurate measure of success or difficulty in solving a set of problems. Just passing along any old tradition or practice isn’t sufficient.)
7. We can attack large problems by breaking them up into a series of smaller problems and then iterate on them. We do our best on generation one, then focus on the hardest parts in the next generation, and so on, until we solve the big problem satisfactorily. This hearkens to the principle of failing fast.

Step by Step

I started the article by asking whether one algorithm can save the world. Could it, if consistently and accurately followed? I believe so. But regardless of your answer to that question, I still believe it can give us some very practical advice. We can use this advice across a broad array of decision makers and on a large scope of problems.

We can identify many more principles and practices implied by the boosting algorithm. There is danger in stretching the metaphors too far, but I believe we can learn and solve a lot from this simple, elegant algorithm. I would love to see what others you come up with. Share your metaphors in the comments section and help your fellow "stubs"!