For better or worse, we all now experience progress. Technological, medical, economic and social progress are part of modern life. New, faster, smaller, cheaper products are being developed and sold to us. Even people in the most remote areas of the planet are affected. It is a huge force in our lives, but do we understand it? Is the dream of progress waxing or waning? Do we control progress, or does it control us?
Until the seventeenth century, the English word ‘progress’ just meant ‘the action or an act of journeying or moving onward’1. For most people, history was cyclical. Kingdoms rose and fell, but there were no underlying improvements to the average person’s quality of life. However, things were changing in the small, antagonistic and relatively backwards countries on the very western edge of the Eurasian land mass. One of the first people to notice and write about these changes was Francis Bacon. He recognized the significance of the growing number of new discoveries during this period and wanted to replace “the veneration of Aristotle with direct investigation of nature2.” A new awareness of this growing knowledge was developing and it was becoming associated with the word ‘progress’.
Bacon was a visionary and a promoter. In his time, the evidence of progress was marginal. However, according to David Wootton, by “the middle of the eighteenth century, progress had come to seem so inevitable that it was read backwards into the whole of previous history3.” It had become a defining feature of the Enlightenment and the modern age.
During the seventeenth century, many different people wrote about the new discoveries of the period and developed theories to explain the changes they saw. Encyclopedias were published to provide the public with easy access to old and new learning. One of the most ambitious of these was the Encyclopédie, which was co-founded and edited by Denis Diderot. Its aim was “to gather all available knowledge, to examine it critically and rationally, and to use it for social advancement.” Diderot wanted "to change the way people think" and thought he could do this by providing all of the world’s knowledge in one set of books. It was a huge collaborative project. Research, production and publication took over 40 years4. The final volume appeared in 1772, and by then the Encyclopédie consisted of 17 volumes of text and 11 volumes of illustrations comprising of 71,818 articles and 3,129 illustrations5 respectively. Diderot probably did not know the scale of what he was taking on. It marked the last time you could realistically think that all human knowledge could be captured in one set of books. The rate of discoveries and progress was already too fast.
Diderot’s work straddles another threshold. The Encyclopédie was battling superstition and advocating rationality. In early modern Europe, it was estimated that between 40,000 and 100,000 women were executed for witchcraft6. How people arrived at the answer to the question of whether magic existed mattered because it was used to justify or condemn the morality of killing so many women for witchcraft. The Enlightenment had one foot in an age of superstition and the other in the beginning of the modern age. Its thinkers were concerned with questions about truth, morality and breaking free from superstition. Adam Smith is famous today for writing the Wealth of Nations which focused on economic progress. However, he also wrote the Theory of Moral Sentiments which described his ideas on moral judgement.
You can think of progress as being like a car engine. The car is designed to transport you to your destination, but it is the engine that provides the power. An engine has components such as a starter motor, lubrication system, etc. Each component provides a discrete function and benefit, but the power of the engine comes from all of these parts working together. The Enlightenment writers made a good start with their ideas about progress, but they came too early to complete their theories. They had some of the parts of the engine, but they were missing some key ideas. We now have over 200 years of additional experience, and many new ideas have been thought up and discoveries made. This book is a synthesis - it uses the advances of the last 200 years to add the missing parts to the engine and show how those parts work together.
Continuing the metaphor, this book uses a symbol to represent the engine of progress. Each chapter is a component of the engine and a part of the symbol. The chapters build on each other and the symbol pulls them together to provide a single narrative. You move around the labels of the symbol a little like you would move around the numbers on a clock face. The following summary gives you an idea of how the symbol works and provides a structure.
The label on the top of the symbol is Goal. It is at the top of the symbol because having an objective determines your direction, creates context and defines what you should measure progress against. Once you know where you are going, you can measure your progress towards your destination. So, in a sense, the goal defines progress.
In order to achieve anything, you need to have an idea of what you want to do. As Kwame Nkrumah said, “Action without thought is empty.” The top inner circle represents the world of ideas. It is intertwined with the physical world, representing the process of applying an idea and the resulting changes in the physical world due to the idea’s application.
Progress is about the future: ‘Life must be lived forwards’(Søren Kierkegaard). You set an objective for some point in the future and then work towards it. For an idea to be useful and help it must make a prediction. It must tell you something about the future you are interested in. However, just like the weather forecast. It is not enough to just make a prediction - anyone can do that - the forecast has to be accurate, and reliable.
Progress needs truth or to put it the other way round if you apply false ideas you are very unlikely to make progress and much more likely to regress. The next two labels of the symbol are about the truth. The first focuses on testing the truth. Ideas that predict some events and exclude others can be physically tested. You can check that they are logically consistent and how accurate and reliable their predictions are.
Results that match the prediction support an idea’s truth and those that contradict it indicate it is untrue. Progress need truth, so it is important to be honest about the results. All the results are relevant but negative results are particularly significant, and they should not be discarded. However, it is very human only to look for supporting results, and this temptation needs to resisted.
In theory, the test results should decide the truth and what you believe: seeing is believing. However, people often have a preconceived belief and the new evidence may not be compelling. The challenge is that progress needs us to be open to new evidence. As each result comes in belief needs to be updated. If it is more supporting evidence, then our belief becomes a little stronger. However, if the evidence contradicts our current belief, then we need to give it more significance and reduce our level of belief accordingly. More and more supporting evidence may be reassuring, but it is not very informative. You might already have been very confident an idea was right, and now you are ever so slightly more confident. In the practical world, this increase makes no difference to how you will act. However, negative results are much more significant and informative. Just one completely unexpected result can question the truth of the idea. Although it may be unsettling, we learn more from negative results.
Either way, once you believe an idea is true and can help you the next thing to consider is how to apply it. Progress is largely a social enterprise: people working together towards a shared goal. In a social enterprise, individuals need to discuss and agree on the goal, test and agree which ideas will help, develop an implementation plan and then work together to apply it. The team that forms in this process is much more effective than uncooperative individuals and can achieve great things.
Having introduced the key parts of the progress, if you now take a step back and look at the symbol as a whole, new features emerge:
● One top level goal can be split into sub-goals. Each sub-goal can have its own set of ideas for achieving the goal. The symbol can be used within itself or nested.
● The symbol describes a self-correcting process. Ideas are physically tested, and those that fail are eliminated.
● Ideas and actions build on themselves. There is a cumulative effect with each new idea and application opening adjacent possibilities.
In conclusion, the maximum rate of progress is achieved when all the components of the engine of progress work together and overall progress is how far the application of the idea moves you towards the goal.
The components of progress can also be expressed mathematically. At the core of much of machine learning are mathematical algorithms for updating belief as new information is presented (learning) and in these algorithms belief is a probability calculation. The original example that shows this pattern is Bayes Theorem.
Although the language is different and it uses mathematical notation Bayes Theorem and the symbol for progress have the same shape or form:
● Idea = Hypothesis
● Results = Observation
● Belief: Prior Probability = Belief before the new test results, Posterior Probability = updated belief after including the new results
Testing is the gold standard. Executing a well-designed and rigorous experiment provides the most informative data. Bayesian statistics complements this by providing a mathematical method which uses the data to assess truth and update belief. Machine learning and Bayesian statistics are powerful techniques for handling data, and when they are harnessed to the right goals, they can drive progress.
Overall this book is a short exploration of what factors drive progress and how they work together. It is intended to illustrate how understanding the mechanics of progress can provide an explanatory and useful framework for the big and small questions of life.
1 Definition from The New Shorter Oxford English Dictionary
2 Jorge Luis Borges, The Enigma of Shakespeare (1964)
3 David Wootton, The Invention of Science, Loc 238
5 Wikipedia - Encyclopédie
6 Brian Levack (The Witch Hunt in Early Modern Europe) multiplied the number of known European witch trials by the average rate of conviction and execution, to arrive at a figure of around 60,000 deaths. Anne Lewellyn Barstow (Witchcraze) adjusted Levack's estimate to account for lost records, estimating 100,000 deaths. Ronald Hutton (Triumph of the Moon) argues that Levack's estimate had already been adjusted for these, and revises the figure to approximately 40,000. - Wikipedia