Fooled by Randomness
Nassim Nicholas Taleb
‘… it is not natural for us to learn from history… It is a platitude that children learn only from their own mistakes; they will cease to touch a burning stove only when they are themselves burned; no possible warning by others can lead to developing the smallest form of cautiousness… In a well-known psychology case the Swiss doctor Claparede had an amnesic patient completely crippled with her ailment. Her condition was so bad that he would have to reintroduce himself to her at a frequency of once per fifteen minutes for her to remember who he was. One day he secreted a pin in his hand before shaking hers. The next day she quickly withdrew her hand as he tried to greet her, but still did not recognize him.’
FH notes: Only the people who are hurt by a crash learn from it and get prepared for the next black swan.
Take an example of a retiree. He has an expected excess return of 15%, with a 10% error rate per annum (volatility). A 15% return with a 10% volatility per annum translates into a 93% probability of success in any given year. But seen at a narrow time scale, this translates into a mere 50.02% probability of success over any given second as shown below:
Scale
1 year
1 quarter
1 month
1 day
1 hour
1 minute
1 second
A minute-by-minute examination of his performance means that each day (assuming eight hours per day) he will have 241 pleasurable minutes against 239 unpleasurable ones. Over a short time increment, one observes the variability of the portfolio, not the returns. When I see an investor monitoring his portfolio with live prices on his cellular telephone or his handheld, I smile and smile.
Finally, I reckon that I am not immune to such an emotional defect. But I deal with it by having no access to information, except in rare circumstances (e.g. when price changes in excess of its usual volatility).
Regardless of what people claim, a negative pang is not offset by a positive one (some psychologists estimate the negative effect for an average loss to be up to 2.5 the magnitude of a positive one); it will lead to an emotional deficit.
Accordingly investors, merely for emotional reasons, will be drawn into strategies that experience rare but large variations. It is called pushing randomness under the rug. Psychologists recently found out that people tend to be sensitive to the presence or absence of a given stimulus rather than its magnitude. Therefore strategies that have small regular losses but large infrequent gains tend to be undervalued.
FH notes: rare but large variations e.g. fixed income instruments like bonds, structured products with high fixed rate returns, real estate.
One conceivable way to discriminate between a scientific intellectual and a literary intellectual is by considering that a scientific intellectual can usually recognize the writing of another but that the literary intellectual would not be able to tell the difference between lines jotted down by a scientist and those by a glib nonscientist.
FH notes: Smart people can recognize each other. I can tell apart 周顯 and Nassim from the other finance topic authors / commentators.
We do not need to be rational and scientific when it comes to the details of our daily life – only in those that can harm us and threaten our survival. Modern life seems to invite us to do the exact opposite; become extremely realistic and intellectual when it comes to such matters as religion and personal behavior, yet as irrational as possible when it comes to matters ruled by randomness (say, portfolio or real estate investments).
I was once asked in one of those meetings to express my views on the stock market. I stated, not without a modicum of pomp, that I believe that the market would go slightly up over the next week with a high probability. How high? “About 70%.” Clearly, that was a very strong opinion. But then someone interjected, “But, Nassim, you just boasted being short a very large quantity of SP500 futures, making a bet that the market would go down. What made you change your mind?” “I did not change my mind! I have a lot of faith in my bet! [Audience laughing.] As a matter of fact I now feel like selling even more!” The other employees in the room seemed utterly confused. “Are you bullish or are you bearish?” I was asked by the strategist. I replied that I could not understand the words bullish and bearish outside of their purely zoological consideration. My opinion was that the market was more likely to go up, but that it was preferable to short it, because, in the event of its going down, it could go down a lot. Suddenly, the few traders in the room understood my opinion and started voicing similar opinions. And I was not forced to come back to the following discussion.
Event
|
Probability
|
Outcome
|
Expectation
|
Market goes up
|
70%
|
Up 1%
|
0.7
|
Market goes down
|
30%
|
Down 10%
|
-3.00
|
Total
|
-2.3
|
Karl Popper:
There are only two types of theories:
1. Theories that are known to be wrong, as they were tested and adequately rejected (he called them falsified).
2. Theories that have not yet been known to be wrong, not falsified yet, but are exposed to be proved wrong.
A theory that falls outside of these two categories is not a theory. A theory that does not present a set of conditions under which it would be considered wrong would be termed charlatanism – it would be impossible to reject otherwise.
Popper’s falsificationism is intimately connected to the notion of an open society. An open society is one in which no permanent truth is held to exist; this would allow counter-ideas to emerge. Popper believed that any idea of Utopia is necessarily closed owing to the fact that it chokes its own refutations. This is also the difference between an open and a closed mind.
If the science of statistics can benefit me in anything, I will use it. If it poses a threat, then I will not. I want to take the best of what the past can give me without its dangers. Accordingly, I will use statistics and inductive methods to make aggressive bets, but I will not use them to manage my risks and exposure. Surprisingly, all the surviving traders I know seem to have done the same.
FH: Observe facts including past history to form an idea to trade on. Do not trade on large leverage on a bet that requires the absence of black swan.
I do not deny that if someone performed better than the crowd in the past, there is a presumption of his ability to do better in the future. But the presumption might be weak, very weak, to the point of being useless in decision making. Why? Because it all depends on two factors: The randomness content of his profession (e.g. pianist and dentists are professions which has little randomness) and the number of monkeys in operation (e.g. the initial pool of people entering the industry, the higher the number the higher the chance someone will have good track record out of pure luck).
The most intuitive way to describe the data mining problem to a nonstatiscian is through what is called the birthday paradox, though it is not really a paradox, simply a perceptional oddity. If there are 23 people in a room, what is the chance of there being 2 people with the same birthday? About 50%. For we are not specifying which people need to share a birthday; any pair works. Implication -> when one throws the computer at data, looking for just about any relationship, it is certain that a spurious connection will emerge, such as the fate of the stock market being linked to the length of women’s skirts.
Assuming you are standing in 1900 with hundreds of investments to look at. There are the stock markets of Argentina, Imperial Russia, the United Kingdom, Unified Germany, and plenty of others to consider. A rational person would have bought not just the emerging country of United States, but those of Russia and Argentina as well. The rest of the story is well-known; while many of the stock markets like those of the United Kingdom and the United Stated fared extremely well, the investor in Imperial Russia would have no better than medium-quality wallpaper in his hands. I wonder if those “experts” who make foolish (and self-serving) statements like “markets will always go up in any twenty-year period” are aware of this problem.
FH notes: This ties back with 周顯’s observation that 50% of blue chip companies in HSI falls out of HSI after a few decades.
People who are as close to being criminal as probability laws can allow us to infer (that is, with a confidence that exceeds the shadow of a doubt) are walking free because of our misunderstanding of basic concepts of the odds. Equally, you could be convicted for a crime you never committed, again owing to a poor reading of probability.
One Harvard lawyer used the specious argument that only 10% of men who brutalize their wives go on to murder them, which is a probability unconditional on the murder. The correct way to look at it is to determine the percentage of murder cases where women were killed by their husbands and had previously been battered by them (that is, 50%) – for we are dealing with conditional probability that O.J killed his wife conditional on the information of her having been killed, rather than the unconditional probability of O.J. killing his wife… Arguing that O.J. Simpson has 1/500,000 chance of not being the killer from the blood standpoint (remember the lawyers used the sophistry that there were four people with such blood types walking around Los Angeles) and adding to it the fact that he was the husband of the person and that there was additional evidence, then (owing to the compounding effect) the odds against him rise to several trillion trillion… I am glad to be a trader taking advantage of people’s biases but I am scared of living in such a society.
The following famous quiz was given to medical doctors (which I borrowed from the excellent Deborah Bennett’s Randomness).
A test of a disease presents a rate of 5% false positives. The disease strikes 1/1,000 of the population. People are tested at random, regardless of whether they are suspected of having the disease. A patient’s test is positive. What is the probability of the patient being stricken with the disease?
Most doctors answered 95%, simply taking into account the fact that the test has a 95% accuracy rate. The true answer is:
Number of afflicted persons
Number of true and false positives
Consider that out of 1,000 patients who are administered the test, one will be expected to be afflicted with the disease. Out of a population of the remaining 999 healthy patients, the test will identify about 50 with the disease. Therefore 1 in 51 (close to 2%).
Think of the number of times you will be given a medication that carries damaging side effects for a given disease you were told you had, when you may only have a 2% probability of being afflicted with it!
Wittgenstein’s ruler: Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table you may also be using the table to measure the ruler.
A book review, good or bad, can be far more descriptive of the reviewer than informational about the book itself. Example: the information from an anonymous reader on Amazon.com is all about the person, while that of a qualified person, is all about the book.
FH notes: You don’t have to care so much about comments / advice from other people until their authority / qualification is established.
Harvard psychologist B. F. Skinner constructed a box for rats and pigeons, where the food is delivered at random. He saw quite astonishing behavior on the part of the birds; they developed an extremely sophisticated rain-dance type of behavior in response to their ingrained statistical machinery. One bird swung its head rhythmically against a specific corner of the box, others spun their heads counterclockwise; literally all of the birds developed a specific ritual that progressively became hardwired into their minds as linked to their feeding.
This problem has a more worrying extension; we are not made to view things as independent from each other. Our bias is immediately to establish a causal link.
FH notes: Superstition is in our nature. It is conscious effort to filter out random coincidence / noise from signals / genuine causal links.
Say you own a painting you bought for $20,000, and owing to rosy conditions in the art market, it is now worth $40,000. If you owned no painting, would you still acquire it at the current price? If you would not, then you are said to be married to your position. There is no rational reason to keep a painting you would not buy at its current market rate – only an emotional investment.
FH notes: One rational reason of not acquiring at current price is that there are other investment opportunities which are more under-valued than the painting. Also this only applies to assets with no fixed income.
from
The Prophet
by Khalil Gibran
Then a woman said, Speak to us of Joy and Sorrow.
And he answered:
Your joy is your sorrow unmasked.
And the selfsame well from which your laughter rises was oftentimes filled with your tears.
And how else can it be?
The deeper that sorrow carves into your being, the more joy you can contain.
Is not the cup that holds your wine the very cup that was burned in the potter’s oven?
And is not the lute that soothes your spirit, the very wood that was hollowed with knives?
When you are joyous, look deep into your heart and you shall find it is only that which has given you sorrow that is giving you joy.
When you are sorrowful look again in your heart, and you shall see that in truth you are weeping for that which has been your delight.
Some of you say, “Joy is greater than sorrow,” and others say, “Nay, sorrow is the greater.”
But I say unto you, they are inseparable.
Together they come, and when one sits alone with you at your board, remember that the other is asleep upon your bed.
Verily you are suspended like scales between your sorrow and your joy.
Only when you are empty are you at standstill and balanced.
When the reassure-keeper lifts you to weigh his gold and his silver, needs must your joy or your sorrow rise or fall.
How to win friends and influence people
Most of our so called reasoning consists in finding arguments for going on believing as we already do.
You cannot teach a man anything: you can only help him find it within himself. - Galileo
Let the other feel the idea is his or hers.
The magic phrase that would stop arguments, eliminate ill feeling, create goodwill, and make the other person listen attentively - I don't blame you one iota for feeling as you do. If I were you I would undoubtedly feel just as you do.
Dramatise your ideas
Give the other person a fine reputation to live up to
Seven Pillars of Wisdom - T. E. Lawrence
A rebellion which stood still or went back was lost.
The Arabs said that each man believed his ticks to be gazelles.
