Schmnalysis: How to use Analytics
Updated: Jul 14, 2019
In my last postabout hacking the Fed, I presented a model to predict their rate decision. I thought I’d review that model’s performance, expound a bit on why I find value in an analytical approach and how, in general terms, I think model’s and analysis can be applied in the most efficient manner.
Prediction, + 20bps, Actual, 0. So much for a fancy-pants random forest model, huh? So what went wrong? In some sense, nothing. Models aren’t definitive, like any forecast, they have some probability of being correct, and some of being incorrect. In this case, a small data set was our biggest weakness. In a groundbreaking paper in 2001, Microsoft engineers showed that, essentially, the type of model was not nearly as important as the quantity of data. Here’s a recent link from the Google AI blogthat expounds on that point.
So, if you want a good model, you need lots of data. This shouldn’t be a surprise. It is what explains, in many different career areas, why experienced veterans outperform rookies; they have more data stored from their experiences. More data, better mental model, better decisions, better performance. Think Charlie Munger.
Analytics are everywhere. You can’t escape using one or being influenced by them throughout your day. But should you use them? Should you trust the analysis they present? Especially models used to make financial decisions? I build and use models in my own investing so I have a unique view.
In the Navy, I drove a minesweeper for four years. Over that time period, I learned every inch of that ship and understood every system. So, when I took the conn and drove the ship, I had a very subtle understanding of how to make the ship perform. Models and making decisions based on models are very much the same.
When building and testing a model, it is much more than just banging out some code. Almost anyone can be tasked with that. There needs to be an understanding of the meaning the model is trying to capture and a frank analysis of the model’s strengths and weaknesses. No one but the designer can understand and use that model as effectively.
So, does that mean you shouldn’t use or take into account models you didn’t build? Not necessarily. It pays to ask lots of questions and be thoughtful. For instance, when interviewing quantitative traders, I would focus on out-of-sample results; models with fantastic Sharpe ratios and R2’s in-sample would fall apart out-of-sample, like the picture below. You “eat” out of sample, ie, the future trades the model makes, so that’s what counts
If you ask enough questions and can intuit and understand a model, (and usually the simpler the model, the better), then adding it to your decision-making process should add value. How models add value is a bit more interesting.
We humans have a surprising number of mental biases built into our mushy brains. These biases did an extraordinary job of ensuring survival when being chased by lions 200,000 years ago, but now might not be so useful. Multiple studies and entire books have been written on this subject, so I won’t go into too much depth. But models primarily help with two biases I frequently see in investing.
Recency bias. We tend to think the last thing that happened will persist far into the future. Analytics help us see, and use, the breadth of data through time to balance a decision. If you reacted to every market movement you’d buy at the top and sell at the bottom. And while that might make your broker happy, it won’t do you any good. Tools that help avoid recency bias give you the knowledge to stand against the crowd.
Second, when a person forms a viewpoint they tend to look for facts that support their position and ignore those that refute it. This is confirmation bias. This is why firms tend to subconsciously hire people just like themselves. Sometimes this works, but mostly it doesn’t. As many studies have shown, the best decisions are made by diverse groups of people. Analytics provide unbiased information and can help break the chains of confirmation.
I provide as much code as I can in my blog posts. Please tear it apart, use it, become comfortable with it, or, like some of you have, point out ways I can improve it (thanks!). My hope is that it helps you become comfortable with analytics. Whether we like it or not, models are all around us, and if you can’t use them effectively, you’ll end up being used by them.