Now I want to talk about models versus data. Let me remind you of what we've done thus far. We have estimated a logit model to predict whether a firm is investment-grade or speculative grade. I'm going to repeat the results. The old results using the 11 predictors that we had discussed in the past, the 11 credit risk APIs. I'm going to put the precision, recall, and F1-score for our logit model here just so we have a benchmark for comparison. To refresh your memory, remember precision is the probability of correctly predicting an investment-grade outcome, conditional on predicting investment-grade more generally. When we predict investment-grade, we can either get it right or we can get it wrong. We're getting a right 76.8 percent of the time. Then the recall scores, the probability of correctly predicting an investment-grade outcome when it is an investment-grade outcome in the data. The data can either be investment-grade or speculative grade. We are getting it right 77.3 percent of the time overlapping in the data with our model. Then the F1-score is this harmonic mean or weighted average of the precision and recall, which is about 77.1 percent. What I then did is I looked at several alternative models for classifying speculative grade and investment grade firms. Specifically k-nearest neighbors, decision tree, random forest and a support vector machine. I also did something called cross-validation for all of the models including the logit to ensure I'm not just overfitting on one particular sample. The details of that are not important, let's keep this relatively high level. The point I'm trying to make here is that when I look at alternative models, I do see improvements on different dimensions. For example, the k-nearest neighbors has a very high, well, relatively high recall score of 84 percent compared to all other models. Although not surprisingly, it seems to suffer a bit on precision, at least when compared to the logit random forest and support vector machine. The decision tree, this just doesn't do particularly well along any metric relative to the other models. In terms of overall precision and recall F1-score, we can see that the random forest and the k-nearest neighbors, well, the k-nearest neighbors wins it by a slight margin of 79.6 percent. We do see some variation in the predictive accuracy of different models. We can do better than logit given our current specification of inputs. What I want to talk about now though, is comparing performance not just across different models, but also across different predictors. Remember our original predictors were the 11 credit KPIs. What I then did is I went out and somewhat old Moody's research report detailing some of the predictors that they use in their credit rating model. Now, that's not going to lead to perfectly accurate predictions for a variety of reasons, not the least of which I'm looking at S&P credit ratings, but also because there's a lot more that goes into credit ratings than just the quantifiable P&L and balance sheet data for a firm. There's a lot of discussions, there's industry analysis, competitive analysis, etc. But what I'm trying to see here is, if I push better data, more informative data through the model, will that lead to an improvement? How will that improvement compare relative to the improvement I got by looking at different models? I've put all of this on the screen here. Let me walk through and explain what it all means. The first three columns measure the precision recall and F1-score of our five different models using the 11 original predictors that we talked about from the very start of this series. The last three columns present the same metrics for the same models, but using different inputs, different x-variables, different predictors, which I'll just refer to as Moody's predictors. What I think is important to take away from this table is that the increase in predictive accuracy across all measures, moving from the original data or the original predictors to the Moody's predictors is significantly larger than any increase in accuracy obtained from alternative models. Let's just focus on the F1-score to make things concrete and simple. We can see that the F1-scores range from 71 up to 79.6 percent using the original predictors. But when we use Moody's predictors, these numbers jump up all the way to 88.6 percent in the case of the random forest. What's the message here? The message is data, not models. That's what leads to successful machine learning implementations. Let me repeat that. Data, not models. Models can help. I'm not dismissing the importance of models. I'm simply saying in terms of relative ranking, where you want to spend your resources is on better data, better understanding of the data-generating process. That's what's going to drive successful machine learning exercises. That's what's going to translate those exercises into value accretive, actionable decisions.
