If you're going to be an informed consumer of output from infectious disease transmission models, it's important to understand how to assess key vulnerabilities of models. Some of the most important vulnerabilities to know about include model uncertainty which includes stochastic city, poor data quality and poor model fit changes in parameters over time and experience of the modeling team in this section, we're going to discuss each of these vulnerabilities and why they're important to assessing model value. First, we're going to talk about uncertainty. Every model makes assumptions and its structure and parameters and inputs That we're not 100% sure of. In addition, there are random processes that determine transmission, known as stochastic events, which means that there's always some random process that governs when and where things occur when assessing a model's value. It's important to ask about the major sources of uncertainty in the model assumptions and how uncertainty in the model parameters was handled. The uncertainties in the modeling process, including stochastic city. Mean that the output must also include some kind of representation of that uncertainty. one common way that uncertainty is incorporated into model output is through the use of confidence or credible or prediction intervals around predicted estimates. This figure shows a classic example of what confidence intervals around model output can look like This output is from a forecasting model of COVID-19 mortality incidents in the United States. The blue solid line represents the observed data Around July 10th the prediction of future mortality begins and the purple line with circles represents the model data output. The wide light blue shaded area around the purple line with circles represents the 95 confidence interval of that estimate. The best way to interpret this interval is to say that we can be 95% confident that the true predicted mortality estimate will fall within the shaded range. There are two things to note about this confidence interval first, as you can see, it becomes increasingly larger over time and this is typical of forecasting models where uncertainty over the predictions grows over time. Second, it's important to note that the confidence interval includes possible outcomes of both increasing and decreasing trends suggesting that the forecast of what would happen next was quite uncertain. Finally look at the red line with the triangles. This line represents what happened in actuality the true incidence of death over this predicted time period. The observed values were added in over the predicted values to ascertain how well the forecast was able to predict what would happen next. As you can see here, the observed values match fairly closely with the predicted purple line showing the predictions. This shows that the model did predict well what the future trend would be, even though the estimates were highly uncertain. Stochastic city is another important component of model uncertainty stochastic events are those that we know will happen with some predictable frequency even though we don't know exactly when or where they will happen. Let's look at an example of what I mean by this. Let's assume that there is an infection and we know that the probability of infection on average is 0.5, meaning that 50% of the population on the whole will become infected. We have some confidence that this will be true over a large population, but it's still difficult to predict whether or not any one person will be infected In the figure here. You can see that we start with a population of 10 uninfected lobsters. If we apply the 0.5 probability of infection to each lobster, that's the equivalent of flipping a coin for each lobster to determine whether or not they'll be infected. In this example, you can see that Just by chance six of the lobsters end up infected. They're shown here in red. Now let's look at an example where we have The same starting population of 10 uninfected lobsters and we use the same probability of infection for each lobster a coin flip, but because of random variation in the coin flip process, only two of these lobsters are infected this time. So even though we've used the same assumptions about infection, we got a different answer. Many models include the stochastic nature of infection, which means that there's always some amount of uncertainty in the model output in this figure, you can see output from an infectious disease transmission model that includes uncertainty from stochastic city. The model was built to simulate the trajectory of an influenza outbreak. Each of the gray lines on the figure represent one run of the model or one scenario that is compatible with the model inputs. But the result can change a lot each time you run the model because of stochastic processes here, they ran the model hundreds of times the average values of those model simulations are represented with the red line and the blue line indicates what was actually observed. Another way to handle uncertainty and infectious disease transmission models is through what's called a sensitivity analysis. Sensitivity analyses allow for investigation of just how much uncertainty in a given parameter matters to the transmission process. One common way to investigate uncertainty is to compare model output from a number of different models. Each using different values of the parameter you're unsure about. For example, Assume that are not could vary anywhere from between 2-4 and you're not sure which is the right value. You could run your model using are not estimates of to run it again with are not estimates of three and yet again with are not estimates of four or anything in between. To understand how much the model output changes based on variation and your assumption about the are not value. It will help you understand how important variation and that particular parameter is for your model output. If the output does not really vary across your different models, then perhaps uncertainty about that parameter does not matter much to your model. But if it does, then investing in research or better ways to estimate the true value may be important. You can also use sensitivity analyses to investigate the importance of a given parameter on your model output. In this study, we used a strategic model to investigate how contact tracing programs could reduce the reproductive number of COVID-19. We wanted to know how important it is for contact tracing programs to be quicker and more complete and identifying and quarantining contacts. However, we thought that the answer might depend on whether or not case isolation is widespread and rapid, which is shown on the panel on the left or if it's limited and slow, which is shown in the panel on the right. Each of the colored lines in the figure, represents a certain level of completeness and quarantine of contacts. The yellow line represents a scenario where only 10% of contacts are identified and quarantined. Where the dark purple line represents 90%. Each point on each line represents the average time that contacts are identified and quarantined relative to the onset of the case. So, if you follow the trajectory of each of these lines, it shows you how much you reduce or not by improving the timeliness of contact quarantine. Let's start by looking at the left panel which represents a scenario where COVID-19 case detection in isolation is widespread and rapid. So this contact tracing program is doing very well with case identification and isolation. In this panel on the left, focusing on where the dark purple line meets the y axis highlighted here in the circle. You can see that the model suggests that in this scenario, if 90% of contacts are quarantined at the same time that the case becomes sick, then the reproductive number will be about 1.35, which is a dramatic reduction. You can also see in this left panel that if you identify in quarantine contacts, 14 days after the onset of symptoms in the case, the completeness of quarantine does not matter at all. So, whether you quarantine 10% of contacts or 90% of contacts, there's no change in the reproductive number if your identification of contacts and quarantine of contacts is that late. Now, look at the right panel. In this scenario where case finding and isolation is limited and slow, there's very little impact of contact quarantine on the reproductive number. No matter how complete or quick the quarantine of contacts is. All of the lines are very close to each other. So this sensitivity analysis shows that the impact of contact quarantine on the reproductive number, is highly influenced by the timeliness and completeness of case finding and isolation. So, we were able to use this sensitivity analysis to demonstrate that the utility of quarantine of contacts is sensitive to the performance of case finding and isolation. Beyond uncertainty, another common model vulnerability is the validity of the data used in the model. All models that use data are vulnerable to this because no data set is perfect. However, it's important to understand how imperfect the data are and where the imperfections are. It's important to understand the limitations of the data you have to understand the biases or what is missing. Sometimes the data quality are poor because the surveillance systems producing the data are new, but you expect the data quality will improve with time. That's important to consider for new outbreaks where data sources are nascent. One way that data can be limited is through incomplete capture of cases or infections and surveillance data are notoriously incomplete. Many surveillance systems rely heavily on health care seeking and access. So, in places we're seeking care or access to care are low, missingness will be more important to consider. Models that rely on case counts, may not be hampered by some missingness. But if the missingness is excessive, it could be a problem. Take the example of SARS-CoV-2 infections detection in Sitakunda, Bangladesh. This is a rural area where health care seeking and access to care are limited. From the start of the pandemic through May of 2021, there were a total of 705 clinical cases of COVID-19 reported from facility based surveillance in this area. However, a serial prevalence study in the same area, estimated that 63% of the population had antibodies against SARS-CoV-2 in May 2021. Suggesting that there had been over 200,000 infections in this area. The ratio of infections occurring to cases reported through surveillance was approximately 300 to 1. In this kind of extreme scenario using surveillance data, could be a major model vulnerability. Something else to consider when assessing model performance and value. Is that model parameters and inputs like are not or the infection fatality ratio, the number of contacts per person. Those inputs can change over time and location. This is particularly true for emerging pathogens and is a source of vulnerability for infectious disease transmission models. Let's look at another example from the COVID-19 pandemic to better understand this issue. Early in the pandemic, many studies were conducted to try to understand or not. As you know, there are many factors that will influence or not including biology, human behavior and other cultural factors. Therefore, estimates for are not varied a lot by study and by context. Here, you can see the results from a systematic review of R knot estimates for SARS-CoV-2 by country. An early estimate from the US was 5.9. One from Spain was 6.4. Well, at the bottom of the figure, you can see an estimate from Belgium that was 3.6. Importantly, note that the uncertainty around many of these estimates was high. In this kind of situation, it becomes difficult for infectious disease models to do well because the best value of this important parameter was unclear. Research studies to better estimate these parameters can sometimes be the best option for improving validity of model output and they should be repeated by country and context. Since there may be real differences in how transmission occurs across time and setting. Model validation is one way to ascertain how well a model is able to predict future outcomes. The best way to validate a model, is to compare the model output to observe data. If the historic projections match well with the data that were ultimately observed, then this can indicate that the model produces valid results. This can increase your confidence that the model will produce valid results in the future. However, it's important to keep in mind, that just because the prediction from a model doesn't match what you ultimately observe, this does not always mean that the model was wrong. For example, the goal of many models is to investigate what will happen so that you can take action to change the course of an outbreak. If you take action to reduce transmission, then that can change the course of the outbreak and make the observed case counts much different than the original prediction. This point is important to remember what considering the validity of model output. This slide compares the observed weekly case incidents of COVID-19 in the United States to the first nine rounds of modeled predictions. The observed data are in the solid gray line and each round of prediction is presented in a different color with multiple lines and shaded areas representing the uncertainty of the modeled outputs. If we're comparing the observed data to the model predictions as a way to think about model validation. The first thing to note here is that the true incidents of Covid cases was almost always within the confidence intervals of the model predictions. With the exception of the time around the emergence of the delta variant. When incidents was higher than predicted. The gaps between the predicted incidents and the observed incidents are a result of many factors including biology, changes in population immunity, human behavior evolution of the virus as well as intentional public health efforts to reduce transmission. But it's difficult to tease apart the contribution of each of these at any given time point. I want to briefly mention the issue of model overfitting, this is a particular weakness of statistical models. When models are initially built and validated against data, there is a tendency to use as many data points as possible to get the predictions as close as possible to the historical observed data. With the assumption that the closer the match between the predictions and the historic data, the better the predictions will be in the future. However, this strategy can backfire because there's always a trade off between how well the predictions fit historic data and how general Izabal the model will be to other populations or to future scenarios. So this is just something to keep in mind. Finally, models and model output can be vulnerable to the experience and expertise of the modeling team that produces them. Producing a model that's useful for policy requires technical skill, an understanding of the infectious disease transmission process and good communication between modelers and policymakers. Typically, models are made by teams and each team member may bring a particular expertise to the endeavor. This is not to say that people are teams with more limited experience cannot produce useful models. Rather, I only want to make the point that sometimes experience does matter. For example, experienced teams may have years of technical experience. They may have experience with a specific pathogen or population that's relevant for the policy question at hand. They may have collaborative networks of experts that they can consult when they encounter problems and some experience with successful prior communication with policymakers can also be an advantage. [MUSIC]
