The Hidden Truths in Political Science Models

Political science models are often used to predict and explain political phenomena. However, these models can be quite uncertain. This uncertainty comes from many different choices made during the modeling process. Most studies only look at a few of these choices, like the variables included in the model. But there are many more choices that can affect the results. These include the structure of the model, the type of statistical errors considered, how the sample is selected, and how the main variable is measured. All of these choices can lead to different results. To tackle this issue, a new approach combines two methods. The first method, Extreme Bounds Analysis, looks at many different models to see how robust the results are. The second method, the multiverse approach, considers many dimensions of modeling choices at once. This combined approach allows for a thorough check of model uncertainty across multiple aspects. It was applied to four important topics in political science: how democracies form, trust in institutions, provision of public goods, and the generosity of welfare states. The results showed that model uncertainty is widespread. This means that the results can change significantly depending on the model used. Many independent variables showed both positive and negative effects, depending on the model specification. Three different methods were used to figure out which modeling choices had the biggest impact. These methods were nearest 1-neighbor, logistic, and deep learning. All three methods showed that the choice of variables had a smaller impact compared to the sample selection and how the main variable was measured. This suggests that model uncertainty comes more from how data is collected and measured than from the variables included in the model. This has important implications for how model uncertainty should be assessed in the social sciences. It highlights the need for more careful consideration of sampling and measurement choices. The findings raise important questions about the reliability of political science models. If the results can change so much depending on the model, how can we trust them? This uncertainty should make researchers more cautious about their conclusions. It also suggests that more attention should be paid to the modeling process itself. Researchers should be more transparent about the choices they make and how these choices affect the results. This could help to build more trust in political science models and their findings. It could also lead to more robust and reliable models in the future. It is crucial to understand that models are tools, and like any tool, they have their limitations. Recognizing and addressing these limitations is key to advancing the field of political science.