Wednesday, October 16, 2013

Abusing Models in Economics [Wonkish]

Cameron Murray said some nice, though thoroughly unwarranted, things about me on his blog ‘Fresh Economic Thinking’, so I thought I would add some of my thoughts to his discussion on the use of models in the social sciences. Warning this will get wonkish.

Firstly, I think you need to be specific when discussing the use and abuse of ‘models’ in the social sciences. Economics is a large field and there are thousands of different models, with a large variety of assumptions (realistic and not) and applications. They range from a simple one line OLS estimation to a large set of simultaneous equations. Some are useful, some are not, but making sweeping generalisations about every model in the field only clouds the debate. 

Accordingly, I am going to narrow my part in this discussion to the branch I am most familiar with, and where models are probably most contentious, macro-economics, to talk about what models are used for and how they should be evaluated.

Cameron states that a model “cannot be shown to be of scientific value unless it offers useful predictions”. I think this burden is unfairly harsh – especially when it comes to macro models. There are essentially two types of predictions one could make with a model: forecasts of the impact of a change in policy relative to a baseline scenario, and broad forecasts about future economic outcomes (which usually include some assumption about future economic policy). The former are generally unverifiable. We can make an educated guess about what would happen if, for example, the US were to shut down their federal government, but given the dynamic world we live in it’s impossible to measure the accuracy of this forecast as there is a constant barrage of other events (or shocks) that will affect the US economy at the same time.

Conversely the accuracy of broad forecasts of economy (for example the growth in GDP over the next year) can be measured and compared. However, there is a vast literature that shows that these forecasts are really hard to do. Regardless of which model is used, large-scale DSGE models or small and simple vector auto-regressions (VARs) macro-model forecasts rarely produce informative forecasts. Not only that, but most professional forecasters, groups of highly trained economists using many models coupled with judgement based on years of experience, rarely forecast well either. The point is macroeconomic forecasting is hard. Really hard. I wouldn’t discard a model of the earth’s tectonic plates for falling to predict an individual earthquake, because nobody can predict earthquakes. By the same token it’s ridiculous to expect a macro-model to provide “useful predictions”, because nobody can forecast the economy. It’s an absurdly high standard.

So if a model can’t be used for forecasting then what is it good for? I agree with Cameron that model’s can be useful for telling (hopefully) plausible stories about the economy. Often you need a lot of implausible assumptions to tell these stories, but these assumptions are often necessary to be able to understand a particular mechanism or aspect of how our economy works (the price puzzle is an example of this, where structural assumptions are usually necessary to find the 'correct' casual-link between interest rates and inflation).

There are obviously a lot of commonly used assumptions which are not good representations of how the world really works. Calvo-pricing frictions are not the same as down-ward stick nominal wages, a financial accelerator is not the same as Lehman Brother’s failing, and adjustment costs are not the same as lumpy investment projects.  Each of these assumptions could be improved upon with a more complex approach to better fit reality and the data. However, while this is probably do-able for each assumption in isolation, adding complexity to every aspect of a model at once would quickly make it intractable, and thus useless.

Modelling in the social sciences is not doubt a flawed process. But in areas that don’t lend themselves to experimentation or natural experiments (such as macroeconomics) models are important tool to enable our understanding of the world we live in.

PS Of course if Cameron was referring only to micro-models in his post, then a lot of what I’ve said maybe orthogonal to his point. Though in my experience the controversy over modelling is concentrated in macro.

PPS There are other approaches out there which could handle a more complex set of assumptions, agent-based modelling for example, but these approaches are fairly new and have their own issues.

No comments:

Post a Comment