Why economic modelling is so tricky

When I was studying economics at university in the mid-1960s, I wasn't quite sure what a "model" was. These days, politicians and business people are always using the word, and most of us think we know what they mean.

In case you're not sure, here's an explanation I would love to have seen at uni, provided by Dr Richard Denniss, executive director of the Australia Institute, in his new paper, The Use and Abuse of Economic Modelling.

"A model, be it a model car or an economic model, is a simplified representation of a more complex mechanism. A model is typically smaller, simpler and easier to build than a full-scale replica. A model sheds light on the main features of the reality it seeks to represent," Denniss writes.

"An economic 'model' is not a physical thing, like a model car. Rather, it is a mathematical representation of the linkages between selected elements of the economy."

Thus an economic model is, unavoidably, a simplified version of the economy, or an aspect of the economy. It includes those aspects of reality the model-builder regards as most important in explaining what happens, and leaves out all those aspects that don't seem to make a big difference.

So the results you get from a model are only as good as the modeller's choice of what to include and what to leave out. In practice, the model's predictions will often prove astray because some factor the modeller assumed wouldn't be important turned out to be.

Different types of economic models are used for different purposes. Earlier this week I used Denniss's paper to discuss the input-output model that industry lobbies use to make their industry sound bigger than it is.

Today let's discuss the most sophisticated models economists have developed; "computable general equilibrium" models. These are often used to shed light on the effect of a major policy change on the economy over the next 10 or 20 years.

Economists often analyse only a part of the economy, called a "partial equilibrium analysis", while assuming ceteris paribus (all other things remain equal) in the rest of the economy. This is unrealistic because, in the economy, everything is connected to everything else. Changes in one bit lead to changes in other bits, which then feed back on the first bit. So general equilibrium models attempt to capture all these interactions between different industries and markets. (In practice they can't capture all of them, so they still rely on the ceteris paribus assumption to cover those they leave out.)

If you remember nothing else about models, remember this: their weakness is they are built on a host of assumptions, and therefore are only as good as the assumptions on which they are built. Some assumptions are obviously unrealistic and couldn't possibly hold; some happen to be overtaken by events.

Models are sets of equations with dependent and independent variables. The modeller decides the values of the independent (or "exogenous") variables and the model calculates the values of the dependent ("endogenous") variables. So, if the modeller puts in the wrong independent variables (usually assumptions or guesses about the future), the dependent variables will be wrong, too.

The model-builder also specifies the "elasticity" (sensitivity) of the relationships between the variables. For instance, when the exchange rate rises will the reduction in exports be big or small? Elasticities are partly based on empirical evidence, but they also reflect the model-builder's beliefs about how the economy works. Should that belief be wrong, the model's results will be wrong.

It's common for general equilibrium models to be Keynesian in the short run (up to 10 years) but neoclassical in the long run (20 years or more). That is, key variables such as inflation, unemployment and economic growth are determined by the strength of aggregate (total) demand in the short run, but by the strength of aggregate supply in the long run.

This means the economy is assumed to be at full employment in the long run, and economic growth over the period is assumed to be determined solely by the growth in the labour force (the population of working age and its rate of participation in the labour force) plus the rate of improvement in the productivity of labour.

How do you know what the average rates of growth in population and productivity will be over the next 20 years? You take an educated guess, then plug them in. But here's the trick: once you've done that, you've predetermined where the model's results will end up, regardless of whatever policy changes you simulate happening to the economy in the meantime.

No matter how much some change knocks the economy off its assumed long-run course, the model's specifications assume it will not only get back on course but also catch up to where it would have been.

The economy can take up to 10 years to return to its "steady state".

So, the bigger the initial departure from the long-run trend, the bigger the ultimate bounce back - by design. And, by design, nothing can ever happen that changes our destiny.

Thus the model assumes away "path dependency" - the idea that where we end up is determined by what happens to us on the way; that some developments leave us permanently better off, while some leave us permanently worse off. This is clearly unrealistic.

But see what it means? It means the policy change you're purporting to be testing doesn't stand a chance of making much lasting difference, for good or ill. And that means your test is a sham. You give the appearance of testing some proposition, but the outcome is essentially predetermined.

I think such models should be used only in private by consenting economists. They have a good understanding of the assumptions on which the model's results are built and they know whether they share the modeller's faith that the economy works the way her model assumes it does.

When the results of these models are paraded before the public - by governments and treasuries, as well as interest groups - they can't help but mislead. They appear to be proving some policy change would be good or bad but, in truth, they're coming to predetermined conclusions.

The sign that the sponsors and modellers are out to mislead is shown by their failure to highlight their model's key assumptions in some sort of comprehensible product disclosure statement.

Read more >>

Damned lies and economic modelling

One of my resolutions this year is to spend more time trying to prevent lobby groups from using dodgy economic "modelling" to mislead my readers. Canberra has developed a bad case of modelling mania, but most of it is dubious. The less you know about modelling, the more it impresses you.

Any day of the week you can hear politicians demanding to see the modelling behind some figure the government has produced (even if Treasury did a quick calculation on the back of an envelope).

But the worst offenders are business interests, which pay big money to Canberra economic consultants to produce supposedly independent reports aimed at persuading governments to give them something, or at persuading the public to stop the government taking something from them.

One trick they often pull is trying to make their industry sound bigger and better for the economy than it is. Just last week the Minister for Manufacturing, Kim Carr, was defending the government's grants to the car industry by claiming it provides employment to 46,000 workers and "more than 200,000 in associated jobs".

But the industry that's been trying hardest to bolster its economic importance lately is mining. According to a press release issued by the Australian Mines and Metals Association, "213,200 people are directly employed in mining, oil and gas operations in Australia, with an additional 639,600 indirect jobs created by the resource industry".

Did you notice how the second of those suspiciously precise figures was precisely three times the first? "Spurious accuracy" is one of the signs a con job is in progress.

I'm sure you've seen the ads sponsored by the Minerals Council of Australia and others telling "Our Story" about what a wonderful, caring industry it is. The related website says that "across the nation, mining employs 187,400 people directly, and a further 599,680 in support industries". So for every mining job, another 3.2 are created elsewhere.

According to a report funded by Peabody Energy, "the Australian coal industry employs over 32,000 people and indirectly creates an additional 126,000 jobs in Queensland and New South Wales". So that's an employment "multiplier" of 3.9.

Where do these figures come from? Knowing how many people are employed in a particular industry isn't hard. Every month the Bureau of Statistics conducts a giant sample survey of households, asking them about their experience in the labour market. That's where our monthly figures for employment and unemployment come from. Every third month the bureau asks people what industry they work for.

It's obvious that all industries buy materials and other "inputs" from other industries, to which they then apply a lot of labour and equipment to produce whatever goods or services are their "output". So every industry can justly claim its purchases from other industries create jobs in those industries. Many could claim to create more jobs indirectly than directly.

But how do they know how many? They pay an economic consultant to do a report that tells them. How do the consultants know? They look up the industry's multiplier in a model-based document produced by the bureau called the input-output tables.

Trouble is, the tables are subject to significant limitations, which make it easy for them to be misused. All is explained in a paper by Dr Richard Denniss, of the Australia Institute, to be released today, The Use and Abuse of Economic Modelling in Australia: Users' Guide to Tricks of the Trade.

Like all models, the bureau's input-output model is built on a host of assumptions, as the bureau acknowledges in its accompanying documents. This reliance on assumptions shouldn't surprise you. Were you to work out a household budget for the year, you'd have to make many assumptions, including about what will happen to prices in the future.

Denniss reminds us of the key assumptions: that the relationships between inputs and outputs are fixed and so unaffected by changes in technology or changes in the relative prices of inputs; that all the output of an industry is identical, with no differences in quality or features; and that increasing the quantity of the industry's output would yield no economies of scale.

These unrealistic - but unavoidable - assumptions greatly limit the use you can make of employment multipliers without misleading yourself or others. You can't assume that doubling the size of an industry (such as mining) would also double the number of jobs it created directly and indirectly. Nor can you assume that a significant reduction in the size of an industry (such as car making) would mean the same reduction in total employment.

Another problem is that the employment multipliers involve double counting - more than one industry taking the credit for "creating" a job in some other industry. Denniss finds that if you used the multipliers to calculate the jobs directly and indirectly created by each Australian industry, then added them up, the total would be 187 per cent of all the jobs in the nation.

Yet another problem is the implication that the significant expansion of an industry would do nothing but add to employment in the industry and elsewhere. This assumes such an expansion would have no effect on wage rates, skill shortages, the exchange rate and much else.

Denniss quotes the results of some quite different modelling commissioned by the proponents of what would be one of the world's largest mines, the China First mine in Queensland. It found that, though the mine would lead to 6000 new jobs, it would also lead to the loss of about 3000 jobs, most of them in manufacturing.

We should be highly sceptical about claims made by interest groups on the basis of reports

from "independent" consultants with no acknowledgment of all the hidden assumptions.

Read more >>

Swan's sleight of hand hides mining concession

Tony Abbott is right. Julia Gillard and Wayne Swan have grossly misled the public on the cost of their abject surrender to the three big mining companies over the former resources super profits tax.

They claimed that almost halving the rate of the tax - from 40 per cent to an effective 22.5 per cent - and making various other concessions demanded by the companies would reduce tax collections by just $1.5 billion over its first two years, a mere 12.5 per cent of the originally budgeted $12 billion.

How was that unbelievably small cost achieved? Partly by shifting the goal posts. As we now know, the revenue to be raised by the new version of the tax was estimated using higher prices for coal and iron ore than were used in estimating the revenue to be raised by the original version. The new estimates also used different assumed production volumes.

To what extent do these "parameter" revisions cause the revenue cost of the policy changes to be understated? Gillard and Swan are still refusing to say. Apparently, this is none of the electorate's damn business. So we're forced to rely on estimates by people not in full command of the facts.

These suggest the government's figure of $1.5 billion over the first two years understates the value of the concessions to the big miners by

$1.6 billion (according to sharemarket analysts at Goldman Sachs JBWere), or $3 billion (according to mining tax consultants quoted by David Uren of The Australian, who deserves special mention for pursuing this issue).

Let's be clear: there's nothing wrong with the government using more up-to-date parameters when it redoes its budget figuring. No, the crime is to do so without acknowledgement, let alone without indicating the value of the parameter changes. Swan not only failed to acknowledge the change, he avoided answering a direct question on whether he had changed any of the assumptions that underpinned the revenue estimates. (These figures have not been made public - you won't find them on Swan's website - but merely "circulated" to gallery journalists, presumably because Swan had something to hide.)

Coming from a treasurer, this isn't tricky behaviour, it's dishonesty. If you can't trust the Treasurer to be honest about the cost of measures, who can you trust? I can't think of a previous treasurer who betrayed our trust so badly.

But the other part of the sleight of hand is to change the tax in ways that have implications over many years, then tell us only about the first two. Telling us more would involve making assumptions about commodity prices and exchange rates, but that's just as true of the four years of estimates produced for every budget and budget review.

It's a weak excuse that could be overcome if the government wanted to do so. So again we're forced to rely on figuring by outsiders lacking the Treasury's knowledge. The Goldman Sachs analysts estimate that, on a like-with-like basis and using quite pessimistic assumptions about commodity prices, the cost to revenue of the changes imposed by the big three will total about $35 billion by 2019-20.

With the original tax package (that is, including all the tax concessions on which the resources tax revenue was to be spent) we were given no idea of whether it was revenue neutral beyond the first two years. It may not have been because the loss of revenue arising from lifting the superannuation guarantee to 12 per cent by 2019-20 will be huge.

But whatever the position originally, it's a safe bet it will be worse now the chief payers of the minerals resources rent tax have been allowed to redesign it.

Even so, there are a few points to make. Few people have noted that, according to Swan's figures, revenue will be $1 billion higher in the first year, but $2.5 billion lower in the second. These differences partly reflect the secret parameter changes, but they also seem to reflect the choice companies were given between writing off their assets at book value at an accelerated rate over five years (36 per cent in the first year, 24 per cent in the second), or writing them off at market value over 25 years (4 per cent a year).

Since we can be sure the companies will pick the method that favours them, this choice will end up reducing the amount the tax collects. In the early years, however, those companies opting for market value will pay more tax rather than less.

But it doesn't follow that all the tax saved by the big companies equals the amount lost by the taxman. Why not? Because Gillard and Swan have allowed the big three to rejig the tax in ways that suit them at the cost of the smaller miners, particularly those in the early years of their projects and those mining ventures yet to be born.

The original tax's now-abandoned guarantee to pick up 40 per cent of losses was of little value to the big boys, but (despite their claims to the contrary) of great value to the new small boys (as was also the now-abandoned plan to give a refundable rebate rather than a simple deduction for exploration costs).

Whereas under the original tax 2500 firms would have been affected, now only about 320 will be. But this means about 2200 mining companies will now not be relieved of paying state royalties. And those remaining in the tax net will get only a deduction against profits (and a carry-forward in the event of losses), not an automatic refund.

This greatly reduces the economic efficiency gain from the new tax because so many miners will remain subject to royalties based on volume or price, not profits. Well done.

Read more >>

Model way of conning us all

A new prime minister but the same old problem: the mining industry claims the resource super-profits tax would damage it and the economy, whereas the government claims it would be great for the industry and the economy.

And both sides have "independent modelling" to support their claims.

If that doesn't make you sceptical about the use of modelling in the political debate, it should. But if you need more, try this: the two seemingly diametrically opposed modelling exercises were undertaken by the same commercial firm, KPMG.

It's taking people - even those close to the political action - a long time to wake up to the truth that the use of modelling in political arguments is just a way of conning the electorate. The less you know about economic models and how they work, the more impressed you are by their seemingly authoritative results.

The economy is a highly complex mechanism, which economists don't understand all that well. When you construct a mathematical model of the economy, you end up with a hugely oversimplified version of the real thing.

Often you can't test what you'd like to test - and what the punters assume you tested - because the model isn't sophisticated enough or because the data series you'd need don't exist. You end up with a model full of "proxies" (the best substitutes you can find). You can't model shades of grey, so you make do with black and white.

In other words, you have to make lots of assumptions. Economists don't know how the economy works; they just have rival theories about how it works. So their models are based on one theory or another.

The results thrown up by models are based heavily on the assumptions used. Use this set of assumptions, get that result. Use a different set, get a different result. Tell them what results you'd like and competent modellers can find the assumptions that produce what you want.

Economists don't accept the results of someone else's modelling until they know what assumptions were used and decide whether they consider them realistic or consistent with their own prior beliefs. Ideally, they want to determine which particular assumptions are driving the results.

Honest use of modelling results highlights the key assumptions used. But that is never the way modelling results are used in the political debate. Rather, the people who paid for the modelling quote a version of the results as impressive as possible and quite unqualified. The assumptions on which the results are based are never mentioned. They're trying to con the uninitiated.

The government paid KPMG Econtech to model the long-run effects on the economy of the resource super-profits tax and the cut in the rate of company tax. The government says the results were a "reform dividend" of a 0.7 per cent increase in long-run gross domestic product and a long-run increase in real average after-tax wages of 1.1 per cent.

If the long run is 15 or 20 or 30 years (we're not told), that's a pretty modest dividend. And the key assumption? Apparently, that the changes would make the tax system more economically efficient (because economic theory says they would).

Get it? If you thought the modelling was testing whether the changes would be good for the economy, you were conned. All the modelling tells us is by how much the changes would benefit the economy if they're economically efficient as assumed ... given all the other assumptions.

The modelling KPMG prepared for the industry lobby group, the Minerals Council of Australia, was begun in September last year - so much for the claim the industry was "ambushed" by the government.

It found that the impact of a higher effective tax rate and funding costs above the long-term government bond rate would be to reduce the net present value of new mining projects under evaluation. "This is likely to result in mining companies deferring or cancelling Australian mining projects in the short to medium term."

But what were the key assumptions used to achieve this result? I can tell you thanks to an evaluation of the study by Professor Paul Frijters, of the University of Queensland, found on the clubtroppo.com.au blog site.

He says that, rather than looking at what the new tax would do to all possible future projects, the report looks at the "second quartile" of all projects. That is, not the 25 per cent of most profitable projects but the 25 per cent after those.

"This is, of course, because the first quartile will go ahead anyway and the third quartile will probably see increases in net present value due to the cost-sharing in the resource super-profits tax. It loads the dice towards the negative to focus on only 25 per cent of all considered future projects," Frijters says.

Even so, Frijters says the study relies on two tricks to get its low net present values. In four out of six cases it assumes new projects have to obtain their equity capital at a cost of 15 per cent a year. And it assumes all projects last 30 years, even those that soon fail.

The 15 per cent required return is based on actual returns to equity over the past 30 years (including capital gains) which are unlikely to be repeated in the coming 30 years (you can't go on growing by 15 per cent a year forever), rather than the cost of obtaining capital.

Frijters says huge mining companies should be able to use corporate bonds to borrow capital for about 8 per cent. Clearly, inflating the assumed cost of capital makes projects appear less profitable.

Given this inflated cost of capital, the assumption that even failed projects last 30 years hugely reduces the value of the new tax's guarantee of a 40 per cent rebate on all losses, because firms have to wait up to 30 years to receive their rebate, with the value of that rebate indexed by only the long-term government bond rate.

I noted, too, the assumption that the guaranteed 40 per cent rebate on losses doesn't affect the cost of equity capital, the cost of borrowed capital or the proportions of each that are used.

So a key attraction of the proposed tax is effectively ignored as, remarkably, these economists assume businesses don't respond to incentives.

Frijters concludes that, in his mind, the report carries a big sticker saying "some poor competent modeller was told to make up a set of assumptions that would help the cause of a rich client".

Read more >>