How the west was really won: by manipulative data vizualization.

One of most well-known historical theories about America is the “frontier thesis,” advanced by Frederick Jackson Turner in 1893. It states that the long existence of a “frontier,” a zone between the settled and unsettled regions of the United States, is responsible for distinctive aspects of the American character: independence, self-reliance, egalitarianism, a certain disdain for high culture and learning, etc. The United States was, at independence, a fairly narrow strip of land east of the Mississippi, but the frontier was pushed continually west until eventually it reached the Pacific and vanished altogether.

You can see this movement on a sequence of population density maps, based on decennial census results, published in the 1898 Statistical atlas of the United States. Here’s the 1790 to 1860, when they didn’t even both showing the west:



What is big data?

People often ask me

What is big data?

The answer I usually give is

Any data too large to process using your normal tools & techniques.

That’s a very context dependent answer: last week in one case it meant one million records, too big for Stata on a laptop, and in another it meant a dataset growing at about 1TB per day. To put it another way, big data is anything where you have to think about the engineering side of data science: where you can’t just open up R and run lm(), because that would take a day and need a terabyte of memory.


What is the legacy of economics, so far?

Steve Landsburg asks us to step back and gain a little perspective in his defence of economics and of economists’ failures to predict or address the 2008-and-beyond crisis. He points out that we don’t, after all, consider the failure to predict the current Ebola outbreak a failure of medicine.

His argument is summarised by the following:

If you care about human well-being, recession-fighting is small potatoes. It’s that long-term upward trend that matters. And economists, fortunately, understand a lot about what it takes to nourish that trend — things like well-enforced property rights, the rule of law, free trade, sound money, limited regulation and low marginal tax rates. Even more fortunately, economists have managed, however imperfectly and with fits and starts, to impress that understanding on the minds of policymakers. As a result (and going back, at least, to the repeal of the Corn Laws), we’ve had better policies and greater prosperity.

I don’t find this a compelling defence. He’s absolutely right that the long term trend far outweighs any short-term fluctuations (although let’s not pretend that GDP equals wellbeing: it doesn’t seem controversial that an employed person in 1980 would be happier than an unemployed person in 2008, notwithstanding the latter’s better life expectancy, iPhone, etc.).

The problem for me is that he doesn’t make the case that the particular insights or arguments of economists have really made much difference to the trend. The industrial revolution was already well on its way when the Wealth of Nations was published. It’s even less clear that 20th century economics has had much impact on the long term growth rate. The productive frontier is determined by technological innovation: perhaps we’d actually have been better off if all of the 20th century’s economists had trained as engineers? Meanwhile, economics has tried, since 1945, to push lagging countries towards the frontier. The results have been disappointing at best.

I could, perhaps, be convinced that without Smith and Ricardo we’d have less specialisation and trade; that without Keynes we’d still be locked in the currency crises of the late 1800s; and that without Schumpeter there would be no iPhone 6. But I’m skeptical. The advance of effective economic policy seems to have been at least as much a result trial-and-error, of cultural and political evolution, of empire and conquest (both military and commercial) and of the empirical reality of growth and stagnation, as it has been the result of wise economists offering counsel to receptive rulers.

This doesn’t mean I think economics is a waste of time. We still do meteorology, astronomy, cosmology – even history – although those disciplines focus largely on observation and ex-post explanation. Nobody expects a weather forecaster to intervene in a hurricane. Economists, though, have created and traded on the belief that we can successfully shepherd economies, over the short and long run. Economics has become the dominant language of policy debate, even as it is more often used as a fig leaf for pre-existing ideology. We have sewn the seeds of our own inevitable failure.

So unlike Landsburg, I do think this crisis matters. I think economics should take from it a good measure of humility. In the medical analogy, we are still somewhere in the era of the barber-surgeon.