ABC’s Q&A and (lack of) political party bias

Q&A is a live panel discussion show, filmed before a studio audience, produced by Australia’s ABC. It is virtually identical to BBC’s Question Time for British readers. A few months ago I noticed that all the transcripts are posted online, and I thought this would be an interesting way to analyse the political bias, and representativeness, of the show.


UK Hansard Archive Bulk Download URL File (or When is Open Data Not)

I am currently working on a project that involves large scale analysis of various countries’ Hansards (this is, transcripts of parliamentary debate). In general, this is messy data. Recent transcripts have been produced, possibly natively, in a variety of XML or SGML formats. Earlier transcripts have, where available, been digitised from printed archives.

The UK Parliament has such a digitised archive, here.

Frustratingly though, although these zipped XML files are available, there is no bulk download option or simple FTP archive of them. Instead, the files are listed in a paged format. Worse, the pages are generated by a form submit using client side javascript, so standard spidering options like curl won’t work.

So, to save anyone else the pain, here is a link to a file I built that contains links to every file in this archive. I used the handy FormRequest feature of Scrapy, my favourite, heavily used, scraping tool.


You can use this directly with wget -i urls.txt, although be warned, it has nearly 3000 files of just over 1MB each. You’re welcome.

Women in Australia’s Parliament

Is there an easy explanation for Tony Abbott’s male-dominated cabinet? Not that I can see.

Australia’s Prime Minister-elect, Tony Abbott, has received a lot of criticism for selecting only a single woman amongst his 19 cabinet ministers. ‘The cabinet of Afghanistan now has more women (three) in it’ noted acting Labor leader Chris Bowen. Or as the Guardian put it, somewhat more sedately, ‘Tony Abbott unveils experienced, male-dominated cabinet.’

This plays nicely into the Labor opposition’s continuing narrative that Abbot has a problem with women. But I’m unwilling to ascribe to malice what can be otherwise explained, so I wondered: perhaps the Guardian had inadvertently uncovered the cause. Perhaps, I thought, the Coalition simply has very few experienced female parliamentarians who would be suitable for cabinet positions.

It’s not a completely ridiculous idea: the proportion of female MPs has been steadily rising, so the median years of parliamentary experience of a female MP is certain to be less than that of a male MP. Could this explain Abbott’s cabinet?