Age of Analysis

As someone who has been both a custodian and user of national public data – both gamekeeper and poacher – the unfolding story of public data in 2010 has been momentous.

At the end of the year data.gov.uk references over 5400 data sets, not at all bad in a year since the formal launch on 21^st January. With the “Show us a better way” (..to use the data) campaign rewarding applications for local recycling, cycle paths, school catchment areas and postbox locations. So lots about using the data, and loads of super apps built in quick time. The really strong emphasis to emerge has been around the data visualisation.

get the graphic

Perhaps the more subtle barometer is the Guardian Data Blog, and the extent to which the headline visuals for each story have become data visualisations. There are now 800 posts over the last two years. Of the last page of posts to the end of December 2010 nearly 50% of the visuals (7 of 15) were data visualisations. For the last 15 posts of 2009 this was just over 25% (4 posts), and for the first full page of 15 posts at the start of 2009, this was less than 7% (1 post).

So it’s certainly been the year of the visualisation. Perhaps epitomised by the BBC Joy of Stats documentary in December, broadly constructed around the dynamic visualisation of Prof. Hans Rosling showing the global heath and wealth trend over the last century, in 200 countries over 200 years using 120,000 numbers, and in four minutes. Plus of course all the design visualisations which have become more mainstream, especially those commissioned or produced by the Guardian, extending further by the flick fan club with 900 members and nearly 500 publically produced visualisations to date.

This is all great progress, but not of course and end in itself. The real point is to create better understanding from which better decisions can be made, whether for personal benefit, for the public purse, for probity or national policy. In the age of austerity decisions are tougher, but can be made easier and made more confidently with the insight that comes with the right analysis and interpretation of that data.

So for me perhaps the most significant step on the on the mainstream evolution on our civilisation’s relationship with its data is the publication by the UK Statistics Authority in July on “The value of statistical commentary”. In short – actually only a two page statement – it places clear emphasis on the messages that come from the analysis of the data, and that statistical commentary are equally important enough to be public property, just like the data. So for me that’s one step closer to the messages which is the real treasure in any analysis of any data.

So the real insight and understanding comes from a blending of the analysis and visualisation. There's been a great data and visualisation effort, with more to come on the analysis and insight activities. So here's my "effort index" on how I see the relative effort to date.

get the graphic

So if 2010 saw the full Dawn of the Data Designer, then 2011 may well start the Age of the Analyst.