Should the government be doing more with the information it holds on us?

The paradox of the modern age is that whilst we are apparently more willing than ever to disclose our private thoughts to public scrutiny, the fear of what ‘Big Brother’ might do with our personal information is also high.  The reaction to recent revelations about Facebook, Google and others sharing information on users with the US government via PRISM, plus GCHQ tapping into fibre optic cables, have sparked a public debate about what we can reasonably expect to remain private.

Leaving aside the question of whether governments should be allowed to access our personal communications, concerns about the use that will be made of our data are not limited to terrorists, nor the information held by technology companies.  When survey respondents are asked whether they are content for their answers to be linked to data held on them by government departments, the proportions opting out range from 5 per cent to more than 50 per cent.  Is this reluctance to allow society to learn lessons from government-held data due to a healthy suspicion of anything that might increase our vulnerability to potential security breaches, or a knee-jerk reaction based on ignorance of the value of such data in informing evidence-based policies?

Despite the potential risks, it is my belief that it’s in the public interest to ensure that governments make full use of the information they hold on us, because ignorance costs money.  For example, following the 2011 riots, David Cameron announced a programme to help Britain’s ‘120,000 most troubled families’.  The policy intervention to turn these families around, led by Louise Casey, aims to increase participation in schooling, improve educational attainment, get adults into work and tackle health problems, crime and anti-social behaviour.  All this comes at a total cost to the taxpayer in excess of £1 billion.

How will we know whether this money has been well-spent?  To answer this question, we need to know what would have happened to these families if they hadn’t been assisted by the programme.  Surveying families who received help under the programme at different stages gives us an idea of how things changed for them over time.  But we can only survey a small proportion of the families targeted by the programme – partly because of cost and partly because not every family will participate.

Analysing the administrative data that government departments already hold on these families offers a cost-effective way of establishing whether the programme improves outcomes for those it aims to help.  Across government there exists information that can be used to track individuals over time and to observe whether anything changes as a result of participation in the programme.  I’ll be using these data to see whether the programme has had a positive impact on the range of outcome it is designed to effect.

Because departments are obliged to record certain types of information e.g. whether an individual is claiming Jobseeker’s Allowance, key fields on administrative datasets are generally well-recorded and are not subject to the problems of recall that can affect survey data.  Survey findings can also be affected by non-response bias.  For example, it’s generally harder to arrange a time to interview someone who is out at work all day than someone who is unemployed.  When it comes to monitoring longer-term impacts from government policies, the difficulties of maintaining participation by survey respondents who lose interest over follow-up surveys, or become difficult to trace, can make it virtually impossible to draw meaningful conclusions about the impact of an intervention on the full range of people subject to it.

The case for learning from the data that government departments hold on us is clear when faced with a steady decline in survey response rates, but what are the prospects for making better use of these data in the future?  The UK Administrative Data Taskforce, which reported in December 2012, made a strong case for removing limitations on access to anonymised data for the purposes of carrying out research firmly in the public interest i.e. conducted by publically-funded researchers, rather than private-sector bodies.

Perhaps the thought of a myriad of researchers trawling our data is a scary prospect, but arguably the dangers of limiting access to those working directly for a particular department are greater.  In recent years some departments have increased the extent to which analyses of the effectiveness of government policies are carried out in-house, rather than by external researchers.  In an environment where the underlying data used in these evaluations are not available for public scrutiny, this potentially leaves the independence of the findings open to question.

At present, one of barriers to greater use of administrative data to evaluate the effectiveness of government policy is that government departments differ in their ability to share information with each other.  The report by the Administrative Data Taskforce proposed setting up ‘a generic legal gateway’ to overcome this problem.  This is important, because it is only by being able to link up various bits of information on individuals, contained within datasets held by different government departments, that we start to gain a full picture of the circumstances facing individuals.  Education may affect employment prospects, but we can’t observe both of these things in a single administrative dataset, so we need to be able to collate information across multiple sources.

The security of our data must not be compromised – we all have an interest in ensuring that the highest standards of protection are maintained.  There can be few researchers who have been granted access to government data in recent years who haven’t shuddered at the potential consequences of failing to ensure its security.  The Administrative Data Taskforce acknowledges the importance of engaging with the public over the use of administrative data and ensuring that the reasons for wanting to make better use of this resource are understood.  But it is hard to justify the types of restrictions on access to data that limit the ability of policy-makers to make the right decisions and the ability of the public to hold them to account.  In a world where we are increasing conscious of the costs of big government, perhaps we should all be asking why, given that governments hold so much data on us, they aren’t making full use of it to improve policy-making?