It was May 13, 2009, the first day at work for Britain’s first Director of Digital Engagement, Andrew Stott. Dubbed the “Twitter Tsar” by the local press, he was to help the country’s civil servants use social media to engage with citizens more effectively.

“It was my first day on the job – I thought I would be on Twitter all day,” the now retired civil servant told Challenge during a recent trip to Singapore.

Instead, he received a letter from then Prime Minister Gordon Brown’s office, informing him that Sir Tim Berners-Lee, the computer scientist credited with inventing the Internet, had been urging the government to create an “open data” platform so that data that can be freely used, re-used and redistributed. (See “Why open data?”)

Was this feasible? Could it be done in the next six months? The Prime Minister’s office wanted to know.

“In a way, they said: ‘There’s no particular hurry about it, but could we have a full project business plan by close of today?’” recounted Mr Stott with a burst of laughter.

So instead of firing away tweets, he spent the day drawing up a plan. This would form the foundation of data.gov.uk, the British government’s central online repository of non-personal public datasets.

A month later, Sir Berners-Lee was appointed to oversee the creation of data.gov.uk while Mr Stott was tasked to make the open data portal a reality. The Guardian newspaper called him the “perfect choice” as he had worked on IT projects and in communications for many years in the government.

Changing mindsets

By January 2010, the public beta site of data.gov.uk was launched with 3,000 data-sets. This was three times the number of datasets first released by the United States’ data.gov, the world’s first national open data site. But it had been tough convincing public agencies – the data owners – to unlock their data. “The problems are not technical. It’s policies; it’s the mindsets of officials,” said Mr Stott. Hence, one of the first things he did was to appoint “open data champions” in ministries to change mindsets. Such a person must be of sufficient seniority to talk “as an equal to the person who could authorise release of data”, he said. The data.gov.uk team also aimed for the low-hanging fruit – “data that was already published but wasn’t available as re-usable data” – to score quick wins.

To qualify as open data, datasets have to be in machine-readable formats, such as spreadsheets, and not in PDFs. Converting data to reusable formats was the first step in unlocking public data.

Show, don’t just tell

Another strategy was to show data owners how their data could be used to their advantage. “The policymakers were absolutely delighted to see how their data could be used to further the department’s objectives,” said Mr Stott of the closed-door hackathons organised with the British Home Office, which looks after policing, terrorism and immigration issues. During these sessions, software developers, who signed non-disclosure agreements, coded programmes using the agencies’ data.

A successful outcome was the hyperlocal police.uk portal. Enter your postcode, town or street name and you will see details of the local police team, the crimes reported and their outcomes. Users can also track crime levels over time.

The portal is also “designed to get citizens to engage more with the police and to take part in setting local neighbourhood policing priorities” – an outcome that dovetailed with Mr Stott’s digital engagement mandate.

Create bespoke solutions

When faced with resistance, the data.gov.uk team dug deeper and found two primary concerns:

First: the nail-biting uncertainty of what the data would be used for and the deeper fear that people would use the data to criticise the government. In response, Mr Stott would tell the data owners: “That’s not your concern. It’s not your data, it’s the public’s data in the first place; they paid for it to be collected.”

Second: the fear that the published data is inaccurate. “They are afraid that people will send them corrections and they wouldn't know what to do,” he explained.

To address these concerns, a unique Open Government Licence was introduced to give users the right to freely re-use published data, even for commercial purposes, but it also protects data owners from any errors or omissions in their data.

“One thing is apparent,” said Mr Stott. “Releasing data has generated more pressure and support internally to improve data quality much more than before.”

Now more public agencies are willing to put their data to the test through crowdsourcing. For instance, when the national database of bus stops was shared on Open Street Map (a Wikipedia of maps), the site’s volunteers validated the 300,000 bus stop locations and found 18,000 that were inaccurate. The national database was promptly corrected and updated.

Encourage the re-use of data

While it is critical to unlock datasets, it is just as important that governments actively encourage the use of the data, said Mr Stott.

This can be done by releasing data on matters people care about, such as healthcare costs, or by allowing users to fill in simple online forms to request and suggest new datasets. Open data must be genuinely open so users can use it freely, for any purpose, as long as it is lawful. Onerous terms and conditions will stifle the re-use and potential, he stressed.

These days, Mr Stott advises on open data and transparency issues as a member of Britain’s Public Sector Transparency Board, and travels widely to speak on open data.

He may no longer be the Twitter Tsar, but he hasn’t stopped tweeting from his @DirDigEng account, created when he took on the digital engagement job. On August 28, 2013, he tweeted to the world: “data/gov/UK has passed 10,000 data-sets this morning – well done everyone! #opendata”.