Data fragmentation: How government can finally connect the dots

Despite years of progress in digital service design, government data remains fragmented – locked in departmental silos, bound by legacy infrastructure, and impeded by a lack of shared standards. A panel at Think Data for Government saw experts (pictured) from both the public and private sectors agree that it’s time to finally confront the fragmentation problem and design solutions that serve the public better.

“We’re ready to call time on data fragmentation in the public sector,” said Gavin Freeguard, chairing the panel. “The digital transformation agenda has matured thanks to GDS, the Central Digital and Data Office (CDDO), and others. But the underlying data that powers public services still isn’t connected in the way it needs to be.”

Fragmented by design

The roots of the problem, according to Fay Cooper, chief product officer at Scrumconnect, lie in how services have been built.

“Traditionally, governments really designed in silos. For one perspective, there are no cross government service design groups that stand today. There are no cross government service design roles or product roles.”

Cooper pointed to a change in public perception over time: “Five years ago… there was a lot of feeling concern about trust with data. ‘I don’t want you a joined-up experience. I don’t like this, but where does my data go? Where did you get that from? Where’s it going to go?’… But I think in the past couple of years, we’ve seen a big shift when users … wanting one interaction, and they don’t understand why we make it so complex.

“Technically, there’s still a way to go. How do we achieve that? How do we make it safe and simple? … How do you make it safe and secure?”

How can we cross-collaborate?

David Nelson, lead data engineer at the Department for Education, reflected on his 12 years in government: “There’s a lot of fantastic work that goes on across government at both project and programme level… but it is just that within their own programmes. It’s working out, how can we cross, collaborate across those different programmes?”

In his experience, the public sector is transforming data “for either analytical, meaningful insights internally or for external users. And we’re all building components that could, in theory, be reused.”

But reuse is rare due to structural issues:

“There’s also an issue here around funding, in that funding is programme specific,” he said. “One of the things from the recent Spending Review, is that admin costs are being cut considerably across departments, but actually a lot of admin staff is what underpins all of our data divisions.”

He added: “We don’t want sacrifice delivery, but we also don’t want to create amounts and a technical debt.”

Recommendation for an API hub

James Freeland, senior architect at the Department for Science, Innovation and Technology (DSIT), said departments are progressing at different paces.

“There are some departments who have made progress, and you will find some other departments… who have got legacy systems, legal issues which make it impossible for them to move forward with this.”

Yet opportunities are on the horizon, he said. “There is a detailed platform that is built it would enable everyone [like] local authorities to have access to the same data… that kind of solution would be progress made in terms of sharing data.”

Here, Freeland stressed the role of APIs: “One of the recommendations… came up with the idea of an API hub. We sustained it up, and we think the API hub will get things going in terms of allowing people to set up APIs in quick time.”

Presenting data that’s easy to understand

Simon McLellan, head of data engagement at the Met Office, said while the organisation’s data doesn’t involve citizens, fragmentation still creates challenges. However, the Met Office has made progress collaborating internationally:

“How can we make our data available so that to help other people where there is an increasingly important part of people’s decision making… through purposeful data and intelligence,” he said.

“We need to find ways of describing it that means something to people that don’t have that domain knowledge of a PhD in meteorology,” he added.

Time to call time on data fragmentation

Meanwhile, Sam Hazeldine, COO of Great Wave AI and a partner at Scrumconnect, said the rise of AI is making data challenges more visible.

“I think data fragmentation and data quality is becoming more transparent to everyone as they try and utilise generative AI… it’s time to call time on data fragmentation.”

He also urged pragmatism: “There’s still work that can be done around generative AI and using that technology for kind of point solutions.”

But he added: “We need serious leadership at the top to go, ‘This stuff needs to happen’.”

Ultimately, said Cooper: “It’s not joining up everything. It’s joining up where it matters, where it makes a real impact for users.”