Fragmented data weighing down public sector’s digital transformation plans

Fragmented data is hampering the public sector’s ability to analyse and draw meaningful insights, according to a new State of Digital Government review.

Only 27 percent of technical and non-technical leaders across 120 public organisations believe their current data infrastructure enables a comprehensive view of operations or transactions. Seventy percent say their data landscape is not well coordinated, interoperable, or enables a unified source of truth.

In a separate recent digital maturity survey completed by 76 local authorities, using and managing data was scored as the lowest maturity area with the median response neither agreeing nor disagreeing that they had a mature approach to using data.

“Fragmented data is an issue both within and across organisations. For instance, Cabinet Office’s ‘Better data for a better Cabinet Office’ strategy highlights the need for integration, yet legacy systems and skill gaps hinder effective data use and quality control. Defra faces similar problems from fragmented data having to manage overlapping digital services and data for imports, exports and fisheries, which have similar functions but are maintained separately due to differences in policy,” said the report.

“These issues are recognised externally: Citizen’s Advice highlighted specific opportunities to better harness fragmented data held across charities and the public sector to better predict and prevent public detriment, such as using one-off discretionary housing payments as a predictor of homelessness.”

The report also noted that while councils hold a wealth of data on each resident, it is a challenge to ensure that systems and data are interoperable due to barriers by legacy suppliers and/or the high costs of APIs which significantly hinder digital transformation.

But it also paid tribute, despite these limitations, to “the ingenuity of public servants has led to noteworthy results such as DfE using supplier-held data to overcome the friction of sharing between schools and enable real-time attendance visibility. Data harmonisation efforts have also delivered successes. The NHS Federated Data Platform, adopted by Northamptonshire University Hospitals, has improved theatre scheduling, and at Chelsea and Westminster Hospital, has helped to track cancer patient workflows. Home Office and DVLA developed a ‘Photo at the Roadside’ API service used by 41 police forces to confirm driver identity by retrieving the photo from drivers’ licences on a handheld device, meaning roadside checks are up to 66 percent faster and saving over 500,000 police hours. The MOJ has developed an open-source data matching product called Splink, which can link a million records a minute for justice use cases (e.g. identifying pathway of offenders), and has been adopted by public and private sector organisations around the world.”

Perfect storm of challenges creates data fragmentation

The report said data fragmentation results from a combination of technical limitations, risk-averse cultures, unclear regulations, and different governance standards. Legacy systems can also pose a major challenge to real-time data sharing, such as COBOL systems in HMRC, which require additional software to share data via APIs.

“A strong sense of data ownership and reluctance to aggregate personal data further inhibit sharing,” it said. “Departments such as Defra report resistance from ALBs that view their data as proprietary, creating silos even within the organisation. In extreme cases, data provided by one organisation will not even be ‘shared back with them’. These issues are often tied to concerns over data accuracy, limited capability to manage data appropriately, and fears of reputational damage if shared data results in errors. Organisations are also concerned that aggregating large, sensitive datasets, such as salary information from DWP, immigration status from Home Office, and address data from the Ministry of Housing, Communities and Local Government, could increase fraud risks and lead to disputes over responsibility for detection.

“Legislative frameworks such as the Digital Economy Act 2017 and the Data Protection Act, have tried to overcome barriers. However, data sharing agreements and access point APIs are often laboriously agreed, and there is also limited ability to enforce them. The ONS Integrated Data Service programme aims to improve accessibility of official data, and is accredited under the DEA 2017, but has yet to fully launch its service.”

Data fragmentation is just one area of concern highlighted in the in-depth report, which identifies five root causes for the challenges the public sector is facing. These comprise leadership, fragmentation, a lack of consistency measurement of digital performance, talent and funding.