Putting the spotlight on the data profession

Stuart, what’s your role and how did you get into data?

I operate as a CTO or CIO (and sometimes Chief Engineer). Through my career in software engineering and cloud computing, I’d say I’ve always done a lot of data work – building data platforms, data integration layers, working with CMS metadata, open data, geospatial data, and I was very early into data dashboards (having built a SaaS data analytics product many years ago).

In the last few years, I’ve specialised more in pure data work and started to view the world through that lens – meaning I’ve got more into data engineering, data pipelines, analytics, data science and AI. This has coincided with moving more towards Azure (having heavily worked with AWS previously), which has a plethora of potentially great data and AI services to work with.

What’s your view of today’s data profession?

There has been a growing realisation that after years, possibly decades, spent talking about data driven decision making that we still trust the view of a person over a machine. This is because we still don’t have trust in our data – it’s often not high quality, accurate, contextualised, timely or in the right format to be truly useful to us; and the recent hype and potential of AI has shone an even brighter light on the importance of being good with data, in all senses of the word good.

Data as a profession, in my opinion, has lagged software, cloud, and other disciplines, which take the headlines; it is not quite as mature and well explored, there is a proliferation of tools, techniques, and thinking that is yet to evolve and mature, but there is so much untapped potential.

Government has, however, made efforts to embrace data. It’s restructured into DDaT teams, is aiming to provide training and support for staff to develop their data understanding and skills, and is improving sector-wide data literacy. This should have a positive impact and is something many commercial sector organisations would struggle to justify investing in.

In your opinion, what’s the ideal data team model?

Right now, the data team operates largely in a silo, and I’m yet to see data professionals and specialists forming part of multidisciplinary teams. Although we’re very far off, I would expect to see organisational structures that are designed to support a data first approach to the delivery of products and services. Currently the thinking is not there, but we’ve seen this pattern before e.g., when testing teams were siloed, or IT operations teams, and UX design teams.

A lot of people from the software world think, like I used to, that they are already experts in data. But pure data work is nuanced and arguably more complex and gnarly as you must think about software and data, and the governance and workflows are different. This absolutely necessitates a different set of skills to your typical multidisciplinary team.

How is the hype around AI having an impact on the data profession?

Everyone wants to do AI; however, the real value of AI is in the data generated and owned by a government department, which necessitates being good with data engineering – reducing technical debt, promoting interoperability, fixing data quality issues, and being able to trust your data are vitally important things. No amount of AI is currently able to fix those problems for you.

Data scientists are not able to spend as much time on their core duties e.g., downstream processing of data because of issues with data quality, data pipelines, and source systems; and this means that data scientists and other data roles are increasingly having to get involved in data engineering work.

What this tells me is that the focus on ‘creating AI solutions’ needs to be repositioned to ‘enabling AI with high-quality, timely data’ – and that is where the effort needs to go, it’s the hard yards of reducing technical debt, digitising services, data engineering, data operations, and data governance. This results in data that can be trusted and productised to fuel the delivery of business outcomes using AI or other appropriate technologies/tools.

I’d like to see a pivot away from AI looking for problems to solve to viewing AI as another viable tool to solve a problem or deliver an outcome. Treating it in isolation and as the end state risks creating another cottage industry, which takes away from solving genuine user needs.

Why do you believe data projects fail and how do we remedy them?

Data can sometimes have a bad reputation; everyone has heard a story about a failed data warehouse or data lake project (nearly as often as a failed CRM effort!), and the reason this happens can be because of approach i.e., taking on too much, not breaking the problem down, centralising solutions, and not being iterative, or not being agile. Solving this is about operating models, delivery models, user centricity, and decentralised architecture – these are skills that should be focused on as they can’t be easily bought from a supplier.

It’s far better to take a high-value, smaller data asset, productise it and run it using robust engineering practices (data ops) than to take on too much and fail. Engineering practices are critical to build at scale, and whilst you can’t apply the same engineering practices from software engineering to data (as you need to think about both software and data on a data project), there is some overlap, and you can apply the same thinking to data assets.

When talking about a deeply technical subject like data, it’s easy to forget the importance of softer skills, but these can often have the biggest impact, particularly when it comes to one of data’s key challenges – elevating the role of data in an organisation. This  is about demonstrating impact and value, storytelling, influencing, and being able to articulate ROI, building momentum and finding reasons to create change.

Pivotl are a sponsor of Think Data for Government. You can register here.