Reaching Data Health Through the Covid cloud

The global pandemic has forced organisations to make decisions at an incredible pace, where every single decision has an impact on their future success. Companies must anticipate and prepare for different business scenarios with the growing Delta variant, contributing to the variety, velocity, and volume of data. Cloud migration has seen a tremendous acceleration because organisations need to share all types of data remotely. The rapid pressure for digital transformation has challenged existing technology capabilities, contributing to a Covid cloud. The difficulties accessing, organising, and sifting through the data deluge exacerbate the situation, causing new challenges using data to drive business decisions and outcomes. 

Sifting through dark data

For decades, managing data focused on the mechanics: collecting, processing, and storing the data.  It turns out this was the wrong problem to solve. The problem isn’t that we don’t have the data but that we are not using the data for any definite purpose. So much of this data ends up unused or unleveraged that most organisations end up with a digital landfill of messy, toxic dark data. They don’t have a good idea of what data they have and where it resides or even exists.

Neglecting this dark data causes a host of problems. In addition to the competitive disadvantage of not using potentially valuable data in the business and high storage cost, there is an increased risk of security breaches and significant privacy violations, and reputational damage resulting from them. It may sound overwhelming, but there are solutions to get your data out of the ‘dark’ and find new business value.  

Get your head above the cloud

The preoccupation with the mechanics of data management has created enormous challenges. Most organisations deploy a piecemeal approach to managing, integrating, and storing data, creating silos. Not only is this expensive and difficult to manage, but it also makes dark data where analysis cannot penetrate. There’s no way for the people and systems on the front lines to quickly validate if the data fueling day-to-day business decisions is reliable, timely, and compliant.  

For the most part, software, and platforms for moving, collecting, preparing, and storing data are not helping companies gain a deeper understanding of the data they have or driving better data outcomes. When you move data and even store data without consideration for quality or health, you build a digital landfill of corporate information. Instead of solving your customer and business problems, you make it harder to sort through the chaos. 

It seems some data management companies believe that storing more data or moving it to the cloud will help a business overcome these fundamental challenges. While efficiencies and new analytics platforms in the cloud are assisting companies in reinventing themselves and becoming more data-centric, more and more businesses realise that a piecemeal approach to data management – even if the data is in the cloud – is not sufficient. It’s not enough to simply collect, move, and prepare data more efficiently in the cloud. Focusing on data health becomes even more critical in this ever-growing cloud-first environment.   

Defining and maintaining healthy, high-quality data

Business data traditionally comes from internal and third-party sources and is often collected through manual entry, making it error-prone and unmanageable. In addition, when extracted from its initial environment, data loses its original context and is removed from the habits, workarounds, and best practices of its regular users, which often remain undocumented. This inevitably leads to problems when it’s applied to new purposes (i.e., analytics) where the data could be of low quality (inaccurate, out of date, incomplete, etc.) for its new purpose or, worse yet, not authorised (personally protected information).    

I see similarities between physical health and data health. Maintaining both requires discipline, logistics, and equipment. Just as human practitioners – doctors, nurses, etc. – are critical for a quality health system, so too are data professionals (and other business users) part of the solution to data health. Good data health concerns every employee, so our approach must be pervasive.  

Prognosticating data health and success

The building blocks of a sound data health system require: 

• Identification of risk factors (e.g., organisational apps or suppliers) to prevent security issues  
• Prevention programs that standardize data for easy digestion 
• Proactive “inoculation” using ML to train systems to recognize insufficient data 
• Continuous monitoring for quick data diagnosis and correct assessments 
• Protocols for persistent prognosis that improves data quality 
• Efficient treatments that balance risks and benefits accordingly 

Making data health a reality

The axiom, ‘An ounce of prevention is worth a pound of cure,’ still stands the test of time. Establishing data health is a new professional muscle and will require a lot of work and new ways of thinking. By building a culture of continuous improvement, backed by people equipped with the best tools and software available for data integrity, integration, governance, and accessibility, you protect your company from the most significant and costliest risks while optimising the value data delivers. 

Investing in data health now will enable you to rise above the Covid cloud and achieve a holistic system of preventive measures, effective treatments, along with a supportive culture that realises and sustains the true promise of actionable and timely data insights.

Krishna Tammana is Talend CTO