Challenge
Forecasting weather requires deep scientific knowledge and vast amounts of computing power. As world-leading experts, the Met Office constantly improves their forecasting process, helping us stay safe and thrive. But how can they measure the success of a new weather model? It’s one thing to know that your model is more scientifically accurate. But when it comes to how users respond to and understand that information – it’s a whole different ball game.
By automating the sentiment analysis of vast amounts of user feedback, we allowed the Met Office to extract patterns and trends with an efficiency that was previously impossible. In turn, we knew we could then combine this with traditional methods of user research that provide in-depth, qualitative insights into user behaviours.
Our data science team helped the Met Office interrogate hundreds of thousands of individual pieces of user feedback from millions of daily users of the Met Office public website and mobile apps. The challenge was extracting a needle of insight from this haystack of information to demonstrate if a new weather model was more trusted than previous iterations.
Approach
When we joined, a sole team member was reading hundreds of comments every day, on top of their day job, trying to extract insight from multiple spreadsheets and data sources. Our data scientists quickly recognised this tiresome task was a perfect opportunity for AI.
To begin, there were conversations around which technology to use. Should we jump on the large language model (LLM) bandwagon, or use a simpler, more tailored solution? Despite the hype in the industry, we knew an LLM was not appropriate for this use case as the complexity of these systems, their unpredictable behaviour and the large cost associated with the compute would be excessive. Instead, we recommended that in these early stages, the use of well-established natural language processing (NLP) techniques to tackle the problem.
“Following the advice of our Made Tech colleagues who are experts in this field, we opted for a tried and tested NLP model to categorise the thousands of items of accuracy feedback we receive. The efficiency of this system and the amount of time it has saved extracting insight from our feedback has been invaluable,”said User Journey Manager, Fleur Layzell.
To get started, we broke this task into three problems:
- How do we collect and process so much data?
- What insight can we extract from user comments?
- How do we feed this insight back to stakeholders?
In order to process the hundreds of thousands of pieces of user feedback, we built an automated analytics pipeline. This pipeline had to be easy to maintain and robust enough to support computationally heavy machine learning (ML) tasks. It also had to safely strip any personal information that users might have submitted.
Building this platform centralised a single source of truth for all user feedback. Made Tech analysts and data scientists were then able to begin sifting through comments and developing analytics processes. It was in this space that the majority of data cleaning, enriching and preparing processes were carried out.
This central analytics environment instantly began closing the feedback loop between stakeholders and users, empowering the Met Office to begin asking for bespoke pieces of analysis and digging into feedback following feature releases. We were able to present their user feedback in a completely new light – visualising user distributions, targeting specific use cases and reporting on different weather scenarios as they happened. Something that had not been possible before.
We brought in junior developers to the team to provide valuable experience on at greenfield project. By aligning to the Met Office’s strong culture of excellent people, culture and developing capabilities, the team were able to provide a supportive space to learn, upskill and share knowledge across the team and wider Consumer Digital Programme.
Results
“Through their hard work and dedication, our colleagues at Made Tech have created a robust data pipeline. And in doing so, a more efficient way for the Met Office to not only monitor accuracy and cost reporting, but open the door to other areas where accuracy and potentially more general feedback can be monitored – increasing our validity as an organisation where user-centred experience is at the heart of everything we do,” said Layzell.
Improving the response to user feedback
Thanks to the development of the feedback analytics platform (pictured above), the Met Office can now actively monitor user feedback and measure the perceived success of new weather models in real time. Stakeholders and product owners can track user sentiment throughout the rollout and precisely quantify the impact of any changes they make to the service.
They’re able to monitor the impact of different weather systems, check model performance across different regions and environments and rapidly respond and iterate on issues as they arise – reducing the feedback-to-development loop from weeks to days. All focused on providing the best service possible for users.
User feedback analytics platform
The new analytics platform empowers the Met Office to conduct more types of data analysis. We’re now able to use powerful NLP techniques to categorise comments, allowing a shift from anecdotal to actual evidence. We can visualise and monitor feedback as it lands, and slice it however needed. For example, if the weather science team needs to know how users in Region X responded to forecast Y, this is now achievable with the click of just a few buttons. By presenting this analysis in responsive, live dashboards, Made Tech has enabled an interaction with Met Office user data like never before.
Moving forward
AI technology continues to progress at a lightning pace. Smaller, open-source generative AI solutions are now far more available, with performance and reliability nipping at the heels of the larger closed-source models. The next steps for the Met Office will be to further enhance this analysis platform by generating automated, summary reports, condensing a never-ending river of user feedback into easy-to-understand, direct points of action.
This is an excellent example of delivering value to our public sector clients. By working in partnership with us, we have led the Met Office to the adoption of new technologies that harness the power of their own data.
