Data is at the core of today’s digital economy. It is generated every time you make a transaction, take a photo or even enter the tube. Data powers everything from smartphones to governments, and yet one essential element of it often goes overlooked: the strength of the underlying data foundation.
In different contexts, data can mean anything from customer behaviour and financial metrics to sensor readings and social sentiment. But in general it refers to information about people, objects, and systems. However, without structure, consistency, and governance, this information remains just raw input. To unlock its full value, organisations need more than just storage solutions — they need a clear, scalable approach to managing and activating their data. That’s where the concept of a data foundation becomes critical.
A data foundation refers broadly to the tools, infrastructure, processes, and standards that ensure data is accurate, accessible, secure, and aligned with business goals. It enables companies to move from reactive decision-making to predictive, more data-informed strategies. And in an era of rapid AI integration, it has become somewhat of a necessity.
We spoke with Raman Kapoor, Head of Data at Innovate UK, about the role of data infrastructure in modern world:
The progress artificial intelligence helped to achieve in the previous couple of years is nothing short of a new industrial revolution. It transformed the way we live, work, and interact with each other. At the same time, there has been a massive increase in the amount of data generated and used. The volume and variety of data continue to grow exponentially. If done right, data and technologies like AI can boost the economy and transform public services.
Organisations that once relied on intuition are now using machine learning models and advanced analytics. But none of this would be possible without a solid data foundation. So how can companies build one that supports their growth and innovation? And what separates those that succeed from those that struggle to make sense of their data?
Unlocking this potential starts with strong foundations and continues through responsible governance, open collaboration, and forward-thinking strategies. Here are the essential factors for building the ultimate data foundations from Raman Kapoor.
Every effective data strategy requires a solid foundation of data infrastructure. The foundation requires organisations to develop structured interoperable systems which maintain accurate reliable usable data throughout its complete lifecycle. Organisations that fail to establish foundational work will end up making decisions through the use of faulty or insufficient information.
Every sector faces multiple challenges including inconsistent standards and outdated legacy systems along with fragmented data infrastructure and a lack of skilled professionals. Small businesses along with numerous organisations struggle to manage their data at a high standard because they do not have sufficient resources. The inconsistent quality of data leads to lost opportunities for innovation.
A solution to these problems begins with establishing a clear framework. A comprehensive Data Quality Framework, paired with a Data Maturity Model, helps assess and elevate data practices. Flagship projects can demonstrate real-world impact, while a centralised team of experts and cross-sector collaboration ensures consistency in standards and legal interpretation.
Data is only powerful when it’s accessible at the right time, in the right format, and to the right people. Ensuring that, while respecting privacy and security, is fundamental for achieving the benefits of the digital age.
Open data, like the EV charging network information that helps drivers plan routes and authorities improve infrastructure, demonstrates how sharing public data can unlock tangible benefits. Meanwhile, private data sharing, when done securely and ethically, allows businesses to collaborate, reduce risk, and improve services. Globally, data sharing supports cross-border research, economic growth, and joint action on challenges like climate change or pandemics.
But accessibility also demands responsibility. Legal frameworks like GDPR ensure that personal data is only shared with consent and under strict conditions. Trust must be built into every level of data handling — from encryption to anonymization, from transparency to the right to revoke consent.
Unlocking the full value of data is not the job of one sector alone. It requires coordinated action across government, business, and society:
Responsibility is not an obstacle to innovation — it’s what makes innovation sustainable. Informed consent, secure systems, and transparent practices are key to building a data-rich future that benefits everyone.
Legacy systems can slow down even the most advanced organisations. At Innovate UK, during cloud and technology migration outdated tools like Talend, WSO2, and manual reporting processes were leading to errors, inefficiencies, and a lack of trust in the data.
To fix that, Raman Kapoor, Head of Data at Innovate UK, led a multi-phase programme to modernise the entire infrastructure. As a result, the department moved from fragmented legacy systems to a centralised and scalable environment on AWS.
Raman Kapoor’s solution included replacing MySQL with PostgreSQL, consolidating data in AWS Data Lake, automating ETL processes with AWS Glue, and switching from WSO2 to AWS API Gateway for more reliable integrations. Outdated reporting tools were replaced with cloud-native dashboards, allowing teams to access real-time insights.
The implementation rolled out in five phases: auditing the existing setup and legacy inventories, migrating and integrating systems, redesigning data models and fixing data parity issues during migration, thorough testing, and staff training. Each step was grounded in feedback and careful planning to ensure minimal disruption.
Only through this collective effort can we harness the full potential of data while safeguarding the rights and trust of the public. Establishing a solid data foundation from the outset — one that prioritises accuracy, accessibility, and interoperability — is not just a technical requirement, but a strategic imperative. When every step, from auditing to implementation, is approached with intention and collaboration, the results go far beyond operational improvements. By embedding these values into the very architecture of our data systems, we don’t just solve today’s challenges — we create the conditions for sustained, meaningful innovation.
The impact of the modernisation was clear: data quality improved, reporting became more efficient, and costs went down. Most importantly, the new infrastructure laid the groundwork for making reliable, data-informed decisions at scale.
The journey of Innovate UK illustrates what’s possible when organisations commit to doing data right: higher quality, greater trust, and the agility to meet new challenges head-on. The principles are the same both for the public and private sector: strong foundations today lead to smarter decisions tomorrow. The future will be shaped by those who treat data not just as an asset, but as a shared responsibility. And in that future, the organisations with clear, purpose-driven foundations will lead.
