By Neil Sandle, Product Management Director, Asset Control
The ability to track and visualise data lineage is becoming increasingly important in financial services. When valuing portfolios or modelling risk, or investigating referential data in regulatory reporting, firms need to trace data back to its source, demonstrate who has looked at it, what quality verification it has undergone and what tests it has passed.
The need for firms to explain and document data lineage in this way has become an integral part of data quality frameworks deployed across the sector and is enshrined in many recent regulations. In addition, from an information security or content licensing perspective, lineage is also needed to connect data to licensing policies and/or access restrictions. Yet, many firms still struggle to get it right.
Financial services organisations often lose track of individual data elements in the overall flow. Many neither fully record the data sources they have used to make calculations nor document parameter or business rule changes.
The main issue firms face is that the data management applications they use don’t trace the data, underlying the master copy, or its connections with the approved data used downstream. Added to this, many businesses operate over a dispersed infrastructure. Different departments often retain their own data stores, including dedicated copies of data that might also be available elsewhere.
Organisations often worry that consolidating all their data into one system would lead to scaling issues and performance deterioration. Different departments, working with dedicated budgets, frequently want their own separate private data stores.
Taken together, all this means that the process at which values used in internal and external reporting are arrived at involves a journey through different applications – and contextual information is often lost.
In summary, the lack of capability in many data management systems, together with the development of convoluted architectures that encourage the retention of multiple data copies, leads to information uncertainty and quality degradation.
To mend their broken data chains to meet regulatory requirements and achieve strategic goals, firms will need to track the contextual information around a data point. That means finding a way of pinpointing the sources of data; understanding and capturing the selection process that led to a particular source being picked, and what the rules were to derive a piece of data based on underlying data. Moreover, they need to get better at keeping this information close to the data and tracking changes to the rules around data sources.
The second major opportunity presented to firms is around rationalising their infrastructure and cutting back on the number of applications they have. Beyond this, financial services firms need to focus on making the infrastructure simpler. That means fewer handover points where data leaves one application and enters another. It also means bringing users closer to the data rather than having to transport the data itself every time.
Technology can help firms achieve their goals in these respects. It has a role to play in tracking information, putting data models in place and ensuring that the metadata that is captured, retained and made accessible is sufficiently rich to keep sources together with the metadata, keep the relations in place between sources and final data, and ensure that relevant rule parameters are in place.
In a sense, therefore, technology needs to track all the data but also needs to be able to scale to guarantee easy access to fast growing data volumes and make certain that users can get their hands on that data and have an effective path to it. Users that trust this access path are less inclined to build their own databases and their own duplicate copies of data.
Data lineage is becoming more important for financial services organisations today. It is increasingly hard-wired into regulations and data quality frameworks that impact firms across the sector. Data lineage is central to firms growing need to deliver ‘explainability’. If a bank, for example, values something at £25 on a given day, it needs to explain why it valued it at this amount, how it came to that decision, what data points it used and what else it took into account.
All this context increasingly needs to be tracked. This need for firms to ‘explain’ how they arrive at certain decisions has been enshrined in recent regulations. But data lineage is not just needed for regulatory purposes, it can also help in the process of carrying out diagnostics to improve data quality and in content licensing, in knowing the origins of a particular piece of data. In light of recent regulations like GDPR, it can also be crucial in maintaining and managing client records in a compliant way.
Overall, the need for data lineage is making financial services firms increasingly data-centric in their approach. They have to connect all the various pieces in their infrastructure and trace the journey of data from A to B. In doing that, they need to put the right processes and systems in place. That’s crucial already today and it is likely to become ever more so in the future.