Opensee: Innovative hybrid cluster capabilities
By Eric Téa, Tech Lead, and Christophe Rivoire, UK Country Manager
- Security and compliance concerns over handling sensitive client data are holding back the complete migration by banks to Cloud-based services.
- Most banks are already exploring the Cloud or at least starting the Cloud migration.
- The challenges posed to banks by hybrid infrastructure handling very large datasets are here to stay, and the industry needs to strive to ensure all data can be accessed quickly and securely.
The evolution of the modern world: embracing modern hybrid infrastructures
Security and compliance concerns over handling sensitive client data are holding back the complete migration by banks to Cloud-based services. As regulators cautiously review Cloud usage, we expect hybrid infrastructure of Cloud and on-prem datasets will be the ‘norm’ for any large financial institutions for the foreseeable future.
The benefits of the Cloud for banks, like any other sector, are incontestable in terms of flexibility, agility, scalability and costs. Most – if not all – financial institutions have already begun exploring the Cloud or at least starting the Cloud migration. COVID 19 has accelerated this trend, making it a clear priority for the industry. McKinsey & Co estimates that more than 75% of banks store less than 25% of their data on the cloud today, yet in five years’ time more than 50% plan to have more than 50% of their data on a cloud.
The complexities and challenges of hybrid infrastructures
This hybrid situation for very large datasets creates challenges for IT and business teams alike. At Opensee we have seen first hand the host of challenges it presents in terms of regulatory and financial reporting, while curbing the insights and intelligence they can gain to improve operational performance and client service.
For large international banks with local branches or entities in multiple jurisdictions, however, the situation is more complex, since local regulators have different approaches to the Cloud for the same type of data. Taking the example of risk datasets, a large group of local regulators are strongly encouraging local banks or branches of multinational bank under their supervision to migrate to the cloud, however, simultaneously, others of their peers are making it clear to the banks or entities they supervise if they are permitted to migrate the same data to the cloud or differentiate between cloud providers.
In such a regulatory area banks running global operations have ‘hybrid’ datasets stored on the Cloud or on-prem, where potentially about 80% to 90% are eligible to migrate to the cloud and 10% to 20% are not. This puts global financial institutions in a complicated situation in meeting their requirements to calculate, analyse and report global numbers across all their different entities, especially when it comes to Value at Risk (VaR) or market risk-related capital requirements (FRTB) calculations. Risk managers cannot allow the complexity of a hybrid infrastructure to compromise the speed and optimisation they require when dealing with these massive and fragmented amounts of data.
How to deal with hybrid infrastructure challenges
From a bank’s data management and analytics perspective, it is clear that hybrid infrastructures require a good understanding of regulatory requirements and local specificities with the appropriate flexible technology to enable users to perform their own analytics seamlessly.
From a technical perspective, the on-prem/cloud data segregation can be seen as a distributed data case. Hence, this kind of data layout can be managed with usual MapReduce approaches for metrics computation. MapReduce is a programming model for processing large datasets with a parallel and distributed calculation at the cluster level. Opensee uses a distributed and replicated database for non-hybrid datasets. But to achieve regulatory data segregation in terabytes of hybrid datasets, Opensee’s team has introduced an innovative extra distribution layer at the database level to process MapReduce flows efficiently.
The need to strive during digital transformation with the help of hybrid infrastructures
The challenges posed to banks by hybrid infrastructure handling very large datasets are here to stay, certainly for as long as uncertainty over security and regulatory compliance persist. We firmly believe the industry needs to strive to ensure financial institutions can access all of their data – even fragmented – quickly and securely so business users can run and create their self-service analytics on 100% of the data regardless of the hybrid structure of the data.