Nov 29, 2022

Data environments at many investment firms have not changed for a decade, and firms use tools that have simply not kept up. In order to support the business with the data that firms require, firms are seeking a platform that provides the agility required for data pipelines and self-service analytics. As firms modernize their investment tools, they also find themselves simultaneously struggling to modernize their data platforms. Cloud-native tools are replacing legacy systems, and asset managers, which have long valued data as a strategic asset, have embarked on a mission to develop “a modern data platform” to enable their data-driven organizations. So what does a modern data platform look like within an investment firm?

Since a picture is worth a thousand words, please refer to the following infographic …

The Modern Data Platform

Cutter defines a modern data platform as a collection of cloud-native tools centered on a cloud data warehouse or data lakehouse. Together, these two elements comprise a best-of-breed data platform that enables a data-driven organization.

A modern data platform can drastically shorten the time required to get data into your organization, support self-service analytics, and drive significant improvements in response time and performance.

At its heart, a modern data platform centers on a data warehouse and/or data lakehouse offering data storage and a query engine, along with tools to help orchestrate data pipelines and access.

Some Vendors You Should Know

As asset managers move to cloud data storage, they are expanding beyond the traditional use cases normally associated with data warehouses and data lakes. Firms traditionally used data warehouses for storing transactional data. They considered utilizing data lakes for use cases where they needed raw, unstructured, and semi-structured data. Data scientists and data analysts often preferred data lakes for data exploration and sandbox analytics.

Today, however, several vendors combine the characteristics of both data warehouses and data lakes and offer integrated analytics capabilities. If your firm wants to modernize its data platform, you should familiarize yourself with the following solutions:

Snowflake, used by many firms, is known for its ease of use and powerful performance. One popular Snowflake feature provides the ability to share data. FactSet, MSCI, S&P, and other vendors can make their data instantly available without needing to send files. State Street, BNY Mellon, and BlackRock also have started to provide clients with access to their data through Snowflake.

  • Some firms use Amazon S3 and Redshift with AWS to provide a complete toolkit, including Athena (SQL access to S3 stores), Glue (integration and prep), and Sagemaker (data science) to enable analytics and data science capabilities.
  • Many firms use Databricks, which provides a wide set of tools for data access and analytics, including its lakehouse, data sharing, and data science tools.
  • Tools like Dremio, Denodo, and Databricks provide SQL access across multiple underlying stores, including disparate database technology as data lakes.
  • Azure Synapse and Azure Data Lake are tightly coupled with Microsoft tools, such as ADF, AD, Power BI, and Purview.
  • Delta Lake provides an open-source storage framework that works with many vendors’ tool sets.

Asset managers have traditionally used EDM or ETL tools for ingestion and transformation, but many of these tools have not modernized their technology. Firms now use integration and transformation tools like Fivetran, dbt, and Matillion to integrate and transform the data by utilizing the flexibility of the new data stores.

As asset managers incorporate more investment data in addition to alternative and ESG data sets into their research process, a modern platform must integrate with data science and analytics platforms like Databricks and Azure ML, as well as BI tools like Power BI, Tableau, and Looker for reports and dashboards. To provide self-service access to all this new information, firms also use AI-driven search tools like ThoughtSpot.​

Other Tools That Can Help

Other tools available as part of a modern data platform include data catalogs, which offer metadata management and automatically classify data across the organization. Meanwhile, data quality tools give firms the ability to identify, understand, and correct flaws in data. Several vendors such as Ataccama, Collibra, Informatica, Precisely, and Talend may offer solutions inclusive of both data quality and data governance (including a data catalog), as well as master data management functionality.

Data observability tools like Monte Carlo and Databand monitor, track, and troubleshoot the end-to-end health of enterprise data systems, with the goal of eliminating “data downtime.” And many of these tools use AI to analyze and detect issues, simplifying monitoring of data pipelines. Access and security tools like Privacera support the management of data access, privacy, and security across your firm’s data architecture.

Modern times require modern technology. For more information about modern data platforms, please see Cutter’s July 2021 research report “A Guide to Effective Data Storage in Investment Management”.