Getting Your Data & App Ready for Precision Medicine

Biomedical research institutes and healthcare providers are dealing with an enormous growth of data, mainly unstructured, that is flowing from many sources – faster and faster. All types of data need to be captured, labeled, cleaned, stored, managed, analyzed, cataloged, protected and archived. The disparate data sources, types, ownership and governance create silos that impede data access, drive down efficiency, drive up costs, and slow times to insight and discovery.

The volume and complexity of data also drives the adoption of modern analytical frameworks such as big data (Hadoop and Spark) and AI (machine learning and deep learning) and applying it for the thousands of research and business applications (e.g. genomics, bioinformatics, imaging, translational and clinical). Supporting big data demands for rapidly evolving frameworks and workloads along with the collaborative nature of biomedical research requires comprehensive storage and compute capabilities.

Because of these challenges, it is imperative for the infrastructure and underlying architecture to become agile, data-driven and application-optimized – in short – becoming ready to advance precision medicine.

The Art of Possible

In 2014, I created and made public IBM Reference Architecture for Genomics to take on these challenges. It has evolved by taking on more workloads (eg medical imaging and clinical analytics) and expanded to include AI and Cloud. Since 2018, we renamed the architecture to IBM High Performance Data & AI (HPDA). It grew out of healthcare and life sciences industry use cases and leveraged IBM’s history of delivering best practices in high-performance computing, artificial intelligence and hybrid-multi Cloud. In fact, the basic HPDA framework was used to construct Summit and Sierra – currently two of the world’s most powerful supercomputers designed for data and AI. The architecture is designed to help life science organizations easily scale and expand compute and storage resources independently as demand grows, to ensure maximum performance and business continuity. It supports the wide range of development frameworks and applications required for industry innovation with optimized hardware as a foundation – without unnecessary re-investments in technology.

The HPDA reference architecture is deployable into software-defined infrastructure (SDI) that offer advanced orchestration and management capabilities. Currently it can support major computing paradigms such as traditional HPC, data lakes, large-scale analytics, machine learning and deep learning.

These capabilities then become the foundation for developing and deploying applications for fields such as genomics, imaging, clinical, real-world evidence and Internet-of-things. The HPDA architecture can be implemented on-premise in a local data center or off-premise in a private or public cloud. Our team and clients have also demonstrated and deployed advanced use case and platforms in hybrid cloud.

The architecture has a “Datahub” layer designed to manage the ocean of unstructured data that is siloed in disparate systems using advanced tiering functions, peering, and cataloging. The advanced capabilities allow the data to be captured very rapidly, stored safely, accessed easily and be shared globally in the most secured and regulation-compliant way wherever and whenever the data is needed.

The second layer is the “Orchestrator” which brings efficient scalable computational capabilities based on a shared infrastructure to schedule millions of jobs and deploy policy-driven resource management with critical functions like parallel computing and pipelining for faster time to insights and better outcomes. With the advancement of Cloud technologies such as containers and container orchestration, Orchestrator is now fully capable of deploying and manage Cloud-ready or Cloud-native workloads.                                                           

                          

The capabilities of the Datahub and Orchestrator that were designed as two separated abstraction layers can be extended all the way to the cloud. They work together moving data to balance and control the dispatching of workloads avoiding bottlenecks that may cause jobs to run slower. In the latest edition of HPDA, we introduced Unified Data Catalog use case support to further improve the integration of the two layers. Imagine now that every applications will have its data, metadata, provenance and results recorded and tracked automatically. This meta-information will then be fed into a governance catalog or platform to facilitate data exchange, ensure privacy compliance and secure access to data and information.

The logic behind this reference architecture was to free workloads from hardware constraints: assign the optimal resources needed (CPU, GPU, FPGA, VM) and address unpredictable workload requirements. This architecture is portable to different infrastructure providers, deployable to different hardware technologies and allows the workloads to become reusable through validated and hardened platforms.

This true data-driven, cloud-ready, AI-capable solution is based on deep industry experience and constant feedback from leading organizations that are at the forefront of precision medicine.

The Values to Reality

Users and infrastructure providers are achieving valuable results and significant benefits from the HPDA solution.

The key values for users:

• Ease-of-use: self-service App Center with a graphical user interface based on advanced catalog and search engines that allows users to manage the data in real-time with maximum flexibility.
• High-performance: cloud-scale data management and multicloud workload orchestration allows users to place data where it makes sense and provision the required environment for peak demand periods in the cloud, dynamically and automatically, for as long as needed, to maximize performance.
• Low cost: policy-based data management that can reduce storage costs up to 90% by automatically migrating file and object medical data to the optimal storage tier.
• Global collaboration: allows multi-tenant access and data sharing that spans across storage systems and geographic locations enabling many research initiatives around the globe with a common reference architecture to establish strategic partnerships and collaborate.

The key values for infrastructure providers:

• Easy to install: a blueprint that compiles best practices and enables IT architects to quickly deploy an end-to-end solution architecture that is designed specifically to match different use cases and requirements
• Fully tested: IT architecture based on a solid roadmap of future-ready proven infrastructure that can easily be integrated into the existing environment protecting already made investments, especially the hardware purchase and cloud services.
• Global Industry Ecosystem: wide ecosystem to align with the latest technologies for hybrid multicloud, big data analytics and AI to optimize data for cost, compliance and performance expected by end users.

Today, there are over 100 enterprise deployment of HPDA in world's largest cancer center, genome center, precision medicine projects, and research hospitals. Some large pharma and biotech have also started to adopted the architecture in their multi-Cloud infrastructure.

To learn more about HPDA and its use cases and adoption in precision medicine and drug discovery, please download and read my IBM Redbook  at http://www.redbooks.ibm.com/abstracts/redp5481.html