Learn how GE Healthcare used AWS to create a new AI model that interprets MRIs

MRI images are understandably complex and data intensive.

For this reason, developers Training large language models (LLMs) for MRI analysis had to split captured images into 2D. However, this only results in an approximation of the original image and thus limits the model’s ability to analyze complex anatomical structures. This leads to challenges in complex cases Brain tumorsSkeletal diseases or cardiovascular diseases.

But GE Healthcare appears to have overcome this daunting hurdle and is introducing the industry’s first full-body 3D MRI (FM) research foundation model at this year’s show AWS re:Invent. For the first time, models can use full 3D images of the entire body.

GE Healthcare’s FM was built from the ground up on AWS – there are very few models designed specifically for medical imaging like MRIs – and is based on more than 173,000 images from over 19,000 studies. The developers say they were able to train the model using five times less computational effort than before.

GE Healthcare has not yet commercialized the foundation model; it is still in an evolutionary research phase. An early reviewer, Mass General Brighamwill start experimenting with it soon.

“Our vision is to put these models in the hands of technical teams across healthcare systems, providing them with powerful tools to develop research and clinical applications faster and more cost-effectively,” said Parry Bhatia, Chief AI Officer of GE HealthCare, told VentureBeat.

Enables real-time analysis of complex 3D MRI data

Although this is a groundbreaking development, generative AI and LLMs are not new territory for the company. The team has been working with advanced technologies for more than 10 years, Bhatia explained.

One of its flagship products is AIR Recon DLa deep learning-based reconstruction algorithm that enables radiologists to obtain crisp images faster. The algorithm removes noise from raw images and improves the signal-to-noise ratio, reducing scan times by up to 50%. Since 2020, 34 million patients have been scanned with AIR Recon DL.

GE Healthcare began work on its MRI FM in early 2024. Because the model is multimodal, it can support image-to-text search, link images and words, and segment and classify diseases. The goal is to give Healthcare professionals More detail in a scan than ever before, Bhatia said, leading to faster and more accurate diagnosis and treatment.

“The model has significant potential to enable real-time analysis of 3D MRI data, which can improve medical procedures such as biopsies, radiation therapy and robotic surgery,” Dan Sheeran, GM of healthcare and life sciences at AWS, told VentureBeat.

It has already outperformed other publicly available research models on tasks such as classifying prostate cancer and Alzheimer’s disease. It has demonstrated up to 30% accuracy in matching MRI scans to text descriptions in image retrieval – which may not sound particularly impressive, but it’s a big improvement over the 3% capability of similar models.

“It’s gotten to the point where it’s producing some really solid results,” Bhatia said. “The impact is enormous.”

Do more with (much less) data

The MRI process requires a few different types of datasets to support different techniques for mapping the human body, Bhatia explained.

For example, a so-called T1-weighted imaging technique highlights fatty tissue and reduces the water signal, while T2-weighted imaging increases the water signals. The two methods complement each other and create a complete picture of the brain to help doctors detect abnormalities such as tumors, trauma or cancer.

“MRI images come in all different shapes and sizes, just like you would have books in different formats and sizes, right?” said Bhatia.

To address the challenges posed by different data sets, the developers introduced a “sizing and adapting strategy” to allow the model to handle and respond to different variations. Additionally, data may be missing in some areas – an image may be incomplete, for example – so they have trained the model to simply ignore these cases.

“Instead of getting stuck, we taught the model to jump through gaps and focus on what is available,” Bhatia said. “Think of it like solving a puzzle with some pieces missing.”

The developers also relied on semi-supervised student-teacher learning, which is particularly useful when data is limited. In this method, two different neural networks are trained on both labeled and unlabeled data, with the teacher creating labels that help the student learn and predict future labels.

“We’re now using a lot of these self-supervised technologies that don’t require large amounts of data or labels to train large models,” Bhatia said. “It reduces dependencies so you can learn more from these raw images than in the past.”

This helps the model work well in hospitals with fewer resources, older machines and different types of data sets, Bhatia explained.

He also emphasized the importance of the multimodality of the models. “Many technologies have been unimodal in the past,” Bhatia said. “It would just look at the image, at the text. But now they’re becoming multimodal, they can go from image to text, text to image, so you can bring in a lot of things that were done in the past with separate models and really unify the workflow.”

He emphasized that researchers only use datasets to which they have rights; GE Healthcare has partners who license anonymized data sets and ensure compliance standards and guidelines.

Using AWS SageMaker to address compute and data challenges

Undoubtedly, there are many challenges in creating such sophisticated models – such as the limited computing power for gigabyte-sized 3D images.

“It’s a huge volume of 3D data,” Bhatia said. “You have to get it into the memory of the model, which is a really complex problem.”

To overcome this problem, GE Healthcare built on it Amazon SageMakerwhich provides high-speed networking and distributed training capabilities across multiple GPUs and leverages Nvidia A100 and Tensor Core GPUs for large-scale training.

“Because of the size of the data and the size of the models, they can’t send it to a single GPU,” Bhatia explained. SageMaker allowed them to customize and scale operations on multiple GPUs that could interact with each other.

Developers also used Amazon FSx In Amazon S3 Object storage, which enabled faster reading and writing of records.

Bhatia pointed out that another challenge is cost optimization; Amazon’s Elastic Compute Cloud (EC2) enabled developers to move unused or infrequently used data to lower-cost storage tiers.

“Using Sagemaker to train these large models – primarily for efficient, distributed training across multiple high-performance GPU clusters – was one of the critical components that really helped us move faster,” said Bhatia.

He emphasized that all components were developed with data integrity and compliance in mind, taking into account HIPAA and other regulatory requirements and frameworks.

Ultimately, “these technologies can truly streamline, help us innovate faster, as well as improve overall operational efficiency by reducing administrative burdens, and ultimately lead to better patient care – because now you are providing more personalized care.”

Serves as a basis for further specialized fine-tuning models

While the model is currently limited specifically to the MRI field, researchers see great opportunity for expansion to other areas of medicine.

Sheeran pointed out that AI in medical imaging has historically been limited by the need to develop custom models for specific conditions in specific organs, which required expert annotation of each image used in training.

However, this approach is “inherently limited” due to the different ways in which diseases manifest in individuals and presents problems with generalizability.

“What we really need are thousands of such models and the ability to quickly create new ones as we encounter new information,” he said. Additionally, high-quality, labeled datasets are essential for any model.

With generative AI, instead of training individual models for each disease/organ combination, developers can now pre-train a single base model that can serve as the basis for additional specialized, fine-tuned models downstream.

For example, GE Healthcare’s model could be expanded to areas such as radiation therapy, where radiologists spend a lot of time manually marking organs that may be at risk. It could also help reduce scanning time for X-rays and other procedures that currently require patients to sit still in a machine for long periods, Bhatia said.

Sheeran marveled: “We’re not just expanding access to medical imaging data through cloud-based tools; We are transforming the way this data can be used to drive AI advances in healthcare.”