Radiology Lab - Medical Imaging Segmentation Lab

Alright. My name is Hayden. For the last 5 years, I've been working at a cardiac MRI software company. And what we have been, basically building is software that looks at our pulse sync, which which is called FASTSYNC. And so what it is is called a strain encoded sequence. What you're looking at is you're looking at multiple cross sections of the heart and you're seeing how the tissue actually contracts. So you're sending 2 different frequencies, low tune and high tune and where you're able to measure from that is actually how much the tissue itself is contracting. So we would build deep learning models from scratch on tens of thousands of our images to actually predict where the heart is on there and then we would actually quantify from those segmentations how the heart is performing on a regional basis. But 1 of the downsides is it only works on FAST sync itself. It doesn't work on any other cardiac modalities and I would have to get a lot of training data to try to extend it beyond that. And so that's where I started building this web app called Radiology Lab And it is extended to cine data, it's extended to brain MRI data, it's extended to cardiac CT data and I have it integrated with what's called SAM 3. SAM 3 is a very popular segmentation model. It's a transformer based model so it's a little bit heavier and here I'll load a data set right here. What this takes in is a DICOM data set which is a medical imaging files format that is very popular and you'll see right here that I have a what's called a volumetric renderer. It's basically a ray marching based shader all written inside, React 3 and this is a cine heart modality. And so there's a bunch of different pulse sequence that you can look at which all show different representations of the same data. So this is 1 scan for 1 patient and multiple different heart modalities and this is the pulse sequence of the company that I worked at. This is FastSync. So when you press play you can actually see it change colors and what that represents is the actual tissue contracting itself. While on CIDI data, you also have the motion but it's much higher resolution and our FAST sync models couldn't actually predict on this CENI data. And so I set up a API on Hugging Face for SAM 3, and I'll enter the key, connect it, And the nice thing about SAM 3 is it allows for a lot of different types of prompts. So it doesn't just take image in segmentation out, you actually have to guide it. So you can either show it a shape or you can actually type what it's supposed to draw. And so I'll search right here for myocardium which is actually like the tissue in the heart and after a little bit it's a pretty heavy model you'll see that it's actually contouring the myocardium right here. And you can do this for multiple planes, which you can see on the right right here, and add to your datasets. But if you look on this on a 2 d basis, you'll notice that it's kind of drawing stuff right at the, the top of the heart. And that's probably not something that I would want the model to actually draw. So what you'll do is you'll click on the object and you'll actually try to erase that section, accept your changes, and run it and you'll see that it actually adjusts all the images throughout. The nice thing about this transformer based architecture, it's able to use context throughout and updates its history. So 1 other dataset that I wanted to show was Yeah. Yeah. So the objective is to help people label data so they can build training data to train other AI models and then they can hook those AI models into the same platform and then it will be geared and fine tuned to their heart modalities or their brain MRI whatever their imaging is. So reloading this I'll open a brain MRI data set which typically has a lot more planes and a little bit higher of a resolution. This is actual data. Yes. So this dataset was open sourced through the, the cancer archive online, the cancer imaging archive. And so I just downloaded that. And as you can see, this is called a flare imaging modality. And you can kinda see these bright regions, which would be anything from inflammation or edema or something along those lines but they're typically they're unhealthy regions. And I'll re hook the SAM 3 and what I'll do for this 1 is I can't just describe this object. It's a little bit more difficult to describe this object. So I'll actually draw an example first. So I'll create a new object. I'll paint this object all in that red region that I want it to propagate through time. And so you'll notice there's a little bit of holes and whatnot. And I'll accept this. I'll change it to run for the entire view, the entire group of planes. And then you'll see it from the side. You'll see just 1 plane right here, but then you'll see it propagated that through the entire view. Since there isn't actually multiple frames for each of these planes, you can just see it throughout. Now that I have this these segmentations at every single plane across this, what I did is I actually hooked up what Ron was describing, Gaussian splatting. And so now that I have all of these segmentations, I actually want to try to create a mesh around it. And sometimes you have planes that are very sparse, they're very far apart and what you actually want to do is you want to kind of fill in the gaps. And so what I used is the G Splat library. It's a Python library that actually is PyTorch kind of based and it actually trains to the images themselves. And when I say that is it will actually fit and predict everything that is within inside this mask itself. So it's predicting the images and the mask at the same time, Then what happens is you get that entire volume and then you predict a mesh that encloses that using what's called ray marching. And I'll hook up my Gaussian splatting API as well. And you'll see So as you are kind of trying to, I'll say it for the company that I worked at is we wanted to identify where the heart is. So if I had this platform and I could draw a bunch of examples where the heart is, I could take the big model and fine tune it so that way it will learn that representation. So this is just to help people build training data on a more general basis. It won't just work on 1 imaging modality, it will work on any sort of DICOM format. And but this specifically is you're actually training to predict where Gaussians are in the actual, brain. And so when I switch over to the mesh, you'll see just the mesh itself and right now it is trying to fit a bunch of ellipsoids to represent the actual underlying, bright intensity values that I segmented. And so as you can see right here you actually have a object that is a lot more higher fidelity and you actually have a mesh that can then be exported for let's say you have a neurosurgeon that wants to actually quantify how much, how big a tumor is. They're able to use that. Well you also have the data that's inside of it as well. All the galaxies, all the intensities, all the different fields that you can export as well. So the next step is to take this mesh and what's called tetrahedralize it. Basically just make it a volumetric mesh, not just stuff on the outside but stuff on the inside. So that way researchers can actually model it and whether they're finite elements kind of analysis software and whatnot.