Carotid Artery Vessel Wall Segmentation Challenge¶
Endorsed by SMRA 2021 and MICCAI 2021¶
Organizors: Chun Yuan, Li Chen, Niranjan Balu, Mahmud Mossa-Basha and Jenq-Neng Hwang from University of Washington, United States¶
Updates (08/02/2021)¶
Link to test data has been posted in the Results page
Updates (7/30/2021)¶
Quantitative scores have been posted in the Results page
Updates (7/14/2021)¶
The docker container submission deadline has been extended to July 22nd, 2021.
Updates (7/9/2021)¶
Example notebook on how to write to the CASCADE format for submission has been added. Participants, please see data submission page for link.
Updates (6/24/2021)¶
The docker container submission deadline has been extended to July 15th, 2021 (previously July 1st)
Updates (5/13/2021)¶
The Society of Magnetic Resonance Angiography is providing three cash awards to the top three teams. The award is sponsored by TS Imaging. Award amounts will be as follows: First place: $4000; Second place: $1500; Third place: $500
Background¶
Atherosclerosis, a leading cause of death worldwide, is a systemic disease that leads to plaque formation or luminal narrowing in multiple vascular beds, including carotid arteries. Atherosclerosis develops in the walls of the artery (vessel wall) and therefore it is important to measure the thickness of the vessel wall to differentiate normal and diseased vessels. Vessel wall (VW) magnetic resonance imaging (MRI), using black blood imaging, has been effective at visualizing normal and diseased arteries and characterizing atherosclerotic lesions. VW MRI has previously been used in research settings, with careful and comprehensive manual segmentation of the vessel wall. However, manual segmentation is labor intensive and requires a high degree of training in vessel wall review. On the other hand, automatic segmentation is also challenging for complex atherosclerotic lesions and in complex arterial geometries.
Vessel wall imaging (VWI) with MRI of the carotid artery has been recognized to be able to identify atherosclerotic lesions which pose increased risk of causing clinical events. However, traditional axial acquisition MRI sequences require a long scan. To ensure patient compliance and diagnostic image quality, a fast 3D carotid black blood MRI sequence (3D Motion Sensitized Driven Equilibrium prepared Rapid Gradient Echo, 3D-MERGE) has been developed, which allows large coverage of carotid arteries with submillimeter isotropic resolution in coronal acquisition, and is able to depict atherosclerotic lesion burden, severity, and luminal stenosis. This rapid sequence, which can complete a carotid scan in 2 minutes, has potential clinical application in identifying patients with advanced lesions but its application is limited due to the complexity of 3D image review, the large number of images available, and the lack of trained radiologists with extensive experience in the evaluation of carotid vessel wall thickness.
Challenge¶
In this challenge, the task is to segment the vessel wall from 3D-MERGE image with high accuracy and robustness. While the challenges of segmentation in different body regions are different, all vessel wall segmentation requires the basic steps of identifying the artery (localization) and lumen and outer wall segmentation. Then the wall thickness (difference between the lumen and outer wall contours) can be measured. Other clinically usable measurements such as lumen area or percent stenosis can also be derived from the vessel wall segmentation. Therefore, this challenge focuses on the important first step of vessel wall segmentation.
Process¶
Training data release: Jan, 2021
Docker container submission deadline: July 1st, 2021 July 15th,
2021
Results on test set run by organization team
Announce top 10 candidates: August 1st, 2021
Report submission: September 1st, 2021
Presentation at SMRA 2021: Sep 10-12 2021
Participating teams will submit their docker container containing their algorithms through email or Dropbox link provided in the challenge website. The container will be applied to the hidden test set by the organizing team to evaluate the algorithm’s performance on the test set.
Only the organizers of the challenge will have access to the test cases until all the docker containers have been submitted (July 2021). After that, the images and labels in the test set will be released (August 1st 2021) and can be used by the participants to prepare their reports. The top 10 teams will be chosen towards mid-August 2021.
Requirements¶
- Use of images will be restricted solely to this challenge, participants must agree not to use the images in any other capacity.
- There are no publication requirements in this challenge. The participants may choose the author list (participating team) for their presentations at SMRA 2021 and MICCAI 2021. Participating teams are free to publish their own results. The challenge organizers will not publish a challenge paper first.
- There is no requirement for participants to open source their code. But publicly sharing your code is greatly encouraged.
Winners¶
The challenge is funded by the SMRA 2021. The panel of judges will not include the sponsors of the meetings/prizes.
Quantitative metrics include Dice Similarity Coefficient (DSC) of the vessel wall region (between lumen and outer wall contours), differences between manual and segmented lumen area/outer wall area/normalized wall index, differences between manual and predicted Hausdorff distance, and run-time (not in the score).
The quantitative metric used for ranking (QuanM) is the weighted average of DSC (50%), lumen area (10%) and outer wall area (10%), normalized wall index (10%), Haussdorf distance (10%), and degree of stenosis (10%). Run-time will not be included in the WQM, but it will be reported in the evaluation results.
Only the algorithms with the top 10 QuanM will be selected for qualitative evaluations.
Qualitative metrics include expert group assessments on clinical usability of the segmented contours. The winning solution should be able to identify the correct artery of interest and segment vessel wall contours robustly and consistently, even near carotid artery bifurcations, be able to segment plaques correctly, and be robust on both high and low image qualities regardless of flow artifacts. Five expert judges from SMRA 2021 with both technical and clinical backgrounds will evaluate the segmentation results and each of the judge will give a QualM score ranging from 0-20, with 0-5 on each of the four factors.
1. Contour smoothness and fitness quality.
2. Segmentation performance near the carotid bifurcation. The segmentation quality should not decrease a lot near the clinically important area around the carotid bifurcation.
3. Segmentation performance on carotid plaques. The segmentation algorithm should perform well on both healthy and diseased arteries. Carotid plaques should not be ignored, or under/over-estimated.
4. Segmentation performance on images with low image quality/flow artifact.
(New) Judge list:
Dr. David Saloner, University of California San Francisco
Dr. Mahmud Mossa-Basha, University of Washington
Dr. Peter Douglas, University of Glasgow
Dr. Jie Sun, University of Washington
Li Chen, University of Washington
The winner will be the algorithm with the highest sum QualM from the five experts (ranging from 0-100).
The QualM scores on each factor will be based on the following criteria:
0: Totally unacceptable.
1: Works for some (>10%) slice/cases.
2: Satisfactory results (No mistakes if expert readers do the segmentation) on about half the slices/cases, or works for most slices/cases but few are ideal (for example, systemically under/over estimating wall thickness)
3. Majority (>75%) of the slices/cases are satisfactory.
4. Mostly satisfactory with only occasional (\<10%) mistakes.
5. Perfectly solving all the challenges/No problems observed.
Top five teams with QualM will be invited to both SMRA 2021 (Sep 10-12, 2021) to present their methods. Participating teams can choose whether their algorithm will be made public. All teams are encouraged to submit a report introducing their methods.
How to download data?¶
Please register your team in the grand challenge website, then find the download link at the Download page (https://vessel-wall-segmentation.grand-challenge.org/Download/)
More details about the data format at the Data page (https://vessel-wall-segmentation.grand-challenge.org/Data/)