With this challenge, we made available a large dataset of 1200 annotated retinal fundus images. In addition, an evaluation framework has been designed to allow all the submitted results to be evaluated and compared with one another in a uniform manner.
REFUGE Challenge consists of THREE Tasks:
- Classification of clinical Glaucoma
- Segmentation of Optic Disc and Cup
- Localization of Fovea (macular center) -- NEW Task added
Imaging Data
All fundus images are stored as JPEG files
Reference Standard
Task 1. The reference standard for glaucoma presence obtained from the health records, which is not based on fundus image ONLY, but also take OCT, Visual Field, and other facts into consideration. For training data, glaucoma and non-glaucoma labels (a.k.a. the reference standard) are reflected on the image folder names.
Task 2. Manual pixel-wise annotations of the optic disc and cup were obtained by SEVEN (was 3 as proposed) independent GLAUCOMA SPECIALISTS from Zhongshan Ophthalmic Center, Sun Yat-sen University, China. The reference standard for the segmentation task was created from the seven annotations, which were merged into single annotation by another SENIOR GLAUCOMA SPECIALIST. It is stored as a BMP image with the same size as the corresponding fundus image with the following labels:
128: Optic Disc (Grey color)
0: Optic Cup (Black color)
The numbers in front of the structures indicate the pixel-wise labels. All other pixels are labeled as 255(White Color).
Task 3. Manual pixel-wise annotations of the fovea (macular center) were obtained by 7 independent GLAUCOMA SPECIALISTS. The reference standard for localization task was created by using the average of selected annotations from the 7 annotations, for each individual images by another independent GLAUCOMA SPECIALIST.
Training and Off-site and On-site Test datasets
A total of 1200 color fundus photographs are available. The dataset is split 1:1:1 into 3 subsets equally for training, offline validation and onsite test, stratified to have equal glaucoma presence percentage. Training set with a total of 400 color fundus image will be provided together with the corresponding glaucoma status and the unified manual pixel-wise annotations (a.k.a. ground truth). Testing consists of 800 color fundus images and is further split into 400 off-site validation set images and 400 on-site test set images.
Submission Guidelines
General Guidelines.By the time of submitting your results, please prepare a single .ZIP file. Inside the ZIP must be: a folder, named "segmentation", with the segmentation results; a CSV file named "fovea_location_results.csv" with the fovea localization results; a CSV file named "classification_results.csv" with the classification results. Please, take into account that submitting unorganized files could lead to a wrong evaluation.
Challenge Task 1: Glaucoma Classification.
The classification results should be provided in a single CSV file, named “classification_results.csv”, with the first column corresponding to the filename of the test fundus image (including the extension “.jpg”) and the second column containing the estimated classification probability/risk of the image belonging to a patient diagnosed with glaucoma (value from 0.0 to 1.0).
Challenge Task 2: Optic disc/cup Segmentation.
The segmentation results should be provided in a “segmentation” folder, as one image per test image, with the segmented pixels labeled in the same way as in the reference standard (bmp (8-bit) files with 0: optic cup, 128: optic disc, 255: elsewhere). Please, make sure that your submitted segmentation files are named according to the original image names and with the same extension.
Challenge Task 3 (Optional Task): Fovea Localization.
The localization results should be provided in a single CSV file, named “fovea_location_results.csv”, with the first column corresponding to the filename of the test fundus image (including the extension “.jpg”), the second column containing the X-coordinate and the third column containing the Y-coordinate. Please, make sure that your submitted segmentation files are named according to the original image names and with the same extension.
Evaluation Framework
This challenge evaluates the performance of the algorithms for: (1) glaucoma classification, and (2) optic disc/cup segmentation. Thus there will be two main leaderboards. The average score across the two leaderboards will determine the final ranking of the challenge. In case of a tie, the classification leaderboard score has the preference.
Classification results will be compared to the clinical grading of glaucoma. Receiver operating curve will be created across all the test set images and an area under the curve (AUC) will be calculated. Each team receives a rank (1=best) based on the obtained AUC value. This ranking forms the classification leaderboard.
Submitted segmentation results will be compared to the reference standard. the disc and cup Dice indices (DI), and the cup-to-disc ratio (CDR) will be calculated as segmentation evaluation measures. Each team receives a rank (1=best) for each evaluation measure based on the mean value of the measure over the set of test images. The segmentation score is then determined by adding the three individual ranks (2xDI and 1xCDR ). The team with the lowest score will be ranked #1 on the segmentation leaderboard.
Please NOTE that Task 3 (fovea localization) is OPTIONAL for MICCAI ONSITE challenge and NOT counted into the final leaderboards. However, it is ESSENTIAL for the ONLINE challenge on the TEST dataset. The evaluation criterion is the Average Euclidean Distance between the estimations and ground truth, which is the lower the better.
MICCAI 2018 and On-site part of the Challenge at OMIA Workshop
The REFUGE challenge will be hosted at the MICCAI 2018 conference in conjunction with
OMIA workshop. There will also be an on-site part of the challenge when the second part of the test set will be released. The participants will have 1 hour on the day of the challenge to provide the results on the "on-site test set". Papers submitted to the REFUGE will be automatically considered for the challenge-part of the OMIA workshop unless otherwise stated by the participants. Each team can have up to 2 names appear in the author list of the challenge review paper (edited by the organizers with the papers submitted from the participants) .
A paper must be submitted together with the off-site validation results by Jul 28 for the teams intent to attend the on-site challenge.
In the manuscript please describe the methods used, the novelty of the methodology and how it fits with the state-of-the-art, and a qualitative and quantitative analysis of results on the training/validation data (and on other reasonable settings).
