Evaluation Metrics

#1

Good evening Nick,

I’m currently running some tests using the KiTS19 dataset. I have all segmentation results now saved in necessary .nii.gz files. I would appreciate any guidance in how I can go about the best route for computing my evaluation metrics.

Distilling from the thread here I notice discussions about an evaluator in Docker and the use of evaluation.py. Since I’m using nnU-Net there is also a different evaluator.py script.

My question is; is there a location where I can find the Ground Truth segmentations of the test cases for KiTS19 (case_00210 - case_00299)?

Best,

Andres

#2

Hi Andres,

The ground truth for cases 210 - 299 are being held private in order to preserve the integrity of the leaderboard at https://kits19.grand-challenge.org/evaluation/challenge/leaderboard/

In order to obtain your test set performance, I would recommend that you submit your predictions to that leaderboard by following the instructions here.

Best,
Nick

1 Like
#3

Ahh totally!! I thought since the competition ended that they were available and I was missing something… Appreciate again the super-fast response.

Take care,

Andres

1 Like
#4

Hi Nick,

Sorry to bother you again. I submitted my results and was wondering if there is any chance of knowing the individual Dice Scores for each case? I can imagine this being kept private as well to maintain challenge integrity so I completely understand if this can’t be shared.

(I should state that my intention is not to rank up on the leaderboard, but more of personal research)

Thanks again for your sharing your limited time,

Andres

#5

Hey Andres,

I’m sorry but that information has also been kept private for the same reason. Perhaps there’s some way I could give you the information you need without divulging too much? If you’re looking for things like min/max/spread, I could probably provide that.

Best,
Nick

#6

Understood. And yeah, that would be useful to know!

Truly appreciate it Nick.

#7

Hi Nick, I know you’re super busy on KiTS21! Just wanted to follow back to see if there is any chance of getting this data. Thanks again.

Best,

Andres

#8

Sure! Sorry I forgot to follow up on this. Looks like there are two leaderboard entries for you. Would you like info on the one from Jan 23 or Feb 13?

#9

Hi Nick,

No worries, and the one from January would be great!

1 Like