Test Data Release and Rule Change

Announcements
#1

Hi all,

The test data has been pushed to the GitHub repository and submissions have been enabled!

Our original policy was to keep all scores private until after the deadline in order to avoid fine-tuning on the test set. However, as was recently discussed, we have recieved considerable feedback both publicly and privately asking that we make at least some scores available to the participants prior to the deadline. We understand the concern here, and we certainly want to avoid a situation where due to a book-keeping error, a team effectively misses their chance to compete for the official MICCAI leaderboard.

As a compromise, we will be allowing teams to see the approximate scores of at most two submissions prior to the deadline. By approximate, I mean the aggregate score for a randomly selected 45 of the test cases. We hope that this will ease some concerns while maintaining a high-integrity leaderboard.

The mechanism for seeing these scores is as follows: Within 24 hours of your submissions, you will receive an email with the unique Submission ID, asking whether you would like to hear an approximate score. A response of “yes” will be honored no more than twice, but you are free to keep submitting even after hearing your second score. Your most recent submission will be used for the leaderboard, regardless of whether you have been told the score or not.

Thank you to everyone who provided us this valuable feedback. Once again, if you haven’t already, please fill out the survey! It really helps us to better serve the participants of this challenge and will inform our challenge design in the future.

Thanks, and happy inference!
Nick

#2

Hi neheller,
Thank u fou u and u team’s work for this.
And a question,
we can know 2 times our score, but we cant know the leaderboard before the deadline?
And another question,
the randomly selected 45 of the test cases used for the 2nd score is same as used in 1st score?
waiting for u reply, thanks

#3

Correct, you will not know the leaderboard before the deadline. The randomly selected 45 will differ each time you submit.

#4

Hi Neheller,

I have submitted the test result yesterday. But still i didn’t receive my score.

Thanks,
Sabari

#5

Thanks for letting me know. Please email me at helle246@umn.edu with your grand-challenge.org username. I thought I was all caught up but it’s possible I missed something.

1 Like
#7

"Please note that regardless of which scores you choose to view, only your most recent submission will count towards the MICCAI leaderboard.
"

It means , if my last submission score is low, you will display the low score. Most of the competition they only take the best score from the user submission to display in the final leaderboard. (Kaggle, evalai.cloudcv.org )

#8

The official leaderboard for the MICCAI event is meant to contain unbiased performance estimates for the method used by each team. If we were to take the highest of all submissions made by a team, this would bias the metrics upward and incentivize teams to make a large number of submissions, even if they’re only marginally different.

After the 7/29 deadline, another leaderboard will be created in closer to a kaggle style, but for the official challenge we will only consider each team’s most recent submission.

2 Likes