Rules: Automatic or manual modification to the training data

#1

Are these possible regarding the rules?

–

  • Fix ground truth labels for training data in an automated but heuristic process.

I believe this is absolutely possible because it is a part of the automated segmentation system and it is reproducible.

–

  • Manually fix ground truth labels for training data.
  • Manually add some ground truth labels (e.g. liver) for training data.
  • Manually identify some cases or images and remove them from the training process.

Since these only modify the training data, the Manual Prediction clause does not prohibit them.
However, taking the External Data clause very strictly, these may violate the rules especially if we do not share the fixed/added information.

–

  • Use some Grand Challenge data which requires registration for downloading.

Is it regarded as “publicly available” if registration or authorization is required to obtain the external data?

3 Likes
#2

Thanks for the great questions!

_

Using an automated but heuristic process is certainly allowed.
_

The third bullet about removing certain cases is allowed. We don’t want to be forcing you to use data if you’d prefer to exclude it.

Let me discuss with my fellow organizers about the other two. I’m leaning towards allowing them since other teams could also do this if they chose to, therefore it’s not an unfair advantage like an existing private dataset would be. On the other hand, it may hinder the objective comparison of methods that we were hoping for.

Other participants: please let us know if you have an opinion on this!
_

You may use other Grand Challenge data requiring registration as long as registration is open to all other participants as well.
_

As a general note, if you find problematic data, we’d much prefer that you submit an issue prior to April 5 and we’ll take care of it by April 15! There are a handful of fixes already in progress.

1 Like
#3

The organizing team met today and we decided that manually fixing ground truth labels and adding ground truth labels are both allowed. Thanks again for the good clarifying questions.