Evaluation Error

#1

Hi,

I submitted the predictions and the got the following message:

Unfortunately the evaluation for the submission to KiTS19 failed with an error. The error message is:

evalutils.exceptions.ValidationError

What should I do to deal with the error message ?

#2

Please send me an email (helle246@umn.edu) with the job ID. I can take a look at the file you submitted and try and see went went wrong.

#3

Hey,

I also got the evalutis.exceptions.ValidationError and sent you an email with my job id.

I already double checked that the predictions and the volume have the same shape, as well as that there are not any typos in the file or zip name.

The predictions were created with the following code:

# seg : numpy matrix
# cid : case id
# output_path : prediction directory (e.g. "predictions")
# Write a segmentation prediction into in the NIFTI file format
def save_segmentation(seg, cid, output_path):
    # Resolve location where data should be written
    if not os.path.exists(output_path):
        raise IOError(
            "Data path, {}, could not be resolved".format(str(output_path))
        )
    # Convert numpy array to NIFTI
    nifti = nib.Nifti1Image(seg, None)
    # nifti.get_data_dtype() = seg.dtype
    # Save segmentation to disk
    nib.save(nifti, os.path.join(output_path,
                                 "prediction_" + str(cid).zfill(5) + ".nii.gz"))

If we can track down the problem here in public, it may help others to avoid it.

Cheers,
Dominik

#4

Good idea, @dmueller.

I think the issue is that you provide None for your affine matrices. The prediction affine matrices should be the same as for their corresponding image files.

#5

Thanks for your help, @neheller.

With your email, I noticed that prediction_00268 was a prediction with incorrect shape. I was able to track and solve the bug. Looks like I was too hasty with my assumption that all my predictions have the correct shape, sorry :confused:

I tested a prediction submission without affine matrices and it succeeded. Still, should we add it?

#6

That’s interesting, apparently the affine matrices are not as important to the evaluation as I had thought…

I’m glad your issue is resolved!

#7

Dice scores should be independent of the geometry. As long as the pixel arrays match then the geometry can be discarded (or even different between image and prediction). My guess is that that’s why affine=None didn’t matter.

Keep in mind that if you are loading both into an image viewer like slicer or MITK then they will not be aligned if the geometry is inconsistent.

#8

You’re right in that the Dice scores are computed given two numpy arrays which have no concept of geometry. I had thought that the numpy arrays extracted from the geometrically oriented Nifti1Image objects were transposed to a standard orientation, but apparently not.

Luckily this has not been an issue for any of the submissions yet.

#9

So, Is it confirmed that we need not worry about affine matrices ? Currently we are passing np.eye(4) for all cases.
Let us know if we need to take care of this.

Thanks
Bharadwaj

#10

Yes, this is confirmed. What matters is that your prediction data blocks (i.e. numpy arrays) have the same orientation/shape as the corresponding imaging.

1 Like