Dear organizers!
I did not find any information about the memory and time constraints of the model submission. Do we have some restrictions?
Sincerely,
Bohdan
1 Like
Hi @SuperBo,
Good question! We’re actually in the process of confirming these details with the admins at grand-challenge.org and don’t have an exact answer quite yet. Our intention is not to force participants to have to work hard to reduce size/execution time, so the limits won’t be too strict. That said, we do need the containers to run in a reasonable amount of time. I will let you know as soon as this information becomes available to me.
Best,
Nick
Tentatively I can say that the containers will have access to a single NVIDIA T4 card (16GB GPU memory) and 4 CPUs with 30GB of CPU memory. We’re not sure about the time yet, though.
We will be doing some experiments over the next few days and should have a final answer on Tuesday next week. Thanks again for the good question.
Thank you for the detailed answer. We are waiting for the final info.
Have a nice weekend!
1 Like
Hi @SuperBo,
My apologies for the delay. I don’t quite have an answer yet but I promise I haven’t forgotten, and I hope to have more info tomorrow or Wednesday.
Best,
Nick
1 Like
We have decided to set a time limit of 20 minutes per case on the aforementioned hardware. In order for you to test your Docker container against this constraint, we be changing the “sanity-check” set (described here) to three of the largest cases from the training set, and your inference should run to completion on this set in less than one hour.
Hi, I’m wondering how many such GPU cards will the container be able to access? I assume more than 1 since you wrote “cards”? or was it a mistake?
and also, when will we be able to perform the sanity check? I assume it’s not possible yet, or have I missed anything?
Thanks!
My apologies for the lack of clarity. You will be given access to a single NVIDIA T4 card.
The sanity check will become available when submission opens on August 30th. Between the time that you submit your containers and the final results announcement, only the sanity check results will be available to you.
Thanks for the clarification
I have a few more technical questions.
when submission starts, let’s say for example I submit a solution right away and the sanity check takes 50 minutes. will I get the answer after 50 minutes?
what if it takes 20 minutes so I conclude that I have “room for improvement” and I can maybe change something in my submission to make it take longer and still meet the requirements but be more accurate. can I do it? I can do it as many times as I want during the submission period?
Here it says that the sanity check will be available prior to the submission period, so I assumed until then we could do multiple checks, and then submit one final submission and that’s it.
But now you said that the sanity check will only become available when submission opens, so I’m wondering how it will work exactly.
Thank you!
Hi @rollthedice,
You will receive your sanity check results as soon as they become available. The delay will include your runtime plus a small amount of overhead time for the instance to start and the evaluation code to run. We’re still determining what the overhead time will be but I think it will be on the order of several minutes.
You may submit to get sanity check results as many as 15 times per day during the submission period. This is just to prevent abuse, and we anticipate that most teams will submit far less than this.
I apologize for that sentence on the website. It is out of date and I will change it.
Thanks for the good questions!
1 Like
Hi @neheller ,
I’m wondering if there’s a known limit on the expected docker container file size?
Thanks
I’m not aware of a size limit, but I wouldn’t be surprised if they cap it somewhere just for security reasons. I will inquire and get back to you.
It looks like they cap the containers at 10 GB after gzip compression.
1 Like