Since there will be no leaderboard, can people just talk about their current best score? We are getting about .83 AUC with very little effort. We would like to compare it to other people's scores.
If we can submit for one or both of the tasks, how will the contestants be evaluated on AUC? Do we have winners for each task? Or best of the two AUCs?
Hi Claudia - an you provide either (or preferably both): 1) A link to the scoring methodology 2) A simple 10 record example (false data) to demonstrate the score.
Sounds dumb I know but just want to be sure I am on the right track.
Is there a restriction on number of decimals for the score? Is it possible to have PatientID, Score format? (We dropped 'RecordID' from the data to decrease the size of the data set)
Question:
ReplyDeleteFor the INFOMRS contest 2009, there are two tasks. Whether I have to finish both of them, or I can choose one?
Answer:
You can do one or both, as you like!
When is the deadline to submit the results?
ReplyDeleteSince there will be no leaderboard, can people just talk about their current best score? We are getting about .83 AUC with very little effort. We would like to compare it to other people's scores.
ReplyDeleteIf we can submit for one or both of the tasks, how will the contestants be evaluated on AUC? Do we have winners for each task? Or best of the two AUCs?
ReplyDeleteI do not mind if people want to talk about their AUC here. But you may want to clarify your evaluation methodology (test set, 10 fold CV, etc)
ReplyDeleteThere will be two winners - one for each task.
Claudia
Hi Claudia - an you provide either (or preferably both):
ReplyDelete1) A link to the scoring methodology
2) A simple 10 record example (false data) to demonstrate the score.
Sounds dumb I know but just want to be sure I am on the right track.
Hi Claudia - any update on my post from the 2nd of September?
ReplyDeleteI can tell you how to compute AUC.
ReplyDeleteLet P be the set of scores that you give to positive examples (i.e. where field 9 = 2 or field 9 = 20, depending on the task).
Let N be the set of scores that you give to negative examples (i.e. all the rest).
AUC = |{x in P, y in N : x > y}|/|N|/|P|
That is, the fraction of all pairs of positive and negative examples where the positive example gets the higher score.
Hi, John - here is an URL clearly explaining the simple AUC calculation by Hands and Till,
ReplyDeletehttp://www.csd.uwo.ca/faculty/ling/papers/ijcai03.pdf
As the competition deadline is nearing, can you let us know how we submit/upload the results?
ReplyDeleteHere's a 14-line C program to compute AUC.
ReplyDeleteplg.uwaterloo.ca/~gvcormac/auc-trivial.c
The deadline is now 1 week away. It has been several weeks since there has been any response on this blog.
ReplyDeleteIs the contest still on? How do we submit our solutions?
Is there a restriction on number of decimals for the score? Is it possible to have PatientID, Score format? (We dropped 'RecordID' from the data to decrease the size of the data set)
ReplyDeleteAfter submission, do we get the AUC scores back immediately? If not, how do we know we are not making any mistake?
ReplyDelete