Thursday, August 13, 2009

Questions about the submission

Instructions will be posted here and on the competition website in the next weeks.

14 comments:

  1. Question:
    For the INFOMRS contest 2009, there are two tasks. Whether I have to finish both of them, or I can choose one?

    Answer:
    You can do one or both, as you like!

    ReplyDelete
  2. When is the deadline to submit the results?

    ReplyDelete
  3. Since there will be no leaderboard, can people just talk about their current best score? We are getting about .83 AUC with very little effort. We would like to compare it to other people's scores.

    ReplyDelete
  4. If we can submit for one or both of the tasks, how will the contestants be evaluated on AUC? Do we have winners for each task? Or best of the two AUCs?

    ReplyDelete
  5. I do not mind if people want to talk about their AUC here. But you may want to clarify your evaluation methodology (test set, 10 fold CV, etc)

    There will be two winners - one for each task.

    Claudia

    ReplyDelete
  6. Hi Claudia - an you provide either (or preferably both):
    1) A link to the scoring methodology
    2) A simple 10 record example (false data) to demonstrate the score.

    Sounds dumb I know but just want to be sure I am on the right track.

    ReplyDelete
  7. Hi Claudia - any update on my post from the 2nd of September?

    ReplyDelete
  8. I can tell you how to compute AUC.

    Let P be the set of scores that you give to positive examples (i.e. where field 9 = 2 or field 9 = 20, depending on the task).

    Let N be the set of scores that you give to negative examples (i.e. all the rest).

    AUC = |{x in P, y in N : x > y}|/|N|/|P|

    That is, the fraction of all pairs of positive and negative examples where the positive example gets the higher score.

    ReplyDelete
  9. Hi, John - here is an URL clearly explaining the simple AUC calculation by Hands and Till,

    http://www.csd.uwo.ca/faculty/ling/papers/ijcai03.pdf

    ReplyDelete
  10. As the competition deadline is nearing, can you let us know how we submit/upload the results?

    ReplyDelete
  11. Here's a 14-line C program to compute AUC.

    plg.uwaterloo.ca/~gvcormac/auc-trivial.c

    ReplyDelete
  12. The deadline is now 1 week away. It has been several weeks since there has been any response on this blog.

    Is the contest still on? How do we submit our solutions?

    ReplyDelete
  13. Is there a restriction on number of decimals for the score? Is it possible to have PatientID, Score format? (We dropped 'RecordID' from the data to decrease the size of the data set)

    ReplyDelete
  14. After submission, do we get the AUC scores back immediately? If not, how do we know we are not making any mistake?

    ReplyDelete