Programmer and NLP researcher.

Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Sep
24
awarded Autobiographer
Jul
11
awarded Supporter
May
23
comment Correct way to evaluate correlation of a computer model with multiple human annotator scores
There is also another thing that bothers me in the first approach. When you average first (especially in the case of a large number of humans), you end up to get very similar numbers for every product. I think the second approach sidesteps this problem, right?
May
23
comment Correct way to evaluate correlation of a computer model with multiple human annotator scores
Thank you for the detailed and clear answer. The computer scores are always fixed, so in the 2nd case you just repeat the same computer numbers for every annotator and every product. And I in practice the number of annotators and the number of products are quite high (40 and 300, respectively), not just 3 as in my example here.
May
22
comment Evaluating correlation with multiple human annotators
I don't follow why you think the second case does not distinguish annotators, can you explain a bit more? The annotators are enumerated sequentially instead of all together (as in the first case), but there are three distinct annotators nevertheless. I guess the right question to ask is in the case of multiple human annotators (so multiple "gold standards" for evaluation), what is the best way to aggregate them in order to evaluate the correlation of some model. I understand you are saying that first method (averaging the human judgements first) is more sensible, right?
May
22
asked Correct way to evaluate correlation of a computer model with multiple human annotator scores
May
21
revised Evaluating correlation with multiple human annotators
deleted 2 characters in body; edited tags
May
21
comment Evaluating correlation with multiple human annotators
Hi, I think the question is clear: I need the correlation between model's scores and human scores (the way I got the model scores is irrelevant). If we had only one human, the solution is trivial. Now we have more than one human, what is the appropriate way to get the correlation of all human scores with model scores?
May
21
asked Evaluating correlation with multiple human annotators
Apr
24
awarded Yearling
Apr
16
awarded Student
1 2 3 4 5