Programmer and NLP researcher.

Jul
11
awarded Supporter
May
23
comment Correct way to evaluate correlation of a computer model with multiple human annotator scores
There is also another thing that bothers me in the first approach. When you average first (especially in the case of a large number of humans), you end up to get very similar numbers for every product. I think the second approach sidesteps this problem, right?
May
23
comment Correct way to evaluate correlation of a computer model with multiple human annotator scores
Thank you for the detailed and clear answer. The computer scores are always fixed, so in the 2nd case you just repeat the same computer numbers for every annotator and every product. And I in practice the number of annotators and the number of products are quite high (40 and 300, respectively), not just 3 as in my example here.
May
22
comment Evaluating correlation with multiple human annotators
I don't follow why you think the second case does not distinguish annotators, can you explain a bit more? The annotators are enumerated sequentially instead of all together (as in the first case), but there are three distinct annotators nevertheless. I guess the right question to ask is in the case of multiple human annotators (so multiple "gold standards" for evaluation), what is the best way to aggregate them in order to evaluate the correlation of some model. I understand you are saying that first method (averaging the human judgements first) is more sensible, right?
May
22
asked Correct way to evaluate correlation of a computer model with multiple human annotator scores
May
21
revised Evaluating correlation with multiple human annotators
deleted 2 characters in body; edited tags
May
21
comment Evaluating correlation with multiple human annotators
Hi, I think the question is clear: I need the correlation between model's scores and human scores (the way I got the model scores is irrelevant). If we had only one human, the solution is trivial. Now we have more than one human, what is the appropriate way to get the correlation of all human scores with model scores?
May
21
asked Evaluating correlation with multiple human annotators
Apr
24
awarded Yearling
Apr
16
awarded Student
Apr
16
awarded Scholar
Apr
16
comment Determining and enforcing linear dependence
Thank you, that was very informative.
Apr
16
accepted Determining and enforcing linear dependence
Apr
15
comment Determining and enforcing linear dependence
It is cosine distance, sorry but since I mentioned "direction" I assumed this as evident.
Apr
15
comment Determining and enforcing linear dependence
Thank you very much, please read the modified question. I need to retain as much as possible the direction of the original vectors, I don't want them to be orthogonal.
Apr
15
awarded Editor
Apr
15
comment Determining and enforcing linear dependence
@rschwieb Thank you for pointing this out, I have modified my question
Apr
15
revised Determining and enforcing linear dependence
added 147 characters in body
Apr
15
asked Determining and enforcing linear dependence
Feb
17
awarded Caucus
1 2 3 4 5