# dkar

Programmer and NLP researcher.

 Nov 29 comment Steps to generate parse tree from CYK Algorithm (Natural Language Processing)Glad I helped :) Nov 16 answered Steps to generate parse tree from CYK Algorithm (Natural Language Processing) Nov 16 awarded Caucus Apr 24 awarded Yearling Mar 22 awarded Enlightened Mar 22 awarded Nice Answer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Sep 24 awarded Autobiographer Jul 11 awarded Supporter May 23 comment Correct way to evaluate correlation of a computer model with multiple human annotator scoresThere is also another thing that bothers me in the first approach. When you average first (especially in the case of a large number of humans), you end up to get very similar numbers for every product. I think the second approach sidesteps this problem, right? May 23 comment Correct way to evaluate correlation of a computer model with multiple human annotator scoresThank you for the detailed and clear answer. The computer scores are always fixed, so in the 2nd case you just repeat the same computer numbers for every annotator and every product. And I in practice the number of annotators and the number of products are quite high (40 and 300, respectively), not just 3 as in my example here. May 22 comment Evaluating correlation with multiple human annotatorsI don't follow why you think the second case does not distinguish annotators, can you explain a bit more? The annotators are enumerated sequentially instead of all together (as in the first case), but there are three distinct annotators nevertheless. I guess the right question to ask is in the case of multiple human annotators (so multiple "gold standards" for evaluation), what is the best way to aggregate them in order to evaluate the correlation of some model. I understand you are saying that first method (averaging the human judgements first) is more sensible, right?