Validating the web-based evaluation of NLG systems

Alexander Koller, Kristina Striegnitz, Donna Byron, Justine Cassell, Robert Dale, Sara Dalzel-Job, Johanna Moore, and Jon Oberlander

In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics/4th International Joint Conference on NLP of the AFNLP (ACL/IJCNLP), Short Papers, Singapore, 2009.

The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internet-based approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.

Download: Download

BibTeX Entry
@InProceedings{give-acl-09,
	author = {Alexander Koller and Kristina Striegnitz and Donna 
		Byron and Justine Cassell and Robert Dale and Sara 
		Dalzel-Job and Johanna Moore and Jon Oberlander},
	title = {Validating the web-based evaluation of {NLG} systems},
	year = 2009,
	booktitle = {Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics/4th International Joint Conference on NLP of the AFNLP (ACL/IJCNLP), Short Papers},
	address = {Singapore}
}

Back: Publications