Variations in relevance judgments and the measurement of retrieval effectiveness

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Variations in relevance judgments and the measurement of retrieval effectiveness

Full text

Pdf (1.20 MB)

Source

Annual ACM Conference on Research and Development in Information Retrieval archive
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval table of contents

Melbourne, Australia

Pages: 315 - 323

Year of Publication: 1998

ISBN:1-58113-015-5

Author

Ellen M. Voorhees

National Institute of Standards and Technology, Gaithersburg, MD

Sponsors

University of Melbourne : University of Melbourne
SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 10, Downloads (12 Months): 105, Citation Count: 52

Additional Information:

references cited by index terms collaborative colleagues peer to peer

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTex EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/290941.291017 What is a DOI?

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Robert Burgin, Variations in relevance judgments and the evaluation of retrieval performance, Information Processing and Management: an International Journal, v.28 n.5, p.619-627, 1992 [doi>10.1016/0306-4573(92)90031-T ]
	2	C. W. Cleverdon, J. Mills, and E. M. Keen. Factors determining the performance of indexing systems. Two volumes, Cranfield, England, 1968.
	3	Cyril W. Cleverdon. The effect of variations in relevance assessments in comparative experimental tests of index languages. Technical Report Cranfield Library Report No. 3, Cranfield Institute of Technology, Cranfield, UK, October 1970.
	4	Gordon V. Cormack, Charles L.A. Clarke, Christoher R. Palmer, and Samuel S.L. To. Passage-based refinement (MultiText experiments for TREC-6). In Proceedings of the Sixth Text REtrieval Conference (TREC-6}, pages 171-186, November 1997. Draft for conference notebook.
	5	C. A. Cuadra and R. V. Katter. Opening the black box of relevance. Journal of Documentation, 23(4):291-303, 1967.
	6	Donna K. Harman, The first text retrieval conference (TREC-1) Rockville, MD, U.S.A., 4–6 November, 1992, Information Processing and Management: an International Journal, v.29 n.4, p.411-414, July/Aug. 1993 [doi>10.1016/0306-4573(93)90037-E ]
	7	Stephen P. Harter, Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science, v.47 n.1, p.37-49, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I]
	8	K.L. Kwok, L. Grunfeld, and J.H. Xu. TREC-6 english and chinese retrieval experiments using PIRCS. In Proceedings of the Sixth Text REtrieval Conference (TREC-6), pages 96-102, November 1997. Draft for conference notebook.
	9	M.E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4:343-359, 1969.
	10	G. Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1971
	11	Linda Schamber. Relevance and information behavior. Annual Review of Information Science and Technology, 29:3-48, 1994.
	12	Karen Sparck Jones. Automatic indexing. Journal of Documentation, 30:393-432, 1974.
	13	Karen Sparck Jones, Information Retrieval Experiment, Butterworth-Heinemann, Newton, MA, 1981
	14	Alan Stuart. Kendall's tau. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, volume 4, pages 367-369. John Wiley & Sons, 1983.
	15	M. Taube. A note on the pseudomathematics of relevance. American Documentation, 16(2):69-72, April 1965.

CITED BY 52

	David Bodoff , Pu Li, Test theory for assessing IR test collections, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	David Bodoff , Pu Li, Testing algorithms is like testing students, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Baolong Liu , Douglas W. Oard, One-sided measures for evaluating ranked retrieval effectiveness with spontaneous conversational speech, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Mark Sanderson , Ian Soboroff, Problems with Kendall's tau, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Mingfang Wu , Michael Fuller , Ross Wilkinson, The role of a judge in a user based retrieval experiment (poster session), Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.331-333, July 24-28, 2000, Athens, Greece
	I. E. Kuralenok , I. S. Nekrestyanov, Evaluation of Text Retrieval Systems, Programming and Computing Software, v.28 n.4, p.226-242, July-August 2002
	Ian Soboroff , Charles Nicholas , Patrick Cahan, Ranking retrieval systems without relevance judgments, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.66-73, September 2001, New Orleans, Louisiana, United States
	Jimmy Lin , Dina Demner-Fushman, Will pyramids built of nuggets topple over?, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.383-390, June 04-09, 2006, New York, New York
	Andres R. Masegosa , Hideo Joho , Joemon M. Jose, Effects of highly agreed documents in relevancy prediction, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Javed A. Aslam , Emine Yilmaz, Inferring document relevance from incomplete information, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Ben Carterette , James Allan, Incremental test collections, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany
	Kazuko Kuriyama , Noriko Kando , Toshihiko Nozue , Koji Eguchi, Pooling for a Large-Scale Test Collection: An Analysis of the Search Results from the First NTCIR Workshop, Information Retrieval, v.5 n.1, p.41-59, January 2002
	Eero Sormunen, Liberal relevance criteria of TREC -: counting on negligible documents?, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland
	Jimmy Lin, Evaluation of resources for question answering evaluation, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Ben Carterette , James Allan , Ramesh Sitaraman, Minimal test collections for retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Ellen M. Voorhees , Dawn M. Tice, Building a question answering test collection, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.200-207, July 24-28, 2000, Athens, Greece
	Gordon V. Cormack , Thomas R. Lynam, Statistical precision of information retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Inderjeet Mani , David House , Gary Klein , Lynette Hirschman , Therese Firmin , Beth Sundheim, The TIPSTER SUMMAC Text Summarization Evaluation, Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics, June 08-12, 1999, Bergen, Norway
	Robert L. Donaway , Kevin W. Drummey , Laura A. Mather, A comparison of rankings produced by summarization evaluation measures, NAACL-ANLP 2000 Workshop on Automatic summarization, p.69-78, April 30-30, 2000, Seattle, Washington
	Martin Braschler , Peter Schäuble, Using Corpus-Based Approaches in a System for Multilingual Information Retrieval, Information Retrieval, v.3 n.3, p.273-284, October 2000
	Massimo Melucci, On rank correlation in information retrieval evaluation, ACM SIGIR Forum, v.41 n.1, p.18-33, June 2007
	Harr Chen , David R. Karger, Less is more: probabilistic models for retrieving fewer relevant documents, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Benjamin Piwowarski , Georges Dupret, Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM), Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Ian Soboroff , Donna Harman, Novelty detection: the TREC experience, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.105-112, October 06-08, 2005, Vancouver, British Columbia, Canada
	Ian Soboroff, Dynamic test collections: measuring search effectiveness on the live web, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	David Hawking , Nick Craswell , Peter Bailey , Kathleen Griffihs, Measuring Search Engine Quality, Information Retrieval, v.4 n.1, p.33-59, April 2001
	Douglas W. Oard , Julio Gonzalo , Mark Sanderson , Fernando López-Ostenero , Jianqiang Wang, Interactive Cross-Language Document Selection, Information Retrieval, v.7 n.1-2, p.205-228, January-April 2004
	Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia
	Anna Ritchie , Simone Teufel , Stephen Robertson, Creating a test collection for citation-based IR experiments, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.391-398, June 04-09, 2006, New York, New York
	Mark Sanderson , Hideo Joho, Forming test collections with no system pooling, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
	Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece
	Tetsuya Sakai , Karen Sparck-Jones, Generic summaries for indexing in information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.190-198, September 2001, New Orleans, Louisiana, United States
	Iain Campbell, Interactive Evaluation of the Ostensive ModelUsing a New Test Collection of Images with Multiple Relevance Assessments, Information Retrieval, v.2 n.1, p.89-114, February 2000
	Jane Reid , Mounia Lalmas , Karen Finesilver , Morten Hertzum, Best entry points for structured document retrieval: part I: characteristics, Information Processing and Management: an International Journal, v.42 n.1, p.74-88, January 2006
	Jaana Kekäläinen, Binary and graded relevance in IR evaluations: comparison of the effects on ranking of IR systems, Information Processing and Management: an International Journal, v.41 n.5, p.1019-1033, September 2005
	Paul McNamee , James Mayfield , Christine Piatko, Haircut: a system for multilingual text retrieval in java, Journal of Computing Sciences in Colleges, v.17 n.3, p.8-22, February 2002
	Steven M. Beitzel , Eric C. Jensen , Abdur Chowdhury , David Grossman, Using titles and category names from editor-driven taxonomies for automatic evaluation, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA
	Ben Wellner , Lisa Ferro , Warren Greiff , Lynette Hirschman, Reading comprehension tests for computer-based understanding evaluation, Natural Language Engineering, v.12 n.4, p.305-334, December 2006
	Jack G. Conrad , Xi S. Guo , Peter Jackson , Monem Meziou, Database selection using actual physical and acquired logical collection resources in a massive domain-specific operational environment, Proceedings of the 28th international conference on Very Large Data Bases, p.71-82, August 20-23, 2002, Hong Kong, China
	Viviane Moreira Orengo , Christian Huyck, Relevance feedback and cross-language information retrieval, Information Processing and Management: an International Journal, v.42 n.5, p.1203-1217, September 2006
	Yuri Kagolovsky , Jochen R. Moehr, Current Status of the Evaluation of Information Retrieval, Journal of Medical Systems, v.27 n.5, p.409-424, October 2003
	Jack G. Conrad , Joanne R. S. Claussen, Early user---system interaction for database selection in massive domain-specific online environments, ACM Transactions on Information Systems (TOIS), v.21 n.1, p.94-131, January 2003
	Alistair Moffat , Justin Zobel , David Hawking, Recommended reading for IR research students, ACM SIGIR Forum, v.39 n.2, p.3-14, December 2005
	W. S. Wong , R. W. P. Luk , H. V. Leong , K. S. Ho , D. L. Lee, Re-examining the effects of adding relevance information in a relevance feedback environment, Information Processing and Management: an International Journal, v.44 n.3, p.1086-1116, May, 2008
	Jialun Qin , Yilu Zhou , Michael Chau , Hsinchun Chen, Multilingual Web retrieval: An experiment in English–Chinese business intelligence, Journal of the American Society for Information Science and Technology, v.57 n.5, p.671-683, March 2006
	Peter Ingwerswen, Users in context, Lectures on information retrieval, Springer-Verlag New York, Inc., New York, NY, 2001
	Jane Reid, A Task-Oriented Non-Interactive Evaluation Methodologyfor Information Retrieval Systems, Information Retrieval, v.2 n.1, p.115-129, February 2000
	Inderjeet Mani , Gary Klein , David House , Lynette Hirschman , Therese Firmin , Beth Sundheim, SUMMAC: a text summarization evaluation, Natural Language Engineering, v.8 n.1, p.43-68, March 2002
	Eric C. Jensen , Steven M. Beitzel , Abdur Chowdhury , Ophir Frieder, Repeatable evaluation of search services in dynamic environments, ACM Transactions on Information Systems (TOIS), v.26 n.1, p.1-es, November 2007
	Pia Borlund, The concept of relevance in IR, Journal of the American Society for Information Science and Technology, v.54 n.10, p.913-925, August 2003
	Wessel Kraaij , Jian-Yun Nie , Michel Simard, Embedding web-based statistical translation models in cross-language information retrieval, Computational Linguistics, v.29 n.3, p.381-419, September 2003
	Ellen M. Voorhees , Donna Harman, The text retrieval conferences (TRECS), Proceedings of a workshop on held at Baltimore, Maryland: October 13-15, 1998, October 13-15, 1998, Baltimore, Maryland

INDEX TERMS

Primary Classification:
H. Information Systems
H.3 INFORMATION STORAGE AND RETRIEVAL
H.3.3 Information Search and Retrieval
Subjects: Retrieval models

General Terms:
Design, Management, Measurement, Performance, Theory

Collaborative Colleagues:

Ellen M. Voorhees: colleagues

Peer to Peer - Readers of this Article have also read:

M⁴: a metamodel for data preprocessing Proceedings of the 4th ACM international workshop on Data warehousing and OLAP
Anca Vaduva , Jörg-Uwe Kietz , Regina Zücker
Data structures for quadtree approximation and compression Communications of the ACM 28, 9
Hanan Samet
A hierarchical single-key-lock access control using the Chinese remainder theorem Proceedings of the 1992 ACM/SIGAPP Symposium on Applied computing
Kim S. Lee , Huizhu Lu , D. D. Fisher
Putting innovation to work: adoption strategies for multimedia communication systems Communications of the ACM 34, 12
Ellen Francik , Susan Ehrlich Rudman , Donna Cooper , Stephen Levine
The GemStone object database management system Communications of the ACM 34, 10
Paul Butterworth , Allen Otis , Jacob Stein

Adobe Acrobat

QuickTime

Windows Media Player

Real Player