The strange case of reproducibility versus representativeness in contextual suggestion test collections

The final publication is available at Springer via http://dx.doi.org/10.1007/s10791-015-9276-9

Detalhes bibliográficos
Autores: Samar, Thaer, Bellogin Kouki, Alejandro, De Vries, Arien P.
Formato: artículo
Fecha de publicación:2016
País:España
Recursos:Universidad Autónoma de Madrid
Repositorio:Biblos-e Archivo. Repositorio Institucional de la UAM
Idioma:inglés
OAI Identifier:oai:repositorio.uam.es:10486/674485
Acesso em linha:http://hdl.handle.net/10486/674485
https://dx.doi.org/10.1007/s10791-015-9276-9
Access Level:acceso abierto
Palavra-chave:Reproducibility
Contextual suggestion
Open vs archived web
Test collections evaluation
Filtering and recommendation
Web IR and social media search
Informática
id ES_9571e9d92cbe15d07a4e1e7907177caf
oai_identifier_str oai:repositorio.uam.es:10486/674485
network_acronym_str ES
network_name_str España
repository_id_str
spelling The strange case of reproducibility versus representativeness in contextual suggestion test collectionsSamar, ThaerBellogin Kouki, AlejandroDe Vries, Arien P.ReproducibilityContextual suggestionOpen vs archived webTest collections evaluationFiltering and recommendationWeb IR and social media searchInformáticaThe final publication is available at Springer via http://dx.doi.org/10.1007/s10791-015-9276-9The most common approach to measuring the effectiveness of Information Retrieval systems is by using test collections. The Contextual Suggestion (CS) TREC track provides an evaluation framework for systems that recommend items to users given their geographical context. The specific nature of this track allows the participating teams to identify candidate documents either from the Open Web or from the ClueWeb12 collection, a static version of the web. In the judging pool, the documents from the Open Web and ClueWeb12 collection are distinguished. Hence, each system submission should be based only on one resource, either Open Web (identified by URLs) or ClueWeb12 (identified by ids). To achieve reproducibility, ranking web pages from ClueWeb12 should be the preferred method for scientific evaluation of CS systems, but it has been found that the systems that build their suggestion algorithms on top of input taken from the Open Web achieve consistently a higher effectiveness. Because most of the systems take a rather similar approach to making CSs, this raises the question whether systems built by researchers on top of ClueWeb12 are still representative of those that would work directly on industry-strength web search engines. Do we need to sacrifice reproducibility for the sake of representativeness? We study the difference in effectiveness between Open Web systems and ClueWeb12 systems through analyzing the relevance assessments of documents identified from both the Open Web and ClueWeb12. Then, we identify documents that overlap between the relevance assessments of the Open Web and ClueWeb12, observing a dependency between relevance assessments and the source of the document being taken from the Open Web or from ClueWeb12. After that, we identify documents from the relevance assessments of the Open Web which exist in the ClueWeb12 collection but do not exist in the ClueWeb12 relevance assessments. We use these documents to expand the ClueWeb12 relevance assessments. Our main findings are twofold. First, our empirical analysis of the relevance assessments of 2 years of CS track shows that Open Web documents receive better ratings than ClueWeb12 documents, especially if we look at the documents in the overlap. Second, our approach for selecting candidate documents from ClueWeb12 collection based on information obtained from the Open Web makes an improvement step towards partially bridging the gap in effectiveness between Open Web and ClueWeb12 systems, while at the same time we achieve reproducible results on well-known representative sample of the web.This research was supported by the Netherlands Organization for Scientific Research (WebART Project, NWO CATCH #640.005.001). Part of this work was supported by the Spanish Ministry of Science and Innovation (TIN2013-47090-C3-2). This work was carried out on the Dutch national e-infrastructure with the support of SURF Foundation.Springer NetherlandsDepartamento de Ingeniería InformáticaEscuela Politécnica SuperiorRecuperación de información (ING EPS-008)20162016-06-01research articlehttp://purl.org/coar/resource_type/c_2df8fbb1AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/674485https://dx.doi.org/10.1007/s10791-015-9276-9reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/6744852026-06-23T12:46:27Z
dc.title.none.fl_str_mv The strange case of reproducibility versus representativeness in contextual suggestion test collections
title The strange case of reproducibility versus representativeness in contextual suggestion test collections
spellingShingle The strange case of reproducibility versus representativeness in contextual suggestion test collections
Samar, Thaer
Reproducibility
Contextual suggestion
Open vs archived web
Test collections evaluation
Filtering and recommendation
Web IR and social media search
Informática
title_short The strange case of reproducibility versus representativeness in contextual suggestion test collections
title_full The strange case of reproducibility versus representativeness in contextual suggestion test collections
title_fullStr The strange case of reproducibility versus representativeness in contextual suggestion test collections
title_full_unstemmed The strange case of reproducibility versus representativeness in contextual suggestion test collections
title_sort The strange case of reproducibility versus representativeness in contextual suggestion test collections
dc.creator.none.fl_str_mv Samar, Thaer
Bellogin Kouki, Alejandro
De Vries, Arien P.
author Samar, Thaer
author_facet Samar, Thaer
Bellogin Kouki, Alejandro
De Vries, Arien P.
author_role author
author2 Bellogin Kouki, Alejandro
De Vries, Arien P.
author2_role author
author
dc.contributor.none.fl_str_mv Departamento de Ingeniería Informática
Escuela Politécnica Superior
Recuperación de información (ING EPS-008)
dc.subject.none.fl_str_mv Reproducibility
Contextual suggestion
Open vs archived web
Test collections evaluation
Filtering and recommendation
Web IR and social media search
Informática
topic Reproducibility
Contextual suggestion
Open vs archived web
Test collections evaluation
Filtering and recommendation
Web IR and social media search
Informática
description The final publication is available at Springer via http://dx.doi.org/10.1007/s10791-015-9276-9
publishDate 2016
dc.date.none.fl_str_mv 2016
2016-06-01
dc.type.none.fl_str_mv research article
http://purl.org/coar/resource_type/c_2df8fbb1
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv http://hdl.handle.net/10486/674485
https://dx.doi.org/10.1007/s10791-015-9276-9
url http://hdl.handle.net/10486/674485
https://dx.doi.org/10.1007/s10791-015-9276-9
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Springer Netherlands
publisher.none.fl_str_mv Springer Netherlands
dc.source.none.fl_str_mv reponame:Biblos-e Archivo. Repositorio Institucional de la UAM
instname:Universidad Autónoma de Madrid
instname_str Universidad Autónoma de Madrid
reponame_str Biblos-e Archivo. Repositorio Institucional de la UAM
collection Biblos-e Archivo. Repositorio Institucional de la UAM
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869413822471602176
score 15,198674

Registos relacionados