The Combination of Decision in Crowds When the Number of Reliable Annotator Is Scarce

Agus Budi Raharjo*, Mohamed Quafafou

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Citation (Scopus)

Abstract

Crowdsourcing appears as one of cheap and fast solutions of distributed labor networks. Since the workers have various expertise levels, several approaches to measure annotators reliability have been addressed. There is a condition when annotators who give random answer are abundance and few number of expert is available Therefore, we proposed an iterative algorithm in crowds problem when it is hard to find expert annotators by selecting expert annotator based on EM-Bayesian algorithm, Entropy Measure, and Condorcet Jury’s Theorem. Experimental results using eight datasets show the best performance of our proposed algorithm compared to previous approaches.

Original languageEnglish
Title of host publicationAdvances in Intelligent Data Analysis XVI - 16th International Symposium, IDA 2017, Proceedings
EditorsNiall Adams, Allan Tucker, David Weston
PublisherSpringer Verlag
Pages260-271
Number of pages12
ISBN (Print)9783319687643
DOIs
Publication statusPublished - 2017
Externally publishedYes
Event16th International Symposium on Intelligent Data Analysis, IDA 2017 - London, United Kingdom
Duration: 26 Oct 201728 Oct 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10584 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference16th International Symposium on Intelligent Data Analysis, IDA 2017
Country/TerritoryUnited Kingdom
CityLondon
Period26/10/1728/10/17

Keywords

  • Annotator reliability
  • Crowdsourcing
  • EM algorithm

Fingerprint

Dive into the research topics of 'The Combination of Decision in Crowds When the Number of Reliable Annotator Is Scarce'. Together they form a unique fingerprint.

Cite this