Behind the article: Recognizing dialog acts in Wikipedia talk pages

From AcaWiki
Jump to: navigation, search

Citation: Oliver Ferschke and Iryna Gurevych and Yevgen Chebotar (2012) Behind the article: Recognizing dialog acts in Wikipedia talk pages. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (RSS)
Internet Archive Scholar (search for fulltext): Behind the article: Recognizing dialog acts in Wikipedia talk pages
Download: http://www.ukp.tu-darmstadt.de/fileadmin/user upload/Group UKP/publikationen/2012/EACL 2012 OF.pdf
Tagged: Computer Science (RSS) Wikipedia (RSS), Wikipedia Talk pages (RSS), speech act theory (RSS), dialog acts (RSS), collaborative writing (RSS), NLP (RSS), corpus-building (RSS), machine learning (RSS), content analysis (RSS), Simple English Wikipedia (RSS)

Summary

This paper uses speech act theory and dialog acts as a theoretical framework for studying collaborative writing in Wikipedia. The overall goal of this line of research (in its early stages) is to understand how Talk pages contribute to article improvement.

The authors use Simple English Wikipedia in order to have significant coverage: This paper analyzes 100 Talk pages, which they describe as "almost 15% of Talk pages from that wiki"; in fact, they ignored all Talk pages with fewer than 4 turns. They used the 2011-04-04 snapshot, which has 69900 articles and 5783 Talk pages; 683 Talk pages contained more than 3 contributions.

It provides:

  1. provides an annotation schema for coordination-related dialog acts
  2. results in a freely-downloadable corpus of 100 segmented and annotated Talk page, called the Simple English Wikipedia Discussion Corpus,
  3. a machine learning procedure for classifying dialog acts on this corpus (seen as "a dialog act classification pipeline")

Annotation schema

The authors have 4 high-level classifications broken down into 17 subclassifications:

  1. Article Criticism
    1. Content incomplete or lacking detail
    2. Lack of accuracy or correctness
    3. Unsuitable or unnecessary content
    4. Structural problems
    5. Deficiencies in language or style
    6. Objectivity issues
    7. Other kind of criticism
  2. Explicit Performative Announce
    1. Explicit suggestion, recommendation or request
    2. Explicit reference or pointer
    3. Commitment to an action in the future
    4. Report of a performed action
  3. Information Content
    1. Information providing
    2. Information seeking
    3. Information correcting
  4. Interpersonal
    1. Positive attitude towards other contirbutor or acceptance
    2. Partial acceptance or partial rejection
    3. Negative attitude towards other contributor or rejection

Coordination-related-dialog-acts-for-Wikipedia-talk-pages.png

Corpus

Page selection

They used three classes, and randomly chose pages from these classes:

  1. 4-10 turns (50 pages)
  2. 11-20 turns (40 pages)
  3. more than 20 turns (10 pages)

They do not report the distribution of turn count; that would be useful.

Segmentation

They take a thorough approach to automatically segmenting Talk pages, both into threads, topics, and individual author contributions, using some new libraries like the Java-based Wikipedia library and the Wikipedia Revision Toolkit, described in Wikipedia Revision Toolkit: Efficiently accessing Wikipedia’s edit history.

Annotation

Annotations were made by two annotators who were trained on 10 (held-out) discussion pages. Annotators could discuss difficult cases and consult the coordinator. A third person reconciled annotations, choosing a value for the gold standard when the others did not agree.

Further discussion of the annotation procedures (e.g. an appendix with the annotation manual) would make this work easier to reuse; the authors comment that some labels (e.g. interpersonal category, "other kind of criticism") were particularly problematic, and should be revisited in the future.

Indirectness is one of the issues mentioned as an issue, and thus they also provide a useful discussion of conversational implicature theory, with some examples.

File formats

They used the open source MMAX2 software and data from their corpus is released in MMAX2's native format as well as UIMA's XMI format.

Machine learning pipeline

Using Weka, they compare three machine learners Naive Bayes, a decision tree algorithm, and SMO (an SVN optimization algorithm), and combine the best performers for each label into a UIMA-based classification pipeline. They describe feature selection and classification results, pointing out some issues with inter-annotator agreement. One interesting topic is that the algorithms can sometimes outperform humans, for instance with the "Other kind of criticism" and "Unsuitable or unnecessary content" classes.

Selected references

Underlying theory

  • Paul Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3. New York: Academic Press.
  • Ilona R. Posner and Ronald M. Baecker. 1992. How people write together. In Proceedings of the 25th Hawaii International Conference on System Sciences, pages 127–138, Wailea, Maui, HI, USA.


  • John R. Searle. 1969. Speech acts. Cambridge University Press, Cambridge, UK.


Dialog act classification in other genres

  • William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to classify email into "speech acts". In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 309–316, Barcelona, ES.

Related research in Wikipedia

Information quality

Talk pages

  • Fernanda Vie ́gas, Martin Wattenberg, Jesse Kriss, and Frank Ham. 2007. Talk before you type: Coordination in Wikipedia. In Proceedings of the 40th Annual Hawaii International Conference on System Sciences, Waikoloa, Big Island, HI, USA.

Wikipedia-specific Tools

Code is the Java-based Wikipedia library and the Wikipedia Revision Toolkit, described in 2 papers:

Theoretical and Practical Relevance

Most important is their free downloadable corpus, the Simple English Wikipedia Discussion Corpus, of 100 segmented and annotated Talk pages.

Their segmentation approaches should be considered as a candidate for the standard way to segment Talk pages.

They also provide a useful discussion of conversational implicature theory, with some examples from their corpus

Useful definitions

The authors provide some useful definitions: We define a turn (or contribution) as the body of text that is added by an individual contributor in one or more revisions to a single discussion topic until another contributor edits the page."

"a topic (or discussion) is the body of turns that revolve around a single matter. They are usually headed by a topic title."

"the thread structure designates the sequence of turns and their indentation levels on the Talk page."

Graphing method

  • Figure 2 uses an interesting graphing approach: it shows F1-scores for the classification pipeline ("best performance") as a bar chart, along with human and baseline performance marked on the individual bars.

Dialog acts

Their literature review section, in "related work", and in "classification results", provides a useful introduction to theory and applications of dialog acts, including some recent work in learning analytics.

Retrieved from "https://acawiki.org/index.php?title=Behind_the_article:_Recognizing_dialog_acts_in_Wikipedia_talk_pages&oldid=7713"