Believe me—We can do this! Annotating persuasive acts in blog text

From AcaWiki
Jump to: navigation, search

Citation: Pranav Anand, Joseph King, Jordan Boyd-Graber, Earl Wagner, Craig Martell, Doug Oard, and Philip Resnik (2011) Believe me—We can do this! Annotating persuasive acts in blog text. AAAI Workshop on Computational Models of Natural Argument (RSS)
Internet Archive Scholar (search for fulltext): Believe me—We can do this! Annotating persuasive acts in blog text
Tagged: Computer Science (RSS) persuasion (RSS), online argumentation (RSS), blogs (RSS), NLP (RSS), natural language processing (RSS), perlocutionary acts (RSS), speech act theory (RSS)

Summary

This paper describes the construction of a corpus of blog posts annotated to highlight persuasion tactics

The authors define persuasion as "instances where an agent attempts to convince another party to adopt a novel belief, attitude, or commitment to act."

Taxonomy

(from Table 2; these come mainly from Marwell and Schmitt or Cialdini, with a few inspired by Walton et al.

Outcomes

  • Threat/Promise
  • Social esteem
  • Self-feeling
  • Outcome

Generalizations

  • Deontic Appeal (to duties/obligations)
  • Moral Appeal
  • Social Generalization
  • Good/bad traits

External

  • Popularity
  • VIP

Interpersonal

  • Favors/debts
  • Consistency
  • Empathy
  • Scarcity

Other

  • Redefinition
  • Reason

Corpus Construction

  • Blogs were selected from the Blog Authorship Corpus (Koppel et al. 2006), collected in August 2004 from Blogger.com, with posts from 19,320 different blogs.

Test set

A separate test set of 30 posts was hand-selected "based on their coverage of tactics and persuasion acts". Interannotator agreement was low due to confusion between opinion and persuasion; situations where it wasn't clear whether persuasion was intended by the author; and "difficulty in distinguish between belief revision and attitude change (Is “smoking is bad for your health” aimed at attitude change or belief revision? What about “smoking is dangerous”?)."

Annotation Guidelines

Annotation guidelines focused on "justificatory text" and "blatant persuasion" (where the author's intent is clear); they are available with the persuasion corpus. Annotators were asked to mark the smallest text span containing a tactic, and to note whether persuasion was used (i.e. "was intended to forestall potential skepticism in the reader")

Annotation Process

Eight annotators were trained, then tested, on subsets of the test set, resulting in a Kappa of >.8.

40 blogs (among those with more than 200 posts) were randomly selected, resulting in 25,048 posts. Each annotator annotated all the posts in 7 blogs, with 20% overlap with other annotators. Persuasive acts or persuasion tactics were found in 4,603 posts in 37 blogs.

Persuasion acts were relatively rare (457 posts; 380 for believe revision, 128 for compliance gaining--51 had both). Agreement was measured with Krippendorff's $\alpha$. Sometimes there were serious disagreements (e.g. Redefinition--to distinguish metaphors and value judgements).


Predicting Persuasion from Tactics

They tested 3 systems to see how well they could classify posts based on features (compared to human annotation). Precision, recall, and F-score were the measures used.

The systems were stemmed unigram; sentiment/causation/insight (using 71 list count features from LIWC and MPQA); topics (using LDA); and tactics (based on the annotation). They also did an "ablation study", testing all 7 possible combinations with a Naive Bayes classifier, then trained RIPPER, a rule-based classifier, over the seven most important features (reason, deonic appeal, outcome, empathy, recharacterization, threat/promise, good/bad traits).

Analyzing False Negatives

2.1% of false negatives were incorrectly annotated 16% were "highly charged, opinionated statements or personal statements" 4% were reviews of movies, books, and CD's (not covered) 68% were "legitimately persuasive on reexamination" 3% were imperatives with "remember", "think", and "imagine"


Selected References

Taxonomy was influenced by

General literature

Argumentation mining

Subjectivity

Corpora

  • Koppel, M.; Schler, J.; Argamon, S.; and Pennebaker, J. 2006. Effects of age and gender on blogging. In In AAAI 2006 Spring Sym- posium on Computational Approaches to Analysing Weblogs.

Theoretical and Practical Relevance

Their persuasion corpus, which contains over 4600 blog posts annotated for the presence of persuasion and related tactics, is publicly available.

Distinguishing Persuasion from Opinion

Examples in the paper distinguish persuasion from opinion, and indicate the persuasive texts attempt to engage the reader.

Retrieved from "https://acawiki.org/index.php?title=Believe_me—We_can_do_this!_Annotating_persuasive_acts_in_blog_text&oldid=7318"