Introduction to Natural Language Processing

Spring 2018

Instructors: Dr. Yoav Goldberg
Emails: yogo / cs.biu.ac.il
Office Hours: By email appointment

Resources

Online Discussion Forums

Managed via Piazza

Assignments

1. Assignment 1

2. Assignment 2, deadline May 16, Midnight.

3. Assignment 3, deadline June 10, Midnight.

4. Assignment 4, deadline: June 29, Midnight.

Machine Learning Reading Material

Students who do not take the Machine Learning course can read the first 3 chapters of the following Book. You are encouraged to read chapters 4 and 5 also (as well as the rest of the book, but this is not needed for our class).

Lectures

1. Intro to Natural Language Processing, and discussion of the sentence-boundary detection task. slides
   sentence boundary example 1 | example 2
   Optional reading: Gillick 2009

2. PP Attachment, Language Models, and simple probabilistic modeling || PP attachment data || Language model generation examples.

   Probability Review

   • Optional reading:

     • PP attachemnt: Collins and Brooks 1995
     • Language Models: Jurafsky and Martin ch. 4
     • Language Models: Mike Collins Lecture Notes
     • Language Models: Chen and Goodman empirical study of smoothing techniques for Language Modeling

   • Possible Excersice:

     • Implement the Collins and Brooks PP-attachment model.

3. Part of speech tagging with HMMs pos slides || tagging slides

   • Recommended reading: Parts-of-speech, from university of Sussex
   • Recommended reading: Mike Collins' HMM notes
   • Probability primer: Jason Eisner's tutorial (video)

4. Discriminative Training, more on Part of Speech Tagging slides || slides2

   • Recommended reading: Any textbook in Machine Learning || Greedy tagging paper || Better greedy-tagging (blog) || MEMM Paper || Structured Perceptron Paper
   • Further reading: Mike Collins' Log-linear models

5. Sequence Segmentation. slides

   • Optional Reading: Twitter Tagset and Tagging || F1 score (wikipedia) || Chunking as BIO tagging with SVMs || NER design and features || Semi-markov CRF (somewhat more complex than discussed in class, but same dynamic-program)

6. Syntax, PCFG, CKY. slides

   • Recommended Reading:
     Beatrice Santorini on Grammar, Constituency, etc |
     Beatrice Santorini on Constituency Tests (these are chapters 1 and 2 of a larger book) |
     Mike Collins' PCFG and CKY notes
   • More Slides Jason Eisner on PCFG and CKY

7. Distributional Similarity. slides

8. Dependecny Parsing. slides

   • The parse trees we've seen in class.
   • Further Reading: McDonald et al's MST-Parser, Joakim Nivre on transition-based parsing, Nivre and McDonald on Stacking, Sagae and Lavie on Voting, Koo et al on Cluster Features
     Kiperwasser and Goldberg Neural Dependency Parsing
   • Resources: Stanford Dependecies - Definition and Extraction Software || Universal Dependencies || Hebrew Dependency Treebank || Hebrew Universal Treebank.
   • Slides on Universal Dependencies. pdf

9. Dependency Parsing (continue from previous class)
   Information and Relation Extraction slides

   • Further Reading for Information and Relation Extraction Ellen Rillof's reading list
   • Resources: Relation Extraction Videos from Stanford NLP: 1 | 2 | 3 | 4

10. Unsupervised Learning and LDA. slides

    • Further Reading / watching:
      -- Emojis Sentiment paper
      -- Clear and practical explanation and derivation of LDA and the collapsed Gibbs sampler by William Darling.
      -- Topic Models and LDA (Steyvers and Griffiths)
      -- Tutorial talk on LDA by David Blei
    • Resources:
      -- David Blei's Topic Modeling Resources
      -- Mallet
      -- gensym
    • Very Optional Reading (derivations, background, etc):
      -- Bayesian (dirichlet) extension to HMM, nice explenation of this type of graphical models. Goldwater and Griffiths
      -- Slides on the Dirichlet distribution by Mark Johnson.
      -- The complete derivation of the LDA integrals, by Bob Carpenter.

11. Coreference Resolution. slides

    • Further Reading: Features for Coreference and the Pairwise model the uiuc system, Stanford's rule-based sieve system, detecting singletons, BIU's predicate paraphrase detection, University of Washington's Neural End-to-End Coref
    • Resources: Stanford Coref System,
      UIUC Coref System
      UW Neural End-to-End Coref

12. Sentiment, Biases in Text, Ethics. main slides, "making a racist AI" .html,.ipynb, Bias In Text slides

    • Further Reading::
      • Sentiment tutorial by Bing Liu (slides, includes tons of references within)
      • Caliskan et al 2017 (embeddings include human biases)
      • Hovy and Spruit 2017 (social impact of NLP / ethics)
      • Kate Crawford The problem with Bias, NIPS 2017 keynote (Video)