Deep Learning for Text and Sequences

Semester B, 2023

Class starts at 17:00.

Instructors: Prof. Yoav Goldberg
Email: yogo / cs.biu.ac.il
Office: Building 216 (virtual only this semester) Office Hours: By email appointment

Online Discussion Forums

Managed via Moodle.

Assignment and Excercises

All submissions are handled through the submission system.

1. Excercise 1: Hypothesis classes and XOR (due March 20, 2023)

2. Excercise 2: Softmax and Gradients (due March 28, 2023)

3. Assignment 1: Gradients and SGD (due 23:59, April 24, 2023). You may find this python (and numpy) intro useful.

4. Excercise 3: Hello Computation Graph (no need to submit)

5. Assignment 2: Window-based Tagging (Due May 16, 2023).

6. Assignment 3: RNN Acceptors and Transducers (Due May 31, 2023).

7. Assignment 4: Implementing an Extractive Question Answering Model (Optional. Due: July 30, 2023).

Lectures

1. Intro, Linear models (1). pdf
   • Optional videos:
     • Standard form for linear equations. video
     • Drawing 2d linear inequalities. video
   • further reading
     • Benefits of depth (Amrami and Goldberg) simple geometric proof
     • chapter 2 in neural nets for NLP book.
     • Stanford 231n on linear classifiers and on optimization.
     • Stanford 231n on babysitting the learning process and more

2. Multi-layer Perceptron (quick recap), representatios, Ngram-LM pdf

   • further reading
     • Dropout paper.
     • Chapters 8, 9 in neural networks for NLP book.
     • Bengio et al neural LM neural language modeling paper.

3. Word Embeddings, pdf

   • further reading:
     • Word Embeddings: Chapters 10, 11 in neural networks for NLP book.
   • Nice Online word2vec learning demo demo

4. Variable Length Seqs, Ngrams, ConvNets pdf

   • further reading:
     • More in-depth look of the behavior of CNNs on text data: Jacovi, Sar-Shalom and Goldberg
     • What information can be extracted from CBOW? (and other architectures): Adi et al
     • ConvNets for Sequences, Hashing Trick: Chapter 13 in neural networks for NLP book.
     • ConvNets in Vision: stanford course notes | stanford course slides

5. Computation Graph pdf || Recurrent Neural Networks (1) pdf

   • further reading:
     • Chapter 14 in neural networks for NLP book.
     • Andrej Karpathy's "unreasonable effectiveness" blog post
     • RNN vs. ngram language models blog post
     • Karpathy et al visualizing RNNs Language Models paper
     • Linzen et al RNNs and Syntax paper

6. Recurrent Neural Networks (2) pdf

   • further reading:
     • Chapter 15 in the neural networks for NLP book.
     • Kyunghyun Cho's class notes (section 4.2, 4.3)
     • Chris Olah's blog post
     • Weiss, Goldberg and Yahav (LSTM vs. GRU) Formal power of LSTM vs GRU
   • Optional reading:
     • Neubig et al, Autobatching in DyNet paper
     • Bradbury et al, QRNN paper
     • Plank Sogaard and Goldberg (biRNN for POS). pdf

7. Recurrent Neural Networks (3), Multi-task Learning, modeling, Seq2seq pdf

   • further reading:
     • Chapters 14 and 16 in neural networks for NLP book.
   • papers mentioned in lecture:
     • Sogaard and Goldberg (MTL at different layers). pdf
     • Klerke Sogaard and Goldberg (MTL with eye-tracking and CCG). pdf
     • Karen Livescu slides (MTL at different layers, speech). pdf
     • Gonen and Goldberg (Semi-supervised preposition sense). pdf
     • Ficler and Goldberg (Coordination modeling). pdf

8. Recurrent Neural Networks (4) - Attention-based Models, Transformers (if time) pdf

   • further reading:
     • More attention slides (from Graham Neubig).
     • Chapters 17 in the neural networks for NLP book (attention)
     • See the papers mentioned in slides.

9. Inverse-problems. LM-Pretraining. pdf

   • further reading:
     • See the papers mentioned in slides.
     • Presentation slides about GPT-3 and its abilities. pdf

10. Continue LM-pretraining (same slides as previous lecture).

    • mentioned in class:
      • Blog post which is partly terrible pseudo-science and partly interesting: Waluigi Effect

11. Embedding Projections, Adversarial Representation Learning, Ethics pdf

    • further reading:
    • Papers appearing on slides.
    • Embedding Projections: Mikolov et al 2013, Artetxe et al 2017,
    • Adversarial Representation Learning:Conneu et al 2017 (embedding projections), Chen et al 2016 (cross lingusl sentiment)
    • Ethics:
      • Kate Crawford Video Lecture re Ethics/Biases in ML, The Problem with Bias, Kate Crawford
      • Cathy O'neil Video Talk Weapons of Math Destruction
      • Margaret Mitchel and Timmit Gebru Podcast episode
      • FATML Conferences

Resources

Books and Tutorials

• Intro to machine learning concepts: Course in Machine Learning. In particular read the first 3 chapters. Some later chapters are also relevant.

• The Deep Learning book

• Primer on Neural Networks for NLP

• Kyunghyun Cho's class notes

Other Courses

• Graham Neubig at CMU. class of 2018 (includes videos)

• Graham Neubig at CMU. class of 2021 (includes videos)

• Roger Grosse and Nitish Srivastava at University of Toronto. class

• Richard Socher at Stanford. class

• Hugo Larochelle at University of Sherbrooke. video lectures