Advanced Topics in Deep Learning (896542)


Staff:

Lecturer: Yossi Keshet <jkeshet@cs.biu.ac.il>

Teaching assistant: Roni Cheriyak <advmlcourses+adv@gmail.com>

Zoom link to the live meeting is https://us02web.zoom.us/j/86973527069?pwd=UkFMVkxzRFR2cUR6R08zNjRBMWJMUT09


Lecture notes

  1. Deep learning theory 1: What is machine learning theory? What are generalization bounds?
    Definitions and introduction.
    Recording of the lecture held on 8/3/2021.

  2. Deep learning theory 2: Re-thinking generalization; Robustness and generalization; Stability and generalization
    The lecture is based on the following resources:
    - Zhang, Bengio, Hardt, Recht and Vinyals, Understanding deep learning requires rethinking generalization, 2017
    - Jakubovitz, Giryes, and Rodrigues, Generalization Error in Deep Learning, 2018
    - Xu and Mannor, Robustness and Generalization, 2012
    - The blog of Mostafa Samir (three parts), 2018
    Recording of the lecture held on 15/3/2021.

  3. Deep learning theory 3: PAC-Bayes theory and generalization
    The lecture is based on the following resources:
    - Belkin, Hsu, Ma, and Mandal, Reconciling modern machine-learning practice and the classical bias–variance trade-off, 2019
    - Jiang, Neyshabur, Mobahi, Krishnan, and Bengio, Fantastic Generalization Measures and Where to Find Them, 2019
    Recording of the lecture held on 6/4/2021.

  4. Adversarial attacks and defenses
    Recording of the lecture held on 12/4/2021.

  5. Variational Auto-encoders (VAE)
    There are two blog posts that I like. The first is Understanding Variational Autoencoders (VAEs) by Joseph Rocca, and the second is Variational Autoencoders Explained by Kevin Frans
    Recording of the lecture held 26/5/2021.

  6. Generative Adversarial Networks (GANs)
    Recording of the lecture held on 3/5/2021.

  7. Wassershtein GANs
    The lecture is based on many resources by the GAN blog series of Jonathan Hui is highly recommended.
    The recording of the lecture held on 24/5/2021.

  8. GANs applications
    The recording of the lecture held on May 31, 2021.

  9. Advanced training methods: multitask learning, adversarial learning, and mixup.
    The recording of the lecture held on June 10, 2021 (corrected link).

  10. The Attention Mechanism
    The Jay Alammar's blog: Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
    The Listen, Attend, and Spell paper.
    Recording of the lecture held on 14/6/2021.


Assignments:

  1. Assignments 1: GANs. Instructions and generate_data.py - due June 13, 2021 at 22:00 via the SUBMIT system. This is not a mandatory assignment.