E1 285o Advanced Deep Representation learning 3:1 (August 2022)

Course Instructor: Pratosh A P, ECE

Syllabus

Recap on Fundamentals of Deep Learning: Empirical Risk Minimization, Divergence minimizations and Likelihood maximization Techniques, Deep Learning Architectures (Convolutional and Recurrent Architectures).

Deep Generative Models: Introduction to Generative models, Autoregressive and invertible models, Latent variable models, Variational inference and recognition networks (VAE, WAE), Adversarial Learning, Generative Adversarial networks and variants (BiGAN, CycleGAN, StyleGAN, WGAN), Normalizing Flows, Score/Diffusion based models

Transfer Learning and Domain Adaptation: Discrepancy-Based Approaches: statistical (MMD) geometrical and architectural criteria, Generative Domain Adaptation: Adversarial and Non-adversarial Methods, Reconstruction based methods, Domain Generalization: Representation, data manipulation and Learning strategy methods

Few-shot and Meta Learning: Introduction to Multi-task and Transfer learning, Meta-learning framework for few-shot learning, Metric learning, comparators and relational networks, Optimization-based meta learning, Generative meta learning

Semi and Self-supervised Learning: Consistency Regularization, Proxy-label Methods, Active Learning, Weakly supervised learning methods, Self-supervised and Contrastive Representation Learning, Contrastive losses, Memory-bank techniques, BYOL, SWAV, SimCLR, MoCo, Hard negative mining.

Applications: Brief Discussions on Applications of each of the aforementioned topi

Textbooks / References

  1. Understanding Machine Learning: From Theory to Algorithms, Shai Shalev-Shwartz and Shai Ben-David, Cambridge University Press
  2. Murphy, Kevin P. Probabilistic Machine Learning: Advanced Topics, MIT Press, 2023
  3. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning, MIT Press, 2016
  4. Deisenroth, Marc Peter, A. Aldo Faisal, and Cheng Soon Ong. Mathematics For Machine Learning. Cambridge University Press, 2020.
  5. Machine Learning from Weak Supervision: An Empirical Risk Minimization Approach, By Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai and Gang Niu, MIT Press
  6. Deep Generative Modeling, Jakub M. Tomczak, Springer 2022
  7. Semi-Supervised Learning, Olivier Chapelle, Bernhard Schölkopf and Alexander Zien, MIT Press
  8. Seminal and Survey papers from Machine Learning Conferences such as ICML, Neurips, ICLR, CVPR, AISTATS etc.

Prerequisites:

  1. Mandatory: A course on probability theory
  2. Mandatory: A course on classical machine learning fundamentals
  3. Mandatory: Moderate programming skills in Python

Grading:

  • 1 Minor and 1 Major 50%
  • Project and assignments (20+20)%
  • Term Paper 10%