Link Search Menu Expand Document

Calendar

Behavior of large language models

Mon Jan 3
Introduction
Lecture
Percy Liang
  1. Why does this course exist?
  2. Language models
  3. Overview of the course
Wed Jan 5
Capabilities
Lecture Discussion
Percy Liang
  1. Adaptation framework
  2. Perplexity
  3. Prompting
  4. Overview of GPT-3 tasks
Discussion paper:
Mon Jan 10
Harms I
Lecture Discussion
Rishi Bommasani
  1. Performance disparities
  2. Social bias and stereotypes
Discussion paper:
Wed Jan 12
Harms II
Lecture Discussion
Percy Liang
  1. Toxicity
  2. Disinformation
Discussion paper:

Data behind large language models

Mon Jan 17
(no class - MLK day)
Wed Jan 19
Data
Lecture Discussion
Percy Liang
  1. Scrutiny of existing datasets
  2. Data documentation
  3. Data ecosystems
Discussion paper:
Mon Jan 24
Security and privacy
Lecture Discussion
Tatsunori Hashimoto
  1. Data poisoning
  2. Differential privacy
Discussion paper:
Wed Jan 26
Legal considerations
Lecture Discussion
Percy Liang
  1. Copyright law
  2. Fair use
Discussion paper:

Building large language models

Mon Jan 31
Modeling
Lecture Discussion
Percy Liang
  1. Tokenization
  2. RNNs, Transformers
Discussion paper:
Wed Feb 2
Training
Lecture Discussion
Percy Liang
  1. Objective functions
  2. Stability
  3. Debugging
Discussion paper:
Mon Feb 7
Parallelism
Lecture Discussion
Christopher RĂ©
  1. Data parallelism
  2. Model parallelism
  3. Pipeline parallelism
Discussion paper:
Wed Feb 9
Scaling laws
Lecture Discussion
Tatsunori Hashimoto
  1. Scaling laws
Discussion paper:
Mon Feb 14
Modular architectures
Lecture Discussion
Percy Liang
  1. Mixture of experts
  2. Memory-augmented (retrieval) models
Discussion paper:
Wed Feb 16
Adaptation
Lecture Discussion
Sang Michael Xie
  1. Probing
  2. Fine-tuning
  3. Lightweight fine-tuning
Discussion paper:
Mon Feb 21
no class (Presidents’ Day)
Wed Feb 23
Environmental impact
Lecture Discussion
Percy Liang
  1. Training and inference costs
  2. Carbon emissions
Discussion paper:

Beyond large language models

Mon Feb 28
Guest lecture
Lecture Discussion
Maxwell Nye
Wed Mar 2
Guest lecture
Lecture Discussion
Margaret Mitchell and Stella Biderman
Mon Mar 7
Project presentations
Student Presentations
Wed Mar 9
Project presentations
Student presentations