Calendar

HolidayLabor Day

LectureLLVM Introduction 1

Saining Xie

LLVM Neural Network Architectures: Past, Present and Paths Forward

LectureLLVM Introduction 2

Saining Xie

LLVM Training objectives: Supervised learning, Self-Supervised Learning, Generative Models and Beyond

Reading materials

• The Illustrated Transformer
• Attention is All You Need (Optional, but recommended!)

LectureLLVM Introduction 3

Saining Xie

LLVM Training objectives: Supervised learning, Self-Supervised Learning, Generative Models and Beyond

LLVM Data, Infrastructure, Model Training, Inference, Complexity, Metrics

• Making Deep Learning Go Brrrr From First Principles.
• NVIDIA deep learning performance guide.

LecturePaper Reading and Reviewing

Saining Xie

How to be a Good Citizen of the LLVM Community

Reading materials

CLIP: Learning Transferable Visual Models From Natural Language Supervision, Alec et al.

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, Li et al.

Reading materials

• OpenCLIP, Cherti et al.
• SLIP, Mu et al.
• FLIP, Li et al.
• BLIP2, Li et al.
• ImageBind, Girdhar et al.
• COCA, Yu et al.

GPT-3: Language Models are Few-Shot Learners, Brown et al.

Paper ReadingPanel Discussion - LLM & RLHF

WebGPT: Browser-assisted question-answering with human feedback, Nakano et al.

Reading materials

• Scaling Laws for Neural Language Models, Kaplan et al.
• OpenAI’s blog on ChatGPT
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Patrick Lewis et al.
• Learning to summarize from human feedback, Stiennon et al.
• Training language models to follow instructions with human feedback, Ouyang et al.

• John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

• GPT-4 Technical Report

• LLaMA: Open and Efficient Foundation Language Models, Touvron et al.
• Alpaca: A Strong, Replicable Instruction-Following Model
• Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality

• LlaMA 2: Open Foundation and Fine-Tuned Chat Models, Touvron et al.

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Wei et al.

Paper ReadingPanel Discussion - Multimodal LLM

Flamingo: a Visual Language Model for Few-Shot Learning, Alayrac et al.

Paper ReadingPanel Discussion - Multimodal LLM

LLaVA: Visual Instruction Tuning, Liu et al.

Reading materials

• IDEFICS, Laurençon et al., HuggingFace
• OpenFlamingo, Awadalla et al.
• MiniGPT4, Zhu et al.
• InstructBLIP, Dai et al.
• LISA, Lai et al.

Diffusion Models Beat GANs on Image Synthesis, Dhariwal et al.

Paper ReadingPanel Discussion - GenAI 1: Image

High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al.

Reading materials

• Classifier-free Guidance, Ho et al.
• Distillation, Cui et al.
• Flow matching, Lipman et al.
• ControlNet, Zhang et al.
• DreamBooth, Ruiz et al.

Imagen Video: High Definition Video Generation with Diffusion Models, Ho et al.

Paper ReadingPanel Discussion - GenAI 2: Video & 3D

Zero-1-to-3: Zero-shot One Image to 3D Object, liu et al.

Reading materials

• Make-a-video, Singer et al.
• DreamFusion, Poole et al.
• ObjectVerse, Deitke et al.

Segment Anything, Kirillov et al.

Paper ReadingPanel Discussion - Open World Perception

DINOv2: Learning Robust Visual Features without Supervision, Oquab et al.

Reading materials

• LISA, Lai et al.
• FastSAM, Zhao et al.
• GroupViT, Xu et al.
• LSeg, Li et al.

Generative Agents: Interactive Simulacra of Human Behavior, Park et al.

Paper ReadingPanel Discussion - LLM Agent

PaLM-E: An Embodied Multimodal Language Model, Driess et al.

Reading materials

• The Rise and Potential of LLM Based Agents, Xi et al.
• Lilian’s blog
• Voyager, Wang et al.
• HuggingGPT, Shen et al., HuggingFace Space
• ViperGPT, Suris et al.
• VisProg, Gupta et al.

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision, Radford et al.

Paper ReadingPanel Discussion - Data

LAION-5B: An open large-scale dataset for training next generation image-text models, Schuhmann et al.

Reading materials

• Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling, Gandhi et al.
• WhisperX: Time-Accurate Speech Transcription of Long-Form Audio, Bain et al.
• DataComp: In search of the next generation of multimodal datasets, Gadre et al.
• LAION-AESTHETICS, Schuhmann et al.
• WebLI - PaLI Dataset, chen et al.

Paper ReadingPanel Discussion - Infra and Training

Flash Attention-2: Faster Attention with Better Parallelism and Work Partitioning, Dao et al.

Paper ReadingPanel Discussion - Infra and Training

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Rajbhandari et al.

Reading materials

• DeepSpeed2
• Pathway, Chowdhery et al.
• MoE, Artetxe et al.
• Megatron-LM
• ZeRO-Infinity
• QLoRA: Efficient Finetuning of Quantized LLMs

• Llama 2: Open Foundation and Fine-Tuned Chat Models
• (With a focus on the safety design and discussions.)

Paper ReadingPanel Discussion - AI Safety, Privacy, Law, Alignment

• Extracting Training Data from Diffusion Models, Carlini et al.

Reading materials

• Extracting Training Data from Large Language Models, Carlini et al.
• Towards Measuring the Representation of Subjective Global Opinions in Language Models, Durmus et al.
• Generative AI is a minefield for copyright law, Mahari et al.
• Frontier AI Regulation: Managing Emerging Risks to Public Safety, Anderljung et al.
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming, Ge et al.
• CSCI-GA 3850-007 PhD Seminar: AI: Challenges in Law, Policy, and Ethics

• Evolutionary-scale prediction of atomic-level protein structure with a language model, Lin et al.

Paper ReadingPanel Discussion - AI4Science

• Magnetic control of tokamak plasmas through deep reinforcement learning, Degrave et al.

Reading materials

• Open Catalyst Project
• AlphaFold 2, Tunyasuvunakool et al.
• A glimpse of the next generation of AlphaFold
• GraphCast: AI model for faster and more accurate global weather forecasting
• Foundation models for generalist medical artificial intelligence, Moor et al.