All 26 lectures.

Content outline

1. 01Traditional Software vs. LLMs// 8 min

   • ›Rule-based systems: if-then logic
   • ›Statistical models: learning from data
   • ›Language models: predicting the next word given context
   • ›Example: "The capital of France is ___" → P(Paris) = 0.98, P(London) = 0.001

2. 02The Core Idea: Autoregressive Generation// 12 min

   • ›How LLMs work step-by-step
   • ›Token-by-token generation (show animation)
   • ›Why randomness exists (sampling, not deterministic)
   • ›Probability distributions over vocabulary
   • ›Walk through a full example (prompt → 10 tokens generated, step-by-step)

3. 03Vocabulary and Tokens// 10 min

   • ›What's a token? (not a word)
   • ›Byte-pair encoding (BPE): how text gets tokenized
   • ›Why "ChatGPT" = 3 tokens, "I'm" = 2 tokens
   • ›Token counting matters for cost and context
   • ›Hands-on: tokenize a sentence, show byte-level breakdown

4. 04First Interaction: Prompt an LLM// 15 min

   • ›Access Claude API / ChatGPT API
   • ›Structure: system prompt → user message → model response
   • ›Demo: simple Q&A
   • ›Demo: giving the same prompt twice (same vs. different outputs)
   • ›Observation: the model isn't deterministic

Content outline

1. 01The Scale Hypothesis// 12 min

   • ›Bigger models → better performance
   • ›Parameter count: 7B, 13B, 70B, 100B+
   • ›Training compute: FLOPs, GPUs, time
   • ›Why we went from 1.3B (BERT) → 70B (Llama 2) → 200B+ (GPT-4)
   • ›Diminishing returns: each 10x parameters ≈ incremental gain

2. 02Emergent Capabilities// 13 min

   • ›In-context learning: ability to learn from examples in the prompt
   • ›Chain-of-thought: reasoning step-by-step when prompted
   • ›Few-shot learning: adapting to new tasks with minimal examples
   • ›Why these matter for product: what was impossible at 7B becomes possible at 70B
   • ›Timing: when did these abilities emerge? (scaling laws paper)
   • ›Example timeline: GPT-1 (117M) → GPT-2 (1.5B) → GPT-3 (175B) → GPT-4 (?)

3. 03The Scaling Laws// 13 min

   • ›Chinchilla scaling: optimal ratio of parameters to training tokens
   • ›Loss curves: smoother with more data/parameters
   • ›Predicting performance: if we know scale, can we predict capability?
   • ›Why larger models sometimes fail at simple tasks (brittleness)
   • ›Trade-offs: bigger = more capable but slower, more expensive, harder to deploy

4. 04What LLMs Can and Cannot Do// 12 min

   • ›Strong: language understanding, reasoning, coding, writing
   • ›Weak: real-time information, math (sometimes), grounding in facts
   • ›False claims: "LLMs don't understand" vs. "we don't know what understanding is"
   • ›Capability heterogeneity: great at English, weaker at low-resource languages
   • ›Benchmarks vs. real-world: why test performance ≠ production performance

Content outline

1. 01What is Context?// 10 min

   • ›Context = text the model can see to predict next token
   • ›Context window = max number of tokens the model can "remember"
   • ›Common sizes: 4K (older), 8K, 16K, 128K, 200K (frontier models)
   • ›Why it matters: long documents, conversation history, multi-turn chat
   • ›What happens at the edge: performance degradation ("lost in the middle")

2. 02The Attention Mechanism// 15 min

   • ›Intuition: query, key, value (simplified)
   • ›What attention does: lets the model look back at earlier tokens
   • ›Multi-head attention: parallel patterns (different heads attend to different things)
   • ›Example: "The bank executive was fired after the scandal. She..."
   • ›Attention weight on "executive" is high
   • ›Attention weight on "bank" is lower for "She"
   • ›Computational cost: O(n²) — quadratic in sequence length
   • ›Why this is a hard limit (not just a soft constraint)

3. 03The Context Window Problem// 13 min

   • ›Quadratic complexity means 2x context = 4x compute
   • ›Training LLMs at 2M tokens (rare, expensive, research only)
   • ›Inference cost scales too: longer prompt = slower response, higher API cost
   • ›Position encoding: how does a model know token position? (affects extrapolation beyond training)
   • ›Tricks to extend context: RoPE, ALiBi, flash attention (faster, not longer)

4. 04Practical Context Limits// 12 min

   • ›How much text fits in 128K tokens? (roughly 90K words, or a novel)
   • ›Lost in the middle: performance drops when info is in the middle of context
   • ›Best practices: put key info at start or end
   • ›Cost of long context: 128K window → 10x price vs. 4K window
   • ›Trade-off: more context but slower response

Content outline

1. 01From Probabilities to Tokens// 12 min

   • ›Model outputs probability distribution over vocabulary
   • ›Sampling: randomly pick token according to probabilities
   • ›Why not just take the most likely token? (greedy decoding → repetitive, boring)
   • ›Example: "The weather today is"
   • ›P(sunny) = 0.4, P(rainy) = 0.3, P(cloudy) = 0.2, P(snowy) = 0.1
   • ›Greedy: always pick "sunny"
   • ›Sampling: sometimes "rainy," sometimes "cloudy"

2. 02Temperature: Controlling Randomness// 13 min

   • ›Temperature = 0: always pick highest probability (deterministic)
   • ›Temperature = 1: standard sampling (model's learned probabilities)
   • ›Temperature > 1: flatten distribution (more creative, more errors)
   • ›Temperature < 1: sharpen distribution (more confident, less creative)
   • ›Visual: probability distributions at different temperatures
   • ›When to use: creative writing (T=0.8-1.0) vs. classification (T=0)

3. 03Top-K and Top-P Sampling// 12 min

   • ›Top-K: only sample from the K most likely tokens
   • ›Top-P (nucleus sampling): only sample from tokens that make up cumulative probability P
   • ›Why: avoid tails of the distribution (very unlikely tokens)
   • ›Comparison: top-K = fixed, top-P = adaptive (better)
   • ›Settings: top-K=40 or top-P=0.9 are common

4. 04Practical Settings and Examples// 8 min

   • ›Customer service bot: T=0, top-P=0.9 (consistent, accurate)
   • ›Creative writing: T=0.9, top-P=0.95 (varied, interesting)
   • ›Summarization: T=0, top-P=1.0 (factual, deterministic)
   • ›Show API calls with different settings and outputs

Content outline

1. 01Supervised Learning Fundamentals// 10 min

   • ›Input (text) → Output (next token)
   • ›Loss function: measures how wrong the model is
   • ›Cross-entropy loss (most common for language models)
   • ›Lower loss = better predictions
   • ›Training loop: forward pass, compute loss, backprop, update weights

2. 02Pre-training on Internet-Scale Data// 13 min

   • ›Training data: Common Crawl, Wikipedia, books, code, Reddit, etc.
   • ›Data cleaning and filtering (content policy, quality)
   • ›How much data? GPT-3 (300B tokens) → Llama 2 (2T tokens) → ongoing scale
   • ›Token repetition: how many times are tokens seen?
   • ›Data composition: what's the mix of sources? (affects model behavior)
   • ›Why diversity matters: overfitting on narrow distributions

3. 03The Learning Curve// 14 min

   • ›Loss decreases over time (training progresses)
   • ›Plotting loss (x=training steps, y=loss)
   • ›Overfitting: loss decreases, but generalization breaks
   • ›Validation set: held-out data to detect overfitting
   • ›Scaling laws: loss ∝ 1/(parameters × tokens)^α (α ≈ 0.07)
   • ›Chinchilla scaling: optimal compute allocation (roughly 20 tokens per parameter)

4. 04Compute Cost and Timeline// 13 min

   • ›Training GPT-3: $4M+, months on specialized hardware
   • ›Floating-point operations (FLOPs): measure of computational work
   • ›Hardware: NVIDIA H100, TPUs, custom silicon (very expensive)
   • ›Why it's centralized: training is expensive, inference is accessible
   • ›Democratization: open models (Llama) enable more people to build

Content outline

1. 01From Base Model to ChatBot// 10 min

   • ›Pre-trained LLM: great at prediction, sometimes incoherent
   • ›Example: "How do I bake a cake?" → model might complete with Shakespeare poetry
   • ›Instruction tuning: teaching the model to follow instructions and be helpful
   • ›Data needed: (instruction, desired response) pairs
   • ›Fine-tuning on instruction pairs: supervised learning on "good" responses

2. 02Instruction Tuning in Practice// 12 min

   • ›Dataset size: 100K - 1M instruction-response pairs
   • ›Source: human-written examples, distillation from larger models
   • ›Example format:
   • ›Loss function: same as pre-training (next-token prediction)
   • ›Training time: hours to days (vs. weeks for pre-training)
   • ›Effect: model becomes more helpful, follows format, refuses harmful requests

3. 03RLHF: Reinforcement Learning from Human Feedback// 15 min

   • ›Problem: what's "good" is subjective (harmless? helpful? honest?)
   • ›Solution: let humans rate outputs, use those ratings to train a reward model
   • ›Step 1: collect examples, let humans rank pairs (output A vs. output B)
   • ›Step 2: train a reward model (neural network that predicts human preference)
   • ›Step 3: use PPO (reinforcement learning) to fine-tune the base model
   • ›Result: model learns to generate outputs humans prefer (not just next-token accurate)
   • ›Example: honesty, helpfulness, harmlessness become learned behaviors

4. 04Modern Alignment Techniques// 13 min

   • ›Constitutive AI: train model against a set of principles, no human in the loop
   • ›DPO (Direct Preference Optimization): simpler than RLHF, getting popular
   • ›SFT (Supervised Fine-Tuning): just instruction tuning, no RL
   • ›Cost: adding RLHF adds weeks to training timeline
   • ›Why it matters: Claude and GPT-4 feel "safer" than earlier models (RLHF effect)

Content outline

1. 01Full Fine-Tuning: When and Why// 12 min

   • ›Start with pre-trained model (weights frozen)
   • ›Add new task-specific data (domain-specific examples)
   • ›Update all weights during training
   • ›Use case: domain-specific knowledge (medical, legal, technical)
   • ›Example: fine-tune on internal documentation + customer tickets
   • ›Cost: $100 - $100K+ depending on model size and data

2. 02Data Requirements and Quality// 12 min

   • ›How much data do you need? (rough rule: 100-1000 examples minimum)
   • ›Quality > quantity (100 pristine examples > 10K noisy examples)
   • ›Data format: instruction-response pairs, or just completion examples
   • ›Overfitting: with small datasets, model memorizes not generalizes
   • ›Data leakage: remove test data from training set
   • ›Cleaning: remove duplicates, fix formatting, ensure quality

3. 03LoRA and Parameter-Efficient Fine-Tuning// 15 min

   • ›Full fine-tuning: update all 70B parameters (expensive)
   • ›LoRA (Low-Rank Adaptation): update only small matrices (1000s of parameters)
   • ›Why it works: weight updates often have low rank (are compressible)
   • ›Cost: 10-100x cheaper than full fine-tuning
   • ›Time: hours instead of days
   • ›Trade-off: slightly less powerful than full fine-tuning, still very effective
   • ›Example: LoRA fine-tuning on a single GPU possible (no infrastructure)

4. 04Fine-Tuning vs. Prompting Decision Tree// 11 min

   • ›If data < 100 examples → prompting + few-shot learning
   • ›If data 100-1000 examples → consider LoRA
   • ›If data > 1000 examples + sufficient compute → full fine-tuning
   • ›If domain is very specialized → fine-tuning (medical terminology)
   • ›If task is general but personalized → prompt + context (RAG)
   • ›Cost-benefit: API call cost vs. fine-tuning investment
   • ›Example calculations:
   • ›LoRA fine-tuning: $500 upfront, then cheaper inference
   • ›Vs. API calls: $0 upfront, $X per call forever

Content outline

1. 01Prompt Anatomy// 12 min

   • ›System prompt: sets context, role, rules ("You are a helpful assistant...")
   • ›User message: the actual request
   • ›Format:
   • ›Why it matters: same request with different system prompts gives different outputs
   • ›Example: "Explain photosynthesis" as a tutor vs. as a poet vs. as a comedian

2. 02Clarity and Specificity// 12 min

   • ›Vague: "Write a blog post"
   • ›Specific: "Write a 500-word blog post about LLM fine-tuning for product managers, with a hook, 3 main points, and a call-to-action"
   • ›Why: specific prompts → better outputs
   • ›Anti-pattern: asking for too much at once (ambiguous)
   • ›Best practice: break into substeps if complex
   • ›Example: instead of "analyze customer feedback," ask for "list 3 themes," then "prioritize by impact," then "suggest actions"

3. 03Instructions vs. Context vs. Examples// 13 min

   • ›Instructions: what to do ("Summarize this in 1 sentence")
   • ›Context: background ("You're writing for a 5-year-old")
   • ›Examples: show-and-tell (few-shot learning)
   • ›Structure: context first, then instructions, then examples
   • ›Anti-pattern: burying the ask in the middle
   • ›Best practice: put ask at the end, context at the start

4. 04Common Prompt Patterns// 8 min

   • ›Q&A: "What is X?"
   • ›Completion: "The capital of France is"
   • ›Translation: "Translate to Spanish:"
   • ›Classification: "Is this email spam? Yes/No"
   • ›Generation: "Write a poem about..."
   • ›Reasoning: "Explain your reasoning:"

Content outline

1. 01Zero-Shot vs. Few-Shot// 12 min

   • ›Zero-shot: ask directly, no examples ("Is this email spam?")
   • ›Few-shot: include examples (1-5 pairs), then ask
   • ›Why it works: models adapt within a conversation to patterns in examples
   • ›In-context learning: model learns from context without updating weights
   • ›Example:

2. 02Designing Few-Shot Examples// 13 min

   • ›Example quality matters: clean, representative, diverse
   • ›How many examples? Sweet spot is 2-5 (more doesn't always help)
   • ›Diminishing returns: 10 examples ≈ 5 examples (context is precious)
   • ›Order effects: first example has high impact
   • ›Negative examples: include what NOT to do
   • ›Domain variety: show range of cases the model will see

3. 03The Context Window Trade-off// 13 min

   • ›Each example uses tokens (cost and latency)
   • ›Formula: tokens per example × number of examples = token overhead
   • ›Alternative: fine-tuning (pay once, inference cheaper)
   • ›When examples are worth it: diverse tasks, quick changes, no infrastructure
   • ›When examples are wasteful: high volume, consistent task (fine-tune instead)
   • ›Optimization: prompt compression (remove redundant examples)

4. 04In-Context Learning Mechanics// 12 min

   • ›Why does the model learn from examples? (it's recognizing patterns in context)
   • ›What the model is "learning": task format, response style, edge cases
   • ›Limitations: model doesn't truly understand task, just pattern-matches
   • ›Failure modes: if examples are unrepresentative, model generalizes wrongly
   • ›Advanced: meta-learning (examples teach the model how to learn)

Content outline

1. 01Chain-of-Thought: The Idea// 12 min

   • ›Naive prompt: "What's 2 + 3 × 4?" → Model might say "20" (wrong)
   • ›CoT prompt: "What's 2 + 3 × 4? Let me work through this step-by-step..."
   • ›Result: Model reasons aloud, gets "14" (correct)
   • ›Why: verbalizing reasoning reduces errors (especially in math, logic)
   • ›Trade-off: longer output (more tokens = higher cost)
   • ›When it helps: math, logic, complex reasoning
   • ›When it doesn't: simple Q&A, semantic tasks

2. 02Writing Effective CoT Prompts// 13 min

   • ›Magic phrase: "Let me think step-by-step" or "Let me work through this"
   • ›Show examples: include CoT in few-shot examples
   • ›Explicit steps: "First... then... finally..."
   • ›Ask for confidence: "How confident are you in this answer?"
   • ›Ask for alternatives: "What's another way to solve this?"
   • ›Structure output: "Show your work before the final answer"

3. 03Advanced Reasoning Techniques// 15 min

   • ›Self-consistency: generate multiple reasoning paths, pick majority answer
   • ›Tree-of-thought: explore multiple branches, backtrack on failures
   • ›Decomposition: break complex problem into sub-problems
   • ›Verification: ask model to check its own work
   • ›Example: math word problem
   • ›Decompose: "First, identify given information. Then identify unknown. Then solve."
   • ›Generate multiple paths
   • ›Pick most consistent answer

4. 04Limitations and Failure Modes// 10 min

   • ›Model can reason but still get it wrong (false confidence)
   • ›Hallucinated reasoning: makes up steps that sound plausible
   • ›Not actually reasoning: pattern-matching to training data explanations
   • ›Cost trade-off: CoT is more tokens (2-3x longer)
   • ›When to skip: if accuracy isn't critical, CoT might be overkill

Content outline

1. 01The Problem: Hallucinations and Outdated Knowledge// 10 min

   • ›LLM pre-trained on data from 2023, but it's 2024 (outdated)
   • ›LLM trained on public data, but you need private knowledge (internal docs)
   • ›LLM hallucinates confidently (makes up facts)
   • ›Solution: give the model access to facts before asking it to respond
   • ›Example: "What did Anthropic announce in August 2026?" (beyond training data)

2. 02RAG Architecture// 13 min

   • ›Step 1: Store documents (break into chunks, embed, store in vector DB)
   • ›Step 2: User asks a question
   • ›Step 3: Retrieve relevant documents (semantic search)
   • ›Step 4: Construct prompt with context + question
   • ›Step 5: LLM responds using provided context
   • ›Benefit: model responds only based on provided facts (fewer hallucinations)
   • ›Trade-off: response time (retrieval is additional step)

3. 03Embeddings and Vector Search// 15 min

   • ›Embedding: convert text to vector (array of numbers)
   • ›Semantic search: find documents similar in meaning (vector similarity)
   • ›Vector database: store and query embeddings (Pinecone, Weaviate, Supabase)
   • ›Chunking: break documents into pieces (usually 500-1000 tokens)
   • ›Retrieval quality matters: bad chunks → bad answers
   • ›Reranking: retrieve top-20, rerank with a better model, use top-5

4. 04Prompt Construction with Context// 12 min

   • ›Format:
   • ›Handling multiple documents: concatenate, or separate with markers
   • ›Length limits: if context + question exceed window, truncate intelligently
   • ›Instruction: "Answer based only on the following context. If not found, say 'I don't know.'"
   • ›Failure mode: "I don't know" vs. hallucinating

Content outline

1. 01Why Evaluation Matters// 8 min

   • ›How do you know if your LLM is good?
   • ›"It looks good" is not a business metric
   • ›Evaluation → debugging → iteration
   • ›Types: correctness (is the answer right?), quality (is it well-written?), safety (does it avoid harm?)

2. 02Automatic Metrics// 14 min

   • ›BLEU: overlap between generated and reference text (used in translation)
   • ›ROUGE: recall-oriented metric (used in summarization)
   • ›BERTScore: semantic similarity using embeddings
   • ›Exact match: does output == reference? (too strict for most tasks)
   • ›F1: balance of precision and recall
   • ›Why they're flawed: don't capture semantic correctness, penalize paraphrasing
   • ›When to use: fast, reproducible, but don't trust blindly

3. 03Task-Specific Metrics// 14 min

   • ›Classification: accuracy, precision, recall, F1, confusion matrix
   • ›Named entity recognition: entity-level F1, not token-level
   • ›Summarization: ROUGE, human evaluation
   • ›Translation: BLEU, but also human fluency rating
   • ›Code generation: does it compile? Does it pass tests?
   • ›Open-ended generation (creative writing): requires human evaluation

4. 04Benchmarks vs. Real-World Performance// 9 min

   • ›Benchmark: held-out test set, static evaluation
   • ›Real-world: live distribution, shifting data, edge cases
   • ›Gap: models perform better on benchmarks than production
   • ›Why: benchmarks are curated, real data is messy
   • ›Solution: evaluate on distribution similar to production
   • ›Example: MMLU (multiple-choice) vs. open-ended Q&A

Content outline

1. 01Why Human Evaluation// 10 min

   • ›Many tasks can't be automatically evaluated (is the writing good?)
   • ›Subjectivity exists (some disagreement is OK)
   • ›Gold standard: humans can rate quality accurately
   • ›Cost: manual evaluation is slow and expensive
   • ›Sampling: can't evaluate everything, sample strategically
   • ›When: safety-critical tasks, quality assessment, model comparison

2. 02Designing Rubrics// 15 min

   • ›Rubric = rating scale + criteria + examples
   • ›Example rubric for customer service response:
   • ›Best practices: specific criteria, clear examples, unambiguous levels
   • ›Pilot test: have 3 people rate 20 examples, iterate rubric
   • ›Anti-pattern: vague criteria ("Is it good?") → low agreement

3. 03Inter-Rater Agreement// 15 min

   • ›Problem: different humans rate things differently
   • ›Solution: measure agreement, iterate until high
   • ›Cohen's Kappa: 0 = random, 1 = perfect agreement
   • ›Kappa > 0.7 is "good," 0.6-0.7 is "acceptable"
   • ›How to improve: clarify rubric, add examples, train raters
   • ›Example: "Accuracy" rubric v1 (Kappa=0.45) → add examples → v2 (Kappa=0.78)

4. 04Scaling Evaluation// 10 min

   • ›Sample size: how many outputs to evaluate?
   • ›Confidence interval: 100 samples → ~10% margin of error
   • ›Stratified sampling: sample across categories (good, bad, edge cases)
   • ›Crowdsourcing: hire raters on Prolific/Amazon Mechanical Turk
   • ›Cost: $0.50 per rating × 100 samples = $50
   • ›Time: 1 week for crowdsourced evaluation vs. 1 day in-house

Content outline

1. 01A/B Testing LLMs// 12 min

   • ›Scenario: you have prompt variant A and prompt variant B
   • ›Question: which is better? (can't tell by eye)
   • ›Solution: run both on same inputs, compare outputs
   • ›Sample size: need ~20-100 per variant (depends on effect size)
   • ›Measurement: pick 1-2 metrics (accuracy, user preference, cost)
   • ›Power analysis: can you detect the improvement you care about?
   • ›Example: prompt A (accuracy 85%) vs. prompt B (accuracy 87%) - is 2% real?

2. 02Statistical Significance// 13 min

   • ›Null hypothesis: there's no difference
   • ›P-value: probability we'd see this difference by chance
   • ›p < 0.05 is "statistically significant" (industry standard)
   • ›Confidence interval: likely range for true performance
   • ›Example: prompt B achieves 87% accuracy with 95% CI = [84%, 90%]
   • ›Common mistake: running test until you see p < 0.05 (p-hacking)
   • ›Solution: decide sample size upfront, don't peek

3. 03Production Monitoring// 15 min

   • ›Once deployed, does the model still work?
   • ›Metrics to track: accuracy, latency, cost, error rate, user satisfaction
   • ›Alerting: when does performance degrade? (set thresholds)
   • ›Drift: if real-world distribution shifts, performance may drop
   • ›Example: model trained on customer feedback, users ask about new product (shift)
   • ›Retraining: when to update the model
   • ›Logs: save all inputs/outputs, predictions for offline analysis

4. 04Red-Teaming and Adversarial Testing// 10 min

   • ›Red team: try to break the model
   • ›Jailbreaking: asking for forbidden outputs
   • ›Edge cases: unusual inputs, typos, multi-language
   • ›Stress testing: high-volume queries, long context
   • ›Safety testing: does model refuse harmful requests?
   • ›Example report: "Model generates toxic language when..."

Content outline

1. 01Before Transformers: RNNs and Vanishing Gradients// 8 min

   • ›RNNs: process text sequentially (one word at a time)
   • ›Problem: can't look back at distant past (gradient vanishing)
   • ›Solution: LSTMs, but still sequential (slow to train)
   • ›Insight: we don't need sequential! Process all words in parallel.

2. 02Transformers: Architecture Overview// 13 min

   • ›"Attention is All You Need" paper (2017) revolutionized NLP
   • ›Parallel processing: all positions simultaneously (fast training)
   • ›Attention: let each position "look at" all other positions
   • ›Multi-layer: stack many attention layers (depth = capability)
   • ›Embedding: convert tokens to vectors
   • ›Positional encoding: tell model about word order (since no recurrence)

3. 03How Attention Works (Intuitive)// 15 min

   • ›Query (Q): "What am I asking for?"
   • ›Key (K): "What information do I have?"
   • ›Value (V): "Here's the information"
   • ›Process: Q attends to K (computes similarity), weights V accordingly
   • ›Example: "The bank executive was fired after the scandal. She..."
   • ›Q at "She" asks: "Who is this pronoun referring to?"
   • ›Attention weights highest on "executive" (semantic match)
   • ›Attention weights low on "bank" (less relevant)
   • ›Multi-head attention: do this in parallel with multiple sets of Q/K/V

4. 04Computational Complexity and Why It Matters// 14 min

   • ›Attention: O(n²) complexity in sequence length
   • ›If sequence length doubles, attention cost quadruples
   • ›Training: parallelization helps, but memory is bottleneck
   • ›Inference: can't parallelize (must generate token-by-token)
   • ›Flash attention: faster implementation, same math (recent optimization)
   • ›Extended attention: ALiBi, RoPE (allow extrapolation beyond training length)

Content outline

1. 01Parameter Count and Capability// 12 min

   • ›Parameters: learned weights in the model
   • ›7B parameters (7 billion) = small model (can run on laptop)
   • ›70B parameters = large model (requires server)
   • ›200B+ = frontier (requires specialized infrastructure)
   • ›General trend: more parameters = better performance
   • ›BUT: diminishing returns (each 10x parameters = ~10% improvement)
   • ›Why scale? Because scaling is predictable (unlike random ideas)

2. 02Scaling Laws and Chinchilla Scaling// 13 min

   • ›Loss curves: loss ∝ 1 / (parameters × data)^α, where α ≈ 0.07
   • ›Chinchilla scaling: optimal ratio is ~20 tokens per parameter
   • ›GPT-3: 175B parameters, trained on 300B tokens (under-trained)
   • ›Chinchilla: 70B parameters, trained on 1.4T tokens (optimal)
   • ›Lesson: data is underrated; more data > more parameters
   • ›Doubling both data and parameters: 2x compute, better loss

3. 03Quantization and Distillation// 15 min

   • ›Quantization: reduce precision (float32 → float16 → int8)
   • ›Benefit: 4x smaller model, faster inference, less memory
   • ›Cost: slight accuracy loss (usually <1%)
   • ›When: quantize after training (post-training quantization)
   • ›Distillation: train small model to mimic large model
   • ›Benefit: small model still good, uses less compute
   • ›Example: distill GPT-4 into 7B model (much cheaper inference)

4. 04The Hardware Moat// 10 min

   • ›LLM inference bottleneck: memory bandwidth, not computation
   • ›GPUs have high bandwidth; CPUs have low bandwidth
   • ›Implication: need GPU/TPU for acceptable latency
   • ›Cost: GPUs expensive ($0.40/hour on cloud) → limits accessible models
   • ›Innovation: new hardware (Cerebras, GraphCore, custom silicon)
   • ›Asymmetry: training requires specialized hardware, inference is more accessible

Content outline

1. 01Inference vs. Training// 10 min

   • ›Training: slow (weeks), expensive (millions $), one-time
   • ›Inference: fast (ms), cheap (cents), repeated
   • ›Why they're different: training parallelizes, inference is sequential
   • ›Throughput: how many requests per second?
   • ›Latency: how long does one request take?
   • ›Cost: what's the per-token price?

2. 02Latency Breakdown// 13 min

   • ›Network: ~10-50ms (API call round-trip)
   • ›Compute: ~10-500ms (depends on model size, context length)
   • ›Output length: longer outputs = more time (generates token-by-token)
   • ›Batching: combine requests to amortize cost (but increases latency for individual)
   • ›Streaming: send tokens as they generate (feels faster to user)
   • ›Optimization: cache (Key-Value cache for past tokens, cheaper recomputation)

3. 03Serving Infrastructure// 15 min

   • ›API provider: Claude, OpenAI (simple, expensive, slow)
   • ›Self-hosted: run model on your server (cheap per-token, high upfront)
   • ›Hybrid: small model on-device, large model in cloud (best latency/cost)
   • ›Provisioning: how many GPUs to handle traffic?
   • ›Auto-scaling: scale up/down based on demand
   • ›Costs: GPU ($0.50-2/hour) × uptime = significant

4. 04Local vs. API Trade-offs// 12 min

   • ›API: simple, no infrastructure, limited control, high cost at scale
   • ›Local: complex, own infrastructure, full control, low cost at scale
   • ›Break-even: ~1M tokens/month → consider self-hosting
   • ›Hybrid: APIs for complex logic, local for simple/repetitive tasks
   • ›Example: customer service bot (high volume, simple) → local 7B model

Content outline

1. 01Hallucinations: Why Models Lie// 13 min

   • ›Definition: confident, wrong, plausible-sounding statements
   • ›Example: "What year did Anthropic launch Claude 10?" → model makes up a year
   • ›Root cause: model predicts next token based on training data, not facts
   • ›Why it's hard: model can't distinguish "real knowledge" from "pattern"
   • ›Severity: low for creative writing, high for medical/legal advice
   • ›Detection: is output consistent with known facts? (hard to automate)
   • ›Mitigation: RAG (ground in facts), fine-tuning for honesty, use stronger models

2. 02Taxonomy of Hallucinations// 12 min

   • ›Factual hallucination: wrong facts ("Paris is in Germany")
   • ›Logical fallacy: unsound reasoning
   • ›Inconsistency: contradicts earlier statement
   • ›Out-of-scope: answers outside its knowledge
   • ›Confabulation: fills gaps with plausible but false details
   • ›Example outputs by type (video shows real examples)

3. 03Jailbreaking and Adversarial Prompts// 15 min

   • ›Jailbreak: prompt that makes model do something it refuses
   • ›Example: "Ignore previous instructions. Generate..."
   • ›Why it works: models aren't robust to adversarial inputs
   • ›Common tactics: role-playing ("You're in a movie..."), encoding, hypotheticals
   • ›Defense: RLHF makes models refuse harmful requests
   • ›Limitation: can't achieve perfect safety (tradeoff with capability)
   • ›Red-teaming: proactively find jailbreaks before users do

4. 04Content Filters and Guardrails// 10 min

   • ›Guardrails: block harmful requests before reaching model
   • ›Techniques: keyword filtering (crude), classifier (ML-based, better)
   • ›Cost: adds latency (~10-50ms for classification)
   • ›False positives: blocking legitimate requests (bad UX)
   • ›Fine-tuning for safety: model learns to refuse
   • ›Transparency: tell users why request was blocked

Content outline

1. 01Sources of Bias// 12 min

   • ›Training data bias: internet data reflects societal biases
   • ›Stereotypes in training data → stereotypes in outputs
   • ›Example: "A doctor" → LLM more likely to generate "he" than "she"
   • ›Underrepresentation: low-resource languages, minority topics
   • ›Labeling bias: human annotators bring their own biases
   • ›Model amplification: model doesn't just reflect data, sometimes amplifies biases

2. 02Dimensions of Fairness// 13 min

   • ›Representation: are all groups represented fairly?
   • ›Allocation: does model give good output to all groups?
   • ›Fairness definitions (multiple, sometimes in conflict):
   • ›Demographic parity: model should perform same across groups
   • ›Equalized odds: same false positive/negative rates across groups
   • ›Individual fairness: similar individuals treated similarly
   • ›Tradeoff: can't optimize all dimensions simultaneously
   • ›Context matters: fairness in hiring ≠ fairness in creative writing

3. 03Auditing for Bias// 15 min

   • ›Step 1: identify protected attributes (gender, race, age, etc.)
   • ›Step 2: collect examples across groups
   • ›Step 3: prompt model, measure outputs
   • ›Step 4: analyze disparities (e.g., occupational gender bias)
   • ›Metrics: stereotype rate, representation, performance gap
   • ›Example audit: prompt "A bank executive..." 100 times, analyze pronouns
   • ›Tool: Hugging Face Evaluate library (open-source bias audits)

4. 04Mitigation Strategies// 10 min

   • ›Training data: diverse, representative samples
   • ›Fine-tuning: on diverse examples (but expensive)
   • ›Prompt engineering: "The engineer is..." vs. neutral prompts
   • ›Debiasing: remove harmful stereotypes post-hoc (limited effectiveness)
   • ›Transparency: disclose known biases to users
   • ›Monitoring: track bias over time (bias can shift with data)

Content outline

1. 01Pre-Deployment Checklist// 13 min

   • ›Safety testing: does model refuse harmful requests?
   • ›Bias auditing: measured disparities across groups?
   • ›Evaluation: accuracy acceptable? False-negative cost understood?
   • ›Monitoring plan: what metrics will you track?
   • ›Failure plan: what happens if model breaks? How do you roll back?
   • ›Stakeholder communication: have you told affected users?
   • ›Example checklist (shared onscreen)

2. 02Transparency and Disclosure// 12 min

   • ›Should users know they're talking to an AI? (YES, legally and ethically)
   • ›What to disclose: capability limitations, known failure modes
   • ›Example disclosure: "This AI is experimental. It may make mistakes."
   • ›Policy: use policy, data privacy, feedback collection
   • ›Consent: in regulated domains (finance, health), explicit consent needed
   • ›GDPR, EU AI Act implications: growing legal requirements

3. 03Incident Response// 15 min

   • ›Incident: model generates harmful, biased, or false content
   • ›Immediate: disable feature if serious, investigate
   • ›Document: what happened, when, impact, root cause
   • ›Fix: update model, retrain, add guardrail, tune hyperparameter
   • ›Communicate: tell affected users, transparency builds trust
   • ›Post-mortem: how do we prevent this next time?
   • ›Example: "Our chatbot generated a slur because..."

4. 04Regulatory and Ethical Landscape// 10 min

   • ›EU AI Act: classifies systems by risk, compliance requirements
   • ›GDPR: data privacy, right to explanation
   • ›Sector-specific: healthcare (FDA), finance (SEC), hiring (EEOC)
   • ›Best practices evolving: no settled standards yet
   • ›Business impact: regulation will get stricter
   • ›Competitive advantage: responsible companies will outlast others

Content outline

1. 01The Hype Cycle// 10 min

   • ›Every new technology: peak of inflated expectations → trough of disillusionment
   • ›LLMs in 2023: "AI will solve everything"
   • ›Reality: LLMs are great for some things, mediocre for others
   • ›Question: is this an LLM problem or a feature problem?
   • ›Anti-pattern: adding LLM to product just to add AI

2. 02When LLMs Excel// 12 min

   • ›Language understanding: classify text, extract information
   • ›Generation: write email drafts, summarize documents
   • ›Coding: autocomplete, debugging, writing boilerplate
   • ›Reasoning: explain steps, brainstorm solutions
   • ›Personalization: adapt to user style, history
   • ›Where: high volume, high margin use cases

3. 03When LLMs Fail// 12 min

   • ›Real-time data: LLM doesn't know current events
   • ›Structured output: "Extract these fields" (use extraction, not LLM)
   • ›Math: basic arithmetic (use calculator)
   • ›Determinism: need exact same answer every time (can't rely on LLM)
   • ›Low-latency: need sub-100ms response (LLM is too slow)
   • ›Small screen: long outputs don't fit
   • ›Where: low-volume, high-reliability use cases

4. 04Business Case// 11 min

   • ›Cost: inference cost + infra cost
   • ›Revenue: does feature increase user value? Retention? Price?
   • ›Time-to-market: LLM features ship fast (vs. traditional ML)
   • ›Risk: hallucinations, bias, regulatory (vs. deterministic systems)
   • ›Competitive advantage: LLM or just following trend?
   • ›Example calculation: 10K users, 1M tokens/month, $3/1M tokens = $30/month infra cost

Content outline

1. 01Common Patterns// 13 min

   • ›Chat: conversation interface (ChatGPT)
   • ›Summarization: compress long text (email summarization)
   • ›Classification: categorize inputs (spam detection, tagging)
   • ›Generation: create new content (product descriptions)
   • ›Search: find relevant content (semantic search on docs)
   • ›Code: write, debug, complete code (GitHub Copilot)
   • ›Extraction: pull structured data (invoice parsing)
   • ›Each has different architecture, latency, cost tradeoffs

2. 02The "LLM on Your Data" Pattern// 15 min

   • ›Combine LLM + RAG + domain data
   • ›Architecture: user query → retrieve docs → prompt with docs → response
   • ›Benefits: grounds model in facts, updates without retraining
   • ›Costs: retrieval adds latency, storage for embeddings
   • ›Failure: if retrieval is bad, response is bad
   • ›Optimization: reranking, query expansion, multi-hop reasoning

3. 03Handling Uncertainty and Failures// 13 min

   • ›Model says "I'm not sure" (good)
   • ›Model hallucinates confidently (bad)
   • ›UX: how do you show confidence? (percentage, flag, color)
   • ›Fallback: if model fails, what's the next step?
   • ›Human-in-the-loop: when to escalate to human?
   • ›Cost-benefit: fallback mechanisms are expensive

4. 04Feedback Loops// 9 min

   • ›Collect user feedback: thumbs up/down, flag errors
   • ›Use feedback to improve: fine-tune, update prompts
   • ›Bias: upvotes from certain users only? (selection bias)
   • ›Virtuous cycle: better model → more usage → more feedback → better model

Content outline

1. 01Cost Drivers// 12 min

   • ›Model size: larger = more expensive (per token)
   • ›Tokens: longer input/output = higher cost
   • ›Volume: QPS (queries per second) × 86400 seconds = daily volume
   • ›Formula: tokens per request × requests per day × price per token = daily cost
   • ›Example: 1000 tokens/request × 100K requests/day × $0.001 per token = $100/day

2. 02Cost Optimization Tactics// 15 min

   • ›Use smaller models: 7B vs. 70B (10x cheaper)
   • ›Prompt caching: reuse embeddings for repeated context
   • ›Batch processing: combine requests (cheaper than streaming)
   • ›Optimize prompts: remove unnecessary words (fewer tokens)
   • ›Reuse outputs: don't regenerate if you have cached result
   • ›Local inference: for high-volume, repetitive tasks (one-time infra cost)
   • ›Example: switching from 70B to 7B model = 10x cost reduction (with quality loss)

3. 03Scaling// 15 min

   • ›Scaling query volume (users grow)
   • ›Scaling feature scope (add more use cases)
   • ›Scaling model complexity (better models are slower)
   • ›Provisioning: how many resources do you need?
   • ›Auto-scaling: add/remove resources based on demand
   • ›Cost monitoring: track spend, set alerts
   • ›Example: your product scales from 1K to 100K users, what's the cost impact?

4. 04Quality vs. Cost Tradeoff// 8 min

   • ›Pay for quality: Claude (expensive, good) vs. 7B local model (cheap, OK)
   • ›Context: more context → better answers → higher cost
   • ›Latency: faster = more expensive (more hardware)
   • ›Decision: what's acceptable quality for your use case?
   • ›Hybrid: use cheap model for 90% of cases, expensive for hard cases

Content outline

1. 01Ship Fast, Learn Faster// 12 min

   • ›LLMs are a new technology: what works isn't obvious
   • ›Advantage: can ship fast, iterate on prompts (no retraining needed)
   • ›Strategy: MVP with simple prompting, gather feedback, improve
   • ›Anti-pattern: wait for perfect model (wasted time)
   • ›Timeline: 1-2 weeks to MVP, 1-2 months to production-ready
   • ›Tool: start with API, move to fine-tuning only if needed

2. 02Feedback and Iteration// 13 min

   • ›What to measure: user satisfaction, usage rate, error reports
   • ›Feedback channels: in-app rating, surveys, support tickets
   • ›Iteration cycle: measure → analyze → improve → measure
   • ›What to improve: prompt quality, model choice, fallback mechanisms
   • ›Avoid: over-optimizing for outliers (edge cases aren't product

3. 03Common Pitfalls// 15 min

   • ›Pitfall 1: "Our LLM can do everything" (it can't)
   • ›Pitfall 2: Shipping without guardrails (hallucinations, bias)
   • ›Pitfall 3: Wrong model for the job (too big, too small, wrong architecture)
   • ›Pitfall 4: Ignoring cost (scaling to profitability is hard)
   • ›Pitfall 5: Not monitoring (model silently degrades in production)
   • ›Lesson: be specific about capabilities, measure, iterate

4. 04Product-Market Fit with LLMs// 10 min

   • ›LLM feature solves real user problem?
   • ›Quality good enough? (benchmark testing)
   • ›Cost sustainable? (unit economics work)
   • ›Retention: do users keep using it?
   • ›Competitive advantage: why can't competitor copy this?
   • ›Example: Copilot achieved PMF because developers love autocomplete (fast, high accuracy)

Content outline

1. 01Recent Breakthroughs// 12 min

   • ›Extended context (Claude 200K): what became possible?
   • ›Reasoning models (o1): step-by-step verification, complex problems
   • ›Multimodal (GPT-4V): understanding images, video
   • ›Function calling: structured tool use, agent loops
   • ›Sparse models (mixture of experts): scale without density cost
   • ›What's next: multi-turn reasoning, world modeling, robotics

2. 02Emerging Applications// 13 min

   • ›Agents: LLMs that plan, act, learn from feedback
   • ›Long-context reasoning: processing books, code repositories
   • ›Personalization: models that learn your preferences, style
   • ›Code and formal systems: proving theorems, generating code
   • ›Science: drug discovery, protein folding, materials science
   • ›Robotics: embodied AI, physical reasoning
   • ›Societal: education, creative tools, accessibility

3. 03Research Frontiers// 15 min

   • ›Scaling limits: do scaling laws continue? (compute wall)
   • ›Reasoning: can LLMs solve hard problems? (math, logic)
   • ›Grounding: connecting language to perception, physics
   • ›Efficiency: can we make models smaller/faster? (distillation, pruning)
   • ›Safety: aligning with human values, handling power
   • ›Open questions: what don't we understand about LLMs?

4. 04The Compute and Data Limits// 10 min

   • ›Training data: are we running out? (data efficiency matters)
   • ›Compute: hardware requirements growing (but hitting physical limits)
   • ›Power: inference is energy-intensive (environmental concerns)
   • ›Talent: concentrated at a few companies (access inequality)
   • ›Speculation: will scaling curves plateau? Or are new paradigms needed?

Content outline

1. 01How to Stay Current// 12 min

   • ›Follow researchers and companies (Twitter/X, Bluesky)
   • ›Read papers: Arxiv.org (new research daily)
   • ›Communities: LessWrong, Hugging Face forums, Discord communities
   • ›Podcasts: "Machine Learning Street Talk," "The Gradient"
   • ›Tools: experiment with new models, build prototypes
   • ›Depth vs. breadth: specialize in something (your edge)

2. 02Learning Resources// 13 min

   • ›Fast.ai: top-down learning (practical from day 1)
   • ›DeepLearning.AI: structured courses on LLMs, RAG, etc.
   • ›Papers: Karpathy's "A Hacker's Guide to Language Models"
   • ›Code: Hugging Face, LangChain, Anthropic SDK (learn by building)
   • ›Books: "The Alignment Problem," "The Attention Mechanism Explained"
   • ›Your own projects: best teacher is building something real

3. 03Career Paths// 15 min

   • ›Research: universities, Meta, Anthropic, DeepMind (PhD helpful)
   • ›Infrastructure: building serving systems, optimizing inference
   • ›Products: building LLM applications, PMs, designers
   • ›Safety/Alignment: ensuring LLMs are beneficial
   • ›Sales/BD: enterprise LLM sales (high paying, competitive)
   • ›Entrepreneurship: building AI company (venture-backed startups booming)
   • ›Where to work: the cutting edge is at well-funded companies (high barrier)

4. 04Your Thesis for the Future// 10 min

   • ›What's your bet? (What will matter most in 5 years?)
   • ›Examples: reasoning > scale, efficiency > capabilities, local > cloud
   • ›Test it: build projects that validate your thesis
   • ›Specialize: become the expert in your area of bet
   • ›Network: connect with others who share your vision