Course Materials
# Module 1
Getting Started: Google Colab and GitHub
Learning objectives: Open, run, and save a notebook in Google Colab; navigate a GitHub repository and open a notebook from it; describe the course workflow for accessing, editing, and retaining copies of code.
Before class
- Sign in to Google ColabOptional
During class
- Practice opening, running, copying, and saving a course notebook.Required
# Module 2
Tokens and Embeddings
Learning objectives: Tokenize short texts and interpret token IDs; use sentence embeddings to compare semantic similarity; distinguish representation models from text-generation models; explain how next-token probabilities produce generated text.
Before class
- Read Chapter 1, “An Introduction to Large Language Models,” and Chapter 2, “Tokens and Embeddings.”Optional
During class
# Module 3
Text Classification I: Encoder Models
Learning objectives: Distinguish binary, multiclass, and multilabel classification problems; apply task-specific and embedding-based classifiers; evaluate predictions using suitable metrics and structured error analysis.
Before class
- Read Chapter 3, “Looking Inside Large Language Models,” and the Chapter 4 sections on task-specific and embedding-based text classification.Optional
During class
After class
# Module 4
Text Classification II: Zero-Shot and Few-Shot Models
Learning objectives: Compare NLI-based zero-shot classification with prompted generative classification; use label descriptions, few-shot examples, and constrained outputs; test how label wording and examples affect classification performance.
Before class
- Read Chapter 4, “Text Classification,” from “What If We Do Not Have Labeled Data?” through “Text Classification with Generative Models,” including the Flan-T5 and ChatGPT classification examples.Optional
During class
# Module 5
Text Classification III: Evaluation and Error Analysis
Learning objectives: Compare classifiers on the same held-out examples using aggregate and per-class results; categorize errors and disagreements with reference to the original text; recommend an approach while distinguishing development decisions from final evaluation.
Before class
Required preparation: about 30 minutes, including the video and review task below. Reading times are estimates; the textbook is optional.
- Review your Module 3–4 notes and any saved classification results.Required
- Chapter 4, “Text Classification”: revisit the classification report, confusion matrix, and metric discussion in “Using a Task-Specific Model” (estimated 10–15 min).Optional
During class
One-hour comparison workshop: 5 min recap, 10 min comparison checks, 20 min error inspection, 15 min recommendations, and 10 min discussion. Use existing classification outputs; no new model training is needed.
- Inspect a small selection of incorrect and disagreeing predictions. Record the text, reference label, model predictions, and an error category such as ambiguous label, missing context, truncation, or clear model error. Distinguish disagreement with the reference label from evidence that the reference itself is ambiguous.Required
- Produce one comparison table and a brief recommendation supported by specific errors and a limitation. Keep model and prompt choices on development data. Inspect final-test errors to describe limitations; evaluate any subsequent changes on fresh held-out data.Required
After class
Optional consolidation: about 5–10 minutes. Continue the existing Lab 1 assignment according to its published requirements and deadline; this module adds no separate submission.
- Self-check: explain why the highest aggregate score may hide an important weakness for one class. Identify one error worth addressing and the evidence needed to check whether your proposed change helps (estimated 5–10 min).Optional
# Module 6
Text Clustering and Topic Modeling
Learning objectives: Create document embeddings for an unlabeled text collection; cluster and visualize semantically related documents; interpret topic representations while identifying instability, labeling choices, and other limitations.
Before class
Required preparation: about 30 minutes, including pauses and the sketch; reading time is an estimate. The textbook remains optional.
- Chapter 5, “Text Clustering and Topic Modeling”: focus on “A Common Pipeline for Text Clustering” through the BERTopic discussion immediately before “Adding a Special Lego Block.” Pay attention to outliers, inspecting documents, and information lost in 2D plots (estimated 35–45 min).Optional
During class
- Planned guided activity (classroom notebook forthcoming): embed and cluster a small collection of research abstracts, inspect documents from two clusters and the unassigned group, and propose topic names supported by text evidence. Compare a second random seed or clustering setting; record what changes in the groups, keywords, and 2D view.Required
After class
All follow-up resources below are optional. Choose the item that best addresses a question from class.
- Self-check: explain how a cluster differs from its topic label, why an unassigned document is not necessarily bad data, and why a well-separated 2D plot does not prove meaningful topics. Use one example from class (estimated 5–10 min; no submission).Optional
- Chapter 5 extension: “Adding a Special Lego Block” through “The Text Generation Lego Block.” Compare keyword reranking with generated topic names. What could a fluent label hide about the underlying documents? Reading only; no API account or paid model call is needed (estimated 20–30 min).Optional
# Module 7
RAG I: Dense Retrieval and Grounded Generation
Learning objectives: Trace how a question moves through chunking, embeddings, retrieval, and answer generation; use a scaffolded dense-retrieval workflow; construct a grounded-answer prompt and check whether cited passages support the answer.
Before class
Required preparation: about 20 minutes, including the sketch. Read the text-and-images versions of the three short Microsoft Learn units below; the videos and module assessment are not required.
- Chapter 8, “Semantic Search and Retrieval-Augmented Generation”: read “Overview of Semantic Search and RAG,” the opening dense-retrieval example in “Semantic Search with Language Models,” and “From Search to RAG.” Focus on the pipeline rather than advanced retrieval variants (estimated 20–30 min).Optional
During class
One-hour guided session: 10 min pipeline and prompt walkthrough, 30 min retrieval and answer inspection, 15 min discussion, and 5 min buffer. Classroom notebook and prepared outputs are forthcoming.
- Planned guided activity: use a small document collection and a scaffolded notebook to inspect chunks and source IDs, embed a query, and retrieve passages. Complete a prompt with the question, delimited evidence, citation instructions, and an insufficient-evidence response. Compare an answerable question with one the collection cannot answer.Required
- Check one answer against its cited passages. Record the query, retrieved passage IDs, and one supported or unsupported claim. Saved model outputs will support the activity when generation or downloads are slow.Required
After class
Optional consolidation: about 5–10 minutes. Additional project or lab work follows the existing assignment requirements.
- Self-check: explain how an answer can include a citation yet remain unsupported. Identify whether a missing answer came from missing source material, retrieval, or generation (estimated 5–10 min).Optional
# Module 8
RAG II: Evaluation and Improvement
Learning objectives: Distinguish retrieval failures from generation failures; assess evidence relevance, answer correctness, groundedness, and unanswerable questions; compare one retrieval or prompt change using fixed examples and report its tradeoffs.
Before class
Required preparation: about 20 minutes. Build on the Module 7 workflow; advanced retrieval methods are optional.
- Review two question–passage–answer examples from Module 7. For each, ask whether useful evidence was retrieved and whether the answer used it correctly. Bring one proposed improvement, such as retrieval depth or a clearer prompt (estimated 10 min).Required
- Chapter 8: read “Retrieval Evaluation Metrics” for the idea of queries and relevance judgments. The mean average precision calculation is an optional extension (estimated 15–20 min).Optional
During class
One-hour evaluation workshop: 10 min criteria, 25 min comparison, 15 min error discussion, and 10 min Lab 2 connection. Evaluation notebook and comparison outputs are forthcoming.
- Planned guided activity: use a small practice set with answerable and unanswerable questions. Mark whether the required evidence appears in the retrieved passages; separately judge answer correctness, support, citations, and appropriate abstention. Use a simple evidence table and a human rubric.Required
- Compare the baseline with one change, such as retrieval depth. Keep the questions, model, and other settings fixed; inspect both improvements and regressions using prepared outputs or the scaffolded notebook. Treat this practice set as development data and reserve separate held-out questions for final evaluation.Required
After class
Optional consolidation: about 5–10 minutes. Complete Lab 2 on its existing schedule; no additional evaluation framework or API subscription is required by this module.
- Self-check: explain why a grounded answer can still be incomplete or incorrect, and why a correct-looking answer does not establish successful retrieval. Give one example of an unanswerable question your system should handle (estimated 5–10 min).Optional
# Module 9
Context Design and Model Benchmarks
Learning objectives: Select and organize instructions, examples, and source evidence for a task; interpret a model benchmark in terms of its tasks, metrics, and testing conditions; explain what additional application-specific evidence is needed before choosing a model.
Before class
Online module on October 26: complete this section and the asynchronous activities below in about 60 minutes total. The two readings account for approximately 20 minutes of that total; no live meeting is scheduled.
- Chapter 6, “Prompt Engineering”: revisit “The Basic Ingredients of a Prompt” and “Instruction-Based Prompting.” Use the components to analyze your RAG prompt; advanced reasoning and search techniques are optional (estimated 10–15 min).Optional
During class
Complete the following independently as the asynchronous class activity (estimated 40 minutes). Use your earlier RAG work and the linked readings; model calls and a new notebook are not needed.
- Context-design exercise (20 min): take a Module 7–8 question and its source passages. Draft a revised prompt that separates instructions, evidence, and the requested output; justify what you retained or removed. State how the system should respond when evidence is missing. Propose one comparison that would test whether the revision helps; do not claim an improvement without results.Required
- Synthesis (5 min): explain why changing context can change an application’s performance without changing model weights. Write one reason a high benchmark score alone would not settle your project’s model choice.Required
After class
All follow-up below is optional and outside the 60-minute module. No additional submission is introduced.
- Self-check: identify one dataset difference or prompting difference that could make a published benchmark score a poor guide to your own task (estimated 5 min).Optional
# Module 10
LLM Workflows and Tool Use
Learning objectives: Distinguish a fixed LLM workflow from model-directed tool use; trace a tool request, execution, observation, and final response; diagnose a failed task and propose a concrete safeguard or stopping condition.
Before class
Required preparation: about 20 minutes. Read for the workflow and tool interface; no agent framework installation is needed.
- Chapter 7, “Advanced Text Generation Techniques and Tools”: read the opening of “Chains: Extending the Capabilities of LLMs” and “Agents: Creating a System of LLMs” through the ReAct explanation, stopping before “ReAct in LangChain.” Treat the framework code as historical reference (estimated 20–25 min).Optional
During class
One-hour trace workshop: 10 min overview, 25 min trace inspection, 15 min safeguards, and 10 min project discussion. Classroom traces and worksheet are forthcoming.
- Planned guided activity: compare a fixed retrieve-then-answer workflow with a model-directed retrieval-tool example. Trace the tool name and arguments, execution result, information returned to the model, and final answer. Inspect one successful trace and one failed trace; verify what actually happened rather than relying on the final message.Required
- Identify the first consequential failure and propose one targeted safeguard: argument validation, a tool-use limit, a stopping condition, or human approval for a consequential action. Include a case where retrieved text attempts to redirect the task. State how you would check the final outcome.Required
- Project discussion: justify whether your task needs a single model call, a fixed workflow, or model-directed tool use. Describe the extra evidence you would need before adding complexity.Required
After class
Optional consolidation: about 5–10 minutes; optional extension reading is additional. Building an autonomous agent is not a course requirement.
- Self-check: explain why a tool-call request is not proof that an action succeeded. Identify what observation or output would verify success in your example (estimated 5–10 min).Optional
# Module 11
Model Adaptation
Learning objectives: Distinguish changes to prompts and retrieved evidence from updates to model weights; explain the purpose of parameter-efficient fine-tuning; assess a before-and-after adaptation comparison and decide whether adaptation is justified for a project.
Before class
Required preparation: about 20 minutes. This is an introductory adaptation session and project clinic; no training run, GPU setup, or paid API is required.
- Bring your project’s current output, one evaluation result or failure example, and one question about the next improvement. Check that the project repository is accessible (estimated 5 min).Required
- Chapter 12, “Fine-Tuning Generation Models”: read “Supervised Fine-Tuning (SFT),” “Full Fine-Tuning,” and “Parameter-Efficient Fine-Tuning (PEFT)” through “Low-Rank Adaptation (LoRA).” Stop before QLoRA and the training walkthrough (estimated 20–25 min).Optional
During class
One-hour session: 15 min adaptation decisions, 15 min a worked comparison, and 30 min project clinic. The instructor’s comparison example is forthcoming.
- Instructor overview: connect an observed failure to a possible change in instructions, retrieved evidence, or model weights. Distinguish adapting an embedding model for retrieval from adapting a generator’s behavior. Introduce LoRA conceptually; training recipes and QLoRA configuration are optional extensions.Required
- Planned worked example: inspect one base-versus-adapted model comparison using prepared outputs. Check training-data suitability, separation of development and test examples, improvements and regressions, and whether any gain justifies the data and computing requirements. Training loss alone is not evidence of better task performance.Required
After class
Optional consolidation: about 5–10 minutes. Continue the existing project deliverables; this module adds no training assignment.
- Self-check: justify whether your project needs adaptation now. State what labeled or paired data you would need and how you would detect regressions on held-out examples (estimated 5–10 min).Optional
- Optional textbook extension: choose Chapter 10, “What Is Contrastive Learning?” for embedding adaptation, or Chapter 11, “SetFit: Efficient Fine-Tuning with Few Training Examples” for classification adaptation. Read the conceptual explanation; implementation is optional (estimated 15–20 min for one selection).Optional
# Module 12
Project Presentations
Learning objectives: Present the project question, data, workflow, evaluation, and results as a coherent analytical story; use clear visuals or examples to explain model behavior; discuss limitations and responsible use and respond thoughtfully to questions and peer feedback.
Before class
- Confirm that your GitHub repository opens correctly, the README identifies the starting point, and the analysis can be demonstrated during questions.Required
- Submit the required presentation materials through the designated Canvas link before the presentation window closes.Required