Course Materials

Before class

• Sign in to Google ColabOptional

During class

• Sign in to or create a GitHub accountRequired
• Practice opening, running, copying, and saving a course notebook.Required

After class

• Code.org, “How Chatbots and Large Language Models Work” (about 7 min)Required
• Karpathy, “Intro to Large Language Models” (1 hr, listenable as a podcast)Optional

Before class

• 3Blue1Brown, “But what is a GPT? Visual intro to transformers” (27 min)Required
• Read Chapter 1, “An Introduction to Large Language Models,” and Chapter 2, “Tokens and Embeddings.”Optional

During class

• Reference notebook: Introduction to Language ModelsOptional
• Reference notebook: Tokens and Token EmbeddingsOptional

After class

• Hugging Face LLM Course, “Tokenizers”Optional
• Jay Alammar, “The Illustrated Word2vec”Optional
• TensorFlow Embedding Projector: explore nearest neighbors and semantic clusters in a pretrained embedding spaceOptional

Before class

• Kim, Soderman, and Sang, “Sentiment analysis of solar energy in U.S. Cities: a 10-year analysis using transformer-based deep learning”: read sections 1, “Introduction,” and 2, “Materials and approach,” onlyRequired
• Jay Alammar, “The Illustrated BERT”: read “Example: Sentence Classification” through “Model Outputs”Optional
• IBM Technology, “Text Classification: AI Techniques and Real-World Applications” (14 min)Required
• StatQuest, “Machine Learning Fundamentals: The Confusion Matrix” (7 min)Required
• 3Blue1Brown, “Attention in transformers, step-by-step”Optional
• Read Chapter 3, “Looking Inside Large Language Models,” and the Chapter 4 sections on task-specific and embedding-based text classification.Optional

During class

• Reference notebook: Looking Inside LLMsOptional
• Companion notebook: Text ClassificationOptional

After class

• Hugging Face LLM Course, “How Transformers Solve Tasks”: read “Text classification”Optional
• Hugging Face LLM Course, “Working with pipelines”: try the sentiment-classification example with your own sentencesOptional
• Hugging Face Transformers, “Text classification”: fine-tune DistilBERT on IMDb and run inferenceOptional
• Hugging Face SetFit, “Sentence Transformers Finetuning”: see how an embedding model and classification head work togetherOptional
• TensorFlow Playground: run a classification problem and observe how the learned decision boundary changesOptional

Before class

• Read Chapter 4, “Text Classification,” from “What If We Do Not Have Labeled Data?” through “Text Classification with Generative Models,” including the Flan-T5 and ChatGPT classification examples.Optional
• IBM Technology, “What is Zero-Shot Learning?” (9 min)Required
• CodeEmporium, “Few Shot Learning – explained!” (10 min)Required

During class

• Companion notebook: Text ClassificationRequired

After class

• Bhavesh Bhatt, “Zero Shot Classification with HuggingFace Pipeline” (hands-on code walkthrough)Optional
• Hugging Face, “Zero-Shot Text Classification”Optional

Before class

Required preparation: about 30 minutes, including the video and review task below. Reading times are estimates; the textbook is optional.

• scikit-learn, “classification_report”: read the Returns explanation of macro and weighted averages and the first worked example under Examples. Identify a class that performs poorly despite the overall accuracy (estimated 8 min). (opens in a new tab)Required
• scikit-learn, “Common pitfalls and recommended practices”: read “Data leakage” and “How to avoid data leakage,” stopping before the preprocessing example. (estimated 7 min). (opens in a new tab)Required
• Benjy Bodner, “Confusion Matrix, Precision, Recall & F1 — Explained Together!” (6:09). Focus on false alarms, missed cases, and how F1 combines precision and recall. (opens in a new tab)Required
• Review your Module 3–4 notes and any saved classification results.Required
• Chapter 4, “Text Classification”: revisit the classification report, confusion matrix, and metric discussion in “Using a Task-Specific Model” (estimated 10–15 min).Optional

During class

One-hour comparison workshop: 5 min recap, 10 min comparison checks, 20 min error inspection, 15 min recommendations, and 10 min discussion. Use existing classification outputs; no new model training is needed.

• Compare the simple baseline, task-specific classifier, and zero-shot classifier on the same labeled examples. Verify label mappings and inspect class support, macro-F1, per-class recall, and the confusion matrices. Use the existing Lab 1 materials as the working reference.Required
• Inspect a small selection of incorrect and disagreeing predictions. Record the text, reference label, model predictions, and an error category such as ambiguous label, missing context, truncation, or clear model error. Distinguish disagreement with the reference label from evidence that the reference itself is ambiguous.Required
• Produce one comparison table and a brief recommendation supported by specific errors and a limitation. Keep model and prompt choices on development data. Inspect final-test errors to describe limitations; evaluate any subsequent changes on fresh held-out data.Required

After class

Optional consolidation: about 5–10 minutes. Continue the existing Lab 1 assignment according to its published requirements and deadline; this module adds no separate submission.

• Self-check: explain why the highest aggregate score may hide an important weakness for one class. Identify one error worth addressing and the evidence needed to check whether your proposed change helps (estimated 5–10 min).Optional
• Lab 1: Text Classification and Error Analysis. Use the assignment page to check the required comparison, error analysis, and submission files.Optional

Before class

Required preparation: about 30 minutes, including pauses and the sketch; reading time is an estimate. The textbook remains optional.

• StatQuest with Josh Starmer, “K-means clustering”: watch 0:33–7:08 (6 min 35 sec; English auto-captions available). Why can changing the starting points or the number of clusters change the result? Use this as a contrast to HDBSCAN, which we use for text clustering.Required
• Maarten Grootendorst, BERTopic “The Algorithm”: read “Visual Overview” and “Detailed Overview,” steps 1–5; skip the code and optional step 6. Sketch the path from documents to embeddings, UMAP, HDBSCAN, and c-TF-IDF keywords. Which steps group documents, and which describe the groups? (Estimated 20–25 min, including the sketch.)Required
• Google for Developers, “What is clustering?”: read the introduction and “Clustering use cases,” stopping before “Imputation,” for a refresher on unlabeled data and uses of clustering (estimated 5–8 min).Optional
• Chapter 5, “Text Clustering and Topic Modeling”: focus on “A Common Pipeline for Text Clustering” through the BERTopic discussion immediately before “Adding a Special Lego Block.” Pay attention to outliers, inspecting documents, and information lost in 2D plots (estimated 35–45 min).Optional

During class

• Planned guided activity (classroom notebook forthcoming): embed and cluster a small collection of research abstracts, inspect documents from two clusters and the unassigned group, and propose topic names supported by text evidence. Compare a second random seed or clustering setting; record what changes in the groups, keywords, and 2D view.Required
• BERTopic “Visualize Topics”: explore the embedded topic map and the “Visualize Topic Similarity” heatmap; no installation is needed to view the examples. Compare two topics using their keywords and sizes. What additional document evidence would you need before merging or naming them? (Estimated 10–15 min in class.)Required
• Companion notebook: Text Clustering and Topic Modeling. Reference the clustering, topic inspection, and visualization examples; the full notebook and generative-labeling sections are not required to run for this class.Optional

After class

All follow-up resources below are optional. Choose the item that best addresses a question from class.

• Self-check: explain how a cluster differs from its topic label, why an unassigned document is not necessarily bad data, and why a well-separated 2D plot does not prove meaningful topics. Use one example from class (estimated 5–10 min; no submission).Optional
• BERTopic FAQ: read the entries on inconsistent results between runs, reducing topic outliers, and removing stop words. Explain why fixing a seed is different from establishing stability, and why forcing every document into a topic can be misleading (estimated 10–15 min).Optional
• Maarten Grootendorst, “Interactive Topic Modeling with BERTopic”: read the introduction, the opening of “2. Basic Usage” through the topic-keyword example, and “3. Visualization.” Connect the customer-feedback use case to topic interpretation. Read for concepts; use current BERTopic documentation for code (estimated 10–15 min).Optional
• BERTopic “Quick Start”: inspect the topic and document output examples; optionally try the basic workflow in Colab. Model/data downloads and extra runtime are needed if you run it. Skip “Fine-tune Topic Representations” for the core exercise (estimated 10–15 min to read; execution time varies).Optional
• Chapter 5 extension: “Adding a Special Lego Block” through “The Text Generation Lego Block.” Compare keyword reranking with generated topic names. What could a fluent label hide about the underlying documents? Reading only; no API account or paid model call is needed (estimated 20–30 min).Optional

Before class

Required preparation: about 20 minutes, including the sketch. Read the text-and-images versions of the three short Microsoft Learn units below; the videos and module assessment are not required.

• Microsoft Learn, “Understand retrieval-augmented generation”: read the unit and explain what retrieval supplies that the model’s learned parameters may not contain (estimated 5 min). (opens in a new tab)Required
• Microsoft Learn, “Prepare data for retrieval”: read the unit, focusing on chunk boundaries, embeddings, and source metadata. Name one problem caused by chunks that are too small (estimated 5 min). (opens in a new tab)Required
• Microsoft Learn, “Retrieve information and generate a response”: read the unit and sketch question → retrieval → context → answer. Mark the instructions, source passages, citation requirement, and response when evidence is missing (estimated 10 min, including the sketch). (opens in a new tab)Required
• Chapter 8, “Semantic Search and Retrieval-Augmented Generation”: read “Overview of Semantic Search and RAG,” the opening dense-retrieval example in “Semantic Search with Language Models,” and “From Search to RAG.” Focus on the pipeline rather than advanced retrieval variants (estimated 20–30 min).Optional

During class

One-hour guided session: 10 min pipeline and prompt walkthrough, 30 min retrieval and answer inspection, 15 min discussion, and 5 min buffer. Classroom notebook and prepared outputs are forthcoming.

• Planned guided activity: use a small document collection and a scaffolded notebook to inspect chunks and source IDs, embed a query, and retrieve passages. Complete a prompt with the question, delimited evidence, citation instructions, and an insufficient-evidence response. Compare an answerable question with one the collection cannot answer.Required
• Check one answer against its cited passages. Record the query, retrieved passage IDs, and one supported or unsupported claim. Saved model outputs will support the activity when generation or downloads are slow.Required
• Companion notebook: Semantic Search and Retrieval-Augmented Generation. Optional code reference for dense retrieval and grounded generation; running the full notebook is not required. (opens in a new tab)Optional

After class

Optional consolidation: about 5–10 minutes. Additional project or lab work follows the existing assignment requirements.

• Self-check: explain how an answer can include a citation yet remain unsupported. Identify whether a missing answer came from missing source material, retrieval, or generation (estimated 5–10 min).Optional
• Lab 2: Semantic Search, RAG, and Evaluation. Review the workflow and evidence the assignment will require.Optional

Before class

Required preparation: about 20 minutes. Build on the Module 7 workflow; advanced retrieval methods are optional.

• Microsoft Learn, “Evaluate a RAG solution”: read the text-and-images unit. Separate retrieval checks from generation checks, including citation quality and uncertainty; note the quality, latency, and cost tradeoff (estimated 10 min, including notes). (opens in a new tab)Required
• Review two question–passage–answer examples from Module 7. For each, ask whether useful evidence was retrieved and whether the answer used it correctly. Bring one proposed improvement, such as retrieval depth or a clearer prompt (estimated 10 min).Required
• Sentence Transformers, “Retrieve & Re-Rank”: read the pipeline diagram and explanations through “Re-Ranker: Cross-Encoder,” stopping before “Example Scripts.” Consider when reranking might help and what it cannot recover if the needed passage was never retrieved (estimated 10–15 min). (opens in a new tab)Optional
• Chapter 8: read “Retrieval Evaluation Metrics” for the idea of queries and relevance judgments. The mean average precision calculation is an optional extension (estimated 15–20 min).Optional

During class

One-hour evaluation workshop: 10 min criteria, 25 min comparison, 15 min error discussion, and 10 min Lab 2 connection. Evaluation notebook and comparison outputs are forthcoming.

• Planned guided activity: use a small practice set with answerable and unanswerable questions. Mark whether the required evidence appears in the retrieved passages; separately judge answer correctness, support, citations, and appropriate abstention. Use a simple evidence table and a human rubric.Required
• Compare the baseline with one change, such as retrieval depth. Keep the questions, model, and other settings fixed; inspect both improvements and regressions using prepared outputs or the scaffolded notebook. Treat this practice set as development data and reserve separate held-out questions for final evaluation.Required
• Summarize one failure, the proposed fix, and the evidence that would justify keeping it. Relate the comparison to the existing Lab 2 requirements; hybrid search and cross-encoder implementation are extensions.Required

After class

Optional consolidation: about 5–10 minutes. Complete Lab 2 on its existing schedule; no additional evaluation framework or API subscription is required by this module.

• Self-check: explain why a grounded answer can still be incomplete or incorrect, and why a correct-looking answer does not establish successful retrieval. Give one example of an unanswerable question your system should handle (estimated 5–10 min).Optional
• Sentence Transformers, “Retrieve & Re-Rank”: inspect the example scripts if retrieval ranking is a problem in your project. Reading is optional; running a full example requires model downloads and additional time (estimated 15–20 min to inspect). (opens in a new tab)Optional

Before class

Online module on October 26: complete this section and the asynchronous activities below in about 60 minutes total. The two readings account for approximately 20 minutes of that total; no live meeting is scheduled.

• Anthropic, “Effective context engineering for AI agents”: read “Context engineering vs. prompt engineering” and “The anatomy of effective context,” stopping before “Context retrieval and agentic search.” Focus on relevant evidence, clear instructions, and representative examples (estimated 10 min). (opens in a new tab)Required
• Stanford CRFM, “Language Models are Changing AI: The Need for Holistic Evaluation”: read “Holistic evaluation” and its three numbered subsections, stopping before “Findings.” Identify the roles of task coverage, multiple metrics, and standardized conditions. This is a 2022 methods reading, not a current model ranking (estimated 10 min). (opens in a new tab)Required
• Chapter 6, “Prompt Engineering”: revisit “The Basic Ingredients of a Prompt” and “Instruction-Based Prompting.” Use the components to analyze your RAG prompt; advanced reasoning and search techniques are optional (estimated 10–15 min).Optional

During class

Complete the following independently as the asynchronous class activity (estimated 40 minutes). Use your earlier RAG work and the linked readings; model calls and a new notebook are not needed.

• Context-design exercise (20 min): take a Module 7–8 question and its source passages. Draft a revised prompt that separates instructions, evidence, and the requested output; justify what you retained or removed. State how the system should respond when evidence is missing. Propose one comparison that would test whether the revision helps; do not claim an improvement without results.Required
• Benchmark exercise (15 min): use the HELM article’s standardization example to explain why scores on different datasets cannot establish a fair model ranking. Record the task/domain, metric, and prompting conditions you would require for a comparison relevant to your project. Name one important behavior the benchmark would still leave untested. (opens in a new tab)Required
• Synthesis (5 min): explain why changing context can change an application’s performance without changing model weights. Write one reason a high benchmark score alone would not settle your project’s model choice.Required

After class

All follow-up below is optional and outside the 60-minute module. No additional submission is introduced.

• Anthropic, “Effective context engineering for AI agents”: read “Context retrieval and agentic search” to preview how tools can bring new information into a model’s context (estimated 10 min). (opens in a new tab)Optional
• Self-check: identify one dataset difference or prompting difference that could make a published benchmark score a poor guide to your own task (estimated 5 min).Optional

Before class

Required preparation: about 20 minutes. Read for the workflow and tool interface; no agent framework installation is needed.

• Anthropic, “Building effective agents”: read the introduction, “What are agents?” and “When (and when not) to use agents,” stopping before “When and how to use frameworks.” Identify a task that benefits from a fixed workflow (estimated 8 min). (opens in a new tab)Required
• Hugging Face Agents Course, “What are Tools?”: read “What are AI Tools?,” “How do tools work?,” and “How do we give tools to an LLM?” through the calculator example, stopping before “Auto-formatting Tool sections.” Distinguish generating a tool request from executing it (estimated 12 min). (opens in a new tab)Required
• Chapter 7, “Advanced Text Generation Techniques and Tools”: read the opening of “Chains: Extending the Capabilities of LLMs” and “Agents: Creating a System of LLMs” through the ReAct explanation, stopping before “ReAct in LangChain.” Treat the framework code as historical reference (estimated 20–25 min).Optional

During class

One-hour trace workshop: 10 min overview, 25 min trace inspection, 15 min safeguards, and 10 min project discussion. Classroom traces and worksheet are forthcoming.

• Planned guided activity: compare a fixed retrieve-then-answer workflow with a model-directed retrieval-tool example. Trace the tool name and arguments, execution result, information returned to the model, and final answer. Inspect one successful trace and one failed trace; verify what actually happened rather than relying on the final message.Required
• Identify the first consequential failure and propose one targeted safeguard: argument validation, a tool-use limit, a stopping condition, or human approval for a consequential action. Include a case where retrieved text attempts to redirect the task. State how you would check the final outcome.Required
• Project discussion: justify whether your task needs a single model call, a fixed workflow, or model-directed tool use. Describe the extra evidence you would need before adding complexity.Required

After class

Optional consolidation: about 5–10 minutes; optional extension reading is additional. Building an autonomous agent is not a course requirement.

• Self-check: explain why a tool-call request is not proof that an action succeeded. Identify what observation or output would verify success in your example (estimated 5–10 min).Optional
• Anthropic, “Building effective agents”: choose one of the workflow patterns—prompt chaining, routing, or evaluator-optimizer—and relate it to a text-analysis task. Read for the pattern rather than a framework implementation (estimated 10–15 min). (opens in a new tab)Optional

Before class

Required preparation: about 20 minutes. This is an introductory adaptation session and project clinic; no training run, GPU setup, or paid API is required.

• Hugging Face Transformers, “Fine-tuning”: read the opening definition before the login instructions. Contrast continued training on task data with supplying information in a RAG prompt (estimated 5 min, including the comparison). (opens in a new tab)Required
• Hugging Face PEFT, “LoRA”: read the conceptual introduction, diagram, and advantages before “Usage”; skip the code and configuration reference. Identify what remains frozen and what is trained (estimated 10 min). (opens in a new tab)Required
• Bring your project’s current output, one evaluation result or failure example, and one question about the next improvement. Check that the project repository is accessible (estimated 5 min).Required
• Chapter 12, “Fine-Tuning Generation Models”: read “Supervised Fine-Tuning (SFT),” “Full Fine-Tuning,” and “Parameter-Efficient Fine-Tuning (PEFT)” through “Low-Rank Adaptation (LoRA).” Stop before QLoRA and the training walkthrough (estimated 20–25 min).Optional

During class

One-hour session: 15 min adaptation decisions, 15 min a worked comparison, and 30 min project clinic. The instructor’s comparison example is forthcoming.

• Instructor overview: connect an observed failure to a possible change in instructions, retrieved evidence, or model weights. Distinguish adapting an embedding model for retrieval from adapting a generator’s behavior. Introduce LoRA conceptually; training recipes and QLoRA configuration are optional extensions.Required
• Planned worked example: inspect one base-versus-adapted model comparison using prepared outputs. Check training-data suitability, separation of development and test examples, improvements and regressions, and whether any gain justifies the data and computing requirements. Training loss alone is not evidence of better task performance.Required
• Project clinic: show one result and one failure or limitation. Choose one feasible improvement or a clearer explanation of the existing results, and identify the evidence needed for the poster. Fine-tuning is optional for projects.Required

After class

Optional consolidation: about 5–10 minutes. Continue the existing project deliverables; this module adds no training assignment.

• Self-check: justify whether your project needs adaptation now. State what labeled or paired data you would need and how you would detect regressions on held-out examples (estimated 5–10 min).Optional
• Optional textbook extension: choose Chapter 10, “What Is Contrastive Learning?” for embedding adaptation, or Chapter 11, “SetFit: Efficient Fine-Tuning with Few Training Examples” for classification adaptation. Read the conceptual explanation; implementation is optional (estimated 15–20 min for one selection).Optional
• Hugging Face PEFT, “LoRA”: inspect “Usage” if adaptation is relevant to your project. Treat configuration and training as an optional extension requiring additional data, computing resources, and time (estimated 10–15 min to inspect). (opens in a new tab)Optional

Before class

• Review the project presentation requirementsRequired
• Confirm that your GitHub repository opens correctly, the README identifies the starting point, and the analysis can be demonstrated during questions.Required
• Submit the required presentation materials through the designated Canvas link before the presentation window closes.Required