Hands-on Labs

Weight 40% of final grade
Components 2 applied labs
Work Format Individual
Submission Canvas

The hands-on labs are individual computational assignments that ask you to turn course concepts into working text-analysis workflows. Lab 1 focuses on text classification and error analysis. Lab 2 focuses on semantic search, retrieval-augmented generation, and evaluation.

Availability: Each lab will be available three weeks before the corresponding class session.

Begin with the provided notebook and dataset or document collection. Complete the work in the notebook or in one or more Python .py files. Submit the required materials as one .zip archive through Canvas; assignments sent by email will not be graded.

Submission Files

The archive must contain:

  • README.md identifying the primary notebook or script, describing the included files, and providing instructions needed to rerun the analysis. Include required packages and either permitted data or clear links and access instructions.
  • The completed analysis as an .ipynb notebook or one or more .py files. Code must be clean, readable, and runnable. Notebook outputs must be visible; scripts must save the required results.
  • Any supporting files needed to reproduce or evaluate the work, including clearly named tables, figures, modules, configuration files, or permitted data.
Format choice: .ipynb and .py submissions are equally acceptable and are evaluated under the same rubric. No GitHub repository is required. Do not include restricted or sensitive data in the archive; provide access instructions in README.md instead.
AI-use statement: Follow the course AI policy. Unless a lab explicitly provides an exemption, identify any generative or agentic AI tool used, its purpose, the work completed before consulting it, the advice incorporated, and how the output was verified. If no such tool was used, state that clearly.
Late work: Late labs are accepted with a 25% deduction for each day submitted late. Grades and feedback will be returned through Canvas.

Component Weights

Component Percent of Final Grade
Lab 1: Text Classification and Error Analysis 20%
Lab 2: Semantic Search, RAG, and Evaluation 20%

Lab 1: Text Classification and Error Analysis

Due date: Tuesday, September 29, 2026

Assignment

Build a reproducible text-classification workflow using a provided labeled dataset. Examine how tokenization shapes model input, compare a simple baseline with task-specific and zero-shot classification approaches, and use quantitative evaluation and structured error analysis to determine when each approach is useful.

Preparation: You should be able to work with pandas data frames, run provided Hugging Face tokenizer and pipeline code, and interpret common classification metrics and confusion matrices.

Required Work

  1. Inspect the text and labels, document class balance and relevant data characteristics, and explain how tokenizer behavior and truncation affect model inputs.
  2. Create a reproducible development and evaluation design that keeps model, label, or prompt choices separate from final evaluation.
  3. Establish a simple baseline and apply task-specific and zero-shot classifiers, verifying label mappings and organizing predictions consistently.
  4. Evaluate all approaches on the same held-out examples using appropriate metrics, tables, and visualizations.
  5. Analyze incorrect, uncertain, and disagreeing predictions to identify patterns, ambiguity, and missing context.
  6. Recommend an approach for the task, support the decision with evidence, and explain important limitations and what the results do not establish.
Written responses: Complete the supplied README.md template with concise, evidence-based answers to the assigned questions. A separate report is not required.
Open Lab Submit

Lab 2: Semantic Search, RAG, and Evaluation

Due date: November 8, 2026

Assignment

Build a small retrieval-augmented generation workflow over a provided document collection. Test whether the system retrieves useful evidence and produces answers that remain grounded in that evidence, then improve one design choice and compare the results.

Required Work

  1. Prepare and chunk the document collection, explaining the choices that shape retrieval.
  2. Create embeddings and retrieve relevant passages for a set of test questions.
  3. Generate answers that use and cite the retrieved passages.
  4. Evaluate retrieval relevance, answer correctness, groundedness, and behavior on unanswerable questions.
  5. Change one design choice—such as chunk size, embedding model, retrieval depth, or prompt—and compare the results.
  6. Identify hallucinations, missing evidence, bias, or other failure modes and explain their practical implications.
Shared evaluation

How the hands-on labs are evaluated

The same four criteria apply to both labs and total 100% of each lab grade.

35%

Technical Completeness & Correctness

The required workflow is complete, the code runs correctly, and the methods are implemented appropriately for the assigned task.

30%

Evaluation & Supporting Evidence

Claims about model or system performance are supported with appropriate metrics, comparisons, examples, tables, or visualizations.

25%

Interpretation & Limitations

Results are interpreted accurately, and the discussion recognizes important errors, uncertainty, limitations, and practical implications.

10%

Reproducibility & Presentation

The README, primary notebook or Python source files, and any supporting files are organized, readable, successfully executed, and documented well enough for another reader to follow and reproduce the work.

View the detailed evaluation rubric Expand to compare performance descriptions for all four lab criteria.

The performance band indicates the percentage of points earned within a criterion; the Weight column determines that criterion’s contribution to the lab grade.

Criterion Exemplary
95–100%		 Proficient
80–94.9%		 Developing
40–79.9%		 Needs Improvement
Below 40%		 Weight
Technical Completeness & Correctness All required steps are complete and technically sound. The code runs successfully, and the implementation demonstrates a strong understanding of the assigned workflow. The required workflow is complete and appropriate overall. Minor code or implementation issues do not materially affect the main results. Major parts of the workflow are present, but some required steps are incomplete, unclear, or affected by technical errors that limit the analysis. The workflow is substantially incomplete or incorrect, or major errors prevent the submitted code from producing usable results. 35%
Evaluation & Supporting Evidence Evaluation choices fit the task, comparisons are informative, and conclusions are supported by convincing metrics, examples, tables, or visualizations. The evaluation is appropriate and supports the main conclusions, with only minor gaps in metrics, comparisons, examples, or presentation. Some evaluation evidence is provided, but it is incomplete, weakly connected to the task, or insufficient to support important conclusions. Evaluation evidence is missing, inappropriate, or misleading, leaving the quality of the results largely unexamined. 30%
Interpretation & Limitations Interpretations are precise and well supported. Errors, uncertainty, limitations, and practical implications are examined thoughtfully and accurately. Interpretations are accurate overall, and important limitations are identified, though some errors, uncertainty, or implications could be developed further. The main results are described, but interpretations are incomplete, overstated, or insufficiently attentive to errors and limitations. Interpretations are absent or unsupported, and major errors, limitations, or implications are not recognized. 25%
Reproducibility & Presentation The README provides complete rerun instructions, and the notebook or Python source files are clearly organized, successfully executed, and presented with clean code, relevant results, and all required supporting files. The README, primary analysis files, and any supporting files are readable and reproducible overall, with only minor issues in organization, documentation, output, data access, or presentation. The required files are present, but missing output, unclear code, incomplete README instructions, or inadequate data access makes reproduction difficult. The README or primary .ipynb/.py analysis file is missing, the submission is substantially disorganized or undocumented, or the work cannot be reproduced from the available files. 10%