Lesson 6: Retrieval - Exploration

45 minutes

Overview

This Explore lesson introduces the concept of retrieval and guides students through an investigation on how it impacts a chatbot's behavior. Students start by predicting and discussing how retrieval can impact a chatbot’s behavior, and then investigate and modify existing chatbots to confirm or expand their initial guesses. Through guided activities, they will learn how adding specific information to a chatbot’s knowledge base can significantly improve the accuracy and relevance of its responses. Students will practice adding retrieval information to elicit specific responses from the AI and engage in activities that incorporate personal experiences to see the effect of their retrieval on how the chatbot functions.

Agenda

Warm Up (5 minutes)

Activity (35 minutes)
Student Levels
Teaching Guide

Wrap Up (5 minutes)

Objectives

Students will be able to:

Explain how retrieval allows chatbots to access updated information
Modify chatbot responses by adding and removing retrieval statements

Preparation

Run through the slide deck and videos ahead of time
Review the Level Guide Exemplar
(Optional) Print out the slide deck with speaker notes as a resource during class. Click here for steps on how to do this.

Teaching Guide

Warm Up (5 minutes)

Slide	Lesson Guide
	Discuss: Imagine you had a chatbot with the system prompt “Your goal is to give updates on all of the latest entertainment news that happened last week” Do you think this chatbot will work like we’d expect? Why or why not? Discussion Goal: Based on informal experiences with the chatbot so far, some students may correctly identify that chatbots don’t have access to the most recent information. Some more powerful chatbots like Google Gemini or Microsoft Copilot can access the internet if directed, but the chatbot used in this class doesn’t have that capability. As a result, the chatbot wouldn’t be able to solve this problem very well.
	Say: Since all chatbots are pre-trained on data before being released, there’s a limit to how much these chatbots know - the data has to stop sometime. So how can we make sure chatbots are kept up to date? How can we update the things a chatbot knows and can use in its responses? Question of the Day: How can we update the information in a chatbot?

Activity (35 minutes)

Student Levels

Getting Familiar: These are the levels students will complete throughout the lesson. We recommend briefly exploring the levels yourself before reading the rest of the lesson plan. Some levels are meant to be completed independently by students, but others are meant to motivate class discussion or lead to moments of direct instruction in the slides. Very rarely should students be working through levels completely self-paced.

Teacher Exemplars: Verified teachers can access exemplar responses and "what success looks like" tips for each level in the Links section of the lesson plan. If the resource doesn't appear for you, you may need to become a verified teacher - learn more about this process here.

Teaching with Student Responses: Certain level types are designed to be leveraged with our Student Summary feature, specifically: Predict & Discuss and Mid-Lesson Check-In levels. You can read more about how to leverage student voices during these levels in the Teaching Guide - Incorporating Student Responses in Lessons

Explore

2-3	Predict
	2 3

4-6	Investigate
	4 5 6

Modify

Check-In

Make

Teaching Guide

Teaching Tip

Teaching Tip: Predict & Discuss Levels - Levels 2 and 3 in this lesson are predict and discuss levels, which are designed to facilitate a discussion led by the teacher. After students lock in their predictions, the teacher leads a discussion asking for students to share their observations and hypotheses. There are no right-or-wrong answers - they’re just predictions. The teacher should avoid validating any predictions - instead, as the lesson progresses, students will get the chance to investigate their hypotheses and see how well their predictions matched with reality. This means the most important part of these discussions are students feeling comfortable sharing their intuitions and forming opinion, and less about “right or wrong” answers. For more information about these levels, please read the Teaching Guide to Incorporating Student Responses in Lessons.

Slide	Lesson Guide
	Navigate: Go to Lesson 6 Level 1. This Explore level has a preset chatbot about Tucson that students can interact with. Do This: Ask the chatbot questions about Tucson You can demonstrate this level as a full-class, or students can explore individually. This is meant to be a quick level that sets the initial behavior of a chatbot that we’ll start to modify as the lesson continues. Once students have seen some baseline responses, continue to the next level.
	Navigate: Go to Lesson 6 Level 2. In this Predict & Discuss level, students see a screenshot of a series of statements called Retreival and are asked what they notice and what they wonder about the prompt. Say: Here is another new feature for our chatbots: retrieval. We’ll learn more about this as the lesson progresses, but for right now, I just want you to look at this and write down one thing you Notice and one thing you Wonder. There is no right or wrong answer here - it’s a deliberately open-ended question! You could say “I notice there are 5 things” and that would be a totally fine answer, although I hope folks notice some things that are more interesting. Say: Once you type in your answer, press the Run button to lock it in. This will also change your screen to green. Once most people have their answers locked in, we’ll discuss as a class. Circulate: Make sure students are answering the questions. If you plan to share student responses using one of the teaching strategies in the Guide to Incorporating Student Responses, then you should be monitoring responses via the Student Summary feature on this level.
	Discuss: What do you notice? What do you wonder? Ask a few students to share what they wrote in their level. Discussion Goal: This is a purposefully low-floor, open-ended prompt designed to activate student thinking and pay explicit attention to this new feature. Some students may not really know how to respond and will say very obvious things - this is fine and can still be celebrated. Some students may use this as an opportunity to make some interesting observations, such as: - Each item is like a personal opinion or recommendation - Some of the information is similar to the example prompts from the first level As students share responses, encourage students to respond non-verbally if they had a similar thought (for example, by snapping their fingers or shaking their hands). After hearing from a few students, continue to the next slide.
	Navigate: Go to Lesson 6 Level 3. In this Predict & Discuss level, students are asked to make a prediction of what will happen. Say: This is the same image from before, but this time I’m asking you to make a formal prediction - what do you think will happen when we interact with this chatbot? What’s a question you might try asking it, and what do you think the response will be? Circulate: This question is all about students planting their feet and making a prediction that they can test in the next few levels. Make sure students come up with at least one question that they can test. If you plan to share student responses using one of the teaching strategies in the Guide to Incorporating Student Responses, then you should be monitoring responses via the Student Summary feature on this level.
	Discuss: Try to predict how this chatbot will behave when you ask it questions. What's a question you might ask, and what answer would you expect? Discussion Goal: There is still no right-or-wrong answer here, since students are just making predictions based on their intuition. What’s more important is that students lock-in an expectation that they can test in the next few levels. When listening to responses, try not to give anything away or validate responses as “on the right track” - letting students discover if their prediction is right-or-wrong in the next levels is an engagement strategy that hooks students into the lesson. As students share responses, encourage students to respond non-verbally if they had a similar thought (for example, by snapping their fingers or shaking their hands). After hearing from a few students, continue to the next slide.
	Navigate: Go to Lesson 6 Level 4. In this Investigate level, students can now interact with the chatbot they’ve seen screenshots of in the last few levels Do This: 1) Investigate: Test the chatbot with the same prompts as before 2) Discuss: What changed from before? What worked like you expected? What surprised you? Students can complete this level with a neighbor and have the discussion as a small group rather than a full class. Circulate: Listen for students comparing the results of their investigation to their predictions, or the predictions of classmates. If a student seems stuck, have them try the question they wrote in for Level 3 and see how it responds. Students can also try the example prompts provided in the instructions.
	Say: So far, we’ve seen two settings we can adjust on a language model to change its behavior: temperature and system prompt Click the animation Say: Today we’ll be exploring Retrieval, which allows us to add new information to a chatbot. You can think of it like a mail slot in the door of the language model where we can drop in new information for the model to consider. Click the animation Say: Behind the scenes, retrieval is connected to the storage layer of a language model. When we add retrieval information, we’re affecting the underlying structure a model uses to represent words, called embeddings. This lets the model see some words as “closer” to each other than it previously thought in it’s training data.

Teaching Tip

Teaching Tip: Connecting Back to a Previous Lesson - If you taught the Foundations of Generative AI unit, a few lessons emphasized how Embeddings are numerical representations and similar words are “close” to each other in their numerical representation. This can help us understand how the sentence “The food in Tucson is awesome - some great restaurants in Tucson are Poco and Moms, Kingfisher, and Franks.” impacts our model.

In the initial training data, a word like “Franks” was probably “close” to an entirely different set of words. For example: it could be “close” to “hot dog” or “person” or “blunt” because of the phrase “Let me be frank with you…”
This sentence creates a new association between “Franks” and the phrase “great restaurant”, along with other great restaurants like Kingfisher and Poco and Moms.
This helps update the embedding for “Franks” so it is now “closer” to other great restaurants, and more likely to be listed in a chatbot response.

If students need a refresher on embeddings, consider showing the Generative AI: Storage & Embeddings video

Navigate: Go to Lesson 6 Level 5. Students will complete levels 5 and 6, which help students use retrieval in the chatbot.

Do This: Complete levels 5 and 6
- Level 5 shows you how to add and remove retrieval from the chatbot
- Level 6 has you add and remove a piece of information from the Tucson chatbot.

Circulate: Check-in with students and make sure they can add and remove retrieval statements from the chatbot. Treat level 6 as a fluency test for students, making sure they understand how retrieval works in the chatbot and are making sure to press the Update button when appropriate.

Teaching Tip

Teaching Tip: Exemplar Prompts - Verified teachers can access a resource on the lesson plan page that provides exemplar prompts and retrieval that address each of the levels in this lesson. Consider reviewing these prompts or having them available for yourself during the lesson while circulating and checking with students. This resource also includes "teaching tips" for certain individual levels, which offers guidance and tips if a student is stuck on a particular level. This resource is only visible to verified teachers - click here to learn more about becoming a verified teacher.

Navigate: Go to Lesson 6 Level 7. Students will complete levels 5 and 6, which help students use retrieval in the chatbot.

Do This: Update the chatbot so it’s about where you live (or someplace you’ve lived before).

Circulate: Check-in with students as they complete this level. Students need to both update the system prompt and replace the retrieval, which may take a while. Encourage students to copy-and-paste retrieval information when they can.

Navigate: Go to Lesson 6 Level 8. This is a free response level where students answer a question checking-in on their understanding of retrieval.

Do This: Based on what you observed in the last few levels, answer the question on the level.

Assessment Opportunity

Assessment Opportunity: Formative Assessment - This is an open-ended question designed to let students share how they’re conceptualizing retrieval at this moment, in contrast to a system prompt. Any response that connects the idea of supplying specific, new information to a chatbot is an appropriate response. It may be worth checking in with students who can’t articulate the difference between a system prompt and retrieval or struggle to describe how retrieval can help solve a problem. Check the Generative AI - Teaching Guide for Student Responses for more information about how to use Code.org’s teacher dashboard to view this data and potentially differentiate your instruction with students.

Navigate: Go to Lesson 6 Level 9. In this Make level, students can choose different personal scenarios to focus on.

Do This: 1) Choose at least 2 levels to complete
2) Follow the instructions on each level

Circulate: Monitor students as they attempt these levels. Students will construct their system prompts and retrieval from scratch in these levels, which can take time. Pay attention to how students test their chatbots, ensuring that they are making sure their retrieval statements are appearing as expected in the chatbot responses.

Wrap Up (5 minutes)

Slide	Lesson Guide
	Discuss: Reflect on these three prompts related to today: 1) What’s something you already knew about chatbots before class started? 2) What’s something you learned in class today about how chatbots? 3) What’s something you still wonder about chatbots based on today’s class? These could be discussed in small groups, but can also serve as an exit ticket or personal reflection that is checked after class.
	Question of the Day: How can we update the information in a chatbot?