Areas of Celebration and Exploration

After a brief interlude, it’s time to get back to the blog series I started recently about analyzing assessments.

• In the first post, I shared the importance of digging into the questions, not just the standards they’re correlated to.
• In the second post, I talked about how understanding how a test is designed can help us better understand the results we get.
• In the third post, I shared how I learned to organize assessment data by item difficulty and the implications for supporting our students.
• In this post, I’d like to talk about another way to look at assessment data to uncover areas of celebration and areas of exploration.

Let’s get started!

---

In my previous post I shared the order of questions based on item difficulty for the 2018 5th grade STAAR for the entire state of Texas. Here it is again:

According to this ordering, question 9 was the most difficult item on the test, followed by question 18, question 8, and so on down to question 10 as the least difficult item (tied with questions 2 and 4).

Here’s my question: What is the likelihood that any given campus across the state would have the exact same order if they analyzed the item difficulty just for their students?

Hopefully you’re like me and you’re thinking, “Not very likely.” Let’s check to see. Here’s the item difficulty of the state of Texas compared to the item difficulty at just one campus with about 80 students. What do you notice? What do you wonder?

Some of my noticings:

• Questions 8, 9, 18, and 21 were some of the most difficult items for both the state and for this particular campus.
• Question 5 was not particular difficulty for the state of Texas as a whole (it’s about midway down the list), but it was surprisingly difficult for this particular campus.
• Question 22 was one of the most difficult items for the state of Texas as a whole, but it was not particularly difficult for this campus (it’s almost halfway down the list).
• Questions 1, 2, 10, 25, and 36 were some of the least difficult items for both the state and for this particular campus.
• Question 4 was tied with questions 2 and 10 for being the least difficult item for the state, but for this particular campus it didn’t crack the top 5 list of least difficult items.
• There were more questions tied for being the most difficult items for the state and more questions tied for being the least difficult items for this particular campus.

My takeaway?

What is difficult for the state as a whole might not be difficult for the students at a particular school. Likewise, what is not very difficult for the state as a whole might have been more difficult than expected for the students at a particular school.

But is there an easier way to identify these differences than looking at an item on one list and then hunting it down on the second list? There is!

This image shows the item difficult rank for each question for Texas and for the campus. The final column shows the difference between these rankings.

 

Just in case you’re having trouble making sense of it, let’s just look at question 9.

As you can see, this was the number 1 most difficult item for the state of Texas, but it was number 3 on the same list for this campus. As a result, the rank difference is 2 because this question was 2 questions less difficult for the campus. However that’s a pretty small difference, which I interpret to mean that this question was generally about as difficult for this campus as it was for the state as a whole. What I’m curious about and interested in finding are the notable differences.

Let’s look at another example, question 5.

This is interesting! This question was number 18 in the item difficulty for Texas, where 1 is the most difficult and 36 is the least difficult. However, this same question was number 5 in the list of questions for the campus. The rank difference is -13 because this questions was 13 questions more difficult for the campus. That’s a huge difference! I call questions like this areas of exploration. These questions are worth exploring because they buck the trend. If instruction at the campus were like the rest of Texas, this question should have been just as difficult for the campus than for the rest of the state…but it wasn’t. That’s a big red flag that I want to start digging to uncover why this question was so much more difficult. There are lots of reasons this could be the case, such as:

• It includes a model the teachers never introduced their students to.
• Teacher(s) at the campus didn’t know how to teach this particular concept well.
• The question included terminology the students hadn’t been exposed to.
• Teacher(s) at the campus skipped this content for one reason or another, or they quickly glossed over it.

In case you’re curious, here’s question 5 so you can see for yourself. Since you weren’t at the school that got this data, your guesses are even more hypothetical than there’s, but it is interesting to wonder.

Let me be clear. Exploring this question isn’t about placing blame. It’s about uncovering, learning what can be learned, and making a plan for future instruction so students at this campus hopefully don’t find questions like this so difficult in the future.

Let’s look at one more question from the rank order list, question 22.

This is sort of the reverse of the previous question. Question 7 was much more difficult for the state as a whole than it was for this campus. So much so that it was 7 questions less difficult for this campus than it was for the state. Whereas question 5 is an area of exploration, I consider question 7 an area of celebration! Something going on at that campus made it so that this particular question was a lot less difficult for the students there.

• Maybe the teachers taught that unit really well and student understanding was solid.
• Maybe the students had encountered some problems very similar to question 7.
• Maybe students were very familiar with the context of the problem.
• Maybe the teachers were especially comfortable with the content from this question.

Again, in case you’re curious, here’s question 22 to get you wondering.

 

In Texas this is called a griddable question. Rather than being multiple choice, students have to grid their answer like this on their answer sheet:

Griddable items are usually some of the most difficult items on STAAR because of their demand for accuracy. That makes it even more interesting that this item was less difficult at this particular campus.

We can never know exactly why a question was significantly more or less difficult at a particular campus, but analyzing and comparing the rank orders of item difficulty does bring to the surface unexpected, and sometimes tantalizing, differences that are well worth exploring and celebrating.

Just this week I met with teams at a campus in my district to go over their own campus rank order data compared to our district data. They very quickly generated thoughtful hypotheses about why certain questions were more difficult and others were less so based on their memories of last year’s instruction. In meeting with their 5th grade team, for example, we were surprised to find that many of the questions that were much more difficult for their students involved incorrect answers that were most likely caused by calculation errors, especially if decimals were involved. That was very eye opening and got us brainstorming ideas of what we can work on together this year.

---

This post wraps up my series on analyzing assessment data. I might follow up with some posts specifically about the 2018 STAAR for grades 3-5 to share my analysis of questions from those assessments. At this point, however, I’ve shared the big lessons I’ve learned about how to look at assessments in new ways, particularly with regards to test design and item difficulty.

Before I go, I owe a big thank you to Dr. David Osman, Director of Research and Evaluation at Round Rock ISD, for his help and support with this work. And I also want to thank you for reading. I hope you’ve come away with some new ideas you can try in your own work!

Misplaced Priorities

Every spring thousands upon thousands of Texas students take the State of Texas Assessments of Academic Readiness (STAAR for short). It’s a one-day snapshot meant to evaluate a year of learning within a subject area. Even though many disagree with one-time events as assessments of learning, the fact of the matter is that they are a reality for us and our students. Because these assessments carry so much weight, we pore over the data they generate, often looking for standards where our students performed poorly so we can identify what to focus on in our instruction and intervention.

But what if I told you this well-intentioned practice may be sending us in unproductive directions? Rather than focusing on what our students really need, we may be spending time on topics and/or skills that are not the priority.

Let me illustrate what I mean with a story. I was working with a 4th grade team after a district benchmark we call STAAR Ready. Every spring in my district we give our students a released STAAR to gauge readiness for the actual STAAR coming up in May. Afterward, teams analyze the data to determine which topics to revisit and which students to put into intervention groups.

As I met with this 4th grade team, they showed me a list of the low-performing TEKS (Side note: this is what we call our standards in Texas – the Texas Essential Knowledge and Skills, TEKS for short) they had identified after analyzing the STAAR Ready data. One of the TEKS jumped out at me immediately because I was familiar with the test:

TEKS 4.4A add and subtract whole numbers and decimals to the hundredths place using the standard algorithm;

I asked them to tell me more, and the team told me they had identified students who performed poorly on the questions correlated to this standard. They created an intervention group with these students to work on adding and subtracting whole numbers and decimals to make sure they could do these computations accurately.

I followed up with a question, “Have you looked at the actual questions correlated to these TEKS?” Because they were looking at so much data and so many standards, they hadn’t gotten back into the test. Instead they’d just been identifying high-priority TEKS based on student performance on the questions.

I pulled up the test and showed them this question that had immediately come to mind when they told me they were making a group focused on TEKS 4.4A:

Source: Texas Education Agency, STAAR Math, Grade 4, Item 34

Take a moment and analyze the question.

• Can you see how it involves adding and/or subtracting with whole numbers and/or decimals?
• But what other skills are involved in answering this question correctly?
• What features of the problem might have made it more difficult for the students to answer correctly?

As it turns out, this was an incredibly difficult problem for students! When it was given to students on the actual STAAR in spring 2016, only 43% of students across the state of Texas were able to answer correctly. That means 57% of Texas 4th graders, or roughly 209,390 students, couldn’t find the total cost of three items in a shopping basket. That’s…concerning.

In my own school district, we used the 2016 released STAAR as our STAAR Ready in spring 2017. This allowed me to collect data Texas doesn’t make available to everyone. When we gave the test in spring 2017, the problem was nearly as difficult for our students. About 48% of students in my district answered it correctly. I was also able to determine this was the 6th most difficult item on the entire test of 48 questions!

What’s going on? A lot actually, for such a short question. For starters, key information is spread across two sentences. The first sentence of the problem indicates the quantities of items purchased – 1 hat and 2 skirts. The second sentence indicates their prices. This is subtle, but separating that information across two sentences upped the level of difficulty significantly for 9 and 10 year olds. Students who are not reading closely can quickly jump to the conclusion that they only need to add the two prices shown without realizing that one of those prices needs to be used twice.

The second feature of this problem that ups the difficulty is the fact that it is an open response question, not multiple choice. On this kind of question, a student’s answer has to be absolutely 100% accurate. If they’re off by even 1 penny, the answer is marked wrong. No pressure, kids!

I was curious which feature made the problem more difficult for the students in my district, so I dove into the data. One thing I had available that Texas doesn’t release is the actual answers every student submitted for this problem. I was able to analyze roughly 3,600 answers to see what students were doing. Here’s what I found out.

While only 48% of students got this question correct, there was a chunk of students whose answers were in the ballpark. These are kids who likely made a small calculation error. Unfortunately, if I calculate the percent of students who got it right or reasonably close, that only brings it up to 51% of our 4th graders. That’s not terribly impressive.

So what was everyone else doing? Here’s where it gets interesting. I predicted that these students only found the cost of 1 hat and 1 skirt, and it turns out that’s exactly what 33% of students in my district did. Nearly 1,200 students failed to comprehend that the total cost is composed of a hat, a skirt, and another skirt.

Going back to the team I was working with, I asked, “So now that we’ve analyzed this question, do you think the issue is that your students are struggling with adding and subtracting whole numbers and decimals?” We talked about it and they agreed that the bigger issue is how their students read and comprehend word problems.

Looking just at the standards is a very limiting view of analyzing data. There are often many different ways to assess a standard, and if we don’t take the time to look at the exact questions our students interact with, we might be missing critical information. Had this team done an intervention on pure addition and subtraction of whole numbers and decimals their kids would have gotten better at those skills for sure. But is that really what they needed?

Over the past year, I’ve been analyzing assessment data differently than in the past. In follow up posts I’d like to share some of that with you. In the meantime, please dive into your assessments and analyze those questions, not just the standards. You’ll hopefully come away with a truer picture of what’s challenging your students so that you can more accurately target with what and how to support them.

 

The Slow Reveal

This year my colleague Regina Payne and I tried something new as we visited classrooms across our district – numberless graphs. Similar to a numberless word problem, you present a graph with no numbers and proceed to have a rich mathematical discussion as you slowly reveal more and more information on the graph. Early in the school year, I shared a Halloween-themed numberless graph, and I also wrote a blog post about it.

We briefly touched on this work in our session at the 2017 NCTM annual conference, and it’s been exciting to see my #MTBoS colleagues taking the idea and running with it in their schools! In case you don’t follow them – which will hopefully change after reading this post! – I want to share their work so you don’t miss out on all the great stuff they’re doing.

Kassia Wedekind

Kassia has written two wonderful blog posts about how she took our ideas and tinkered with them to create a data routine called Notice and Wonder Graphs. I like this name because it’s more inclusive than numberless graphs. When it comes to graphs, you might hide the numbers, but you could just as easily hide other parts of the graph first. It all depends on your goals and how you want the conversation to unfold. In Kassia’s first post, she shares this graph with students. Notice it has numbers, and little else.

Curious what it’s about? Then check out Kassia’s post. I’m betting you’ll be quite surprised when you reach the final reveal.

I will share this snippet from her post:

I love this routine for many of the reasons that I love Brian’s numberless word problems–it slows the thinking down and focuses on sense-making rather than answer-getting.

But I also love it because it brings out the storytelling aspect of data. So often in school (especially elementary school!) we analyze fake data. Or, perhaps worse, we create the same “What is your favorite ice cream flavor?” graph year after year after year for no apparent purpose.

I’ve decided to make it a goal to think more about data as storytelling, data as a way to investigate the world, and data as a tool for action. In my next two posts (YES, people! I’m firing the ole blog back up again!) I’m going to delve into the idea that we can use data to discuss social justice ideas and critical literacy at the elementary level. I’m just dipping my toe into this waters, but I’m really excited about it!

And Kassia did just that! So far she’s followed up with one post where her students noticed and wondered about a graph showing the percent of drivers pulled over by police in 2011, by race. I love how the graph sparked a curiosity that got her students to dive more deeply into the data. How often does a graph about favorite desserts or our birthday months spark much of any curiosity?

Jenna Laib

Jenna shared a numberless graph that immediately got me curious! This is one she created to use with 6th grade students.

I can’t help but notice a bunch of dots grouped up at the beginning with a just few outliers streeetttcchhiiing across almost to the very end.

Once she included some numbers, my first instinct was that this graph is about ages. Apparently I wasn’t alone in that assumption!

And then there’s the final reveal.

So why did Jenna create and share this graph? What was her mathematical goal?

I especially loved this observation about how her students treated the dot at 55 before they had the full context about what the graph is really about.

Chase Orton

Chase wrote a detailed post about how he worked with 2nd grade teachers to do a lesson study about interpreting graphs.

…there’s so many rich opportunities for meaningful student discourse about data.  That is, if it’s done right.  Most textbooks suck all the life out of the content.  Students need to understand that data tells a story; it has contextual meaning that is both cohesive and incomplete.  Students need to learn how to ask questions about data and to learn to identify information gaps.  In other words, students need to learn to be active mathematical agents rather than passive mathematical consumers.

Chase walks you through the lesson he and the teachers created and tried out in three different classrooms. I love how he details all of the steps and even shares the slides they used in case you want to use them in your own classroom.

He closes the post with a great list of noticings and wonderings about continuing this work going forward. Here are a couple of them about numberless graphs specifically:

• We need to give students more choice and voice about how they make meaning of problems and which problems they choose to solve.  Numberless Data problems like these can be be a tool for that.
• The missing information in the graph created more engagement.

A huge thank you to Kassia, Jenna, and Chase for trying out numberless graphs and sharing their experiences so we can all be inspired and learn from them. I can’t wait to see how this work continues to grow and develop next school year!

If you’re interested in reading more first-hand accounts of teachers using numberless word problems and graphs, be sure to check out the ever-growing blog post collection on my Numberless Word Problems page. I recently added a post by Kristen Acosta that I really like. I’m especially intrigued by a graphic organizer she created to help students record their thinking at various points during the numberless problem. Check it out!

Sink your teeth into data. Don’t just nibble.

Looking for math all around started as a challenge I made for myself and I’m realizing it’s becoming a full-fledged theme for my year. When I had to think of a topic to moderate this week’s #ElemMathChat, I started by asking myself, “What’s a topic we haven’t talked about since the chat started in August 2014?” After some brainstorming, I eventually came up with analyzing data. What a great topic for my theme! I don’t think I could throw a rock without hitting some data in the world around me.

In fact, as I was fleshing out the topic for the chat, I was regularly checking some real-world data online. After a long dry spell, we finally got some rain in Austin. And by “some rain” I mean a deluge. On a couple days last month it just kept pouring and pouring. Throughout each day it rained, I found myself checking our neighborhood weather station on Weather Underground to see how much rain had fallen. By the time October was over, we had received 10.3 inches of rain in my neighborhood! That simple piece of data became the catalyst for tonight’s #ElemMathChat.

I started digging into rainfall data for October, then rainfall data for other months, and finally I expanded my data dive into other cities in and out of Texas. When I was done, I had a spreadsheet full of various tables of data that I wanted to share in my chat. To make this chat work, I realized I needed to be intentional about how I shared the data in order to tell a coherent story. I also wanted to create a variety of data displays that would match the various data displays students encounter across grades K-5. As an aside, I think #ElemMathChat sometimes leans a bit heavy on content for grades 3-5, so I was trying to be mindful to show some graphs that could be analyzed in a Kinder or 1st grade classroom.

It took several nights to research, create graphs, and pull it all together to make a story, and in the end I’m proud enough of the final result that I wanted to capture it on my blog.

Before starting my data story, I shared the following guiding questions that tied into my primary goals for the chat.

My Data Story

Our story begins with the piece of data that started it all. I asked the participants to tell me what they noticed and wondered about this statement.

What do you notice and wonder?

Many people wondered how this amount of rain compared to other cities. Funny you should ask.

What do you notice and wonder as you look at this pictograph?

One thing I noticed is that I accidentally left the key off the graph. Oops! Each picture is meant to represent 1 inch of rain. Despite my mistake, several people liked that the missing key invited students to wonder about what the pictures represent. That sounds like such a wonderful conversation to me that I opted to leave the key off when sharing the picture in this blog post.

I had a little fun with this graph because I had to decide which cities to include. I decided to focus on other state capitals, but the question became, which ones? When I noticed how many start with A, I decided that was more interesting than picking random capitals. It just so happens that all the other capitals on this pictograph are all on the East coast, so I wonder if it would have been better to choose capitals with greater geographic diversity. In the end this is just a fun way to get our story started so I’m okay with what I chose.

Next we moved from cities outside of Texas to cities inside of Texas, specifically along the I-35 corridor from San Antonio to Waco.

What do you notice and wonder as you look at the October rainfall totals for these cities?

Now that I shared two different graphs, what questions could you ask students about these graphs? What math skills can students bring to bear to interpret and further understand the data in these two graphs?

One thing that we often do with graphs found in textbooks and tests is ask one question about them and then move on. How unfortunate! There’s so much rich information to dig into here. One of my key points for tonight’s chat was reiterating something I read by Steve Leinwand about mining data. Ask a variety of questions about data displays. Sink your teeth into them; don’t just take a small nibble.

The one thing that stood out to me and many others in the chat was how little rain San Antonio received. The difference between San Antonio and New Braunfels is quite striking considering how close they are to each other.

Other people felt that Austin’s rain wasn’t fitting with a general trend in the data. I didn’t want to get into it in the chat, but I’ve noticed the rainfall in my neighborhood tends to be less than other parts of the city. Our weather station recorded 10.3 inches for October but others in Austin clocked in at closer to 13 inches of rain. I thought about using the larger number, but because the catalyst for this whole story was my weather station’s data, I opted to stick with that. By the way, I don’t think it’s an issue with our weather station’s rain gauge. Over the years there have been many instances of rainfall in other parts of the city while my neighborhood in north Austin remains bone dry.

Now that we’ve looked at rainfall in and out of Texas, it’s time to drop a bit of a bombshell. With this new information, what story is the data telling so far?

Here’s what I see as the story so far: Austin received 10.3 inches of rain in October, which was a lot compared to areas outside of Texas, but fairly common for our area in Texas. Not only was this a lot of rain, but it also fell in a very short amount of time, 6 days.

Next, I asked for help. Now that you know it rained only 6 days in October, which data display would you choose to represent October rainfall?

Option 1

Option 2

Most people preferred option 2 because it shows the full picture of October. That was surprising to hear. In my mind, because we just saw the picture graph showing that it only rained 6 days in October, I didn’t feel option 2 was needed. I already know it didn’t rain on very many days, so why waste the space with all those days showing 0 inches of rain? Option 1 puts the focus squarely on analyzing the rainfall on the days where it actually rained. In the end there’s no “right” answer, it all comes down to how you justify showing what you choose to show.

We’re nearing the end of our story. There are two more graphs remaining. What does this next graph add to our story? What is one question your students could answer based on this data?

I love looking for relationships so here are the questions I came up with:

• Where do you see the relationship “three times as much” represented in this graph?
• Where do you see the relationship “half as much” represented in this graph?

I especially like wondering what students will come up with because both questions have more than one correct answer.

And now for the last graph. How does this close out our data story?

Here’s a follow up question for you. What could be the sequel to the story I just told? How could you and your students explore and tell the sequel? What other data stories could your students explore and tell?

I closed the chat, and I’ll close this post, with two key points I want everyone to take away from this conversation.