Category Archives: Assessment

Inside the Black Box. Simple and Profound Transformations. #Questions #FormativeAssessment

Posted on by
3

On wait time, the nature of questions, visible thinking, formative assessment and deeper learning:

A particular feature of the talk between teacher and pupils is the asking of questions by the teacher. This natural and direct way of checking on learning is often un-productive. One common problem is that teachers do not allow enough quiet time so that pupils can think out and offer an answer. Where, as often happens, a teacher answers her or his own question after only two or three seconds, and where a minute (say) of silent thought is not tolerable, there is no possibility that a pupil can think out what to say. There are then two consequences. One is that, because the only questions that can produce answers in such a short time are questions of fact, these predominate. The other is that pupils don’t even try to think out a response—if you know that the answer, or another question, will come along in a few seconds, there is no point in trying. It is also common that only a few pupils in a class answer teachers’ questions. The rest then leave it to these few, knowing that they cannot respond as quickly and being unwilling to risk making mistakes in public. So the teacher, by lowering the level of questions and by accepting answers from a few, can keep the lesson going but is actually out of touch with the understanding of most of the class—the question-answer dialogue becomes a ritual, one in which all connive and thoughtful involvement suffers.

There are several ways to break this particular cycle. They involve giving pupils time to respond, asking them to discuss their thinking in pairs or in small groups so that a respondent is speaking on behalf of others, giving pupils a choice between different possible answers and asking them to vote on the options, asking all to write down an answer and then reading out a selected few, and so on. What is essential is that any dialogue should evoke thoughtful reflection in which all pupils can be encouraged to take part, for only then can the formative process start to work.

Inside the Black Box
Raising Standards Through Classroom Assessment
Paul Black and Dylan Wiliam

In those few sentences above, one can find some of the simplest and easiest methods to transform teaching and learning in significant and profound ways.

If you spend time visiting and observing classrooms in this country, you know that we all can improve on the methods suggested above.

= = = = = = = = = =

NOTE: Just about everything that I have read about assessment in the last decade references and/or connects in some way to Black & Wiliam’s article “Inside the Black Box.” Yet I wonder how many educators have taken the time to read it, study it, and strive to implement it. And, I do not mean to sound accusatorial. I genuinely wonder.

My main blog’s 2013 in review – and a brief spur of reflection on “report cards”

Posted on by
3

While I am fascinated, in some ways, with analytics like the one below emailed to me from WordPress, I can’t help but think that “the numbers” only tell part of the story for me – a minor fraction. They certainly don’t tell the most compelling parts of the story, in my opinion. Not by any stretch.

So much more than the number of views or most-read posts, I care about HOW the writing-as-thinking represented here has changed me, and, hopefully, how it has potentially helped change others.

In this regard, such analytics and report cards make me think about what our school report cards lack and suffer from, as story tellers. If I look at my collection of report cards, I see mostly quantitative analytics – proxies for some measurement of my learning and development. Inadequate dashboards claiming to summarize me as a learner of math, English, science, economics, etc.

Perhaps these quantitative measures must play a part in telling my learning story. I’m not convinced. But certainly, we in education can devise better proxies for telling the stories of human development, deep learning, and awe-inspiring growth.

+ + +

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 26,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 10 sold-out performances for that many people to see it.

Click here to see the complete report.

Formative assessment IS design thinking. #DTk12Chat

Posted on by
4

Teachers are designers. Either intentionally or unintentionally, maybe. But teachers are designers.

Some feel that the word, the title, “designer” is being co-opted by too many industries and sectors and professions. But how could one really deny the essence of teacher as designer.

Teachers design with curriculum, learning environment, instructional methodology, and assessment. Together, these elements create pedagogical design.

Because of the heightened attention that design and design thinking are getting, we know more about how great designers design with the needs of the user clearly at the center of the design. Discovery, ethnography, examination, observation, interview – all of these and more are the tools of great design and design thinking.

For the truly intentional, great teachers, formative assessment is an invaluable tool – a system really – to discern the deepest needs of the user… the “student.” Through purposeful use of formative assessment, great teachers – great pedagogical designers – collect critical information by way of discovery (assessment), ethnography (assessment), examination (assessment), observation (assessment), interview (assessment), etc.

But, for these assessments, these tools of discovery and empathy, to be design-employed, the insights gained must be used to inform and transform the pedagogical design for the improvement of the user experience. Better known as “deep learning.”

If an assessment is merely something at the end of instruction to provide a grade for a paper grade book or digital SIS (student information system), then enormous potential is being wasted, underutilized, undervalued. Assessment, used as design tool, can form better design for curriculum, instruction, learning environment, assessment, etc. To reach this potential, though, we need to be intentional as designers.

If you are pursuing design thinking at your school, perhaps you are using the d.School model:

  • Empathize, Define, Ideate, Prototype, Test.

Or perhaps you are using the model from Design Thinking for Educators:

  • Discovery, Interpretation, Ideation, Experimentation, Evolution.

At Mount Vernon, we’ve developed our own model of design thinking:

  • DEEP – Discover, Empathize, Experiment, Produce.

Or perhaps you are working to nurture and build innovators and tracking with such work as Innovators DNA, purposefully infusing the known traits of innovation:

  • Observing, Questioning, Experimenting, Networking, Associating.

Among all of these models, and among the practices of the most highly respected designers and design thinkers, empathy lives at a core – through intentional and purposeful discovery, observation, and ethnography – in order to enhance and improve design for the needs of the user.

Assessment – formative assessment – is essential for one to be a design-intentional teacher.

How are you using assessment as a systemic tool for exceptional design? For the user experience? For the learners?

#ItsAboutLearning

Numbers Count: Contextual Assessment and Quantitative Measures in #PBL #DTk12

Posted on by
3

“He got one out of three!,” said Phil.

“Wow! Can you believe that?!,” responded Ann.

Did the “He” in this short story experience success or failure? Context makes all the difference in the world, doesn’t it?

I can imagine one context: A teacher on a team is reviewing assessment data, and Phil announces to his team that a student “got one out of three.” The tone could be disappointment and disbelief, indicating that Phil thought the student had more command over what had been assessed. The teammate, Ann, knowing how hard the team has been working on the lesson study and assessment echoes Phil’s consternation. In fact, I’ve heard just such a conversation.

I can also imagine a second context: A young boy relatively new to baseball is talking to his mom about a player hitting one out of three at-bats during a season, as the boy figures out what batting averages of .333 mean versus averages of .250. In this context, the exclamations indicate wild excitement at reviewing the success of the young boy’s friend who made the All-Star team. The mom is reflecting the excitement with a big smile on her face, saying, “Can you believe that?!” In fact, I’ve heard just such a conversation.

As schools examine and employ strategies like project-based learning and design thinking, I believe the stories above can be catalysts for talking about quantitative feedback in context. Why is it that the same fraction and decimal is called “failure” in one context and “success” (great success!) in another? Could it be that many of us have a “movie in our mind” playing – one that shapes our beliefs about what it means to get a one out of three based on experience with traditional quizzes or formative assessments? Could it be that we have come to assume that the content and skills on such assessments should be evaluated in such a way that only 70% and above would be considered “passing?” Considering an ed psych concept like Vygotsky’s ZPD (zone of proximal development) might lead us to believe that the scaffolding and instruction is in misalignment with the student’s learning. In context one, many might view one out of three as a problem.

But in the context of baseball, a 33% means something very different. It involves a mental movie that tells us that one out of three is grounds for Hall of Fame induction if the player can do that consistently over a career. Why is 33% so different in this context? Could it be that the high-quality activity of being face-to-face with a pitcher throwing serious heat causes us to shift our expectations and see 33% in an entirely new perspective and point of view? In context two, many might view one out of three as a celebration.

As schools, when we design project-based learning and design-thinking exercises, how might they be informed, in terms of assessment, by the contrasting contexts of taking a quiz versus standing at bat? Are we putting new wine into old wine skins (please forgive the mix of metaphors) when we apply traditional grading practices and certain quantitative measures to more high-quality, intensive contexts that refuse to be assessed with the same mindsets that have historically been applied in the classroom?

How might we be more purposeful and intentional about the interpretation and context of mathematical feedback?

About 14 months ago, I counseled a group of four boys who said to a colleague and me that they had failed.

“Why do you think you’ve failed, guys?”

“Well, Mr. Adams, we only got 2 out of 10 – 20%. In school, 20% is seriously failing!”

“But in your case, through your project, you helped 2 out of 10 unemployed human beings get a job! In your case, your point of view of 20% might need to shift a bit. Just because 20% on a quiz or a test might have indicated real disappointments and ‘disasters’ to you in the past, a 20% employment-bump statistic in your job-fair project could be seen as a wildly successful outcome. It’s more like a batting average than a vocab quiz. That’s how Ms. G and I see it. You positively changed 2 people’s lives this week. Your ‘20%’ will cause ripples that will send significantly positive waves throughout that community.”

When we in schools apply quantitative measures – 100 point scales, 4 point Guskey scales, whatever kind of scales – I believe we need to do so very thoughtfully and carefully. We need to be proactive about our strategic communications surrounding these assessment measures. Students, teachers, parents – we all bring existing mental movies with us into the school setting.

Even if we don’t apply numerical measures – we did not do so in Synergy in the case of the food-desert, job-fair project – we must be aware of the mental movies and previous experiences that students bring with them to these contexts of project-based learning and design thinking. Those four boys did not receive any kind of “final grade” on that project (our course was non-graded, but heavily assessed), yet they applied previous context to a new situation and drew some profound conclusions about their perceived success. It was a powerful learning moment for me. One that has likely taken me the entire 14 months to fully process.

During the past few years, as I’ve consulted with a number of schools, more than a few are applying relatively traditional grading practices to the assessment of skill sets and dispositions. For example, on a report card or progress report, one might find a column or row labeled “Collaboration” and another labeled “Critical Thinking.” Next to the categories one might find an “82” or a “2 on a four-point scale.” One might also see a “B-” in the scoring cell. Or one might see initials like “PG” – “Progressing.”

I realize I am telling a very incomplete story here. I imagine some readers writing to me in the comments or email or Twitter and saying, “Bo, you’re missing the whole point! High-quality PBL shouldn’t even be getting a quantitative measure. It should be performance-task assessed with only narrative, negotiated feedback. No numbers at all! What’s wrong with you?!” With this post, I really mean to provide a catalysts for thinking and doing with those readers and schools who ARE trying to marry quantitative-assessment measures with high-quality PBL and DT. I, too, have serious questions about the “Why?,” and I am also deeply interested in the “How?” if a school just will not consider non-numerical assessment reporting, even for certain courses, strands, projects, assignments, etc.

Are the challenges we are curating or creating causing us to think deeply about the nature of the challenges relative to assessment? Are we orchestrating experiences that are more like the intensive match up between a super pitcher and a batter – ones in which the quantitative measures we apply communicate All-Star results at “33%?” Or are we trying to place new wine into old wine skins and facilitating experiences that challenge kids so slightly that it’s assured most will “pass” or view their Herculean efforts as failure because we’ve neglected to help everyone involved reconceptualize and pivot perspectives on what “one out of three” might really mean in our context?

Are traditional report cards “vanity metrics?”

Posted on by
3

Are traditional report cards “vanity metrics?”

In the business world, we talk about the difference between vanity metrics and meaningful metrics. Vanity metrics are like dandelions – they might look pretty, but to most of us, they’re weeds, using up resources, and doing nothing for your property value.

from “Know the Difference Between Your Data and Your Metrics” by Jeff Bladt and Bob Filbin, Harvard Business Review  |  11:00 AM March 4, 2013

English – 92
Math – 89
Science – 91
History – 88
PE – 93
Art – 90

Such a report card might make the refrigerator. But does it really say very much about the student’s growing capacities in writing (ideas, organization, word choice, sentence fluency, conventions, voice)? And did you assume from that last sentence that I was referring to English, or did you assume that such metrics could cut across the departmentalized curricular landscape and comprise parts of all the subject grades? Do we know if the student asks probing questions and demonstrates curiosity for understanding more deeply? What do those dandelions tell us about the student’s application of such thinking skills as divergent, emergent, and convergent explorations? What do we know of the student’s perseverance, resilience, risk taking, and grit? Can we make any deductions about the student’s observation and experimentation capabilities? Do we know how the student has demonstrated integrity and empathy from those pretty flowers on the refrigerator door?

From that data set, the one that will live indefinitely in the fireproof cabinets and ethernets of schools, what do we know about the student’s growth and emergence in the 4, 5, or 7Cs of 21st century skills? What do we see on that report card about mindset?

What are we grading? What are we measuring? What are we commenting on? What are we collecting and recording and archiving data on? What do we say matters most about our children? Do our report cards shed light on what we say we value most?

Is your school asking these questions? Why or why not? Are these questions among more systemic considerations that you are examining?

#QuestionsAreTheWaypointsOnThePathOfWisdom

#PedagogicalMasterPlanning