Assessment Design
Why do we assess? There are different reasons, but probably the most familiar one is to determine if students have demonstrated the learning outcomes.
According to CAST, other reasons you assess might include:
- accountability to the field/sector, courses that follow, and program goals
- providing students with feedback on their learning progress
- gathering feedback on the effectiveness of your teaching and course progression
According to outcomes-based learning, assessments need to be aligned with the learning outcomes of the course. That is, they should measure what the course outcomes intend to measure. In our previous discussion of learning outcomes, we talked about separating the “means” from the “ends” in our learning goals. The concept of construct relevance in assessments is related. From a UDL perspective, “constructs are the knowledge, skills, or abilities being measured by an assessment” (CAST, n.d.). Ideally, these relevant constructs are taught, and learners are given opportunities to practice them with formative feedback before they are assessed. Sometimes, our assessments contain constructs that are not actually measuring the skills and knowledge we intend to assess. That is, students may require “extra” knowledge, skills, and abilities to perform the assessment task. This can be particularly true when we are teaching and learning online. Extra technical skills may be required to submit assignments through the LMS or to use particular tools. These “extras” are NOT required to demonstrate the learning outcomes for the course, or sometimes even the program outcomes, but are still part of the course in different ways. This indicates that there is a disconnect between what learners are asked to perform versus what is intended to be evaluated. Examples of these “extra” requirements, or irrelevant constructs, might include creating an Adobe file or pulling an MP3 out of a video file.
There are many considerations when looking to remove construct barriers from assessment design and instructions. We want to remove barriers as best we can from assessments in order to get a more accurate picture of students’ learning processes and development. This video summarizes these ideas. As you watch, consider the connections between all three principles to the assessment of learning goals.
An Introduction to UDL and Assessment
>> UDL on Campus. Universal Design for Learning in higher education. An introduction to UDL and assessment.
Reducing learning barriers is of vital importance in post-secondary settings, as learners arrive in colleges and universities with a diverse set of experiences. What may be a preferred learning mode for one individual may be an obstacle for another. In learning environments, individual variability is the norm, not the exception. Instructors of post-secondary courses set goals for learners. Assessment is used in courses to determine how well learners are meeting instructional goals. Goals need to be clear and separate from the means of achieving them. Assessments should offer more than one route for demonstrating mastery.
A critical part of assessment is the construct. Construct is the term used to define the knowledge, skills and abilities to be assessed. Construct relevant factors refer to factors such as type of presentation or response that are related to the construct that the assessment is intended to measure. Many assessments include features that are not relevant to the construct being assessed. These are considered construct irrelevant items. These features may pose barriers for some learners and impact their performance on the assessment. That's where Universal Design for Learning or UDL comes into play.
As a set of three principles for curriculum development, UDL provides a blueprint for creating instructional goals, assessments, methods and materials that work for the widest possible range of learners. This flexible approach to curriculum development can positively impact how learners perform on assessments and result in a more authentic appraisal of learner achievement. The first principle, engagement, focuses on enhancing motivation and persistence. When their achievement is assessed, are they sufficiently but not overly challenged? Do they have options and variety in how they are assessed to help reduce anxiety? The second principle, representation, focuses on the ways in which information, including assessment items, is presented. These can be charts, graphs, images, videos, demonstrations or objects to manipulate. The third principle, action and expression, focuses on how learners demonstrate what they have learned. Do they have the option to write or draw? Do they have the option to present their capabilities through multimedia or a recording?
It is important for instructors to be aware of the inherent variability of learners. The existence of barriers that may impede both the acquisition of information and expressing their competencies. Designing assessments with these challenges in mind will allow for greater flexibility to ensure learner success.
[music]
An Introduction to UDL and Assessment - Runtime 3:07 min
https://youtu.be/TWN4V16Drr0
One important construct that is sometimes irrelevant is time. Many systemic factors reinforce this barrier such as our semestered system, grade entry, the precarity of many teachers’ employment from semester to semester, and student progression from one semester to the next. However, there still might be opportunities within your power to remove irrelevant time constructs. Some students may prefer set due dates whereas others may benefit from flexibility; having the choice allows them to practice expert learning. In the next pair of videos, a student talks about the importance of having time options to deliver her work and a faculty member discusses his changing relationship with due dates.
Sobia is a mom, an employee and a full-time student. Having autonomy over when she submits assignments allows her to manage her own time and meet due dates.
Sobia Rasheed
[ Music ]
>> Sobia Rasheed: Hi there. Let's talk about universal design learning framework, what it is, and how it can affect your studies and your career. Universal design learning framework has helped a lot in terms of giving us the flexibility and the openness to choose what our mode of delivery will be. For example, I'm a mom with three kids and I work full-time. But let me tell you how UDL has helped me in completing this course. Like, my professor for content marketing for digital media marketing, they had given us the choice that we can select any method. We had to write three journals, three e-journals and it was -- they give us the flexibility to choose what mode of delivery we wanted to complete this assignment. It could be a blog. It could be a podcast. It could be a vlog. First of all, not everyone is a speaker, public speaker. Not everyone is a good writer. Not everyone is great in front of the camera. So let's acknowledge our own strengths and weaknesses and let's take this thing forward. The objective of this course was to learn, right? No matter how you want to deliver it, the learning is there. The professors are there to help you out. And the best part about this course is I learnt and my objective was achieved. Another example I would love to quote here is that when the professors -- they keep the midterm or the final exam open. Say for example they open it from Friday up till Tuesday or Wednesday the next week. It really gives me the flexibility of completing it at my own sweet time. So for me, the rest of the days from Friday up till Sunday I work, right? So on Monday, I usually study for the exam and I give the exam on Tuesday and that too at night when my kids -- when I put the kids to sleep because that is the time that I feel that I am -- I can give my undivided attention to this -- to the exam. And this is how I have passed with great, I would say, grades during this entire course and that's about it. Thank you so much for watching.
[Music]
Sobia Rasheed discusses leveraging strengths and time management as they relate to choice - Runtime 2:52 min
https://youtu.be/-ZuNURNyg5Q
In this video, Lazaros reflects on revising his discussion board posting assessment to minimize irrelevant constructs.
Laz Challenges Time as a Construct
[ music ]
>> A central concept in UDL as it relates to assessment practices is the idea of construct relevance. Now CAST defines a construct as the knowledge, skills or abilities being measured by an assessment. For those of us teaching in post-secondary, this sounds a lot like the way we define the learning outcome. Sometimes, what becomes apparent when you look at your course though, through a UDL filtre, is that there are some things that we evaluate or assess that aren't necessarily relevant to the construct and are therefore construct irrelevant. Let's look at an example. Here's a portion of a rubric for discussion activity in a fully online writing course that I teach. You can see the criteria along the left side and the levels of achievement along the top. There are, of course, a couple more criteria that got cut off here, but I just want to focus on these ones right now. Because it's a writing course, I think it's important to look at and evaluate students' grammar and the thoroughness and originality of their work. Looking at the very top, you'll see a criteria for something called promptness and initiative, and this is a real fixture in a lot of online discussion rubrics. But I had to ask myself why I was marking learners on their ability to submit their threads to the discussion on time if this wasn't something relevant to my learning outcomes and if time management skills weren't something that I was teaching them in the course. So in the next iteration of this course and of this rubric, I did away with the promptness and initiative criteria entirely. There are two questions that we have to ask ourselves. First, am I evaluating the knowledge, skills and abilities that I teach? That is, do my assessments match up with the learning outcomes and how I teach them? But UDL forces us to ask another question, namely, am I teaching what I'm assessing? If not, then you may want to reconsider your assessments and rubrics to ensure stronger link between your teaching and what you're evaluating.
[music]
Laz Challenges Time Constructs - Runtime 2:23 min
https://youtu.be/dKO1ZTTz6Qc
Although we’ve seen that action and expression go beyond assessment, assessment design requires consideration of all three principles. This handout titled Creating UDL-Informed Assessments lists some questions to consider when designing or redesigning assessments that take into account barriers, construct relevance, and anti-oppressive practices. You may want to pick one principle or guideline to focus on, see how it works with your students, and go from there. After reading through the prompts, you may find that one area of your assessment is strong whereas another area could use some development. As you consider these prompts, your assessments will become more learner-centred, which is a key focus of UDL.
The Digital Pedagogy Lab Institute has many resources open to educators. Scroll down to Radical Assessment and Ungrading.
Ungrading is an umbrella term for a variety of strategies that nullify the negative effects of grading (Blum, 2020). This movement dovetails nicely with UDL’s focus on students’ agency in their own learning. There are many different ways to approach ungrading within systems such as postsecondary education that still use grades as the currency for progression. To get started with ungrading, you may wish to read Jesse Stommel’s (2018) seminal blog post on “How to Ungrade.”
Myth: If I design my assessments using UDL, no students will require accommodations.