Assessments, Workforce Interviews, and Training with Bob Dolan!

Episode 11 of the ThinkUDL podcast: Assessments, Workforce Interviews, and Training with Bob Dolan. Today Lillian talks with Bob Dolan, Principal at Diverse Learners Consulting and a long-time CAST collaborator. In this episode, Bob outlines how UDL informs assessments, not just in the classroom, but in hiring new employees, learning a new tool or program on the job, and why understanding the principles of Universal Design for Learning can benefit both employees and employers! They discuss construct irrelevant variance and construct underrepresentation, along with fundamental attribution errors, and how poorly made assessments can mislead us. This conversation helps us to understand how important it is to tailor learning to each individual no matter the setting.

Resources

Diverse Learners Consulting: Here is Bob’s website with several helpful articles and information about what he does to match assessments and training with diverse learners in all areas.
CAST: The Center for Applied Specialized Technology explains the UDL guidelines and offer resources for all educators interested in applying the UDL framework.

Dolan, R. P., Burling, K., Harms, M., Strain-Seymour, E,. Way, W. & Rose, D. H. (2013) A Universal Design for Learning-based Framework for Designing Accessible Technology-Enhanced Assessments (Research Report). Iowa City, IA: Pearson. http://images.pearsonclinical.com/images/tmrs/dolanudl-teaframework_final3.pdf
A framework developed by CAST and Pearson used to develop guidelines for incorporating UDL principles into computer-based assessments.

Almond, P., Winter, P., Cameto, R., Russell, M., Sato, E., Clarke-Midura, J., Torres, C., Haertel, G., Dolan, R., Beddow, P., & Lazarus, S. (2010). Technology-Enabled and Universally Designed Assessment: Considering Access in Measuring the Achievement of Students with Disabilities—A Foundation for Research. The Journal of Technology, Learning and Assessment, 10(5). Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1605
Recommendations from a multi-stakeholder research symposium to stimulate research into creating technology-enhanced assessments appropriate for the full range of students through enhanced accessibility. 

Transcript

Will be made available as soon as possible.

[Lillian]  Welcome to Think UDL, the Universal Design for Learning podcast.  Where we hear from the people who are designing and implementing strategies with learner variability in mind.  [Music] I’m your host Lillian Nave, and I’m interested in not just what you’re teaching learning guiding and facilitating, but how you design and implement it, and why it even matters. [Music]

Thank you for joining me for episode 11 of the Think UDL podcast: Assessments, workforce interviews and training with Bob Dolan.  Today, I talked with a longtime CAST collaborator, Bob Dolan, who’s also principal at Diverse Learners Consulting.  In this episode, Bob outlines how UDL informs assessments, not just in the classroom, but in hiring new employees, learning a new tool or program on the job, and why understanding the principles of Universal Design for Learning can benefit both employers and employees.  He explains construct irrelevant variants, and construct under-representation, along with fundamental attribution errors, and how poorly made assessments can mislead us.  This conversation helps us to understand how important it is to tailor learning to each individual, no matter the setting.  Welcome to the Think UDL podcast, Bob!

[00:01:43]

[Bob]  You’re very welcome, and thanks for having me today.

[Lillian]  So, I would love to ask you my first question that I ask so many of my guests here on the Think UDL podcast, and that is what makes you a different learner?  Is there something different, original, quirky, enjoyable about the way you learn?

[00:02:04]

[Bob]  Wow, that’s a great question.  You know, it’s so funny how we want to characterize people as learning one way or another, but what I’m learning more and more is how I learn differently in different circumstances, in different situations.  But, if I have a preferred way to learn, it’s where I can go back and forth, bounce back and forth between different perspectives.  I love reading technical information, for example, but I also love having my hands-on.  So I can start either way.  I love being immersed in learning, so often that will be a hands-on experience, a minds-on experience.  But I love the chance to then have some questions answered that are coming through my mind as I’m learning something new.  Perhaps questions that can best be answered by googling, or when I was a kid going to the encyclopedia, or talking with friends or family or teachers, and then diving back in. I love the back-and-forth.

[00:03:06]

[Lillian]  Wow, what a multi-faceted approach that is, and sounds like you would get a much more nuanced understanding of whatever you were doing if you were able to satisfy all of those desires.  That’s great.

[00:03:21]

[Bob]  Yeah, it was why school was so tough for me so much of the time, because I was supposed to be sitting there patiently absorbing information through a limited number of means, and not getting a chance to really dive in and roll up my sleeves.

[Lillian]  Wow.  

[00:03:35]

[Bob]  When I had shops in middle school, it was such a great experience for me, because I had that opportunity to really do and create and interact physically as part of the learning

[00:03:49]

[Lillian]  That’s something we don’t see very much at all anymore do we?

[Bob]  Sadly, we don’t.

[Lillian]  So, I’m glad to hear, and I bet our listeners are also glad to hear that was that immersive part of Education was something that we used to have a lot of emphasis on.  A lot of students had the chance and the opportunity to be physically immersed in whatever they were learning, and now it’s a lot less so.  As you were saying, it’s a lot of sitting in your seats and listening quietly.

[00:04:20]

[Bob]   Absolutely.  And you know they’ve got to give credit there’s a great interactive teaching that does go on in a classroom, don’t get me wrong.  But for someone like me, I like to be physically moving, and my hands like to physically be moving.  And so when you look at the historical way which we’ve learned new things through like, for example apprentice apprenticeship models.  Yeah, there is that opportunity to really dig in.

[00:04:46]

[Lillian]  And wouldn’t that be great?  I bet you’re going to have some ideas about how we can apply some of those principles in the workforce in our discussions later on, because of what you do.  So, that brings me to my next question and that has to do with how you’ve gotten to what you’re doing now.  You are the founder and principal of Diverse Learners Consulting, and can you tell me a little bit about what got you into founding your own company, which is no small task, and what you do there?

[00:05:20]

[Bob]  Well, I got here by kind of being at both ends of the education technology spectrum.  Starting in year 2000 when I first came to CAST, and I spent several years there as a senior research scientist.  It was a blast.  I learned so much from them, and I was leading a bunch of research projects on accessibility, both in instruction and assessment, and diving deeply into the whole UDL concept.  After I left CAST, I joined Pearson, where I was also doing research, but also doing a lot of product development and looking at ways where we can do really deep consideration of UDL: how do we really address diverse learners?  How do we ensure that we’re designing assessments and curricula and instruction and supporting teachers with professional development that really consider the various ways in which individuals can learn?

[00:06:15]

[Lillian]  Wow, so you had a whole– you really ran the whole spectrum so from one side to the other.  So, I could see that you could move something from the beginning to the end of a project, so how is it then that you ended up moving into kind of business for yourself?  What niche are you moving into here?

[00:06:36]

[Lillian]  Yeah, so about five years ago when I left Pearson, I decided I really wanted to consult, I really wanted to work with some of the various stakeholders that I had been working with for the past several years.  Other publishers and curriculum developers, small not-for-profits, universities, schools, colleges.  And so, I started Diverse Learners Consulting company with an employee size of one, that’s me.  So I’m the founder and principal consultant, and what I wasn’t expecting is how I would be branching out from the mostly k12 and post-secondary education technology arena, to moving into more lifelong learning and workplace assessment and what’s called industrial and organizational psychology.  That came as a surprise, but I realized tremendous opportunities there to apply a lot of the ideas a lot of the principles of Universal Design for Learning in these new arenas.  As we’ll talk, there are still so many assumptions we make about if we want to for example train an employee on using new methods or new software, we make these assumptions about what’s the best way for them to learn.  And they’re pretty monolithic, they’re pretty one-size-fits-all.

[00:07:57]

[Lillian]   What would be one of those assumptions?

[Bob]  Let’s say you have employees need to learn Excel.

[Lillian]  Okay.

[00:08:05]

[Bob]  I would want to make sure that the employees have offered–have a lot of choices in how they can go about that, that take into account what’s the best way for them to learn.  So to what extent is it hands-on, to what extent is it self-paced as opposed to being led by an instructor, can it be both?  Can the individual have up options for going back and forth between the two?  Will they have time to really practice what they’ve learned?  Will they be able to get feedback about how well they’re doing and incorporate that into subsequent decisions about what to do next?  All of these things largely are limited, it’s usually the programs that are available to train employees again fairly monolithic, fairly one-size-fits-all. 

[00:08:53]

[Lillian]  So then a company who wants to train their new employees, or needs their old employees to learn something new, they could contract with you to help them learn this–whatever it is– this new thing that they that they need to learn?

[00:09:13]

[Bob]  Absolutely.  And one of the arenas I’ve been looking into the most, starting from when I was at Pearson, is this whole idea of adaptive learning and intelligent tutoring systems.  In the idea that we can tailor the learning experience to individual learners needs.  It’s hard to do well for a number of reasons, that said, there’s still tremendous promise and there’s been a lot of successes in this area

[00:09:40]

[Lillian]   So, can you tell me about some of the projects that you are doing that employ UDL, and how you are helping companies and workforce readiness in what you’re doing?

[00:09:55]

[Bob]  Sure.  Thinking about one client, which was a fairly large size staffing company but also would do placements for their clients.  And the founder of this company was very interested in increasing the diversity of individuals who are matched with good jobs and given employment opportunities.  And so we talked a lot about how to best do this.  It’s really hard there’s a tremendously high unemployment rate or underemployment rate for individuals with disabilities.  And it’s not because they’re disabilities prevent them from doing the job–well, I mean, to some extent it could be–but largely it’s because of the biases in this whole process by which we select individuals and evaluate their suitability for jobs.  There’s a lot of biases.  There’s something specifically called the fundamental attribution error, which is how we for example, if I were to interview somebody.  We talked about how important first impressions are.   They really are.  So, we get these first impressions and they really jade even what we think of as objective measures henceforth.  So, for example, if I judge that somebody coming in for an interview seems distracted, I’m going to be applying that lens to everything else that occurs during that interview,

[00:11:29]

[Lillian]   Right? 

[Bob]  Its a continuation process.

[Lillian]   And we might be making some assumptions about what that person is doing or what that distraction means, right?

[00:11:39]

[Bob]  Exactly, we could be completely misreading what’s going on.  Or maybe they are distracted, maybe they’re very nervous.  But we’re evaluating them under one specific situation in the interview, and our personalities are not consistent from situation to situation.  So how I am during an interview, or meeting somebody for the first time, is not really predictive of how I am when I’ve got a big task at hand, and there’s lots of distracting information coming in and I’m under a lot of stress, I might be a very different person under that situation.  But because of this fundamental attribution error, we kind of make this assumption of okay here’s Bob, he walks into the office, he seems kind of distracted, he must be a kind of a distracted person hmm.  And again we don’t realize we’ve formed this bias, so we’re going to you know, you could keep asking Bob individual questions to give him a chance to prove himself and explain himself, but our interpretation of his answers is going to be jaded now.  Now extend the situation to people who come through the door with disabilities

[00:12:37]

[Lillian]  Okay

[Bob]  Regardless of how well-meaning we are, we’re going to start making assumptions.  The Individuals with Disabilities Act specifically requires employers to provide reasonable accommodations when somebody has a job to minimize the impact of a disability on somebody’s ability to do the job.  And it also requires them during evaluating potential employees to provide accommodations.  So, back to that excel example, so if I need some assistive technologies in order to work effectively at the computer, and it’s reasonable for my employer to provide those, they have to provide those.  So it might be an alternative keyboard, it might be a screen reader, might be a larger monitor, so I can have things blown up to be bigger.  So similarly, when being-when I’m evaluated for the job, I need to be provided those accommodations.  So if I’m given a test on Excel, so let’s say they give me a task to do during my interview and I have to do it using Excel, but I’m using a computer that does not have those accommodations, not cool for obvious reasons.

[00:13:51]

[Lillian]  Right?  They don’t have the tools that they need to succeed.

[Bob]   Exactly.  So this is point we want to be as objective as possible.  If somebody has a disability that is not going to interfere with their ability to do the job effectively, how can we ensure that perceptions about that disability don’t contaminate our ability to evaluate their true potential?  And it’s hard enough–as I said earlier– it’s hard enough for folks without disabilities to have that objectivity, but with folks with disability that the problem is compounded.

[00:14:29]

[Lillian]  Wow.

[Bob]   And this is where I think that UDL can really be useful, and this is some of the stuff I’ve started doing, is looking at the entire process by which we evaluate candidates for jobs, and making sure we’re providing them with multiple means to demonstrate their abilities, their knowledge, their skills, their attitudes, their beliefs, whatever is relevant to doing the job well.

[00:14:55]

[Lillian]   Okay and what would that look like?

[Bob]  Well, I mentioned the example of Excel.  So for example if you want to know how well I can–how good I am at Excel, you want to be measuring that and not measuring how good I am at manipulating a mouse.

[00:15:09]

[Lillian]   Right

[Bob]   That’s irrelevant.  As long an accommodation is available in inevitably there will be for something like a mouse

[00:15:17]

[Lillian]   Right

[Bob]  Also, if I need to learn a new job, learn a new task at a job as I said earlier, are there options in about how I can go about doing that?  Maybe there’s a mandatory training class that I have to take, but it might be confusing for me, perhaps my listening comprehension skills aren’t great, or my attention span isn’t great when I have a talking head at the front of the classroom.  Are there ways I can be provided with alternate means to engage with the information that’s being provided?  Can I be given some of the information in advance, so I can get up to speed, perhaps self-engage somewhat more?  Can I review the materials afterwards?  Are they going to be being made available?

[00:15:59]

[Lillian]  Right, so just multiple means of representation, one of those main UDL principles as providing multiple means of representation so that workers can get access to that information in any number of ways and at their own timing, sounds like, in their own manner.  Because they’re capable of it, but the company–see now we’re not talking about higher ed or schools– but the employer is only offering one way that may suit some of the company but not every person in the company who are perfectly capable of performing this job, or getting better, or moving forward with the company.  So you’re allowing that workforce to improve.  Whereas it could have been one of those catch points before, if they were kind of thrown to the one particular training session and it’s like the Charlie Brown, you know, guy at the front and you can’t understand what he’s saying, there’s too many people in the room, or there’s no, you know, auditory processing difference or something like that, then that worker or whoever the employee isn’t going to be getting the information that he or she needs.  This is very interesting.  So you’re really focusing on training and assessments that really measure what they’re supposed to measure, and get to where they’re supposed to get.  How very, well, how very UDL of you.

[00:17:39]

[Bob]   That’s exactly it, and there’s this whole notion in the assessment world of validity, right?  So an evaluation or a test, it itself cannot be valid or invalid, but the conclusions we draw from the results from somebody taking that test those can be valid or invalid.

[00:17:59]

[Lillian]  Okay can you give us an example of that?

[Bob]   Yeah so there was that performance task I mentioned with Excel.  So you sit me in front of a computer, and you ask me to do a few things in Excel.  And I have a mouse and I have trouble–I have a limited mobility perhaps in my arm, and I have trouble using a mouse.  I’m used to using a trackpad, for example.  And so, I don’t do well.   I don’t do well on these tasks, and you conclude from my performance– from evaluating my performance– that “nah, Bob isn’t really good at Excel.”  Whereas, in fact, you were measuring my mouse skills, or you were measuring my visual acuity, or you were measuring my ability to engage in a task when there’s lots of distractions because maybe they when I’m doing this excel performance task there’s a lot of busy things going on here.  And I’m already nervous because it’s an interview and I’m under pressure.  So these are compromising the validity of your conclusions.  And we call that compromise the construct irrelevant variance.

[00:19:06]

[Lillian]  Ooh I love that!

[Bob]   Fancy term just meaning that we’re contaminating our measurement, we’re saying that we really don’t want to be measuring.  So we’re getting variants, we’re getting a high performance or low performance across individuals, but not necessarily due to their excel skills, but due to other factors such as ability to handle a mouse or read the screen or do well under distracting environments.  Keeping in mind, however, that well you know the distracting environment thing well perhaps that is a relevant construct, a relevant thing to be measuring if I’m going to be working in a busy crowded office and there’s no ways of accommodating me sufficiently.  I can’t be put into a separate room, for example.  Then one could argue– the employer could argue– well you know we the reason we didn’t provide an accommodation for Bob is because that test mimicked real-life situations.  There could be some argument and the ADA (Americans with Disabilities Act) laws do make it possible for employers to have exceptions for those kinds of situations, again if they’re relevant to the job.  But if again, if there’s an accommodation or support that keep it provided in the workplace, it needs to be provided during the testing, and then you’re going to be testing me more accurately.  You’re going to be reducing construct irrelevant variance.  You’re going to get–you’re going to be able to draw more accurate and more valid conclusions about my abilities.  So one of the main threats of– to validity is contract irrelevant variance, where our measures are getting contaminated by other things we don’t care about.

[00:20:49]

[Lillian]  Right.

[Bob]   Or you shouldn’t care about.  The other main threat to validity is construct under representations.  So construct under representations are when we’re extrapolating somebody’s knowledge and skills based on limited information.  So let’s say I’m teaching you US history, and I have five minutes to come up with a test for you and all your classmates about how much you really know about US history.  And so I come up with a quick set of 10 multiple-choice questions, you know, what year was George Washington born, what color was his white horse, now that’s a giveaway, that wouldn’t be a good question.  How many pounds you know or what was the weight of George Washington’s horse?  Can be little facetious here, basically I can give you a test that just basically consists of recall types of questions

[00:21:43]

[Lillian]   Sounds like very specific examples that could have been one out of thousands.

[Bob]   Exactly and based on your scores on that test, I infer how well you understand and learned US history.

[00:22:03]

[Lillian]   Okay.

[Bob]   That’s construct under-representation.  I’m making big assumptions on very, very limited data.  And there’s a reason why that happens all the time, it’s fast and easy to do those kinds of tests.  If I wanted to evaluate you on your critical thinking skills, that’s a much harder test to develop.   Its a much harder test to ensure it’s actually working properly, because you’ve got to test the tests and make sure they’re doing what you think they are.  They’re harder to score right.   So very often we’ll have a set of standards, including the common core standards, and our tests are–still tend to be kind of toward the low end in terms of depth of knowledge.  And so if I say oh yeah this these students passed algebra and those didn’t, am I really basing that on the full set of information?  I’m relying on correlations largely that yeah I’m saying I know I’m not really testing the depth of the standards, but I have a pretty good guess that if the students don’t know these lower level, if they don’t have these lower level skills and pieces of knowledge, they probably don’t have the higher level ones either.  Yeah it ends up kind of becoming a self-fulfilling prophecy if we relied too much on that.  But anyway, that’s the whole idea of construct under representation as this second major threat to validity.

[00:23:28]

[Lillian]   So it sounds like you help companies and you help people to really get to what they really want, and not trying to test for or assess these irrelevant things that, heretofore, they have been assessing the wrong thing, that they didn’t actually know that they were assessing, is that what you’re finding?

[00:23:53]

[Bob]  Absolutely.  And there’s been a lot of growth in the last few years, last several years.   Employers really wanting to expand the diversity of candidates and expand the diversity of their employees.  For example, considering individuals with autism spectrum disorder.  And for these individuals the standard means by which we select candidates can be really challenging.  You know think about the pressures of the interview.  And so, these employers are looking at different ways to evaluate suitability for the job.

[00:24:36]

[Lillian]  So let’s talk a little bit about those things that happen in the interview that somebody with autism spectrum disorder might find challenging.  If, let’s say, someone with autism spectrum disorder comes in for an interview, those things that often happen in interview are very socially motivated, or there’s a lot of social interaction that has a lot of bearing it seems on whether or not they get the job.  But that may or may not have much of a bearing on the tasks that they have to do, is that correct?

[00:25:13]

[Bob]  Exactly, it may, it may not.  So, I believe it’s AMC Theaters came up with this I believe they call it like a walking interview.  Several years ago, they wanted to better consider hiring individuals with autism spectrum disorder, so they, the interview, rather than being talking heads  across the desk, they would take the candidates on a tour of the theater during normal operating hours so that the candidate could really see what they would be doing.  So this is what you’d be doing if you’re selling tickets, this is what you’d be doing if you’re taking tickets, if you were serving popcorn, if it’s cetera et cetera.  So, rather than have it be this kind of abstract thing, that it’s hard to for individuals to imagine themselves doing, they could really see firsthand okay that’s what I’d be doing.  It turned out to be a much more valid way of evaluating both the candidates’ impressions about the job, as well as their suitability for that position.

[00:26:17]

[Lillian]  That’s great.  Sounds like it’s great for the employer and the employee.  Much better than trying to sit still – and it’s a great way to employ the body at the same time.

[00:26:32]

[Bob]  Absolutely, and so if you go back to the fundamental attribution error, and you think about this person coming into an interview and being pretty agitated by the whole situation, they’re not going to get read well, they’re not going to be evaluated accurately. 

[00:26:52]

[Lillian]  Right there could be plenty of assumptions about– they don’t do well with people, they can’t interact well, they can’t stay on task, and all of those things may have nothing to do with their ability to sell tickets, scoop popcorn, interact with the public, just from that interview.

[00:27:11]

[Bob]  Exactly.  Also, as you know, along with autism spectrum disorder comes some overgeneralizing of course but some pretty phenomenal skills, often quantitative skills, analytic logical skills that many employers are saying “hey we want folks with those skills in our workplace, how do we make sure that we’re not having them culled out of the pool too early for the wrong reasons, and furthermore how do we actually attract folks who might not otherwise even consider applying because they know how onerous and often unfair the selection process can be?”

[00:27:53]

[Lillian]  So, what would be ways that one could attract employees on the autism spectrum disorder, or have a hiring process like the AMC walk that you mentioned, but any other examples that would also be attractive to and helpful in matching the skills and the job with the employee?

[00:28:18]

[Bob]  Well, as always, educating oneself about people and about in this case autism spectrum disorder–what it means, what it doesn’t mean– and training programs are very effective for training folks involved in the selection process to try to identify their biases coming up with more objective means of evaluating candidates.  Those all go a long way in making sure the right people are hired for the right reason, for the right job. 

[00:28:45]

[Lillian]  I’ve recently also read of companies that instead of having a face-to-face interview, something that would be thirty minutes or an hour, might be a series of smaller interviews or tasks like here’s what would traditionally be the kind of problem we need you to solve, and what can you do about this?  Or what– how would you approach this problem and then reporting back kind of instead of a handshake and a face-to-face interview.  Really, that’s a lot about how we get along, or social skills.  It’s more task oriented, that could happen over the course of several days, or two weeks, or something like that where people can get more acclimated to a situation, it becomes more familiar, it’s less distracting, it’s less anxiety producing, that sort of thing, it seems like people are starting to wise up that difference is really great in many cases.  Like you say quantitative abilities, things like that, the ability to remember and process things, are things that we need a lot of and a lot of our data management and things like that. 

[00:30:07]

[Bob]  Absolutely.  And, I wonder what it would take for us to start giving candidates the opportunity to choose so if you were to say “hey Bob, you know, we want to make sure that you can do X Y Z, what’s the best way for us to evaluate that?” 

[00:30:25]

[Lillian]  That would be fantastic. 

[Bob]  We’re still a long way away from that.  But I talk a lot about this in school.  How do we best evaluate individuals.  If you really want to test my understanding of plate tectonics, why not give me a choice, right?  But maybe we’ll get a chance to talk about some of those flexibilities in assessment.  But for now, in terms of the workplace, I wonder what it would be like to give individuals ability to choose, as long as they’re still relevant to the workplace, of course.  So if I’m going to be– if you’re going to put me in a diner as a short-order cook, and I say “hey whoa whoa whoa you’re testing me out here, you know, it’s 7:00 a.m. it’s too busy.  Why don’t you test me how well I cook breakfast let’s say 3:00 in the afternoon when you’re not too busy?” 

[00:31:15]

[Lillian]  Right, that’s when I’d like it.

[Bob]  Yeah and you get the pushback in me and say “Bob yeah I hear you, that’s how you best demonstrate what you want us to see, but that’s not the real situation, right?”  So my point is, if there’s kind of two sides to this, you really still need to evaluate in ways that that have fidelity with what the job’s actually going to entail.  But on the other hand, you don’t want to be considering skills and attributes and characteristics of individuals that aren’t relevant to their ability to do the job well.  It’s a hard thing to do well.  This is not an easy process

[00:31:51]

[Lillian]  Right it sounds like it.  And I’m glad we have folks like you who are really intently focusing on this, that’s going to help so many others to really think about this in their own companies, in their own workplaces, and in their own hiring practices.  I mean this is an– I think– a really important issue that we do need to be thinking about.  You know, my primary area is in higher ed, and the hope is that all of these students go and get jobs, right?  That they are then able to enter the workforce, and yes they all learn differently and do things differently and that hopefully will serve them well, and serve their company, their place of business, their employer well.  But everyone needs to be appreciative and you know of each other’s abilities and work with those.  So we we’re on our way, we’re hoping, you know I can see a movement from where I am to where you are, I’m hoping.  You know you mentioned that idea about assessments and flexibility in assessments, and I do want to ask you about that and what your work in that has entailed?

[00:33:08]

[Bob]  Sure.  When I first came to CAST in the year 2000, and the UDL principles were fairly new at that point, and it was understood that one of the areas where UDL really needed to be applied in education is assessments, not just instruction and curriculum and in standards, but in the assessments.  So, to the extent that we need to or want to evaluate students in terms of what they know, what they can do, we need these assessments to be accurate, whether for summative purposes or for formative purposes.  So I was assigned a task to figure out what that would look like.  What would it look like to make the assessments more accessible.  And I realize there was a lot of low-hanging fruit already there from the world of assistive technologies, instructional technologies, and accessible technologies that could be applied to assessments, especially to the extent that they could be offered online. 

[00:34:07]

[Lillian]  Okay, so what do you mean by them?

[Bob]  Yeah so, for example, it still remains sadly true that many tests that we give students in various subject areas first and foremost test their reading ability, coding and comprehension skills, they’re not supposed to, but they so often do.  So if we’re testing a middle school student on let’s say science tests and they have to read a passage that consists of a few paragraphs talking about well say plate tectonics again, and then answer a few multiple-choice questions.  Well, everybody gets the same test so everybody’s being tested fairly and equally, right?  No, absolutely not.

[00:34:50]

[Lillian]  That’s a loaded question. 

[Bob]  Yeah, totally a loaded question.  So right off the bat, I mean, if you gave me that test and you didn’t give me my reading glasses, it all of a sudden it’s not a science test, it’s a visual acuity test

[Lillian] Right, right.

[00:35:02]

[Bob]  Okay so you’re going to give me this accommodation, shall we call it, my reading glasses, and I know I can now I can see the letters right

[00:35:10]

[Lillian]  And if you gave me that test, and the diagrams had different colors, and I were color blind, then I would have a difficult time understanding what those diagrams meant, too.

[00:35:20]

[Bob]  Absolutely, absolutely.  So, again there’s that construct irrelevant variance working its way in, and eating away at our validity.  But there’s all these supports that we know we can offer students in terms of decoding supports and comprehension supports.  So for example having text-to-speech available, or having just a digitized human voice that can be read aloud, especially if it can be synchronized with the text, and let these be options that the students can choose.  And these are ideally options that the students already using instructionally, and during homework, and during practice.  And ideally what’s being offered during the test this stuff is technologies that they’ve had chance to practice with, not something that you’re –is getting thrown at them for the first time during test day.  If you’re testing a student’s decoding ability, you don’t give decoding supports in the same way if you’re at your optometrists and you’re looking at the Snell eye exam, the Snell acuity chart on the wall, you’re not going to wear your glasses for that

[00:36:20]

[Lillian]  Right, because you need to know what you can’t see.

[Bob]  Yeah exactly, so there is you know, you wouldn’t accommodate that, you– I wouldn’t say to the optometrist “well my vision’s bad, I get to stand at 10 feet instead of 20 feet,” no, no.  So, similarly, there’s big questions about what does it mean to provide a read-aloud accommodation on an English test using deep questions.  Well, is it a test of– is it a fourth-grader’s decoding and comprehension skills your testing, or is it a 12th graders English Lit skills you’re testing, right?  So you’ve got to look at that intersection of the skills and abilities and challenges that students may have, and how that intersects with what you’re actually trying to measure.

[00:36:59]

[Lillian]  Ok.

[Bob]  So, starting this work at CAST almost twenty years ago, again, I realized that there are these lower hanging fruits.  There’s all these things we can do to digital content.  We can transform it and manipulate it in ways that inherently make it more flexible for students, but it goes on– it goes more than just multiple means of representation, there’s multiple means of action and expression.  So, handwriting versus keyboarding, that’s something we explored in our research.  David Rose always talks about that great example of if you want to evaluate a student, and in terms of how well they understand the causes of the Civil War, often we’ll have them write an essay.  Well, let’s open it up, why not let them choose the way that they want to represent, they want to demonstrate their knowledge and skills about the causes of the Civil War, so maybe it’s they want to write a song, maybe they want to do a dramatic recreation, maybe they wanted to create a diorama, etc etc.  And to the extent that institutions and schools will allow for portfolio assessments of students, there is an opportunity for you often for more flexibility for students to show their best work.  Then there is always the biggest challenge, of course, comes from how do you equate these things.  If it’s a multiple-choice test and everybody gets to same thing, it’s really easy to equate it right, you got six out of ten right, I got four out of ten right, you’re better than me.  But if you’re comparing your play that you produced, with my essay, how do you compare those.  So there’s still a lot of hurdles to tackle, that’s a good mixed metaphor.  There’s still a lot of a lot of challenges to solve, but even now with the way we do things, where we think we have– we’re giving students equal opportunity to demonstrate their skills and abilities because everybody’s got the same test booklet, they’re not– it’s not the same situation, again for everything ranging from whether or not the student had brought their glasses that day, to whether or not their decoding skills are working for them on the science test.  So we don’t have as much equivalency as we’d like to believe we have already.

[00:39:20]

[Lillian]  And, are you also incorporating that flexibility in assessments in– what are, what are some of the ways that you are incorporating that flexibility and assessments now in that diverse learners consulting that you’re doing?

[00:39:39]

[Bob]  So, there’s so many reasons why we can, and so many reasons why we do assess, and so often we skip those.  Are we assessing students so that we have an idea of how effective their teaching is, how effectively they’re being taught, are they being given good adequate opportunity to learn?  Or are we evaluating what we should next do for them, what should be the intervention, what should be the end, what are the instructional recommendations.  So I’m talking largely of summative versus formative.

[00:40:08]

[Lillian]  So, a summative assessment would be what the student has learned, and the formative would be what they need to learn next, is that right?

[Bob]  So yeah, and in fact, in formative assessment it’s, I think is a fairly unfortunate term.  It really isn’t assessment, it really is about instruction.  Proper formative assessment is not a test, and then I bristle every time I hear somebody say something like “formative assessments,” no, these are not things.  This is an instructional process by which we are gathering information about student progress, looking at what they’re learning, what learning has occurred, looking for evidence of learning, evidence of challenges, and using that information to inform subsequent decision making, instructional decision making.  That’s true formative assessment, and ideally it’s happening at a very small scales.  Happening in the classroom, with the teacher developing the tools, developing the assessments, and the student’s highly involved in this process as well. So the students are getting feedback.  I mean this happens all the time in gaming, we get instant feedback.  We so often know what we did right, what we did wrong, what we might want to practice, it happens in sports all the time, right.  So, this is all– these are all examples of true formative assessment, to the extent that formative assessment as a process involves a measurement piece, we want to make sure that that measurement is accurate.  In the same way with that we want to for summative assessments.  So that’s where I always start, is looking at why–what are we going to do with the information that we get back, who is it– who is it geared toward, and as important as it is that the evaluation itself be accessible, the way that the information is provided to the stakeholders needs to be accessible.  I’ve seen way too many interim assessment and quote-unquote “formative assessment solutions” out there that provide these very rich and colorful diagnostic reports for educators, and the educators look at it and go “huh?”

[00:42:13]

[Lillian]  “What does this tell me?”

[Bob]  What is it telling me, what am I supposed to do with this information?  And so, as of late– the last several years, I’ve been looking more and more at the teachers’ experience from their perspective of Universal Design for Learning, because not all teachers are the same. 

[00:42:31]

[Lillian]  What?! Not all teachers are the same, too?!

[Bob]  They’re not all going to– so the idea that we can present these incredibly rich dashboards to teachers of all this information about students that that they’re going to know what to do with, it effectively– some will, some won’t.  Some are going to need supports and training and scaffolds, others won’t.  So can we allow teachers to start making choices about what’s the best way of representing this information?  What’s the best way of manipulating and interacting with this information?  What’s the best way of getting excited about this information so that it can be used effectively.  So I really like to apply the UDL lens for starting from 30,000 feet and looking at the entire process, and then sure, one of the places I also often end up is “oh yeah, the multiple choice question, and how do we make that more accessible for students?”  But more and more, I’ve been focusing on the bigger picture, the bigger ecosystem of the assessment, and in the cases where the assessment results are going to be used instructionally, let’s look at the instructional piece as well.  And again, the role of the teacher in this.

[00:43:38]

[Lillian]  Well thank you very much, Bob, for joining me on the Think UDL podcast, and I really appreciate your time talking to us, and letting us know about what you’re doing and what is going on too in the workforce, in higher ed, with assessments, and all the things that we were able to cover today.  So thank you very much for joining us.

[00:43:58]

[Bob]  You’re very welcome, Lillian, thanks for having me.

 [Music]

[00:44:08] [Lillian] You can follow the Think UDL podcast on facebook, twitter, and instagram to find out when new episodes will be released, and also see transcripts and additional materials at the ThinkUDL.org website. The Think UDL podcast is made possible by College STAR, the STAR stands for supporting transition, access and retention in post-secondary settings, and the website provides free resources and instructional aids based on UDL principles. If you’d like to know more, go to the CollegeSTAR.org website. Additional support for the podcast is made possible by Appalachian State University, where, if you call it Appalachian, I’ll throw an apple at you! The music on the podcast was performed by the Odyssey Quartet, comprised of Rex Shepherd, David Pate, Bill Folwell, and Jose Cochez. Our sound engineer is Tanner Jones and our social media coordinator is Ruben Watson. And I’m your host, Lillian Nave, thank you for joining us on the Think UDL podcast.