Welcome to Episode 127 of the Think UDL podcast: Self-Efficacy Research with Ana Redstone. Dr. Ana Redstone has decades of experience as an instructional designer developing online and hybrid courses. She leads UDL and accessibility strategy at Western Governors University and recently completed her PhD in 2023 in Instructional Design and Technology at Old Dominion University with her dissertation entitled, Investigating the Effect of UDL on Learner Performance, Engagement, and Self-Efficacy. In today’s episode, we delve into her research to see what exactly the research shows and if her hypothesis were correct. She looked at student performance through grades, and student engagement and self-efficacy using various LMS metrics. I am thankful for this conversation as I am often asked about the research on UDL in Higher Education and Dr. Redstone brings us one more case study in a burgeoning field.
Resources
King-Sears, M., Stefanidis, A., Evmenova, A., Rao, K., Mergen, R., Owen, L., & Strimel, M. (2023). Achievement of learners receiving UDL instruction: A meta-analysis. Teaching and Teacher Education, 122.(https://doi.org/10.1016/j.tate.2022.103956)
Link to Ana’s dissertation: Investigating The Effect of Universal Design For Learning on Learner Performance, Engagement, And Self-Efficacy
Socials:
Or on X (formerly Twitter): @UDL4HigherEd
Transcript
39:33
SUMMARY KEYWORDS
UDL, learners, instructor, research, learning, self-efficacy, grades, students, work, study, redesign, instructional designer, higher ed, engagement, online, UDL principles
SPEAKERS
Lillian Nave, Ana Redstone
Lillian Nave 00:02
Welcome to think UDL, the universal design for learning podcast where we hear from the people who are designing and implementing strategies with learner variability in mind. I’m your host, Lillian nave. And I’m interested in not just what you’re teaching, learning, guiding and facilitating, but how you design and implement it and why it even matters. Welcome to Episode 127 of the think UDL podcast, self efficacy research with Ana Redstone. Dr. Ana Redstone has decades of experience as an instructional designer developing online and hybrid courses. She leads UDL and accessibility strategy at Western Governors University, and recently completed her PhD in 2023. In Instructional Design and Technology at Old Dominion University, with her dissertation entitled, investigating the effect of UDL on learner performance, engagement and self efficacy. In today’s episode, we delve into her research to see what exactly the research shows and if her hypotheses were correct. She looked at student performance through grades and student engagement and self efficacy using various LMS metrics. I’m thankful for this conversation as I am often asked about the research on UDL in higher education. And Dr. Redstone brings us one more case study in a burgeoning field. And I thank you for listening to this conversation on the think UDL podcast. Thank you to our sponsor Texthelp, a global technology company helping people all over the world to understand and to be understood, it has led the way in creating innovative technology for the workplace and education sectors, including K 12. right through to higher education for the last three decades. Discover their impact at text dot help forward slash learn more, that’s l earn m o r e. So I’d like to welcome my guest Anna redstone to the think UDL podcast today. Thank you, Anna, for joining me.
Ana Redstone 02:38
Thank you for inviting me.
Lillian Nave 02:41
I’m really glad to get the chance to talk about your research. That’s something I get lots of questions about, about the UDL research and then people who kind of misconstrue what the research says. So that’s why I wanted to talk to to you and get the real understanding of your recently published dissertation. And I’m gonna start out with my first question, what makes you a different kind of learner?
Ana Redstone 03:08
Well, first of all, I don’t know that I am a different kind of learner. I think we are all different learners at different times. But I’ll say that I prefer to process information by reading and writing text. I’m mostly an introvert. So I am kind of a nerd in that way that I like to read and write by myself. And part of that is just because of the way that I was raised. I was raised in extreme poverty in a very small town of less than 300 people in southeast Virginia and I was homeschooled for a lot of my elementary and middle school years. The homeschooling that I had was really unstructured. Honestly, my father and stepmother were a couple of hippies. And I was when I was a really curious child. So I became self directed pretty early on. I enjoyed I actually enjoyed reading the textbooks that I had. And because we didn’t have a TV, and we weren’t really allowed to listen to the radio very much. I really just fell in love with books. And the bookmobile visited our little town every Saturday morning, and so Saturday mornings at 10am was like Christmas for me and I would ride my bike over there and just stock up on books for the week. And that was just my way to visit other worlds and my imagination just came alive and I guess that’s probably why that is my favorite way of learning just by reading. I’ll say I also as part of learning I enjoy taking notes in physical textbooks, although I really love the the ease of ebooks on my Kindle. Have, I also really love to read real books where I can take notes and come back later and reread my notes and think who was this person taking, taking these notes and reading whatever this is?
Lillian Nave 05:17
Wow, the you bring back a whole bunch of memories and thoughts to about, you know, love reading, loving having that physical book in your hands. And I loved as I when I was little, I loved the smell of a new book and being able to go to the library or a bookstore, or even an old book, I like an old book smell. It’s such a physical and tactile thing. And it was it was just like very warm feelings for me to have those to think about. And I know you said you were you were homeschooled. And so your, your background was very self directed, which I love. And has reminded me to have one of the things that I did when I was a parent of very young children and did not work at the time. And we would have, like Christmas was going to the library, that was the best thing. And we didn’t have screens. That’s just like, you know, long before the advent of the of the screens that I mean, we had a TV but we didn’t use it. And that was such a wonderful, almost like magical idea was opening up the books and getting those new things every week. So there you just made me very warm hearted. And on the other end, when I was a kid, and I went and I was in school, I do remember the purple mimeograph, speaking about smelling like the the new book and all the thing, there was the mimeograph machine and of course, I’m podcasting so you can’t see me moving my hand and a moat and around motion that I know that I would ask like, Oh, can I do the mimeograph machine, because that purple ink would smell so good. And I would just grab the paper, just a tactile learning memory.
Ana Redstone 07:13
Oh my gosh, we’re such nerd.
Lillian Nave 07:17
So here we are talking about learning. So thank you so much. And so you have been doing more research, and research about learning. And so I wanted to ask you about what I came across your work, your recent project and your dissertation. So why did you decide to do this particular research project?
Ana Redstone 07:43
Um, I’ll say that I, I’ve always had a passion for social justice, civil rights equity. And I’ve attended marches for various causes. And I’ve even brought my kids along when possible, just to teach them that there’s value in standing up and showing up for worthy ideals. So when it was time for me to select a dissertation research topic, I was working at the time as a lead instructional designer at a University Center for Learning and Teaching. And my team had been really grappling with issues of equity and design and accessibility in the online courses that we were developing. So as part of my PhD program, I had to do a lit review, I did a lit review on Universal Design for Learning and was really surprised to find kind of what we were talking about. Before that there’s not a lot of evidence to back up the claim that UDL is really something positive for all learners. So that was pretty surprising for me. And I decided that this topic of UDL and higher education environments was something that could really align well. With my values, and interest in promoting inclusion and equity. I was really drawn to the proactive nature of UDL. You know, in higher ed, we have a lot of accessibility considerations, of sorts of things that have to be done to meet accessibility compliance, UDL kind of shifts that shifts away from that. And it’s more about proactively removing barriers to learning and so I was really drawn to that. That way to kind of holistically shift the focus onto the learner instead of the disability. Um, so I think and just for my research, I think UDL is a really important strategy for higher education institutions. Because, you know, overall All enrollments are declining. But overall online enrollments are actually increasing. And there’s also more disabled students coming into the online space. So I think UDL is a great way for institutions to, to bring in learners and keep them. And it’s going to be pretty competitive. It’s it is pretty competitive in the online space. So having courses that everyone can easily access and that meets their needs is really important.
Lillian Nave 10:42
Yeah, you have hit on a couple points that I’ve been thinking about lately. There is this enrollment cliff that is coming to higher ed, in the next couple of years that just fewer and fewer students are actually just the exist right now in that generation, because it was after a big economic decline, about 18 years ago, little bit, a little bit less but and we really need to be focused and pinpointing like how to teach this next generation because there are plenty of choices, and people can choose something else if it’s not working for them. And also the the point you made about more disabled students in higher ed, higher ed used to be only for a certain kind of student, and it was pretty exclusionary. And I agree that we have far more students coming into higher ed, and they should be, and we should be serving all of those students. And, you know, you and I are both on our UDL bus that you know, thank you. It’s really great. So I appreciate that so much. And yeah, and your your research project, I would love for you to just describe it. How many students how many classes what class or department the length of the study? How you collected the data, if you can just give us a good rundown? That would be great to start us off. Sure.
Ana Redstone 12:10
So the the goal of my research was to find out if a course a whole course, that was intentionally designed with UDL features, would affect outcomes in learners when compared to a course that had not had intentionally designed UDL features. So in effect, I had a treatment group and a comparison group. Specifically, I was looking at their grades their final course grade, their levels of engagement, as measured by interactions within the learning management system, which was canvas and their self efficacy. Yeah, it was really interested in self efficacy because UDL the ultimate goal of UDL, according to cast is to develop expert learners over time, so learners who can self regulate, and I thought it was important to look at self efficacy because self efficacy really leads to self regulation, which can potentially lead to persistence and completion. So important stuff in higher ed. The study was in three phases, I use a multi method research design, which included some quasi experimental, quantitative data and some qualitative data to answer the research questions. It took a long time to complete. I’ll tell you because I first started in the summer and the fall, it took the summer in the fall of 2022. to redesign a course. Before I could even start the study, I worked with an instructor to redesign a 300 level, online computer science survey course to incorporate UDL features. And to do that I turned to the work of a pretty renowned UDL researcher, Kavita Rao, who you may be familiar with. And she created the UDL design cycle published in 2021. And that cycle was developed to be a process for instructors who are developing UDL based online lessons. So I took that as a starting point as a way to redesign the course. But I was working really closely with the instructor as I mentioned, so it wasn’t just the instructor redesigning her own course it was the two of us together. So I adapted rails cycle and I created the UDL redesign cycle. for online courses as a framework for instructional designers to use when they’re working with instructors to implement UDL, as I mentioned, it several phases. Once the study actually started, I had already I had made changes to the course syllabus, the course modules, and one of the assessments to include UDL features. So in phase one, once the course actually started, the instructor taught two sections of the computer science course, in the spring of 2023. One, as I mentioned, was the treatment group that have UDL features. And the other one was the course that the instructor had designed and developed on her own. And she had taught that previously for a few years. So in this phase one, I collected quantitative and qualitative data using a Qualtrics survey from the two groups near the end of the semester. So the survey was basically made up of three parts, there were the demographic items, there was an adapted version of the online learning self efficacy scale. And then there are three short open ended items relating to the participants, achievement, engagement and self efficacy. So there, there were about 50 students at the beginning of the semester in each of the sections. But I ended up getting 23 participants in the treatment group and 20 participants in the comparison group, not nearly as many as I was hoping to get. It’s really, it’s tough, it’s so hard to, you know, first of all, to get an instructor who’s willing to work in this type of situation to work with you. And then to get the participants with two courses that are being taught simultaneously. Very hard, but
Lillian Nave 17:05
But you did it. And it’s also hard to get that many actually, I think that’s a huge amount of student feedback, you know, the evaluation. So that’s to get 20 or 23 out of 50. Is is, I think really good, especially for an online class. Yeah, well, I
Ana Redstone 17:21
worked with what I had. Okay, so phase two took place right after the end of the semester, and I collected the quantitative, final grade, quantitative data, which was the final grade and the participation data from Canvas. And then the last phase was about a week after the course ended, and it was included a semi structured interview with the instructor. Okay.
Lillian Nave 17:55
Wow, that’s a lot. So, what was interesting, too, is that this is computer science. Right? And so, this is, I don’t know, what sort of change you said you made like one change, like in a assessment, or, yeah, something like that. What sort of intervention or change was made in that computer science assessment that was a UDL intervention? Sure.
Ana Redstone 18:25
So as I mentioned, I made changes, actually to three parts of the course, the syllabus, the course modules themselves, and then the assessment. So the assessment that I changed, and I just want to make a note here that because of the comparison, I was looking between the courses, I didn’t want to change the gradings, Ema or, or too much of the assessments to make them as comparable as possible. So the assessment that I changed was students were required to post what was called the instructor called Blog. But it was really just a written assignment, reflecting on the topics covered that week, and it was uploaded to Canvas where only the instructor or the teaching assistants were able to see it. So not interactive at all. And there was actually no student to student interaction in the course. So what I did was I changed that blog and made it a discussion board. And so that was in the treatment group.
Lillian Nave 19:38
Okay, great. Wow. So, okay, so you this is a lot of work to go into the redesign and then pulling your data. So you had a couple of research questions and some hypotheses about this. Can you tell me about those?
Ana Redstone 19:58
So my first research question Didn’t was are there differences in academic achievement measured by final course grades between an online asynchronous course designed with UDL principles and an online asynchronous course without UDL principles, and I hypothesized from the research that the final course grades would be higher in the treatment course the one designed with UDL principles. My second research question was about engagement. Are there differences in engagement between the courses and so um, specifically, it was looking at percentages of assignments turned in on time and Canvas number of page views, the number of participations and so the all those I was counting as engagement. Okay, so I hypothesized that the treatment group would have higher levels of engagement. Okay. My third research question was about self efficacy. So would the treatment group have learners reporting higher self efficacy scores after controlling for GPA, age and prior experience taking asynchronous online courses? And I hypothesize that there would be higher self efficacy in the UDL treatment group. And then the last one was just about the the instructor. And it was just asking what were the instructors perceptions of the course that use the UDL principles compared to the one that didn’t have the UDL principles?
Lillian Nave 21:48
Okay. Did you have a hypothesis about that one?
Ana Redstone 21:53
No, that was a qualitative one. So there was no hypothesis.
Lillian Nave 21:58
Okay. Okay. So here’s the interesting part. I want to know what the results were and what can we learn from the results of this study? All right. So
Ana Redstone 22:09
this is the exciting part. Yeah. So I found that there was really no effect on learner achievement between the groups, they roughly had the same group final grades. There was a positive effect on engagement. And I attribute that to the fact that they were interacting within the within Canvas much more often, because they now had to post their discussion and reply to other learners in the class. And then the greatest finding was that there was a strong positive effect on self efficacy, after even after controlling for, as I mentioned, the age, the GPA and the number of online courses previously taken.
Lillian Nave 23:07
And what did you before he asked about that self efficacy again? And what about the instructor? Like that was your last question, thinking about how did the instructor perceive of this whole thing? What what did you learn from that interview
Ana Redstone 23:27
at the oh, this was really interesting, because part of the redesign, and as I mentioned, I’m not going into that. But that was a really interesting redesign with that, I did collaboratively with the instructor. And the first thing that we did was to identify barriers that learners were going through. And so trying to remove them with the features that were implemented. So she found she she mentioned roughly the same things that she didn’t see that her learners in either course, were struggling particularly, or their grades were any different. She did notice a lot more engagement in the UDL treatment course. And when it came to self efficacy, she also noticed that the learners who were in the UDL treatment course seemed to have much more self efficacy. So for example, they were not coming to her or the TAs with a bunch of questions about how do I do this? Where do I locate this within the course and so on, so they were able to, to really feel more confident and how they were approaching the course.
Lillian Nave 24:53
That is fantastic. That reminds me of Ken Baines book What the Best College students do. He has a whole chapter in that about routine experts and adaptive experts. And it’s just the difference in the students who really just want you to tell them what to do. And then, you know, they’ll do it. But there’s none of that internal motivation. And the adaptive expert is the is really the, the self efficacy in motion, like you are given a new task, and you are curious enough, and you’ve got the tools to figure out how to solve that rather than waiting for someone to tell you how to do it, you know, that it’s got that transferable skill. And of course, that’s what we want in higher education. Like that, to me is one of those things that the college diploma, that credential says to the world is that, you know, I can solve problems like that, like, really, so much for the last 2040 forever. Yours was about okay, well, somebody who’s college educated is somebody who knows how to kind of adapt and has these transferable skills. I’m in the general education program at my university. So it’s those things like communicating effectively, and, and being able to be a critical thinker and that sort of stuff. And it looks like this study has really helped us to drill down into how that happens, instead of just rote memorization, or demonstrating certain skills at a certain time and being graded on those the ability to actually solve a novel problem, which, yeah, also seems to be like the point of computer science, but I don’t know. I don’t know exactly all about computer science, right. Yeah. So, this is interesting. The, the self efficacy part, as you mentioned, is what one of the things that cast who has designed and put out the UDL guidelines says is the is the whole point is to create learners who are motivated, who can solve problems on their own, who can become expert learners become those adaptive experts. And you, really, from the study, I think, made an important difference. And this is the part that I thought was sort of misconstrued when I first came across this on the interwebs. The idea of okay, but the grades were exactly the same. Alright, so here are two different courses. And we’re looking at the outcomes. And the grades are the same. And so some might say, Oh, it made no difference. But I’m seeing it, I understand how you’re, you’re seeing the big difference here. What would you say? If somebody comes and says, Well, I read your dissertation, they all ended up with the same grade. So what’s the point?
Ana Redstone 28:06
Well, I’ll I’ll say what you heard here about self efficacy, and how important it is in college learners, and, and specifically computer science learners. And I’ll say that this is a really small study. And so it’s hard to really see when you’re looking at small numbers, it’s hard to see a big difference. Which makes the big the huge difference between self efficacy even more powerful, you know, seeing that that difference between the groups and such, you know, to such a degree, really speaks to that. And I will say that some research came out 2023, and I’m sorry, I don’t have it here in front of me. But there’s some research out that does point to, you know, that higher higher academic achievement in in learners, it’s a meta analysis that looks at a bunch of empirical studies, and says, you know, what, it does actually help learners achievement, but just not necessarily in this study.
Lillian Nave 29:17
Yeah, gotcha. And we can grab that and put it in the resources for this episode. No problem. Yeah. And what that also signals to me, as you said, you know, this is a small study, and we saw incredible results in that self efficacy. But if we were to do this study again, and we did it on 300 students or 1000 students, we might also see that achievement gap or that or a difference in achievement, bolstered by that self efficacy. And it also makes me think, you know, grades aren’t everything. Grades are not the best determiner of how much somebody has learned. In
Ana Redstone 30:00
a class, certainly not.
Lillian Nave 30:04
And I have lots of people I love talking to about that. But that is another podcast episode. In fact, I was just thinking about that earlier today about how grades really hide a lot of what it’s telling us, you know, it might be telling us how compliant students are, or how timely they were to hand in things, but may not actually tell us everything about how they learn and if they can be that adaptive expert and critical thinker when they when they come out of, of that class or have or out of the university. So, yeah, so we can learn a lot from this study, one of the things I think we’re learning too, is that we need more studies, right? Even with just this small amount, we have great, great results. But there’s so much more. And that’s like one of the reasons I I called you in on this on this call, was to say, look what’s happening now. And we we need more, and to shine a spotlight on this that every study is, I think, really helpful in us understanding why UDL matters. And it is not the easy. It’s not the easy quantitative data that says oh, look at how our DFW rates, right, the D’s and F’s and the withdrawals have improved, because we’ve included UDL which happens. But it’s about that internal complicated, who we are as a learner, how we’re getting better at this expert learner kind of thing. And it’s only with studies like you have designed and done that we really begin to see that. So you, I think this is inspiring, you would inspire me to like, wow, I should do a study like this, we should do this at our university. So if somebody wanted to do this and says, You know what I agree with on? I think we need more research on UDL in higher education, what can I do? What recommendations do you have for implementing UDL, in online courses and for future research, so we could have more of us nerds doing this?
Ana Redstone 32:24
Sure. Well, first, I’d like to say, you know, there are different ways that you can implement UDL, a lot of instructors who are interested in UDL, kind of start off on their own, and start small and add some basic UDL features. And then sort of get feedback from their learners and then adjust. So there’s there’s that approach. But for instructors who are working in higher ed spaces where they have access to instructional designers, or team, you know, there’s often a cycle of design and redesign. And so if an instructor has the opportunity to work with an instructional designer, to redesign the entire course, that could really bring in more UDL features, and offer an opportunity to to really get that feedback from learners and, you know, iterate on the design to make it even better at removing barriers. If, you know if researchers in that could be the instructional designer, it could be, you know, instructors, it could be, you know, faculty, anyone who wants to research UDL, they can use that redesign cycle that I created to to really collaboratively work with an instructor to comprehensively to redesign a course with UDL features, and it does take time in that case. So you know, take advantage of that built in redesign. That often happens in higher ed. So use that time to add UDL features.
Lillian Nave 34:19
You know, you said it takes time that redesign time, but you also said what you found from the research is that when you had your exit interview, are talking with the instructor at the end, those students during the class and there are 50 students in this class did not have to ask more questions like that. So there were there were fewer times when the student had to come in for office hours or send emails which is less time that the instructor has to reteach, whatever the lesson is or clarify I write because the students are catching on, they’re getting it and they kind of know how to do the next steps or they feel empowered to do the next steps. So it seems to me that that would be less time the, the professor or the instructor has to spend kind of rehashing things isn’t Am I getting that? Yes,
Ana Redstone 35:20
absolutely. You know, implementing UDL is like any kind of design project, you put a lot of work in on the front end, but then you reap the rewards, you know, after the course has been launched. And so, yes, that’s you pointed to a really important consideration for instructors is that they do, they will have students who their barriers removed, and they’re able to find what they need to and, you know, get through the course and do well. I’ll just mention too, that, as far as research, you know, we need a lot more empirical research. UDL in higher ed is pretty new, it was a K through 12 initiative for years. And and so in higher ed, especially online spaces, much more Imbert periodical research needs to be conducted. And I think that, you know, this is a starting point for, for UDL research at online spaces, and there’s much more that can be done.
Lillian Nave 36:31
Absolutely. That’s like a call to action. I love it. That’s a, it’s a great way to end our conversation is to say, you know, who’s next, who’s going to bring the next research project and and see what we can do. And, you know, you’re right, it was a total understatement that you said was, there are lots of ways that you can employ UDL. And of course, there’s just millions, you know, so many different ways that you can increase the engagement that you can provide multiple means of representation. I mean, there are just so many interventions, you can start with just a few. And I appreciate how you pinpointed just the few, and how much of a difference that really made for our students. So thank you so much for your time and discussing your research project. And yeah, bringing this for all of us to consider and then, I think spurring us on to what we can do at our institutions and what we can do for our students to lower those barriers. So thank you so much.
Ana Redstone 37:34
Thank you, I really enjoyed this
Lillian Nave 37:43
you can follow the think UDL podcast on Facebook, Twitter, and Instagram to find out when new episodes will be released. And also see transcripts and additional materials at the think udl.org website. Thank you again to our sponsor, textile Texthelp is focused on helping all people learn, understand and communicate through the use of digital education and accessibility tools. Texthelp and its people are working towards a world where difference disability and language are no longer barriers to learning and succeeding, with over 50 million users worldwide. The Texthelp suite of products includes read and write a Casio and orbit note. They work alongside existing platforms such as Microsoft Office and G Suite and enable them to be integrated quickly into any classroom or workspace with ease. Texthelp has changed the lives of millions worldwide and strives to impact the literacy and understanding of 1 billion people by 2030. Visit text dot help forward slash learn more that’s l earn m o r e to unlock unlimited learner potential. The music on the podcast was performed by the Odyssey quartet comprised of Rex Shepherd, David Pate, Bill Folwell and Jose Cochez and I am your host, Lillian Nave. Thank you for joining us on The Think UDL podcast.