The Foundations: Theory of Change, Logic Models, and Evaluation Questions
The first webinar of the three-part series begins with a brief review of the six key elements that should be included in every evaluation plan. The webinar will then dig into the foundational components of a program evaluation: theory of change, logic models, and evaluation questions. We also encourage you to watch Part II: The Methods and Part III: The Finale.
Welcome everyone. Thank you so much for joining us today. We’re so excited to have you with us and hopefully you have a nice hot cup of coffee or a cold cup of coffee with you and ready to learn what I hope will be a lot of great information. before we get started, I do want to take care of a couple of housekeeping items. First, it’s really important to us that you have an opportunity to ask questions, so make sure they use the question function on your computer to be able to do that. I should also note that we have our fabulous intern, Alyce Hopes, with us today to help moderate the Q & A portion which is dispersed throughout the webinar as well.
Let me also introduce myself. I am Lana Rucks, principal consultant of The Rucks Group. The Rucks Group is a research and evaluation firm that gathers analyzes and interprets data to enable our clients to measure the impact of their work. We were formed in 2008 and over the past several years we’ve had the privilege of working with a number of clients, primarily in higher education and grants funded by federal agencies such as the National Science Foundation, Department of Education, and Department of Labor, as well as projects funded by foundations such as The Gates Foundation, and the Kresge Foundation.
If you’re on this call you may have some familiarity with the concept of evaluation, still I think there’s a lot of utility and benefit of providing a definition because there are a lot of different ways that people think of evaluation. The definition that I gravitate to is this one: evaluation is the use of social science methods to systematically investigate the effectiveness of social intervention programs. Now because evaluation comes out of a research tradition, you could kind of understand why there’s a lot of different terms, concepts, and words that are kind of difficult in terms of understanding what they mean, let alone really understanding where they are actually applicable. That makes it very challenging when trying to write an evaluation plan, for grant proposals, particularly, when evaluation has increased in its importance within the grant funding space.
That’s why there are a couple of purposes throughout this webinar series that I want to accomplish. First, I want to be able to organize evaluation terms around a framework – help you to understand conceptually where each term falls, and then using that to be able to provide a formula for actually writing evaluation plans. Taking that together, hopefully the other piece that we’ll be able to do is to reduce some of the evaluation angst. Instead of seeing some evaluation as a nuisance and something that you have to do that’s actually in the way of a project, being able to reduce that evaluation angst so you can see the value of evaluation and how it actually helps you to achieve your goals. Finally, we want to make sure that we’re able to answer your questions.
Let me start by talking about the organizing framework and the organizing framework for an evaluation plan. This is how I think about these elements of evaluation. I think of the first two pieces as “the foundation” and that’s where we’re talking about the theory of change and the evaluation questions and that’s really the purpose of today’s webinar. The next two I think of as “the methods”. That’s the evaluation design, data gathering, and analysis and that’s going to be what we will we will review in two weeks. The final piece will be “the finale”, the use of evaluation findings and the operational approach.
One of the things that you may be thinking, again if you’ve been in this space with evaluations, that I like the concept of things just kind of falling within six elements, but is that really true because it feels like there’s this real kind of variation if I’m working with a small brand versus a large grant it gets really confusing in regards to what i should put in and what I shouldn’t put in. That’s a completely legitimate point. The way that I want you to think about this is to think about an accordion and think about that if you’re working with a small grant and a small scale you push in the elements and we’ll talk about how to do that.
So, for a small scale, for each of those six different components, you may just include a sentence so the resulting evaluation plan may just be a paragraph. For a large scale however, you may kind of pull out and you’ll have more detail in certain elements. So, you’re talking about it with several sentences or a paragraph and you’re adding much more detail. Fundamentally for the evaluation plan, we’re still just talking about six elements, but what’s going to vary will be the amount of detail. I did want to provide at least the way I think about assessing scale so you can understand what I mean when I say small, moderate or large scale (Because scale is really relative it kind of depends on what projects that you’re used to working with or used to working on). When I say “small”, what I’m saying is that the funding amount is about 130 thousand per year and that in that evaluation plan, typically, there are limited words for the evaluation, so there’s not a lot of space for that. When you start getting to moderate scales, somewhere between 150 thousand to one million in terms of funding per year and there may be more explicit requirements for the external evaluator or around the evaluation, and you’re given more space to actually describe what you’re going to do with the evaluation. For a large scale, I’m talking about a funding amount of maybe a million or more and more detailed. Again, you may have a different definition of small, moderate and large, but at least you’ll get a sense of what I’m talking about when I’m referring to scale.
Let’s go ahead and jump in and talk about the first concept, that of the theory of change. When we’re talking about theory of change, what is it that we’re talking about? One way to think about it, it’s the general expected outcomes of an initiative. The other part of that definition I really like, is that it’s the project’s hypothesis or what outcomes will be achieved and how. One way to think about theory of change is in a very simplistic standpoint of what you’re going to do and what are you going to get. Essentially what you’re saying is “if we do this then we will get that”. So, the first step, in terms of formula, is that you want to be able to summarize the theory of change in a sentence. You want to be able to condense down what you’re thinking about the project and really be able to have sentence around that theory of change. So let me walk you through a few examples:
In this first example, this is a this is a Department of Education project that we are working on, it’s funded through the Undergraduate International Studies and Foreign Language program (DOE UISFL) and the project’s theory of change basically asserts that “if the Global Studies program at XYZ College is created then students will be better able to think critically about pressing global challenges and better prepared for post-graduate or a professional career.” Here we have our “if” portion and our “then” portion.
Here’s another example, again, this is a Department of Education Minority Science and Engineering Improvement grant (DOE Improvement) that we would consider a moderate-sized grant. In this project, the project posits that “if early academic assistance through professional and peer mentoring, supported by networks that remove non-academic barriers are provided then there will be increased student performance, graduation, and STEM workforce rates.
Let me wrap this portion up with an example of a large scale. This is the Department of Labor TAACCCT grant that many community colleges were participating in. In this theory of change we wrote it this way, “the theory of change undergirding this project is that if TAA-eligible workers, veterans, and other workers are provided an opportunity to quickly advance their credentials in the field with job openings, in this case as a mechatronics technician or industrial machinery mechanic, then participants will increase their earnings and employability”
So, you can see that in three different types of examples with three different types potential complexity, there was this attempt to really be able to distill what the project is doing down to a sentence, and that’s the first piece of the formula in terms of writing an evaluation plan.
With continuous improvement there are a lot of ways in which developmental evaluation really can help in terms of how to deal with and address new initiatives and new projects. Let me give first a definition of what developmental evaluation is: developmental evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments. There are a lot of words there so let me tease that apart for a moment.
When we talk about innovation for a lot of the projects that we work with, these are new projects – new initiatives in some way. Even if there’s some piece of the project that’s replicating something that’s been done previously, it’s being implemented in a new context because it’s being implemented a new part of the country, or perhaps it involves different individuals or different majors. So, it’s really talking about this newness, this idea that an individual may have, and they’re trying to figure out how to implement.
The other piece is adaptation to emergent and dynamic realities. If you’ve worked in the project, you know that there’s a really different experience in terms of what you write when you’re developing your proposal and what those realities are when you’re actually implementing. A developmental evaluation framework helps to create a way of thinking about how to approach and handle these emergent components. And “emergent” can be big or small. They could be small things like you have a shift in an individual who’s a champion and the organization for the initiative and so that person has left the organization and somebody who comes in is not quite as enthused. Or it could be big like having to deal with COVID-19 and social distancing – how do you deal with those emergent factors? We know that we have a lot of projects right now that have either summer camps for students or project development for teacher and faculty members and are trying to figure out how do we address these emergent and dynamic realities.
The other piece, these “complex environments” is that you’re putting that initiative in the system, it’s organic. When you put in an initiative you have to be sensitive to those complex factors and sensitive to that complexity.
So, one way to help think about this is in framing it in regards or comparing this against the processes that are associated within research and how you think about research. A lot of times you hear this overlap between the commonalities of research and evaluation and there are many commonalities, as I said, in terms of those processes. In a research you have a hypothesis, you develop research questions, you test the hypothesis, and then you analyze the findings (And then importantly, you refine those hypotheses). Well in evaluation it follows a very similar process; you start with a theory of change about how an initiative will work, you develop evaluation questions, you test that theory of change through the project implementation, you analyze your findings, and then you refine your theory of change through the implementation process.
So, in thinking about evaluation from this developmental evaluation lens, the engagement of evaluation and the life cycle of the project will be slightly different. In thinking about the activity life cycle of a project, you have these different decision points and if you’re not thinking about evaluation as being an integral part to help with continuous improvement or form developmental evaluation, you may not be as focused on aligning evaluation to those decision points. So, the evaluation and the decisions of the projects are not in alignment. Whereas, within the developmental context, what you want to make sure you’re doing is within those decision points, you’re also closely aligning the evaluation to be leveraged to help with really good data informed decision making processes.
Let’s look at an example of this:
In another project that we were working with the project team held an outreach event to students and their parents to introduce them to a new major that the school was implementing and it disseminated out the survey at the end. When they disseminated the survey, they had about a 9% response rate. Well, that’s not really the response rate that you really want – you want something much higher than that. So, we had some discussions around that with the project team and about what could be done to change a dissemination method. So, there were a lot of tweaks with that and in a similar initiative that occurred about a month later, we implemented these changes and had about a 40% response rate. Now you could say this is really small because it’s just increasing the response rate on a small initiative, but by having larger representation of their population and the people that were there, they now have better data to be able to make decisions and to incorporate those stronger decisions moving forward and that will help them to achieve their outcomes – and that’s really the point of developmental evaluation.
Very often in dealing with evaluation and evaluation issues, you’re really hopeful that people are enthused and excited about that topic and sometimes they’re just not – not because they don’t want to have that informed information, but for a host of reasons. So, instead of people looking like this, sometimes they’re looking like this. So, the question for people when you’re actually sold on the idea and you believe in using evaluation for continuous improvement purposes, is “how do you get other people to buy into it as well?” Here are a couple of things that you can do:
The first item is in regards to making sure that you’re really linking out the evaluation to a larger frame, making sure that it’s linked to you a greater mission or achievement – that could be the larger mission of what a project is or actually on a much larger scale it could be linked to the mission of the organization as well. When that’s linked to a mission and something bigger, then it’s not feeling as though it’s something extra and something that has to be added on top of everything else that has to be done.
The other item that you can do is a stakeholder analysis. One benefit of an actual stakeholder analysis (and it doesn’t have to be done in a highly formal way it could be slightly informal) is understanding that every stakeholder, has a stake. Every person who’s invested has something that they want out of that and they have slightly different perspectives. You may have the P.I., a project owner, administrator, institutional researcher, program officer, or funder and everyone has a different perspective. So at least being very conscious of what their perspective is and what they may want out of the evaluation, is really helpful in regard to how you frame that message. While this sounds like a lot of extra work, really in practice it’s not. It’s really just taking the perspective of the other person and helping them frame a message in that regards for them.
The other piece I think that’s useful too, is making sure to listen for pain points. Make sure that you find those moments that are “culturable” or learning moments as well. One of the ways that I think about this is in the old adage “You can lead a horse to water but you can’t make him drink.”. I often tease and say, “but you can’t put salt in his hay”. The idea is, once someone has salt in their hay they’re a little more thirsty for water. In the same idea, when there are some pain points with the project and helping to show how evaluation to help in terms of addressing those pain points, it’s much more likely to be able to be persuasive with why evaluations are of value and why evaluation can help for continuous improvement purposes.
I think that we are pretty close to time here so I’m going to go ahead and stop. I just want to say thank you so much for joining us today, I really hope you were able to get a lot of great information from this webinar experience. You will receive an email with information on a survey and if you can complete that survey, we would really appreciate it. We are interested in our own continuous improvement processes and= so I would really like to have that feedback so that we can make sure that we’re offering other resources that would be helpful for you all. I hope you have a great rest of the day. Thank you.
