Evaluators play an important role in framing and facilitating the evaluation but Identifying the right evaluation partner can be quite challenging. As such, there are many issues to consider when you attempt to identify an evaluator such as finding an entity that aligns with the philosophical approach of the project team and the projects’ needs, working with someone who is responsive and overall qualified, and the management of the internal procurement policies. Because of these varied factors, there are many opportunities for the process to go awry, leaving you feeling apprehensive about the entire process.
In this coffee break webinar, Dr. Lana Rucks will draw from her near 20 years of experience in program evaluation to provide five best practices and tips that can aid in the process and ultimately lead to a successful working relationship with an evaluator. All grant writers and project personnel are encouraged to attend!
For the most beneficial experience, we strongly encourage you to review the Now What? How to Use Program Evaluation Findings for Project Continuous Improvement and Developmental Evaluation: What is it? How do you do it? webinars where we introduce key concepts appearing in this webinar.
Additional Resources:
Welcome everyone so excited that you’ve decided to join us today in our next installation of our Coffee Break Webinar, 5 Best Practices for Hiring the Right Evaluation Partner. Before we get started let me go over a couple of housekeeping items:
It’s really important that you have the opportunity to ask questions so make sure to use the question function on your computer to be able to do that. As always, we have Alyce Hopes, our Outreach Coordinator, with us today helping to moderate the Q & A portion as well as addressing any computer issues you may have. If you don’t know me let me introduce myself; I’m Lana Rucks the principal consultant
with The Rucks Group. The Rucks Group is a research and evaluation firm that gathers analyzes and interprets data to enable our clients to measure the impact of their work. We were formed in 2008 and over the past 12 years, we’ve worked with primarily higher education institutions and grants funded by federal agencies such as the National Science Foundation, Department of Education, and Department of Labor, as well as foundation-funded work.
To create the context, or set the stage for our conversation today, I want to talk about some elements that I think make hiring an evaluator challenging; and this is from the evaluator’s perspective of things that I’ve seen. I think one element that makes hiring an evaluator really challenging is just maneuvering through the contracting process. If you’re familiar along with program evaluation you will understand that securing the services of an evaluator is really different than purchasing a pen – it’s a really different type of relationship. The contracting process sometimes really doesn’t map onto that type of relationship that you’ll have ultimately with an evaluator.
Another challenging piece could just be that the project team doesn’t have a lot of experience or knowledge around evaluation so there’s not a lot of understanding of what they even should be expecting in working with an evaluator. Thirdly there’s no certifying organization for
evaluators unlike if you’re looking for an accountant you would know to look for a CPA or a certified public accountant. For evaluators really anyone can hang out a shingle and call themselves an evaluator, so it’s really hard to be able to tease apart the individuals who have the skills to be able to really effectively serve as an evaluator. The final element that I’ve seen to be really a challenge is that the project team doesn’t even know until very late within the process they even need an evaluator.
Now, if those aren’t challenges, let’s say that you know that you need an evaluator and you go in google and you enter in “program evaluator”, you’re not going to get a list of people who do program evaluation; what you’re going to get instead are information about jobs and program evaluation. All of those together really help to create the reason why we wanted to have a webinar talking about the hiring process. During our time together what we’re really hoping to be able to do is a couple of things:
We want to provide some best practices on selecting and working with an evaluator. Towards the end of the webinar, I’m going to highlight some additional resources that you can cross-reference. In thinking about those resources, the webinar will try to bring to life some of the recommendations that are highlighted in those resources and how best to be able to go through the selection process. We also want to demonstrate this process really lays the foundation for a really great working relationship with an evaluator. If you’ve seen any of our previous webinars, we talk a lot about the continuous improvement process because from our perspective, evaluation isn’t conducted just to check a box, it’s conducted to really try to provide value to the implementation process of a project. You may have also seen throughout our webinar that we are always trying to reduce the angst around evaluation. In this context, we want to reduce the angst around hiring and working with an evaluator. And of course, we want to answer your questions too, so make sure to use that chat function.
With that context let’s jump in and talk about the first best practice which is really about starting the process early to allow the evaluator time for selecting and actually onboarding an evaluator. In some ways that really seems slightly self-evident but there are times where you can get really compressed and not able to have the extended amount of time to be able to choose an evaluator and that’s what we want to be able to prevent, so let me give you a couple of examples:
Very often people will reach out and ask if we’re interested in serving as an external evaluator and here’s an example of that from
“Compressed for Time”. We’ll have a little bit of back and forth and I’ll ask you know what type of decision that you make on the evaluator
and sometimes I get this type of response which is “we’ll go with your firm because you were the first to respond”. In this context that they have lucked upon a really great evaluator who will provide some really great insights and recommendations throughout the process.
I completely understand how you can get compressed for time because writing a proposal is very much like building a plane and flying it at the same time and that can be utterly challenging, but we all would admit that this is not the ideal situation in terms of how to select an external evaluator and that the criteria shouldn’t be around how responsive someone is. And that means sometimes we will win a contract and sometimes we don’t. Here’s another example:
Something like this happened where they said thank you for the response, we heard from another evaluator, so we decided to go with that group. Again, we don’t want time to be the criteria, so we want to start thinking about how we function and operate early within the process so that doesn’t happen.
Early is really relative to the stage of the grant life cycle, so let’s talk a little bit about the life cycle and what can you do with each of those
stages that are a little bit on the early side so that you won’t be in the same position as “Compressed for Time” or “In need of Someone Now”. Conceptually, I think of the lifecycle of a grant in this way;
You have your idea generation, funding search, proposal development, award management, and award closeout. I think award closeout is a little bit different than award management in that phase – you can still make changes around the grant and you can still impact the evaluation, but when you get to award closeout, you have fewer options in the types of changes that you can make and so you probably will have to just go in a direction with the evaluation where you just go with the data you have, so that’s why I tease these apart.
During the idea generation phase, one of the things that you can do is to start gathering names of evaluation entities. If you come across an evaluation report just drop that name down, if you talk to other principal investigators or project directors ask them who they’re using as an evaluator and just start to create a list. Another thing that you can do even though you’re in an idea generation is if you come across some names conduct an informational interview. This could be the case particularly if you don’t have a lot of familiarity with program evaluation. Most evaluators will be really agreeable to just have a conversation and really just being able to answer the questions and help to give some information about their philosophy around evaluation.
Moving to the funding search, there are a couple of things that you can do here as well:
One, you can start thinking about the evaluator-project team relationship, and we’ll talk about that more in a moment. Another suggestion that I would offer is that when you’re starting to look at different funding sources and trying to make decisions about whether what you want to accomplish in a project or initiative aligns with that funding source, make sure to search for the requirements for the evaluation. Here’s an example of that:
This is a solicitation for the community and people grant for research hub for the National Science Foundation and what you can do is just pull up that solicitation and do a search on the evaluation or evaluator and you’ll find a couple of really key pieces of information. This is really important because solicitations have varied expectations of how much information you should have about the evaluation and about the evaluator. In this situation you would be expected to actually have the external evaluator named but, in some solicitations, you don’t need to have the evaluator named during the proposal phase. In this situation, you also need to have a detailed evaluation plan and again in some solicitations it’s not necessarily required to have as much detail on that component. As you’re starting to do searches around grant sources, also make sure that you just become aware of any requirements around the evaluation.
During the proposal development phase, you want to if you can, (we’ll get to how you actually procure an evaluator later), involve the evaluator in developing the proposal. Ideally, you want to be able to give the evaluator some lead way that you will be reaching out or that you’ll be giving them a proposal narrative. Just giving them some information that the proposal narrative will be coming is important even if it may be several weeks before that narrative is actually complete. The benefit of giving them a heads up, is that they can make sure that they’re clearing out space and they have the capacity to be able to respond to the request.
During the award management phase, if you did identify someone in the solicitation then you want to make sure to reach out to them as quickly as possible. I know some grants are publicly announced so sometimes we will look to see if the grants that we were a part of were actually awarded so then we can do a reach up very quickly. Other times we’re just not aware of that and the project will be probably aware before we are. Again, just let that evaluator know that it was actually awarded so they can start thinking about the implementation and so you can start working together very quickly. If you don’t have someone identified, then you can go ahead and begin the procurement process right away so that early within the grant life you have an evaluator who’s on the team.
During the award closeout, if you are in need of an evaluator (and there are so many different reasons why you could be in need of an evaluator), one of the things that you can do is to make sure that you’re communicating the time-sensitive nature of the work so that that evaluator will understand that they will need additional capacity to be able to complete the necessary tasks. Also, ensure that there’s alignment in the scope of work with the closeout expectations.
The next best practice is to write up what you would like to see in the evaluation. When I say write out what you would like to see in the evaluation, I’m not necessarily talking about a formal document, I’m not talking about the treaties, I’m just talking about jotting down what your thoughts would be about working with another individual or another consultant. This can be a very informal process that you do just with yourself or maybe the project team does. This is not intended as a formal exercise and you don’t have experience with an evaluator, but you do have experience working with people. This an important step to help start understanding what you may want in terms of working with an evaluator. So, simply start with a question like, “What do I want out of the evaluation? How would I like to work with an evaluator?” In response, you may say you want someone who’s really knowledgeable about the evaluation and can help to use evaluation in terms of the implementation of the project, or that you want somebody who really understands the context in which you’re operating and really understands the nature of your program. Then you may have some other things that just would be really nice; it would be nice if that person could help in terms of making changes along the way to how to evaluate the project or make recommendations in terms of implementation, or it could be just in regard to how responsive you want the person to be.
This is an important exercise for a number of reasons but especially because when you look at some of the resources that I suggest, they will talk about having to make a decision on if you’re going to work with an external evaluator or an internal evaluator, and you’ll see that terminology come up when you look at solicitations. What is an external and internal evaluator?
This varies a little bit by solicitation and by funding program, but in general, an external evaluator is going to be someone who’s not a part of the organization or who’s at the organization but is not considered a part of the chain of command, or not working directly on the project.
An internal evaluator then is going to be the converse; they’re going to be someone who is could be at the institution or that they are part of that chain of command that is actually a part of the project team. Before you start making a decision on if you need an external or internal evaluator, I think that the previous exercise will really help you. You can take that information and create a grid around the types of characteristics that you would like to have, and you can ask “who will have those components that are important to you and your project?”. This step enables you to look at the types of skills that they have and the way that they’re going to approach the evaluation and that process will help you to decide if you need an external or an internal evaluator.
The next best practice is in regard to becoming familiar with the organizational procurement requirements.
One of the things that I have found is that there’s really a lot of variability in the contracting process across organizations and that variability doesn’t always land in regard to the size of the organization and whether it’s private or public. From my own personal experience, I think that some of that has to do with the experience of the institution and working with an evaluator. As we kind of talked about before, the evaluation relationships are a little bit different in terms of procurement and being able to work together. So understandably, there are differences in how to approach that depends on that type of experience. Let me just give some highlights in terms of the different types of processes or different types of approaches:
One could be a sole source and a sole source is when there is not a bid and that an individual has been identified as the only entity that can actually provide the services or deliverables. For federally funded initiatives, they’re guided by the common guidance and common guidance actually does allow for sole-source contracting, but that’s going to be dependent on the policies of the institution. Another approach that’s been quite popular is through a pre-qualification approach. To be able to meet the demands of needing to go quickly with an evaluation, schools will create a pool of evaluators who are already vetted and have already demonstrated that they are qualified, and either they will source that project to someone on that list or a bid will go out.
Then there’s also a competitive bid process that can occur where we’re actually requesting a formal proposal from individuals to be able to serve as an evaluator on a project. You want to know this early in the process because the organizational procurement processes are going to influence how you interact with a potential evaluator. You want to know if you reach out to them and they help you with the evaluation, are they going to be written to the grant, or are they going to have to make a bid on the pro project after the award? So, having some knowledge, whether that’s going to your back-office support or talking to the grant office, just about what that procurement process is will make you feel a lot more comfortable in interacting with the evaluator.
The next best practice is that when you’re actually interviewing or making the selection process, it’s really beneficial to not only have an interview but also have a written document and vice versa – just ensure that you have both of those pieces. When we talk about a written document, you get really different types of information than you get through an interview; in a written document you’re getting really fundamental information. And do I mean by a “written document”? It could be a quote, it could be a proposal, but it could also just be a statement of qualifications or just some element that explains pieces about the evaluation and I’ll go into that in a moment. Now, with an interview you’re gathering slightly different information – it’s more information about how you’re actually going to work together. Again, with evaluation, the rapport that you have with the evaluator is really key and really important so an interview or conversation helps to give you a sense of what it would be like to work with this person.
Here are some of the things you may find in a written document: you may get information about the individual or the firm, you’ll get some previous evaluation experience, and then you’ll also get information about the scope of work and the budget. From the scope of work, you may also get information about how frequently you’re going to engage with the evaluator as well as what types of deliverables that you’ll expect from the person, which is really important to be able to put that in context with the budget. Value is going to come not by what the bottom line or the top line number of the budget is, but that relative piece of how much engagement and how much consultation and the deliverables that will be provided.
During an interview or conversation, you’ll be able to get a little more information about their evaluation philosophy, get more information about their approach to work, and then also I think what’s really key are the types of questions that the evaluator can ask too – those questions can be really informative and this type of dialogue will help both parties really understand if working together is a good fit.
The final best practice is to periodically evaluate what you like to see in the evaluation. Like we said in the beginning you can ask yourself questions, but then ask yourself that question after having a year of experience – what would you now like to see in the evaluator you’re working with? A suggestion around the award management piece is to ask yourself some of these questions and to have an initial review with the evaluator to talk about some of these things in terms of what you’ve seen or what you haven’t seen.
Something else that you may want to do if you’re having a new relationship with an evaluator is to consider just a one-year contract with them so that you both would have a potentially graceful way of exiting, but I should say if you do that make sure that you’re constantly talking to your evaluator about the issues in the and what you want to see out of the evaluation. As the evaluator, I would appreciate that conversation, but I also don’t want someone who is not getting what they need out of evaluation to feel as though there’s not an exit for them as well.
This recording will be posted on our website in a couple of days and you’ll have also had access to a few resources that I think will be helpful to you. Also, join us for our next Coffee Break Webinar series where we’re really excited to have the Project Director for Central State University’s Stem Success Center who is going to talk about program evaluation in practice and really talk about how program evaluation has helped them in the decision-making process and in the evolution of their grant implementation.
As always thank you so much for joining us today if there is a question that you didn’t share please make sure to reach out to us so that we can answer those questions – that’s never a burden, we really appreciate being able to do that. Thank you and have a great rest of the day.
