Evaluation plays a critical role in providing feedback that allows project owners and project teams to make evidence-based decisions about the project and use evaluation findings to persuade decisions with an institution.
In our next coffee break series webinar, The Rucks Group is pleased to partner with Central State University’s Dr. Morakinyo Kuti, Project Director of the U.S. Department of Education’s funded STEM Success Center. In this webinar, we will demonstrate how the elements of evaluation coalesced to enhance the implementation of project. Participants will walk away with an enriched understanding of how evaluation through continuous improvement, can enhance in grant-funded work. These concepts include:
Dr. Lana Rucks:
Welcome, everyone. Thank you so much for joining us today for our next coffee break webinar entitled Program Evaluation in Practice a Case Study of Central State University’s Stem Success Center. If your part of the world is like my part of the world, then hopefully you have a nice hot cup of coffee with you to help you stay warm because it’s cold outside!
Before we jump in let me go over a few housekeeping items:
It’s important that you have the opportunity to ask questions so use the question function on your computer to be able to do that. We have with us today our outreach coordinator who will be helping to moderate the Q & A as well as to address any computer issues you may have. Let me introduce myself – I’m Lana Rucks, Principal Consultant of The Rucks Group. The Rucks Group is a research and evaluation firm that gathers, analyzes, and interprets data to enable our clients to measure the impact of their work. We were formed in 2008 and over the past several years we have worked primarily with higher education institutions and grants funded by federal agencies such as the National Science Foundation, the Department of Education, and the Department of Labor.
If you’ve participated in any of our previous webinars you know we often talk about how there are usually two broad motivations for evaluation; the first is “proving” or using evaluation to essentially check a box and say that it’s done. The second is “improving” which is focused on using evaluation for continuous improvement purposes. It is our belief that “proving” for accountability purposes is usually nested within improving. Today Builds on those concepts by discussing the continuous improvement cycle through an actual example. (For more information on this concept, please view our Now What? How to Use Evaluation Finding for …) Specifically, we want to show how an evaluation finding was used by the project team, so I’m thrilled to have with us today Dr. Morakinyo Kuti of Central State University (CSU) who among many roles, is the Program Director for the Student Success Center. Let me turn things over to Dr. Kuti so he can provide more information about his background as well as information about the project that we will be discussing today.
Dr. Morakinyo Kuti:
Good afternoon Dr. Rucks and good afternoon to our listening audience. As Dr. Rucks said, my name is Morakinyo Kuti. Not only do I work with Central State, but I am a proud Centralian of the class of 1985. I went away to get my master’s and a Ph.D. and came back, and I’ve been working at Central State for 29 years. Although my full-time job is the Director of the Office of Sponsored Programs & Research, I have great interest and passion in ensuring that our students’ success, so I work on several student success programs, one of which is the STEM Success Center which I will describe in a moment. We also have a scholarship program in addition to programs that provide internships and other experiential learning opportunities for students.
The Reason I work at Central State and the reason that I work on the student success program is because I know the power of education to transform human life. As you may know, the single most impactful thing that can predict a person’s future economic status is a bachelor’s degree – regardless of race or nationality, once you have a bachelor’s degree, we can pretty much determine what your social-economic status is going to be. So that is the background for this program and why we know that education is important.
Although I am the one speaking today, the STEM Success Center is a project funded by the U.S. Department of Education. It is under what is called the Minority Science Engineering Improvement Program. I am the program director, and my co-program director is Dr. Ibrahim Katampe and the STEM Success Center manager is Dr. Linda Thomas – she’s the one that “makes the train run on time”.
The Center is designed to improve the quality of honorable underrepresented students, especially female students, to ensure they’re ready to either get a job or go on to graduate school upon graduating from Central State. We have three goals to help students succeed at Central State; one is (which is what we’re going to discuss in detail today) is to establish a comprehensive learning system for students, which means tutoring and providing academic support inside and outside the classroom to address the deficiencies of students taking STEM classes, which are called gatekeeper courses; in order to do that our next goal is to establish a physical location for students to come and enjoy our services; and the last goal for the program is to make sure students enjoy experiential learning opportunities, meaning they can put their hands on what they’re doing either through internships, research opportunities, and other experiential learning opportunities. So the goal of the program is to retain students (making sure they come back to school), persist (making sure they continue onto their education studies), and finally to graduate
Dr. Lana Rucks:
Great, thank you very much for that really great background. So essentially, as I said, the purpose of the webinar is for us to demonstrate how evaluation can inform the continuous improvement process. To do that we’ll first share an evaluation finding and then Dr. Kuti will show how that finding was used, and then of course you’ll have opportunities to be able to ask questions, so make sure to use that function. Let’s go ahead and get started.
For our first evaluation finding, let’s look at how the STEM Success Center impacted grades. As Dr. Kuti noted, part of the STEM Success service and part of the comprehensive learning services is to provide tutoring. So to measure the impact of tutoring on grades, we were fortunate to have a natural experiment in which three different conditions emerged; first, we had students who participated in tutoring and received a stipend; then we had students who participated in tutoring but did not receive a stipend, and then there were students who did not participate in tutoring at all.
When looking at the findings there are several interesting trends that emerge, but I want to also highlight that in regard to withdrawals, students who participated whether they received a stipend or not, experienced a lower frequency of withdrawals than those who did not participate. In a similar vein, that pattern emerged in terms of students who received F’s, such that those who participated in tutoring had a lower frequency of failing than those who did not. That was also intensified for students who received a stipend. For students who are in that condition as well, they also received the highest frequency of A’s. What’s also interesting to point out is that those who participated in tutoring and received a stipend participated in almost five times more tutoring sessions than those who participated but did not receive a stipend. Let me pause there and turn it back over to Dr. Kuti.
Dr. Morakinyo Kuti:
So how did we use the findings that Dr. Rucks just explained?
She mentioned the difference between the people who did and did not receive the stipend, so let me explain what we mean when we say stipend. For a student to get the stipend, they had to participate in a minimum number of sessions during the semester and you could only miss two tutoring sessions in the first half and two tutoring sessions in the second half of the semester. So the more you went to tutoring, the more benefits you got and you received a $100 stipend in the first half and $200 dollars in the second half. In other words, we were trying to incentivize students to go to tutoring. As Dr. Rucks just explained, tutoring was effective in that it increased students’ performance. The interesting thing was that once we got the findings from Dr. Rucks and we showed the faculty members that the students that came to tutoring performed better, some faculty members made tutoring mandatory for class. Some gave a financial incentive, and some gave an incentive in the form of grades. Those faculty member’s students performed better and were able to encourage other faculty members to make tutoring mandatory to provide incentives as well.
Another thing we learned what that faculty engagement is the most critical factor in ensuring success. When faculty that promoted tutoring in their class, persuaded students through incentives, and faculty that followed up with students about tutoring and the impact of their participation on their grades, we were able to convince other faculty members that they were playing a critical role in student success.
So as you know, the whole purpose of the entire exercise is to ensure student success. One way to measure student success is through grades. Are they getting an A, B, or C on their transcript? When students saw an A, B, or C on their transcript, that made them more likely to participate. So, as we said earlier, we want to retain students and make sure they come back, but when they come back we also want to make sure they persist to the next stage. So by students getting good grades and by them coming back to school, it let them know that good grades would mean good grade means academic success towards their stay at Central State University.
Dr. Lana Rucks:
Not to belabor a point, but it is just so very encouraging to see how you glean so much information out of those findings. So, let me pause there and see if there are any questions.
(Q & A Portion not transcribed).
So the next finding that we wanted to be able to share was the impact of the STEM Success Center on retention. To talk about the impact on retention, let me share out how we choose our conceptual framework for the analysis, why we did it, and also how we did it.
So, when talking about grades you remember that there were three different conditions that emerged. What’s also important to remember is that in thinking about students, because we were analyzing this at the level of the course, a student could potentially experience 3 types of courses that offer tutoring in a different way. You could have a student who participated in tutoring and received a stipend, participated in tutoring in another course but didn’t receive a stipend, and for a third course they didn’t participate in tutoring at all – There could be some combination of that across students. So, the analysis was not problematic at the level of the course in terms of how to complete the analysis, because we could essentially split each person into each of those courses. But when thinking about retention now, that becomes slightly problematic because now we’re completing the analysis essentially at the level of the institution – is the person actually coming back to the institution? So we had to re-conceptualize the condition slightly and in this case, we can create three different conditions: did they experience tutoring and received a stipend, did they experience tutoring but did not receive a stipend, or did they not participate in tutoring.
So in this situation that the first student would be categorized as experience tutoring a received a stipend, even though with two other courses they didn’t receive a stipend, at some point they experience receiving a stipend for attending tutoring. In a similar vein, the next student was categorized as experience tutoring but did not receive a stipend. And then, of course, that last student would be categorized as not participating. With that as the conceptual framework, let me share what the findings actually revealed.
In regard to the impact on retention, whether students received a stipend or not, rates were about and 80%. What was interesting is that when they did not participate in tutoring, retention was at 57%. So again, let me turn things back over to Dr. Kuti.
Dr. Morakinyo Kuti:
Thank you. So when we’re talking about all these numbers, what does it all actually mean? We as we said earlier, if you participate in tutoring at a sustained level, you’re more likely to succeed, and success, in this case, is retention and students coming back to school the following semester. There’s a technical definition of freshmen – I don’t want to bore you guys that are not in higher education about what retention is, but in this case, we’re talking about if a student came to Central State in the fall, did they come back in the spring, and if they did, did they come back again in the Fall? That’s what we mean by retention. So, what we showed is that students that receive sustained tutoring came back to school.
Now when you come back to school with an A, B, or C then you see a better attitude about the school and you’re a better student. What we found is that the more “better’ students there are, the better the academic environment for the whole school, meaning students are more engaged in scholarly activities, students go to the library, students participate in social things. More successful students create a more improved academic environment for the whole campus. The more studious students are, the more likely they are to graduate on time.
As you may or may not know, most students don’t graduate in four years. Typically, we measure graduation rates on a six-year basis, but when students perform better, they graduate in four years and so the graduation rate is increased. That’s good for the student because if you graduate on time, you get your scholarship, you don’t have loans and you’re done in a timely fashion. But it’s also good for the institution, for example, the university receives federal financial aid. That is based on whether or not your student loan repayment is above a certain percentage, you lose the ability to give financial aid. The state of Ohio, because Central State is a public school, also reimburses the university based on student completion and success. So, if a student from Ohio graduates, Central State gets twenty thousand dollars for each student that graduates. So, if we had ten students that started and two of them graduate, that’s forty thousand dollars, but if we’re able to get seven students to graduate that 140 thousand dollars, so there’s actually concrete financial benefits.
Also, out-of-state students get three thousand dollars. How does this affect students? Well, the more financially stable Central State is, the more services and support we can offer every student. This is how you go from students participating in tutoring and getting good grades, to actually enhancing the students themselves.
One thing I also wanted to mention is that at Central State, 80% of our students are low-income which means they receive federal financial aid. Remember as I said, the most important predictor of a person’s success is whether you have a bachelor’s degree. So, if you’re a low-income, first-generation student and you get your bachelor’s degree, you are going to earn twice as much money as a high-school drop-out which improves your social-economic status, your family, and society as well because the higher the degree, the more education the person is, the more they earn, and the more they contribute to society.
Dr. Lana Rucks:
To loop back to a question that was asked before about the cost of tutoring and what you shared, can you put a bow on that and turn from your perspective of the cost of tutoring relative to the kind of outcomes and the other financial benefits that you highlighted?
Dr. Morakinyo Kuti:
This is why, and not just because I work in higher education, education is the best investment that individuals and society can make. When you invest in people and they get an education, they’re more likely to make more informed choices which are good for society overall. When we talk about the cost of tutoring, we offer students incentives of 100 or 200 dollars – small change, but for students, this could be useful at the end of the semester. So, the benefit at the end for students personally and society as a whole yields a significant investment beyond the actual dollars we’re talking about. It actually helps the university to fulfill its mission to transform students’ lives but through a nurturing environment by making them more productive members of society.
Dr. Lana Rucks:
Thank you so much for that and thank you also for the individual who asked that question. Why don’t we pause here for a moment to see if there are any other questions?
(Q & A Not Transcribed).
We have just a couple of minutes left so let me just real quickly share from my perspective what I thought the value of evaluation in this continuous improvement process was. One, I think it’s just very fulfilling to be able to witness evaluation in action and to be able to see how it informed the decision-making process. Again, this is something that I often talk about so it would often just make my day to talk to Dr. Kuti and the team and hear about how they’re actually using the evaluation findings and then the other piece I was really appreciative of was the willingness of the project team to implement the data systems to be able to collect the data. As you can imagine, there’s an additional level of work that goes into being able to make sure that you’re gathering the right data and that the data are clean. So that’s some of the things I really appreciated from my perspective. Dr. Kuti, would you like to share your perspectives?
Dr. Morakinyo Kuti:
Yeah, so what I will say is when I first got in this business, people talk about evaluation and assessment – now I really know what they mean is that you have to evaluate as you work on something. When they talk about program evaluation it’s that as you’re implementing the program, we’re looking at the data that is being generated to improve what you’re doing. Because of evaluation, we had to change our tutoring hours because of what students said. This came out of a survey we did where students were able to express that they want a different time or different place to be tutored.
We were also able to increase incentives. We went to my program manager and told her about some of the things that were going on in our program and she gave us the flexibility to increase incentives. That was because of the evaluation. Finally, is the summative evaluation. Because we were able to show the grades and what students were earning, it let us know what was working in the program and what was not working and we were able to demonstrate the effectiveness of tutoring to students and faculty, and to our funding agency.
Dr. Lana Rucks:
Great, thank you. So just really quickly here, in closing, if anyone has any questions or needs any additional information, please feel free to reach out to Dr. Kuti. His contact information is here and also we’re very impressed that you were able to connect with Betsy Devos, the former Secretary of Education, as you can see in this picture. It’s really catching the eye in terms of really important individuals of the project.
Finally, you will be receiving an email in the next day or two with a recording as well as a survey, so please provide any feedback that you may have about this webinar and as well as additional topics. Also, feel free to forward the webinar recording to your network and then mark your calendars for our next coffee break webinar on April 22nd which we’ll talk about annual reporting. So, thank you to all who’ve participated, and thank you Dr. Kuti for being our first guest presenter on our coffee break webinar series.
Dr. Morakinyo Kuti:
Well, you’re welcome. This is not the first time we’ve worked with Dr. Rucks. Our relationship goes back about ten years. We’re just very happy that you’ve provided us your professional services that have improved the project and helped us to deliver in increasing student success, hopefully helping them to succeed, graduate, and improve their lives. So thank you.
Dr. Lana Rucks:
Well thank you for that! I really appreciate it, I know that the team appreciates that as well. Everyone have a great rest of the day.
