A process evaluation tells you whether your program is being implemented as expected. In other words: process evaluations help you see whether the steps and assumptions in your theory of change holdup in practice.
Process evaluations often identify elements of program design that should be modified based on gaps observed in the field. For example, you may run a program that trains veterinary health workers to provide services such as vaccination and deworming to all livestock farmers. A process evaluation may tell you that veterinary health workers are only selling to medium- and large-sized livestock farmers. As a result, you may consider adding monetary incentives for reaching smallholders. Similarly, you may want smallholder farmers to make up at least 50% of the health workers’ clientele, but your process evaluation reveals that it is only 10%. Therefore, it helped you recognize that your program was falling short of a target during implementation, and you may now take up steps to rectify that.
A process evaluation may reveal that it is not possible to implement the program in accordance with the theory of change. For example, if an assumption critical to the success of your program does not hold, and is outside of your control, you may need to consider terminating the program. Say you have a program that offers doorstep banking services for women. Banking agents take deposits from customers and drop the money they collect at the bank daily. However, when you conduct a process evaluation, you learn that the roads have broken down after a recent storm, and rebuilding will take a while. It is not safe for agents to travel on these roads. In this case, you would need to consider terminating the program (or significantly changing how it is set up).
Process evaluations often point out areas for improved monitoring and oversight. For example, if a process evaluation for a program providing extension training to farmers reveals that you are not reaching smallholder farmers, you may want to start regularly collecting customer profile data to track program reach.
Doing a process evaluation involves two steps:
Should you do the process evaluation yourself or hire an expert?
Organizations with the capacity to run surveys, conduct interviews, and analyze data can do process evaluations on their own; however, you may want to hire an external party to reduce the risk of bias.
If you are conducting a process evaluation to meet high-stakes funder requirements, the evaluation would be more credible if it were conducted by an external evaluator, as the risk of bias is lower. If, however, you are conducting a process evaluation for internal learning, you can do the exercise in-house if you have capacity and are confident that you can collect unbiased data.
To run a process evaluation in-house, your organization should be able to collect quality data reliably. Your team will need to design questionnaires and responsibly survey implementers and/or participants on a variety of topics. We generally recommend hiring surveyors externally so as to minimize response bias.
Your organization should also be able to analyze qualitative and quantitative data. Qualitative data analysis requires the ability to organize high volumes of information and the ability to read interview answers with a careful eye. Quantitative data analysis may require only simple summary statistics in a spreadsheet program like Microsoft Excel, or more nuanced econometric analyses using specialized software like Stata or R.
If you do not have internal capacity to reliably collect, analyze, and interpret the data, you should work with an external party.
If you choose to conduct the process evaluation externally, you’ll save on staff time. Note that someone from your organization who is familiar with the program will still need to spend time working closely with the evaluator and supporting them in data collection.
Our step-by-step instructions and template will help you design and execute a process evaluation
Use the Theory of Change Builder to create a clear narrative for why your program will work.
Already created a theory of change? Log in and make sure it is up to date! We’ll refer to it going forward.
You can still proceed without linking your theory of change, if you want to preview the steps or have a theory of change saved elsewhere.
An implementation plan provides the operational details of how a program will run – who will do what with which resources. It also includes when, where, and how frequently these activities will happen. Your implementation plan should also include benchmarks for program success, such as targets for program uptake, service delivery, or program participant satisfaction.
When an implementation plan is not written down in detail, there may be a lot of implicit (and potentially conflicting) knowledge about what should be happening. It is important to document program details so you can compare what’s happening on the ground to what was expected.
An implementation plan is more detailed than a theory of change, and includes the details of how you’ll put your program into effect – who will do what with which resources and materials.
Highlight any important nodes or links you’re unsure of. If your theory of change says that teachers will attend trainings, learn better teaching practices, and apply them in their own classrooms, ask yourself: How will I know if teachers come to my trainings? If they understand the curriculum? If they plan to use the practices we teach them?
You can identify shaky steps by asking:
The Theory of Change Builder automates this process for you! Log in to access a saved theory of change, and we’ll guide you towards identifying shaky steps. If you’re working with an offline theory of change, you can follow these instructions to identify shaky steps. We also provide a template to track shaky steps.
Process evaluations can use quantitative data (anything that can be represented as a number) or qualitative data (harder-to-summarize observations and conversations). Good process evaluations often combine both.
Choose metrics that meaningfully capture the things you care about. For example, a crucial – and uncertain – link in your theory of change might be whether teachers understand and act on the training you provide. You could ask teachers point-blank whether they plan to apply your training to the classroom, but you would get richer and more accurate information by observing them teaching or asking their students.
You can use the indicator recommender to shortlist metrics!
Decide what levels for each metric indicate success or failure. Quantify as much as possible. For example, saying that the program is successful if you see teachers implementing applying what they’ve been trained on in their classrooms is ambiguous. How many teachers need to be acting on the training? How many of the new teaching practices should they be applying? Are “all” practices necessary, or would half of the practices be sufficient to meet our definition of success?
Choose a data collection platform and assemble a team of enumerators. It’s often cheaper to have program staff collect data, but that can be tricky if there any conflicts of interest. If independence is important, consider hiring outside surveyors.
Data analysis for process evaluations is fairly straightforward, since you’re summarizing your data to compare with your expectations for the program, rather than a control group. Analyzing quantitative data can often be done in Excel or Google Sheets. Analyzing qualitative data can be more time-consuming and can take a range of approaches.
Finding weaknesses in your program is good news – it gives you a chance to fix them before investing in an impact evaluation or large-scale growth. If you find that your program is falling short in specific areas, go back to the drawing board! Depending on your findings, you may need to talk more with your target users about their needs; explore how other programs have handled the problem; or run some quick tests of variations of your program.
Not sure where to go from here? Use our guide to frame a question and match it to the right method.
How a process evaluation for Farm to Market Alliance in Zambia identified changes that could make the program more effective
Impact Measurement Guide
The Impact Measurement Guide is still in beta release, we welcome your feedback on which features you like as well as what we should change or improve in future versions.
Impact evaluations compare the people who receive your program to a similar group of people who did not receive your program. Based on how this similar group is chosen, impact evaluations can be randomized controlled trials or quasi-experimental.
A randomized controlled trial (RCT) is considered the gold standard of impact evaluation. In an RCT, the program is randomly assigned among a target population. Those who receive the program are called the treatment group; those who do not receive the program are called the control group. We consider the outcomes from the control group to represent what would have happened to the treatment group in the absence of the program. By comparing outcomes among those who receive the program (the treatment group) to those who don’t (the control group), we can rigorously estimate the impact of the program. The random assignment of the program to the treatment and control group provides the rigor, as it ensures that the selection of people is not based on biased criteria that could affect the results.
When a randomized design is not feasible, there are other, “quasi-experimental,” ways of constructing a valid comparison group.
Matched designs and regression discontinuity designs are just two of many quasi-experimental techniques. J-PAL provides an overview of common methods of conducting an impact evaluation . All such methods seek to identify what would have happened to your target population if they had never received the program, and their success relies on the strength of the assumptions they make about whether the comparison group is a credible stand-in for your program’s target population.
A theory of change is a narrative about how and why a program will lead to social impact. Every development program rests on a theory of change – it’s a crucial first step that helps you remain focused on impact and plan your program better.
You can use diagrams.net to create your Theory of Change.
A needs assessment describes the context in which your program will operate (or is already operating). It can help you understand the scope and urgency of the problems you identified in the theory of change. It can also help you identify the specific communities that can benefit from your program and how you can reach them.
Once you’re satisfied that your program can be implemented as expected:
You have a written plan for how your program will improve lives – great! Make sure to refer to it as you explore the different sections in the Impact Measurement Guide, as it is the foundation for any other method you’ll use.
A process evaluation can tell you whether your program is being implemented as expected, and if assumptions in your theory of change hold. It is an in-depth, one-time exercise that can help identify gaps in your program.
Once you are satisfied that your program can be implemented as expected:
An evidence review summarizes findings from research related to your program. It can help you make informed decisions about what’s likely to work in your context, and can provide ideas for program features.
Once you are satisfied with your evidence review:
A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.
Once you are satisfied with your monitoring system:
A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.
Once you are satisfied with your monitoring system:However, don’t include only academic studies in your review! You should also consult:
The free Zotero plug-in provides an easy way to save, organize, and format citations collected during internet research. Note that Zotero can help you start your annotated bibliography, but it is not a substitute since it does not include any summary or interpretation of each study’s findings.
Start with the 3ie Development Evidence Portal, which has compiled over 3,700 evaluations and over 700 systematic evidence reviews. Steps 2-7 of this example are specific to locating evidence on the 3ie website, but you can also consider looking for a review by J-PAL or Campbell Collaborations or the Cochrane.
For example, suppose your goal is to increase immunization rates in India. Type “immunization vaccination” or other related terms into the search box, and click the magnification lens to search.
The search results include individual studies, which are usually about a single program in a single location, as well as “systematic reviews”, which is what we are looking for because they are more comprehensive. To show only the systematic reviews, on the left of the screen under Filter Results, click on PRODUCTS and check the Systematic Reviews box. We’re now left with 17 evidence reviews related to immunization.
Now you might want to further narrow your search by region or country. In our example, suppose we want to see only those evidence reviews that contain at least one study from India. Click on COUNTRY and scroll down to click on India.
There are still 9 evidence reviews! Now read the titles of each review and start going through the ones that seem applicable to you.
The next page gives you an overview of the study. If it is “Open access”, this means you can read it for free – click “Go to source” below the star rating. If it isn’t open access, you can try some of the strategies in Step 9 to see if you can find the study for free elsewhere.
Clicking on “Go to source” opens a new tab with a PDF of the article. Don’t be intimidated by the length and technical terminology, and start with the summary – these articles usually include an “abstract” and sometimes a “plain language summary” and/or “summary of findings”.
The summary will likely be useful but too vague – dig into the review and look for details about which programs were tried and where, and how well they worked.
a) Go back to your search and see if you can find a PDF posted on one of the authors’ websites – authors often share “working papers”, which might differ only slightly from the final paper, for free on their site.
b) Email the paper’s authors if you can’t find it elsewhere – many researchers are happy to share a copy with people looking to learn from their experience.
For each piece of evidence that you find, there should be a clear justification for including it in the bibliography, such as: it is a landmark study in the topic (i.e. it has a large number of citations or is cited by many other studies in your review), it is relevant to specific aspects of this evaluation (such as measuring similar outcomes, being conducted in a similar context, or evaluating a similar intervention), etc. However, there are no absolute standards for inclusion, and since not all studies will be used in writing up the review, it is better to err on the side of including a study in the annotated bibliography.
Repeat step 9 for every paper from the systematic review that you found relevant!
Process evaluations and monitoring both provide information on how your program is running and whether it is meeting expectations. The key difference is that process evaluations are a one-off activity, while monitoring is ongoing. That means that process evaluations are often more intensive exercises to collect more data and dive deeper into the theory of change. In contrast, ongoing monitoring must not overburden program staff and often tracks just a few high-priority indicators that are critical to program success.
Consider the following questions to help you decide between conducting a process evaluation and building a monitoring system:
1. Are you interested in identifying specific problems or general problems?
Process evaluations typically identify general or systemic problems along the theory of change, whereas monitoring typically identifies specific entities (e.g. service providers or locations)that need more attention.
2. Are you looking to hold program staff accountable?
Both process evaluations and monitoring are implemented for learning – is our program being implemented as planned? If not, at which steps is it breaking down? However, if you are seeking an accountability system, monitoring is better-suited as it is continuous, whereas a process evaluation is a one-time exercise.
3. Do you need ongoing data on how your program is performing?
A process evaluation typically offers a snapshot in time, whereas monitoring involves ongoing data collection and analysis for the entire duration of the program. For example, a process evaluation may do in-depth interviews with program participants on their experiences, whereas a monitoring system might collect data on just a few questions related to beneficiary satisfaction.
4. Do you need comprehensive data?
A process evaluation is typically based on a sample, while monitoring is usually comprehensive. For example, in a teacher training program, you would monitor the training of all teachers (because it is useful to know exactly which teachers did not attend the training), whereas in a process evaluation, you would interview a subset of teachers to understand the reasons why they did not attend the training.
