Like many other development organizations, the World Bank Group often undertakes projects to improve services for underserved communities around the world. Whether these services get utilized by beneficiaries is dependent on many factors – features, cost, convenience, to mention a few. In several instances, even when services are provided for free, utilization is not a given and the desired behavioral outcome may not be achieved. This recurring challenge suggests that we sometimes need to better understand beneficiary behavior, and that opportunities to do so are sometimes missed. Our ongoing work suggests, for example, that entry points that could capture beneficiary feedback often don’t, and that there is insufficient attention to needs or preferences expressed by beneficiaries in project planning documentation. We have also observed that disaggregates for sub-groups of the beneficiary population is a basic, but often lacking, feature in projects with a service delivery focus.
In two earlier blog posts – on “Emerging Lessons from Applying a New Framework for Evaluating Service Delivery” and “Putting Behavior Change at the Center of Development and Evaluation” – we shared preliminary insights from recent work at IEG. These insights help us to better understand how the concepts of service delivery and behavior change interact in development, particularly in the context of the World Bank Group’s operations and projects.
As part of this work, IEG developed evaluative frameworks for service delivery and behavior change that are being integrated into three ongoing sector evaluations. Using our new framework for the evaluation of service delivery, as well as a framework for evaluating behavior change that we call CrI2SP, we are extracting lessons, and testing the frameworks through engagements across the World Bank Group, working closely with colleagues at the Global Delivery Initiative, the Mind, Behavior and Development team, Behavior Change Community of Practice, and others. In this blog post, we discuss how these new frameworks can help practitioners to enhance project planning and associated outcomes by placing the beneficiary at the center of the planning process.
Broadly speaking, the concepts cover the supply (service delivery) and demand (behavior change) sides of World Bank Group operations. For example, incentivizing the use of health services through conditional cash transfers (demand-side, behavior-oriented) only works when there is a functioning health system (supply-side, delivery-focused) in place. Our work has shown that service delivery and behavior change critically dovetail at the beneficiary level. In each of the frameworks we have developed, the beneficiary is key to both effective delivery and use of services designed to impact their well-being.
Framework for the evaluation of service delivery
Our framework for the evaluation of service delivery is comprehensive, emphasizing the systems aspect of delivery. It spans the spectrum from enabling conditions (e.g., political economy, procurement, regulation), through inputs, implementation models, outputs, and service outcomes (e.g., coverage, quality, affordability, reliability). The framework provides evaluators with a lens to help assess the extent to which the many factors involved in a service delivery effort have been taken into consideration in design and implementation.
The framework also alerts evaluators to the critical function of beneficiary feedback in any delivery system. For services to be effective, it is necessary to not only describe intended beneficiaries in detail (e.g., by income, gender, age), but it is also necessary to ensure systems to support iterative beneficiary feedback throughout the process. As our friends at Feedback Labs succinctly put it, feedback is necessary to ensure we know and understand what people want to make their lives better, that we are helping them get it, and, if not, what we should be doing differently.
As part of IEG’s efforts to ensure strategic focus on the poor in our evaluation practice, we also developed a framework for the evaluation of behavior change, which we call CrI2SP (for Communication, Resources, Incentives and Information, Society, and Psychology). The framework is designed to evaluate demand-side interventions aimed at inducing behavior change. CrI2SP is designed to help evaluators systematically assess the degree to which projects identified beneficiaries’ behaviors, diagnosed barriers to adopting a desired behavior-- such as handwashing or breastfeeding -- in order to achieve a desired outcome, and monitored and evaluated behavior change interventions. Questions an evaluator might ask include:
- How was information communicated to beneficiaries?
- Were resources provided?
- What incentives or information was provided to beneficiaries?
- How were societal dynamics used to shape peoples’ behaviors?
- Were simple psychology-based methods used to influence individuals’ decision-making?
The utility of this framework is not limited to post hoc evaluation. In fact, these questions, as well as questions associated with the framework for service delivery (relating, inter alia, to the political economy, financing, and human resources), could just as readily be asked by a Task Team Leader engaged in reflection on the adequacy of project design, prior to finalizing project appraisal.
IEG’s evaluative frameworks for service delivery and behavior change intersect at the critical point of understanding the needs of beneficiaries. In that regard, both frameworks examine whether diagnostic work -- such as analytical services and advisory work, formative research, beneficiary analysis, focus groups, surveys, interviews, field visits or observation – informed project design and implementation.
Our work on service delivery and behavior change challenges our own approaches to evaluation in terms of methodology, resources, and scope. However, we have found it necessary to push the boundaries of our practice in response to the emphasis on delivery and the citizen that is evident in the World Bank Group strategy. We are interested in engaging with colleagues to further explore concepts and practice, so please contact us or otherwise share your thoughts with us about our work on service delivery and behavior change. We look forward to hearing from you!