Testing for Impact at Scale – Lessons in designing an A-B test from Dignitas Kenya
Dignitas provides innovative training and coaching to school leaders and teachers in Kenya, designed to enhance teaching quality and boost children's learning.
In 2022 Dignitas launched an app-based version of their programme, called LeadNow. LeadNow takes the insights, expertise, and evidence gathered from more than a decade of supporting school leaders and teachers in-person and applies that to a virtual world of training and coaching.
Dignitas joined Global Schools Forum’s Impact at Scale Labs programme to design and implement a scale-up strategy for LeadNow in 2023. One of the first steps of the scaling process is to conduct testing. This article outlines Dignitas’ testing of the LeadNow app, from selecting a testing methodology, to designing and adapting it to best fit their contextual realities. In doing so, it offers insight into the ways an organisation may navigate the testing stage of the scale-up process.
Selecting an Appropriate Testing Methodology
As part of the Impact at Scale programme, we understand that testing an innovation is crucial to ensure that we have the most impactful and efficient version of our innovation to scale up. In our scaling framework, we call this ‘desirability’. Testing our innovation allows us to ensure:
- Good fit with the identified problem and implemented solution, including reaching and engaging the intended audience
- Context and cultural relevance of the innovation, especially when scaling up to a new geography or user group
- The best iteration of the intervention for achieving the intended impact
- The best iteration of the intervention for efficiency and viability for implementation at scale
Achieving these four aims requires different testing approaches at different stages along the journey to scale.
For Dignitas and the LeadNow app, phase one of this journey had been completed, with a Theory of Change developed and initial concept testing conducted with teachers, which gave the Dignitas team good insights to get their innovation ready for testing with a larger group. The team’s burning question was ‘Which combination of intervention components drives the highest levels of teacher engagement and achieves the best outcomes, at what cost?’. This aligned with Phase 2 of testing, aimed at building the best iteration of the innovation.
A-B Testing
Based on their research question and their results to date, Dignitas decided to use an A-B testing methodology as this would give them robust evidence on both the impact and cost of each variation of their LeadNow programme to support their decision-making on which variation to take to scale.
A-B testing is a systematic method used to evaluate two variations of an intervention to determine which variation to develop further. The essence of A-B testing involves presenting two versions of an intervention component (such as a teaching method, a piece of educational content, or a user interface design in an app) to two comparable groups. The performance of each version is then measured and analysed to identify which one achieves the desired outcomes more effectively.
This method allows us to make evidence-based decisions about which version or iteration of an innovation is best achieving the intended outcomes most effectively and therefore gives direction to innovation adaptation. By carefully controlling the variables, this approach ensures that any observed differences are due to the intervention itself rather than external factors. This precision is crucial for scaling up educational innovations, as it helps to guarantee that resources are invested in approaches that truly make a difference.
Expert Guidance from Experienced Testers
In pursuit of designing a solid A-B test for the LeadNow programme, Dignitas sought expertise from several fronts. Guidance was provided by the Global Schools Forum and Youth Impact, with Phillip Okull, the Evaluation Director at Educate!, offering specialized advice on crafting a rigorous A-B testing framework. It is important to identify when there is a need seek expertise to ensure that the testing is as rigorous as possible.
Based on expert guidance, a critical step identified was the precise determination of the sample size needed for meaningful results. To achieve this, Dignitas utilized STATA software to calculate the optimal sample size for conducting the test in two distinct counties. These locations were specifically chosen for their contrasting environments: one in an Arid and Semi-Arid Land (ASAL) and the other in a rural setting. This strategic selection was aimed at ensuring the app's effectiveness across diverse geographic and socio-economic contexts.
The methodology adopted for sampling was multistage cluster sampling. This approach was instrumental in achieving a sample that accurately represented the sub-county and zonal demographics. To mitigate potential biases and control for interactions between the different groups, schools within the same zone were allocated to a single test arm. This random assignment was crucial in maintaining the integrity of the A-B testing process, ensuring that the findings would provide a clear direction on the most effective components of the intervention for enhancing teacher engagement and educational outcomes.
Adapting the Approach to the Dignitas Context
The primary goal of the A-B test was to identify which combination of programme components is most effective for teacher engagement and impact on teaching quality. Additionally, the test aimed to evaluate the cost-effectiveness of each component when used in conjunction with LeadNow.
For this purpose, Dignitas structured a three-arm test to compare different approaches:
- LeadNow app only: This group engaged with the LeadNow platform exclusively.
- LeadNow + Coaching: The second group used the LeadNow app, as well as coaching via phone calls from trained education officers.
- LeadNow + Professional Learning Communities: The third group accessed the LeadNow app, and were also given training and resources to run school-based Professional Learning Communities (PLCs).
The focus was on teachers of Foundational Literacy and Numeracy (Grades 1, 2, and 3) and the Instructional Leaders responsible for overseeing the program's implementation in schools. A purposeful sampling strategy was employed for these target grades, recognizing that the same teachers often taught both literacy and numeracy subjects. The school's Instructional Leader, typically the deputy headteacher, played a pivotal role in this setup, ensuring a focused and informed approach to implementing the test components effectively.
Practical Considerations in Designing a Complex Test
A critical takeaway from the A-B testing process is the importance of addressing practical challenges, particularly those related to geographical, technological, and financial factors. To ensure the success of testing, it is important to consider and learn from these practical challenges in the design and implementation of future tests.
Internet connectivity emerged as a significant consideration, especially given the reliance on online learning resources like LeadNow and support mechanisms such as coaching phone calls. Another obstacle was the necessity for smartphones to access the online learning modules. This requirement excluded some school leaders from direct participation in the test if they lacked access to such devices.
Budgetary constraints also played a pivotal role in shaping the test's scope. The initial plan to conduct the test across three counties had to be scaled down to two, a decision directly influenced by budget limitations. This adjustment underscores the budget's influence not only on determining the sample size but also on selecting test locations, illustrating the interconnectedness of financial resources and research feasibility.
A key learning was the importance of an induction meeting, which was an essential component in familiarizing participants with the study's objectives and methodologies before beginning data collection or implementation phases. This ensures that the team is aligned and prepared for the task at hand.
Future Directions for Dignitas
Moving forward, Dignitas has completed the implementation of the first round of A-B testing, and is currently analysing the results to decide which programme variation to take forward into further testing. By building this testing and learning capacity within the team, Dignitas can continue testing and refining their LeadNow app as they scale their programme in Kenya.
About Dignitas
For more information on Dignitas and their LeadNow programme:
Dignitas Project – Vibrant schools. Thriving children.
Leadnow by Dignitas Project – Vibrant Schools. Thriving children.
About Impact@Scale Labs
For more information on Global Schools Forum Impact@Scale Labs: https://www.globalschoolsforum.org/page/GSFImpactScaleLabs
Related Tools in Scale Toolkit