In 2016, the Luminos Fund launched its accelerated, catch-up learning program in Liberia to help address the country’s urgent education needs – including one of the world’s highest recorded rates of out-of-school children. To date, Luminos has helped 12,650 Liberian children catch up on learning and reintegrate into local government schools. In addition, Luminos has trained 497 young adults on our pedagogy and model, and supported them to deliver the catch-up program in classrooms.
During the 2021-22 school year, the Luminos program increased children’s oral reading fluency (ORF) by 28 correct words per minute (CWPM), with girls progressing 3 CWPM more than boys. Students also made substantial gains in numeracy, with a 28 percentage point improvement in addition and a 20 percentage point improvement in subtraction. Our latest report, “Liberia 2021-22 Endline Evaluation Report,” summarizes results from the 2021-22 Luminos program endline evaluation conducted by Q&A Services. [1]
In 2021-22, the Luminos program ran for 9 months—from November to August— in line with the Ministry of Education’s 2021-22 official academic calendar; this calendar was shifted slightly compared to a standard, September – June calendar due to COVID-19. Luminos students attended class for 7 hours per day from Monday to Friday, with approximately 5 hours per day devoted to reading and 2 hours to numeracy.
Luminos supported 3,150 out-of-school students across 105 classes and five counties (Bomi, Bong, Grand Cape Mount, Margibi, and Montserrado) in Liberia. Every year, Luminos works closely with a small group of community-based partners, each of which manages a cluster of classrooms, to deliver the program.
The results of the evaluation show that the Luminos Fund’s Liberia program positively impacted student reading and math outcomes across all EGRA and EGMA subtasks in the 2021-22 school year.
Evaluation Overview
The evaluation aimed to demonstrate the impact of the Luminos Liberia program on student literacy, numeracy, and socio-emotional outcomes during the 36-week 2021-22 program. Q&A Services assessed the literacy and numeracy levels of a random sample of students across all Luminos classes in the first two weeks of the program (baseline) and again in the final week of the program (endline). The RTI/USAID-developed Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) tools, adapted for Liberia, were used at both baseline and endline to assess students on a variety of early grade reading and math skills. A socio-emotional learning (SEL) assessment was also conducted with a subset of the student sample using the International Social Emotional Learning Assessment (ISELA) tool. For more details on the evaluation and methods used, please see the full report summary.
Overall Results
The results of the evaluation show that the Luminos program positively impacted student achievement in both reading and math.
Literacy
On reading, students showed improvement across every EGRA subtask, including an improvement of 50 percentage points on letter identification, 46 percentage points on oral reading fluency (ORF) of Grade 2 level text, 39 percentage points on familiar words, and 33 percentage points on reading comprehension. For ORF, students could read 29 CWPM at endline, compared to 1 CWPM at baseline, an improvement of 28 CWPM.
Numeracy
On numeracy, students again showed improvement across every single EGMA subtask, including an improvement of 35 percentage points on number identification, 33 percentage points on number discrimination, 28 percentage points on addition, 20 percentage points on subtraction, and 22 percentage points on word problems. While the program impacted student achievement on mathematics, improvement was less significant than for literacy. This makes sense given that 5 hours of the Luminos school day (approximately 70% of instructional time) is devoted to literacy and 2 hours each day (30% of instructional time) is devoted to numeracy.
Conclusion
The results of the evaluation show that the Luminos Fund’s Liberia program positively impacted student reading and math outcomes across all EGRA and EGMA subtasks in the 2021-22 school year. Results show that the average student improved by 28 CWPM within the 9-month program, with girls improving 3 CWPM more than boys. These results are incredibly impressive given the short (9-month) timeframe for the Luminos program. Results for the SEL assessment show improvement on self-concept, particularly for girls, suggesting possible impact of the Luminos program on broader student development; however, further research is required. When compared with similar programs in Liberia and globally, year on year the Luminos program is showing strong learning outcomes, particularly on literacy.
To read the full report summary, including additional background on our Liberia program and a more detailed overview of the evaluation and methods used, click here.
References:
Simpson, A. “Luminos Fund Endline Evaluation 2021-22, Liberia,” Q&A Services, December 2022.
At the Luminos Fund, our work is driven by a commitment to help children learn an incredible amount in a short time: three years of school in just 10 months. As the Country Manager for our Liberia program, my team and I set ambitious targets to ensure all our students build foundational learning skills — and we measure their progress along the way, in real-time, often using data averages to gain quick insights.
While averages are helpful, we’ve learned it is critical to dig deeper to uncover who might be falling behind and why, because addressing these subtle differences ensures no student is left behind.
James Earl Kiawoin, Luminos Liberia Country Manager
Proving that Every Child Can Learn
In Liberia, one of our most challenging targets for the 2021/22 academic year was that every student could read a minimum of 40 correct words per minute (WPM) after 10 months.
At the start of the year, internal baseline data showed that children were entering the program reading just one WPM on average. Only three months later, reading abilities varied quite dramatically (this is not necessarily surprising, given our students come from very different backgrounds).
Seven months into the school year, we were entering the final stretch. With just three months left, our team made one last push to ensure all students had a real chance to succeed and could move closer towards the goal of 40 WPM. We doubled down our efforts to understand who was struggling and why.
What Impacts Student Learning?
First, we looked at the data: There was remarkable progress compared to day one, but roughly 50% of students were reading below our benchmark of 30 WPM for this point in the program. Within this group, a majority of students were with new, first-year teachers.
Second, our community partners visited the classrooms to identify what other challenges might exist that data alone could not reveal. For example, were these students struggling because they lived in areas where English, the language of instruction, is not widely spoken? Was student attendance a challenge? Were parents and caregivers supportive and engaged?
Overall, we needed to target different schools for different reasons, but the data helped Luminos and our community partners identify the classrooms with students falling behind and develop tailored strategies to support them and their teachers. This included a range of techniques:
Providing more frequent and tailored support for new teachers:
Our community partners in Liberia typically visit up to five classes a day. In order to better support new teachers, our partners began to spend more time in fewer classes, providing teachers with deeper, relevant, timely, and actionable feedback. One cross-cutting challenge they identified was related to phonics, breaking words down into sounds and combining letters to form words. This approach is not taught in Liberian government schools, and for most of our students, this was the first time they encountered phonics. For the same reasons, our newer teachers benefitted from additional training and support in this area.
Sharing and discussing data with teachers:
While data helps Luminos think about program design, training, and curriculum updates at a high level, teachers can use data daily to help them steer their approach in the classroom, understand which students are struggling, and continuously improve their craft. Our community partners worked to share and discuss student assessment data with teachers, ensuring teachers were equipped with practical strategies to adjust their teaching to help all students learn joyfully and effectively.
Providing additional one-to-one and small group support to students:
During regular lesson times, teachers provided additional one-to-one support to students who were performing lowest in the weekly assessments. Teachers also paired higher-performing students with those who are lower performing so that they can help to provide additional support during lessons. Additionally, teachers identified students who would benefit from small group instruction that provided targeted lessons on specific skills.
Regrouping students and teachers:
In communities with more than one classroom, we moved students to other classes based on their reading level, usually having an experienced teacher available to ensure they can catch up more effectively.
The Results
Looking back on the school year, there was no silver bullet — but there was a process, and that yielded significant learning improvements for our students. At the end of the school year, 67% of students were reading at or above 30 WPM—an increase of over 17% from month seven of the program—thanks to the efforts of our community partners, teachers, and yes – data.
One of the joys of working at Luminos is that data is centrally important to us. It helps us identify challenges and respond to the needs on the ground in real-time.
Luminos is constantly striving to find the right balance – not over-complicating what we ask community partners and teachers to do, but also having high expectations and aspirations for what we can and should be achieving for the benefit of our students.
As I close in on one year at Luminos, I have learned that a hands-on approach to teacher training and a strong focus on collecting real-time data is key. Taken together with our passion for helping every child to succeed, we can support children everywhere to unlock the light of learning and fulfill their potential.
Luminos is constantly striving to find the right balance – not over-complicating what we ask community partners and teachers to do, but also having high expectations and aspirations for what we can and should be achieving for the benefit of our students.
James Earl Kiawoin is the Country Manager for the Luminos Fund in Liberia where he manages day-to-day operations, overall program delivery, government and stakeholder engagement, and supports staff development. Previously, James worked as a Strategy Consultant at Dalberg Advisors in Rwanda where he completed projects on higher education financing and digital ecosystem development including e-government services.
Lindsey Wang is a Program Analyst at the Luminos Fund where she is instrumental in program monitoring, evaluation, and reporting. She joined Luminos in 2016 as a Mechanical Engineering graduate of MITand is currently pursuing a Master in Public Policy at Harvard Kennedy School.
Why build a data dashboard?
COVID-19 has interrupted students’ learning all around the world. Now, more than ever, the international education community needs effective tools to analyze data in real time and spur equitable solutions to close learning gaps. As Luminos’ Program Analyst, I ensure that rigorous data collection and analysis are at the foundation of program management and support efficient service delivery because, in the end, each data point represents an individual or a community. With over 1 billion students slowly returning to school due to COVID-19 (UNESCO), we need faster feedback loops to identify and address learning gaps and better meet the needs of every student and family.
One year ago, I set out to develop a tool for program managers to leverage the wealth of data collected from the field to drive program delivery. I sought to capture a real-time snapshot of the state of our Second Chance program by integrating both quantitative and qualitative data into our model, thus ensuring that vital institutional knowledge and first-hand observations could be shared across far-reaching geographies. Our solution: a data dashboard for program management with three simple objectives to help us monitor program results in real-time and deploy program resources more efficiently:
Monitor Key Performance Indicators (KPIs)
Capture a holistic view of the program and drill down to granular insights
Identify struggling students and facilitators
With the data dashboard, we can easily contrast how students performed on the Early Grade Reading and Math Assessments at the beginning and end of our program. The vast majority of students enter our program unable to read even a single word. In 2018-19, students graduated from our program reading an average of 40 words per minute, well above the national average.
A dashboard for program management
Given my past experience as an engineer applying user-centered design principles to develop products to support low-resource communities, I believed it critical that we prioritize our intended users (program staff) throughout the dashboard development process to ensure we built a tool that met their needs. To do so, I built iteration and feedback into the design process: at each stage of development, I solicited feedback and co-created elements alongside program staff.
My constant engagement with program staff in Liberia and within the Luminos HQ informed the creation of four distinct dashboard reports:
The Program Overview captures an up-to-date snapshot of the program by pairing student enrollment information with internal field reports and spot checks. Program staff can drill down and filter data by program year, region, and implementing partner.
The Student Assessments dashboard captures the distribution of scores for literacy, numeracy, and words per minute (WPM) in each phase and allows users to disaggregate data by region, implementing partner, student demographics, or classroom rating.
The Classroom Observations dashboard enables users to review a log of classroom observations from field visits conducted by program coordinators. KPIs include facilitator performance and internal measurements of attendance and words read per minute.
The Baseline and Endline dashboard compares the results from our external baseline and endline EGRA/EGMA surveys from the program level down to the individual student level.
With the Student Assessments dashboard, we can investigate the distribution of assessment scores across three subjects: literacy, numeracy, and words read per minute (WPM). Our program staff can deep dive all the way to the classroom level to identify struggling students.
Dashboards in practice
The potential applications of the dashboard are vast. Here is a taste of what I hope to achieve once this new tool is implemented:
Diagnose barriers: Imagine we notice a classroom in which most students scored below the program average. The dashboard allows us to examine this datapoint in context. Has there been an economic shock in the community that caused parents to withdraw their students to work? We can compare the performance of the classroom in question against other classrooms in the same or neighboring communities to determine if this is a shared phenomenon. Perhaps the issue lies with the facilitator. We can review the classroom observations logged by program coordinators over the prior weeks to determine if the facilitator is struggling to grasp the principles of Second Chance’s activity-based pedagogy.
Map trends: With data stored in a centralized database, we can combine external baseline and endline data with internal midline and phase-level assessments to create a picture of students’ learning trajectories.
Promote equity: Luminos disaggregates data by region, implementing partner, and student demographic information — such as gender — to promote equity in program delivery. The success of our Second Chance program has always depended on strong partnerships with leaders and advocates in the community who help us localize the program to meet students and families where they are. With this dashboard, we can easily assess how different sub-populations are performing and address their specific barriers to learning.
As Second Chance classes gradually reopen from months of school closures and interim distance learning efforts, our team is committed to supporting our students, classroom facilitators, and communities. I am currently training our program coordinators to use the dashboard to inform their management practices in anticipation of classes resume. We have already begun the orientation process and will dive deeper into each of the dashboard visualizations in the coming months. My hope is that eventually our program staff will bring their own creativity and curiosity to the dashboard and derive unique insights from the data.
The Classroom Observations and Ratings report helps our program coordinators catalog and review their notes from field visits to classrooms.
So, you want to build a data dashboard….
Design is fundamentally an iterative process. Here are some of the lessons we have learned through many rounds of feedback:
Build for your end users: Who will use the dashboard, and how will they engage with the tool? User interviews and feedback testing are great ways to make sure you build something your users need rather than what you think they need.
Identify KPIs early: It is easy to try and incorporate too much into one dashboard. Enumerating your KPIs early will help avoid scope creep.
Address varying levels of data literacy: Make sure to assess the data literacy of your end users and tailor your dashboard to their comfort level with data visualizations.
Track your data sources: Especially as you begin to combine datasets, it is critical to track where your data come from, how and by whom they were collected and cleaned, and how often the data are updated.
If you’re interested in learning more about the development of the Luminos dashboard, please contact info@luminosfund.org.
