This is a summary of the findings of a formative evaluation carried out by Mr. Uttam Sharma, a doctoral student at at the Department of Applied Economics at the University of Minnesota. The evaluation was carried out for OLE Nepal's internal purpose. The results are expected to help learn about the shortcomings of the current test phase of the OLPC project so that necessary fine tuning can be done before next year's larger deployment in more districts. Mr. Sharma can be reached at sharm061@umn.edu.Â
Formative Evaluation of OLPC Project Nepal: A Summary
June-August 2008
Background:
The Nepal government’s Department of Education (DoE) and the Open Learning Exchange Nepal (OLE Nepal) had selected Bishwamitra Ganesh Secondary School and Bashuki Lower Secondary Schools, both in Lalitpur district of Nepal, as test schools for the One Laptop per Child (OLPC) initiative in Nepal. The laptops were distributed to all 135 students in grade 2 and 6 in those two schools on April 25, 2008. The students were also allowed to take laptops home. However, charging of the laptops could only be done in school. The implementation of the initiative has focused on four areas: digital content development, teacher preparation, network and power infrastructure development, and government capacity building.
Study design and method:
This study was initiated after about two months of program implementation. The objective of the present study is formative evaluation, which assesses on-going project activities. This evaluation focuses on how the initiative is being implemented in the two test schools and looks at how the implementation can be altered or improved.
The study period was between mid-June to mid-August 2008. The first week was spent preparing survey questionnaires. The first round of field visits was conducted between June 22 and 27. The first three days were spent in Bishwamitra Secondary School, while the last three days were in Bashuki Lower Secondary School. The 2nd round of field visits was conducted between July 27 and August 31, 2008.
This study is based on the information collected from surveys with teachers, head-teachers, students in grade 6 and their family members, some School Management Team members, school records, the school censuses from the two test schools, as well as detailed discussions with OLE Nepal officials and postings in the OLE Nepal blog (http://blog.olenepal.org/). One-on-one meetings with teachers were also conducted in addition to filling out the survey questionnaires.
The questionnaires were designed to elicit as much information about how the initiative is being run. Because of time constraints, we tried surveying only those students whose roll numbers were odd (or only even-numbered in other cases). All the regular teachers who were present in both schools were surveyed. In total, 31 students from grade 2, 45 students from grade 6, 19 teachers (including 2 head teachers), and 20 households were surveyed for the purpose. We also had discussions with many OLE Nepal officials, which aided us in putting our observations in proper context.
Teacher training:
The teacher training was divided into two parts: residential training and in-school training. A four-day residential training was conducted from March 29 to April 1, 2008, and a four-day in-school training followed after a couple of weeks and was completed on May 2, 2008. The residential training included teachers from both schools together and was held in Panauti, in the outskirts of Kathmandu Valley (35 kilometers from Kathmandu), while the in-school trainings were separately conducted in their respective schools. Ten teachers from Bashuki and 11 from Bishwamitra participated in the residential training.
One of the goals of the training was to make teachers comfortable integrating laptop use into regular teaching. When the teachers were asked whether the complete training was sufficient for enabling them to properly integrate laptops into the regular classroom instruction process, 50% of the teachers’ responses were positive. This response underscores the need to have refresher training on a regular basis, as most of the teachers are using computers for the first time.
Residential training:
Teachers found the following aspects of the residential training useful:
Among those who said there were parts of the residential training that needed to be reduced or removed next time, all of them cited that a lot of time was spent on going over theoretical parts, which limited time spent on practical aspects of using laptops.
With regards to whether the materials in the residential portion of the training were adequate, we obtained conflicting answers from the two schools. While the overwhelming majority of the teachers in Bishwamitra (89%) thought that materials were adequate, less than one-third (29%) of teachers in Bashuki said that it was adequate. Among those who said it was inadequate, they wanted the focus to be more on practical rather than on theoretical aspects. Some teachers wanted more examples on how to teach using laptops. Though the majority of teachers at Bashuki found the materials inadequate, more than two-thirds of the teachers there (71%) agreed that the residential training was delivered in an understandable manner that was easy to follow. For Bishwamitra, the corresponding number was 89%.
When the teachers were asked whether it was a good idea to run residential and in-school trainings separately, almost everyone (15 out of 16) said that it was a good idea and wanted it to be that way in the future as well.
In-school training:
Teachers found many aspects of the in-school training useful, such as:
To the question of what aspects in the lesson planning practice they found particularly useful, they mentioned:
All 17 teachers found it useful to review lesson plans together in a group and to attend laptop classes conducted by colleagues and observe the class.
Classroom teaching-learning process:
Teaching-learning digital materials:
All the teachers said that the materials in the laptops were, in general, consistent with the interests of the students in their classes. However, when asked whether the materials in the laptops, in general, were consistent with the level or capacity of the students in their classes, the answer was rather mixed. Teachers were generally satisfied with the grade 2 contents but had diverse views on grade 6. The teachers found many initially uploaded activities for grade 6 either too easy or too difficult. To the question about whether the materials in the laptops are able to meet the aims and objectives of your class as prescribed in the curriculum, the teachers feel that the recent digital contents are more relevant in this front.
The teachers mentioned that they generally spent one period to go over one activity. This means that if the expectation is that the teachers use laptops for three days a week, then there should be about three activities prepared for each week.
The teachers also mentioned that there were not sufficient exercises, especially for grade 6 students to work on. Since the teachers are so used to following the chapters in the book, the teachers suggested that the teaching materials in the laptops be sequenced according to the chapters in the textbooks. This way, they will not have to spend a lot of time figuring out which activity corresponds to the relevant chapter in the book.
The teachers also suggested that there be multiple-choice question answers and practice questions. Some teachers proposed having different levels in the activities so that students develop a competitive spirit and try harder to go to a higher level. This could motivate them to study harder.
Classroom teaching-learning:
It is important to understand how classroom teaching-learning has been affected by laptop use in classrooms. All the 17 teachers in both schools feel that the use of computers has helped their teaching. All (relevant) teachers feel that their lectures are now more organized and that it is easier to teach students new concepts. They also believe that the use of laptop-based instruction has enabled students to learn concepts more easily. They are also of the opinion that laptop-based instruction has made it easier to give students more practice exercises. In addition, they also feel that the use of laptop-based instruction has made the classes more interactive. All the teachers are unanimous in saying that the use of laptop-based instruction has increased the students’ interest in their studies.
In Bishwamitra, teachers said that they generally use laptops 2 to 3 days a week, while the number, according to the teachers, is at least 3 days in Bashuki. When they use the laptops, they mostly use them between ¼ and ½ of the period. The grade 6 student survey, which was taken independently, also largely confirms this assertion. The mode in Bishwamitra was 3 days, while it was 4 in Bashuki.
Students in grade 6 said that teachers mostly used laptops around the middle of the class. When some of the students forget to bring their laptops, the teachers ask those students to look at the laptop of the student who is sitting next to them.
The students in Bishwamitra generally say that the teachers give them homework every time the laptop is used in the classroom, while those in Bashuki said that they are sometimes given homework. According to the teachers, those homework assignments are mostly of review type to be done on laptops on a regular basis. The teachers ask students to do certain activities in the laptop for that day. The teachers, at times, look at the record in the computers to see whether the student has at least opened that activity at home or not, as one could find those details in the laptop.
When the students were asked how learning has been affected with the introduction of laptops in their classrooms, more than 95% of grade 6 students find learning using laptops easier.
As to why it is easier, the most common response was that you can do the exercise or activities as many times as you want. The second most common one was that these activities were very enjoyable to learn from. The students also appreciated the fact that you could do these activities at your own pace. All these options were cited by more than 50% of the students.
When the students were asked whether other family members also use the laptops, more than eighty percent said yes. Siblings are the ones who use the laptop the most, followed by fathers and mothers. This information is also confirmed with data from the household survey, where many siblings said that they use the laptop regularly.
One question that might reveal a lot about how the students are using the laptop was about what the students did the most on the laptop. In Bishwamitra, 68% of the grade 6 students (15) said they mostly use the laptop for educational activities, while 23% (5) said to take pictures. The number for Bashuki was 32% for laptop activities, while 42% was to take pictures.
When the grade 6 students were asked how effectively they are able to use digital activities in the laptop, most of them said they can use it well. 83% in Bishwamitra said that they can use it well, while 17% said they can use it somewhat. In Bashuki, the corresponding figures were 75% and 25%, respectively. No one said they cannot use the digital materials, even though it was one of the options given.
Some teachers feel that the introduction of laptops has made managing the classes more difficult. Many teachers feel that the noise level has substantially increased. This has to do with the fact that students have difficulty controlling their excitement. The teachers hope that the noise level will subside once the students get used to using laptops. The teachers also admit that they are, at times, having difficulty managing time because they are not used to integrating laptop use in classrooms.
Class preparation:
When teachers were asked, “Compared to the effort you had to put into the teaching-learning process before receiving the laptops, how much effort do you put in now?”, the overwhelming majority of the teachers feel that they put in more effort now.
Some of the reasons cited by the teachers on why the effort has increased:
Laptop and network issues:
Almost all teachers (16 out of 17) find the laptop and its layout easy to use. They think it is very intuitive and also feel that the students should not have much difficulty. Most of the teachers say it takes about 1.5 hours to fully recharge the computer and that the charge lasts for about 3 hours when the computer is in use.
We had also asked the teachers, students, and their family members whether they had any problems with different aspects of computer use. The biggest problem, mentioned by more than 50% of the respondents, was the jumpy cursor. The dust, the humid climate, and the students’ sweaty hands as a result of running around might be contributing to the situation. The problem seems to be slightly more acute in Bashuki.
More than one-third of the teachers thought the sound quality needs to be improved. There either is some problem with the sound quality or that some teachers might have difficulty adjusting the sound. Since earphones are not available, difficulties arise for students and teachers when using sound-related activities. They largely ask students to listen to the sound from the teacher’s laptop while teaching.
The school, students, and their family members, in general, are carefully protecting the laptop. At home, the students keep the laptop away from fire and water. Those who have a closet at home keep the laptop there when not in use. Other students securely keep the laptop in the bag. The family members are aware that the laptop should be stored securely.
Difference between the two test schools:
There are reasons to believe that the two schools are different. For example, teachers in Bashuki say that the community members there do not value education as much. The standardized tests taken in the early days of laptop provision also show that the test scores in Bishwamitra are much higher than those of Bashuki. In addition, the attendance of students in Bishwamitra is more regular than in Bashuki.
There are differences in responses to some of the critical questions in these two schools, which could be because of the reasons identified above. So, it is very essential to identify the root causes behind these differences. One could argue that these differences are hinting at the fact that there might be certain enabling conditions that need to be in place to make sure that the program’s objectives are fully met. It could also mean that it might take time in some places to fully benefit from the initiative, while we could see tangible results in some schools right away. For the pilot study, OLE Nepal might want to seriously consider whether all the schools in the pilot phase need to be treated similarly or whether some activities, like sensitizing the community about the merits of ICT-in-education, need to be conducted in certain areas beforehand.
Some Suggestions and Recommendations:
E-library and internet access:
I think OLE Nepal should try to upload all the books that are on the website to student laptops sooner rather than later. This serves at least two purposes. First, students who want to read those books could easily read them. Second, the users (teachers, students, and their family members) might offer some suggestions that might help further improve the way the e-library is accessed (e.g., layout of the books). Needless to say, the parents and teachers are really looking forward to using the e-library.
Without internet access at home, convincing others that one laptop per child is preferred to, say, a well-planned computer lab in the school might be tougher, especially when the cost difference between these two alternatives can be substantial.
Workload on teachers:
Most teachers in both test schools feel that their workload has increased significantly. This is a serious concern. If this trend continues, it is likely that they will revert, to a large extent, to the traditional way of teaching and ignore the materials in the laptop once the initial enthusiasm for laptops fades away. If this happens, the effect of laptops with educational content on student learning might be minimal. Steps to further reduce workload and encourage laptop use in classrooms need to be devised carefully.
Some ways to decrease teachers’ workload include:
It is apparent that teachers don’t have much free time in the two test schools. Most of them have to teach for six periods (out of seven) in Bashuki. The case is not much different in Bishwamitra. Moreover, for the teachers at Bashuki, almost all the teachers there have to walk more than 2 hours a day to reach the school. All these factors, coupled with extra effort in laptop use, might tempt teachers to revert to the style they are most comfortable with.
Chargers:
I strongly feel that OLE Nepal should take steps to allow students to take chargers home so that the laptop can be used longer and that other family members or community members can also benefit. If the charging is only done at school, there is concern that students might not be able to use the laptop adequately at home once more activities are uploaded to the laptop. If OLE Nepal expects students and their family members to use the laptop for materials in the e-library at home, it makes more sense to allow students to take chargers home.
I think OLE Nepal should go ahead with the plans about providing 2 chargers per student with laptops, so the students can take one charger home, and the other one can be permanently kept at school. This way, the students do not need to worry about carrying a charger every day. The chances of losing or misplacing one’s charger are also minimized.
Books by Janak Shikshya Samagri (official books used in classrooms) in the laptop:
One concern that was raised by teachers and family members in Bishwamitra School was that the load the students carry is huge. It is particularly dangerous during the monsoon season. Many suggested having soft copies of the books used in the school included in the laptops. This would also be one of the immediate and tangible results of the laptop provision. Disruptions in classrooms due to late delivery of books, which happened this academic year, would also be minimized.
Allowance and recognition for teachers:
Though I consider providing regular compensation for extra work resulting from laptop use as highly impractical, I however think some arrangements should be made to compensate teachers when they attend programs like teacher trainings. In addition, OLE Nepal might also want to provide them some recognition (e.g., letters) that teachers, for example, could use for professional development.
Conclusion:
Considering the fact that this is the first time such an initiative is being implemented in Nepal, the test phase in the two schools in the outskirts of Kathmandu Valley is running very well. The teachers, family members, and the SMC members all see great promise in this initiative to improve student learning. The head teachers in both schools consider the pilot program as very effective and see great promise in reducing the disparity in educational quality between private and public schools.
Almost all the teachers and parents think that there is a positive relationship between laptop