Reports - Evaluation Report

Summary of the Evaluations of the Britain Nepal Medical Trust’s Work During the 1998 to 2002 Period

During the 1998 to 2002 period, BNMT implemented three separate programmes. These were the community health and development programme (CHDP), the drug scheme programme (DSP), and the tuberculosis and leprosy control programme (TLCP). All of these programmes had evolved from earlier initiatives. This paper provides a summary of the three separate evaluations of these three different programmes. Each of the evaluations assessed progress in achieving the stated purposes of each of the programmes.

Lessons from the evaluations of this 5year period of work have led BNMT to integrate all aspects of its health work. In addition, it has prompted the Trust to develop more effective ways to work with local partners as well as to focus its efforts more effectively to improve health for more disadvantaged groups of men and women in the Eastern Region of Nepal. The three separate evaluation reports are listed under references at the end of this document and are available from BNMT in Kathmandu for those who want to know more.

1. The Community Health and Development Program (CHDP)
The purposes of the CHDP programme for the 1998 to 2002 period were to:

	Increase through collaboration the capacity of different partners to be involved in health and health related service delivery (HMG/N, INGOs, NGOs, CBOs and communities)
	Develop people’s abilities to take action for improving their health.

1.1 The Evaluation Process
The objective of the CHDP evaluation (1998-2002) was to verify the impact of the programme’s work. An external consultant who worked with a team of staff and partners led the evaluation. They assessed impact and progress in relation to previously defined qualitative and quantitative indicators. A participatory methodology and set of tools were used. Methods included focus group discussions, workshops, village health mapping, and semi-structured interviewing. Findings were crosschecked with existing programme documentation. A variety of people were interviewed from the Regional Health Director to VDC staff, various health workers, local NGOs and schoolteachers. In addition, 16 focus group discussions were carried out with local programme participants of whom 75% were women. Key areas assessed in-depth were the partnership development approach, resource utilisation, capacity building, empowerment and the behaviour change of local people involved in the programme. Assessment work was carried out in August 2002. No baseline data was provided or contextual analysis of districts where the work went on making it difficult to judge the extent of change brought about by the programme.

1.2 What work the programme did from 1998 to 2002
The CHDP programme evolved from BNMT community health programme efforts going back to 1979. During the 1998 to 2002 period, CHDP work was carried out in 10 village development committees (VDCs) of Sankhuwasabha, plus Khadbari Municipality and two nearby VDCs, and 2 VDCs in Khotang. In these locations, CHDP staff worked with a total of 278 community organisations (CO) that involved nearly 29,000 people. Many of these COs were women’s groups. COs were supported in assessing the causes of poor health in their communities using a Participatory Health Analysis and Action Process (PHAAP). Based on community assessment results, COs were then supported to develop and implement action plans to improve community health.

1.3 Some of the major findings
About 60% of all programme participants said they had learned how to use the PHAAP process and to prepare participatory health action plans. Engagement in these analytic health processes as well as on-going CO activities supported them to acquire new knowledge about specific health issues and this in turn influenced them to change their behaviours. The COs have started making annual plans that reflect their priority health needs and the services they want. Other findings are that

Knowledge and behaviour: More than half of the participants said they had acquired knowledge of a range of health issues such as household sanitation, family planning, health service provision and essential drugs. This they said resulted in behaviour changes. The changes mentioned during the evaluation were greater use of toilets and clean water (mentioned by more than 80% of those interviewed) and more frequent cleaning of the household compound, including the covering household waste pits and food and water containers (mentioned by more than 60%), plus increased use of family planning.

The evaluation also found that members of the COs have changed their health seeking behaviour. More than 80% of the people now visit a health post or other health facility to demand services from them. Visits to Dhami/Jhakri (traditional healers) and hospital visitors continue among a third of all people, as does a strong belief in god. Very few participants (less than 10%) sought services from female health volunteers.

About 70% of participating women said their new knowledge and involvement in the groups has substantially helped them in reducing sickness levels within their families particularly of common diseases like diarrhoea and dysentery by about 25%. This the women said has helped them to substantially reduce family health care costs.

The women groups are supported by their husbands: Almost all husbands supported the participation of their wives in CO activities. This they said gave them the opportunity to learn about health and sanitation issues from their wives. About 50% of the men said that as a result they themselves have improved cleanliness in the household.

The women’s groups have established mechanisms to strengthen group dynamics. This includes regular meetings and small-scale savings schemes. In addition, 30% of women participants said they now practice vegetable cultivation.

Capacity building/Empowerment: The COs have learned participatory planning, monitoring and evaluation techniques. COs with some support are now able to develop and implement their own action plans based on a self-assessment of their health needs. The COs most commonly use local resources when implementing their health plans though some seek support from locally based resource agencies i.e. health facilities and the VDCs and DDCs. While some government health offices responded to CO demands others did not because of budgetary and staff constraints. In fact besides a few health camps sponsored by the VDCs, little was done by government agents.

Resource demands and utilization: A number of COs made demands or requests to others. These included construction of health posts and toilets, free distribution of medicines and training on health issues. As noted above, there was limited response. Thus, most people went and constructed latrines by using locally available resources. They use sufficient water for flushing and cleaning surfaces and built their own waste pits. People are now growing vegetables, beans and other vegetables to get nutritious food.

Partnership: A culture of partnership has just started to emerge and is now gaining momentum. However much work is needed to make them more effective.

1.4 Conclusions and recommendations
The evaluation confirmed that most of the planned outputs were achieved except the one on partnership. Partnership development needs special consideration in the next phase and work to integrate partnership into the participatory development approach used (PHAAP needs to be done. Specific recommendations are that BNMT:

	Continue to strengthen the capacity of partners so that they are better able to respond to the growing health service demands of local people.
	Give priority to partnership development with high potential partners and ensure they are involved from the preparatory stage.
	Continue to use the Participatory Health Analysis and Action process (PHAAP). The momentum gained by PHAAP must not be lost and the capacity of COs and other partners to engage in such a process needs to be strengthened.
	Integrate natural resource generation activities with all relevant agencies including those from health, agriculture and other providers and reflect these activities in the annual plans of COs through the PHAAP process.
	Continue to improve coordination efforts with other actors engaged in health and development work so that the programme becomes more effective.
	Ensure that the priorities of poorer people are addressed in any health action plans and that issues of gender, caste and ethnicity are considered and ways to promote equity advanced.

2. The Drug Scheme Program (DSP)
The purposes of the DSP programme for the 1998 to 2002 period were to:
- Assist in the development, implementation and management of an affordable and sustainable cost sharing drug supply system at local health institutions
- Improve people’s access to essential drugs
- Improve the quality of prescribing, dispensing and storage of drugs at services outlets and promote the rational use of drugs among providers and consumers.

2.1 The Evaluation Process
Two senior staff members of BNMT with advice from a few external advisors led the evaluation. They worked with other staff and partners to carry out a cross sectional survey in 31 health institutions (i.e. District health centres, Primary Health care centres and Health Posts) across the 6 districts in which the programme had worked. These health institutions were randomly selected from a list of health centres where work had gone on. In addition, focus group discussions were completed with 31 health committees and 93 groups of patients, mothers and disadvantaged people. In total focus group discussions involved over 3064 people of which over 60% were women. Structured interviews were carried out with another 3,100 households. A document review was also done. Both participatory and traditional survey methods were used. There was no baseline position against which progress was measured but this was a new scheme and the combined methods, large number of people surveyed and thorough analysis of the findings provide a very complete picture of what was done by the programme and the results it achieved. The fieldwork was done from April to June 2002.

2.2 Work done by DSP from 1998 to 2002
BNMT has implemented a drug scheme programme (DSP) since 1980. Up to 1997, the DSP programme was mainly run by BNMT in the Eastern Region with local drug retailers. In 1998, a new strategy was developed to improve the supply and use of essential drugs in government health institutions. This was to ensure wider availability of drugs to local communities at affordable prices. Such a strategy was developed as drug supplies to government health institutions in 1998 and currently last for only 3 to 5 months.

The new work was run very closely with Ministry of Health staff and local communities. It involved supporting community people to form local health support committees (LHSC) to manage the supplies and funds at each health institution, training committees to strengthen the knowledge and skills of community people in managing drug supplies and funds, and collaborating with government district health staff on regular supervisory visits and meetings to each LHSC. Support was also provided for each committee to establish a revolving drug fund (RDF).

As of 2002, the new scheme was operating in 125 health facilities (51 sub-health posts and 74 higher-level health institutions) in six hill/ mountain districts in Eastern Nepal. LHSCs were engaging the wider community to participate in the management of the programme through community meetings, committee meetings, and trainings. BNMT, the District Health Co-ordination Committee (DHCC) and the District Health Office (DHO) were carrying out supervisory/ monitoring of the DSP jointly. Drug manufacturers and distributors were supporting the programme by providing quality drugs on time.

2.3 Some of the major findings
The findings clearly show that community people through the LHSCs are able to manage drug supplies and the revolving fund; most of them are satisfied with the services provided by the DSP; drugs are now available in participating health facilities year round and to all people at low cost, and the rational use of drugs has improved. Specific findings are that:

Community perception towards the programme: About 61% of households said that all people irrespective of economic class, caste or gender equally benefited from the drug scheme run under the LHSCs. The main reason reported for this was the impartiality in the way the programme was administered with around 80% of households saying that all people irrespective of caste or ethnic group equally benefited. About 19% of households said that the poor benefited the most.

On average, 77% of the informants were happy with the work and approaches of DSP. The disadvantaged groups of the community were the happiest (82%) followed by patients (77%), mothers (75%) and health committee members (74.5%). The main reasons for this satisfaction was that the cost of drugs remained low, the programme was administered fairly (everyone was treated in the same way) and the availability of quality essential drugs was brought closer to their homes.

Programme management: The LHSC jointly managed the scheme with the staff of the local health institution. The SWOT analysis with health committees showed that their strengths in managing the drug schemes were regular meetings, regular drug supply to the health institution, drug transportation, management of physical facilities, establishment of the RDF and regular monitoring. The skills they acquired have created an opportunity for them to carryout public awareness activities with the support of local partners; raise RDF from alternative sources; revise the user's fee; facilitate community groups to participate in monitoring drug scheme activities, and manage the fee exemption for the poor.

Areas they said need improvement are auditing of RDF, more regular staff availability at health institutions, and expansion of the scheme to any remaining health facilities in these districts. They said that the main threats for the continuity of the programme were any stops in the government’s annual drug supply, a worsening of the security situation, and the misuse of the revolving drug fund.

The RDF: Financial contributions for the RDF were available from Village/ District Development Committees (VDCs/DDCs), government (in terms of the annual supplies of drugs), patients (from fees for dispensed drugs) and BNMT (a subsidy on drugs). The RDF has increased each year, and on average, the RDF per health institution was NRs. 57,000 in 1999, NRs. 89,000 in 2000 and NRs. 112,000 in 2001. Currently, the fund is sufficient to maintain a one-year drug supply at each health facility where the scheme has operated for three years..

User fees and service utilization: Health committees have decided the cost of drugs for community people. In 2001, average cost per prescription was about NRs 29 ($0.38), for which patients paid about NRs.18 ($0.23). The committees have monitored the cost of the drugs and the fees paid by patients. Patients, who were unable to pay, often received support from their respective Village Development Committee (VDC).

About 90% of the informants were willing to pay for drugs dispensed at health institutions. The main reasons for their willingness to pay were cheaper/ reasonable price of drugs (29%), effectiveness of drugs (23%), reliability of drugs and services (12%), availability of quality drugs (11%). The reasons why people were unwilling to pay were their belief that drugs should be available free of cost (45%), lack of money (15%), disease not being cured (11%), unavailability of all prescribed drugs (8%) and drugs not being available on credit (5%).

Essential drug supplies: The list of essential drugs to be supplied to the health institutions involved in the scheme was developed through meetings with the DHO/PHO. On average, drug availability in 1999, 2000 and 2001 was 83%, 87% and 90% respectively. In all programme-implemented districts the drug supplies were provided using a private-public approach.

Drug prescribing and dispensing: An analysis of the carbon copies of prescriptions showed that since 1994 till 2001, prescribing habits have improved considerably. The average number of items per prescription has decreased gradually every year (from 2.3 in 1994 to 1.9 in 2001); the percentage of prescriptions with at least one antibiotic has decreased; and the percentage of prescriptions with at least one injection has decreased significantly (from 17.2 to less than 5). In all types of health institutions, AHWs were prescribing most days of the year. Peons, who are not trained in prescribing, were also involved in prescribing at all health facilities. At each health facility, AHWs were involved the most in dispensing drugs. Peons were also found dispensing drugs.

More than 70% of all informants said their first choice for treatment is to go to a local health facility. Of these, 84% received drugs from the health facility during their last visit. Those patients who did not receive drugs at the health facility bought drugs from drug retailers (76%) and from health workers (12%). About 96% of the informants said that they received quality drugs from these health institutions, and the remaining 4% said that the quality of the drugs were poor.

Co-ordination and advocacy: Information and the experiences of the scheme were included in government reports. Based on its experience, DSP provided active support and advocacy in the development of the National Drug Financing Policy-2001, in the revision of training materials for the community drug programme (CDP), in the drafting of National Guidelines on Health Insurance Policy, in preparing National Guidelines on Drug Disposal, and in forming CDP networks at the national level.

2.4 Conclusion and recommendations
The evaluation found that the service users (the communities) and service providers (the health committees, health workers and the district health offices) appreciate the scheme. Financial sustainability of a locally managed drug scheme can work if a cost sharing approach is followed; though not all of the poorest households can benefit . Regular supervision, monitoring and training promoted rational drug use and the LHSC’s management and growth of the RDF. Such processes were found to be crucial for ensuring the sustainability of the scheme. Since, LHCS committees are involved in all decisions about how the scheme is run and they monitor its implementation, finances and management, the programme is community owned and more likely to be sustainable especially where MOH staff carry out their important patient treatment and drug administering roles.

Specific recommendations are that:

	All health facilities that have implemented the DSP for three years be phased out from BNMT support
	The DSP be continued under the CDP Section, Logistics Management Division (LMD)/ Department of Health Services (DHS), Ministry of Health of His Majesty’s Government
	Even after the completion of the handover process, BNMT support the DHOs and DHCCs to monitor the programme for a few years.
	Any future model of the programme, continue to carry out activities that will encourage the rational use of drugs as well as the community empowerment approaches used over the last few years
	The DSP be expanded to all remaining health institutions in these 6 districts plus other districts in Eastern Nepal.
	The possibility of including other partners be explored and included in the chain i.e. CBOs, social institutions and the private sector.

3. The Tuberculosis and Leprosy Control Program (TLCP)
The purpose of the tuberculosis programme for the 1998 to 2002 period was to assist the development of the government’s National Tuberculosis Programme (NTP) by integrating the work of the BNMT TLCP programme.

3.1 The Evaluation Process
The purpose of the evaluation was to assess how the previously BNMT implemented TB/leprosy control programme was functioning after its integration within the government health structures. Three external consultants led the evaluation process. In-depth evaluation work was done at a sample of two primary health care DOTs centres and six sub-tuberculosis centres. At these locations, in depth interviews were carried out with a wide variety of health staff including public health officers, laboratory in charge staff, stores in charge staff and district health officers. In addition, DOTs committee members and male and female TB patients were interviewed, as was the regional health director, the NTP director, BNMT staff and all 8-district health officers. Methods used included semi-structured interviewing, focus group discussions, field observation at TB service centres and a document review. No baseline information was provided or contextual overview of how government TB services were operating prior to this integration work and the type of support HMG itself gave to TB in Eastern Nepal over this period. This makes it difficult to judge the significant of change brought about by the TLCP programme.

3.2 What was done by the TLCP programme from 1998 to 2002
BNMT has carried out TB control work in Nepal since 1969. Until November 1997, BNMT’s work on TB/Leprosy was done by BNMT itself as a vertical program in eight hill districts of the Eastern Development Region of Nepal. As of 1998, HMG/N and BNMT entered into an agreement both at the centre and district levels to promote integration of BNMT’s run TB services into government health facilities. The roles and responsibilities of HMG/Nepal and BNMT were set out in MOUs and agreed by both parties. After a pilot integration phase was successfully completed in Ilam District in 1999, integration into government health facilities in the other seven districts followed. This was a gradual process with much training of HMG staff on the technical aspects of TB control and many meetings with government health staff to encourage provision of high quality TB services.

By 2000, the process of integration was completed at 113 different health facilities in the 8 hill districts. After that BNMT worked only as a supporting agency. There after and in co-ordination with the District Health Officer (DHO) it provided training to government health staff, monitored the quality of service to patients and record keeping, maintained hostels in some districts until the end of 2002 and transported TB & leprosy drugs and laboratory materials from the Regional Medical Stores of Biratnagar to respective district headquarters.

Some brief comment is required on leprosy. I think that the facts were that very few cases o f leprosy were identified and treated, and support for leprosy care was handed over to a Dutch organisation.

In addition, it organised District level DOTS workshops; participated in regional and national level planning and reporting workshops and TB control network meetings; and provided NTC support in organising TB orientation and refresher training and TB/HIV orientations for their staff.

3.3 Some of the major findings
The district health authorities – the DHOs and PHOs think that the integration approach has worked. They all understand the concept of integration. The interviews and discussions with each of the 8 hill DHOs and some PHOs found that they have assumed overall responsibility for implementing the integrated TB programme in their district. Specific findings are that

Integration of TB Services: TB centre treatment in charge staff say they carry out the integrated TB programme with minimum support from outside. They take care of persons suspected of having TB and arrange for their examination, administer medicines to hostel residents and outside cases, identify late patients and/or defaulters, keep required records up to date, and order drugs and essential materials from the district stores. They work closely with respective District TB Leprosy Assistants, HP staff, district hospital laboratory staff, district storekeepers and BNMT district staff.

The integration of the TB programme within the regular government health structure requires that all health institutions are fully staffed. However, overall staffing levels in the health institutions where the TB programme was integrated showed a slight decline in the number of positions filled from 70.9% in 1997 to 68.9% 2001. This average hides a wide discrepancy between districts. For instance in Dhankuta 83% of all posts were filled while in Khotang only 40% were filled. Staffing levels were always higher in the more accessible districts.

Case finding data: For the 8 districts combined, the total cases found have increased from 590 in 1997 to 620 in 2001.Passive case finding varied between 17% and 46% for pulmonary TB by district, reflecting varying prevalence. This is quite an achievement given the constraints faced by government health structures.

Cure Rates: Examination of cure rate trends over the last 4 years (1998-2001) showed that since 1998 the cure rate for all districts combined has increased from 76.9% in 1998 to 82.7% in 2001. By district, cure rates increased in Panchthar, Terhathum, Dhankuta, Bhojpur and Khotang. Cure rates declined in Taplejung, Ilam and Sankhuwasabha. These cure rates are still higher than the cure rates for other districts of Nepal. Cure rates are not standardised for severity of infection.

Treatment success rate: The treatment success rate for all 8 districts increased. It was 82.3% in 1998 and it increased to 83.7% in 2001 although in 1999 and 2000 the corresponding figures were higher. The treatment success rate increased in Panchthar, Terhathum, Dhankuta, Bhojpur and Khotang. However, it declined in Taplejung, Ilam and Sankhuwasabha.

The smear conversion rate increased between 1998 and 2001 in Panchthar, Terhathum, Sankhuwasabha, Dhankuta, Bhojpur and Khotang. They however declined in Taplejung and Ilam.

DOTS cohort analysis: Initial results of cohort analysis of DOTs records showed a satisfactory cure and completion rate of 86.9%. The sputum conversion rate among new smear positive cases, two months after the completion of treatment is 85.8%. These results are equivalently to the targets set under the National Tuberculosis Programme (NTP).

Satisfactory performance of TB microscopy work was another success during this tertiary period. This was due to better quality assurance standards being met on microscopy work. The results of microscopy work have achieved more than a 95% overall agreement rate and less than 5% (2.1 and 2.9%) false positive and false negative results respectively.

The NTC Director and the Regional Director, EDR Health Directorate recognise the heavy and positive involvement of BNMT in the development of NTP guidelines and other TB control programmes in Nepal.

3.4 Conclusions and Recommendations
BNMT’s TLCP work was integrated at most government health service facilities in these 8 hill districts. Overall the standard of the service is running high with the TB cure rate-improving year on year. However, shortage of government staff at various health centres does affect the quality of the service. Transfer of staff from one district to another also affects the standard of service, as new staff must be trained to manage TB work. This is also complicated by the fact that the integration of BNMT’s TB work has increased the workload of government staff. With staff shortages and turnover, BNMT staff often stepped in to substitute and fill gaps. This helped maintain quality services but it is not sustainable in the long term. Given on going staffing constraints, NTC and others interviewed asked that technical and logistical support from BNMT continue and only gradually be phased out.

Specific recommendations are:

	Further integration support be provided by BNMT to strengthen the quality of HMG TB services. This work needs to focus on training and capacity building of HMG health staff.
	This work by BNMT be set against a current baseline and indicators agreed with HMG to verify progress and the agreed point when BNMT support should be phased out.
	Sustainable ways to continue laboratory, drug supply transport and hostel support still need to identified and processes put in place for these responsibilities to be absorbed by HMG.
	A public health education TB campaign needs to be continued in these districts, as all TB patients interviewed were unaware of the symptoms of TB until they suffered from them.

References

Chapa, D. R. 2002. Britain Nepal Medical Trust Community Health and Development Programme: Final Evaluation Report 2002. Britain Nepal Medical Trust, Kathmandu.

Karkee, S. B. and Tamang, A. L. December 2002. An Evaluation of Drug Schemes Programme in Eastern Nepal. The Britain Nepal Medical Trust, Kathmandu.

Karki, Y. B., Malla, R. and Mishra, N. 2002. Integration of TLCP with NTP/HMG in Eastern Development Region of Nepal: An Evaluation. Britain Nepal Medical Trust, Kathmandu.