Lifang Zhang, Yan Jia, Yanchun Zhang, Ning Wu, Jiangmei Qin

Performance evaluation indicator system for the implementation of essential drug system in community health service institutions

Lifang Zhang1, Yan Jia2, Yanchun Zhang1, Ning Wu1, Jiangmei Qin1

Objective:This research aims to develop a more scientific and reasonable performance evaluation indicator system for the implementation of an essential drug system in community health service institutions.

Methods:The Delphi method was used to establish an indicator system based on three rounds of expert consultations, and the fuzzy comprehensive evaluation method was used to determine the weights of the indicators.

Results:The participation in the three rounds of consultations were 100% (10/10), 90% (18/20), and 85% (17/20), which showed that the experts had real enthusiasm for participating in this research. The authority coefficients of the first-, second-, and third-level indicators were 0.75, 0.76, and 0.76, respectively, which showed that the consultation results were dependable. The concordance coefficients of the second and third rounds were 0.489 and 0.487, respectively (P＜0.001), indicating that the expert opinions were highly consistent. The performance evaluation indicator system consisted of three first-level indicators (supporting, implementation, and effect indicators), nine second-level indicators, and 21 third-level indicators.

Conclusion:In this new performance evaluation indicator system, the selected experts were representative, the consultation results were dependable, the constructed evaluation indicator system was reasonable, and the setting of weights was scientific.

Essential drug system, Performance evaluation indicator system, Community health service institution, Delphi method

Introduction

The essential drug system has been officially implemented in China since August 2009, and plays an important role in promoting reasonable drug use, easing the economic burden of basic medications for patients, and helping to establish a new grassroots operating mechanism. The establishment of the essential drug system is one of five priorities of the health care system reform. The progress and performance of the essential drug system have thus become key targets for monitoring. However, most of the monitoring indicators in the“Monitoring Plan for Health Sector’s Five Priorities of Health Care System Reform”and the “National Essential Drug System Monitoring and Evaluation Indicators” issued by the Ministry of Health were developed at the provincial level, and few of the indicators were issued by grassroots health care institutions. As the main sector responsible for implementing the essential drug system,grassroots health service institutions are crucial to its sustainable implementation. It is therefore important and necessary to strengthen the performance evaluation of the grassroots health service institutions’ implementation of the essential drug system.

Methods

Design of the indicator system

Performance evaluation differs from monitoring and assessment. Monitoring and assessment focus more on timely, continuous, and systematic tracking of the internal and external factors involved in implementing the essential drug system in order to evaluate its performance, and identify and solve problems to guarantee its smooth implementation and realization of the expected objectives. In contrast, performance evaluation focuses more on the evaluation of the tasks accomplished by the grassroots health service institutions in implementing the system and on formulating the corresponding incentive measures.

The current study involved urban community health service institutions and developed a performance evaluation indicator system by selecting three links (essential drug provision, essential drug use, and zero profit sale) related to the tasks of community health service institutions. A number of aspects of the essential drug system, including catalogue content development, production, and supply, procurement and delivery, reasonable use, price management, payment and reimbursement, quality control, and monitoring and evaluation were covered. Three levels of indicators were involved, including supporting indicators (policy support and human resources), implementation indicators (process indicators), and performance indicators.

Literature review

The performance evaluation indicator system was preliminarily developed after reviewing the literature on performance evaluation to learn the methods and principles involved in developing a performance evaluation indicator system. The literature on community health care development was also reviewed, and consideration was given to the characteristics of the community health service institutions.

Analytic hierarchy process

The current study developed a three-level performance evaluation. The first-level indicator covered three dimensions (supporting, implementation, and performance). The supporting indicators mainly included supporting policies and human resources for ensuring the implementation of the system in the community health service institutions. The implementation indicators comprised essential drug provision, essential drug use, and zero-profit sales. The performance indicators included the effects of system implementation on knowledge, cost control, reasonable drug use, and satisfaction. The second-level indicators were the results of the decomposition of the firstlevel indicators, while the third-level indicators were more specific than the second-level indicators.

Delphi method

An expert consultation table was designed and three rounds of consultations were conducted. In the first consultation, the experts selected the performance evaluation indicators. In the second consultation, the experts gave scores to the importance, sensitivity, and feasibility of each indicator and provided suggestions for the modification of the indicators. The performance evaluation indicators for the community health service institutions’ implementation of the essential drug system was developed based on the suggestions of the experts.

Statistical methods

Expert consultation tables were developed and the database was established using Epidata or Excel. Questionnaire retrieval and expert information were analyzed using SPSS 17.0. The expert scoring was analyzed using statistical methods, and the indicator weights were determined using the fuzzy comprehensive evaluation method.

Results

Expert information

Based on the authority, representativeness, and professional fields of the experts, and taking into consideration the trade-off between the number of experts and difficulties in conducting the consultation, 10 experts were selected for the first consultation and 20 experts were selected for the second and thirdconsultations in the study. The expert information is given in Table 1.

Enthusiasm, authority, and concordance coefficientsThe enthusiasm of the experts refers to the number of expert consultation tables retrieved, which indicates how concerned the experts were about the study. Generally, if at least 50% of the expert consultation tables were retrieved, the tables were used for analysis and report but retrieval of 70% of the expert consultation tables was preferred [1]. In the current study, 100% (10/10), 90% (18/20), and 85% (17/20) of the consultation tables were retrieved in the first, second, and third rounds of consultations, respectively, which indicated that the experts were very concerned about the study.

Authority coefficient

The authority coefficient was determined by two factors [2], i.e., expert’s judgment with respect to the questions (coefficient of judgment) and expert’s familiarity with the questions (coefficient of familiarity). The authority coefficient was the arithmetic average of the two as follows, with a value range of 0–0.95: authority coefficient=(coefficient of judgment+coefficient of familiarity)/2. Generally, an authority coefficient ≥0.70 was considered acceptable [3]. The theoretical values for experience, knowledge, and intuition between domestic and foreign counterparts were 0.8, 0.6, 0.4, and 0.2. The values for very familiar, familiar, unfamiliar, and completely unfamiliar for the coefficient of familiarity were 1.0, 0.8, 0.5, and 0.0, respectively. After calculation, the authoritycoefficients of the experts for the first-, second-, and third-level indicators were 0.75, 0.76, and 0.76, respectively, i.e., all ＞0.70, which indicated that the experts had high authority.

Table 1. Basic information of the experts

Concordance coefficient

The concordance coefficient indicated the agreement of the experts over the indicators. The concordance coefficient can be used to determine the level of disagreement among the experts over each indicator. The concordance coefficient simultaneously indicates the reliability of the consultation results [4]. The concordance coefficient is between 0 and 1; the greater the coefficient, the higher the degree of agreement among the experts. The Delphi method indicated that, after two to three rounds of consultations, the concordance coefficient was between 0.4 and 0.5 [5]. In this study, the Kendall method was used to calculate the concordance coefficient. The concordance coefficients of the second and third rounds of consultation are 0.489 (χ2=764.028,P＜0.001) and 0.487 (χ2=787.761,P＜0.001), respectively, which indicated that the experts shared a high degree of agreement in the two consultations, and the results were therefore valid.

Indicator weights

The influence of the experts’ familiarity and judgment meant that the experts’ scoring of the importance, sensitivity, and feasibility of the indicators was ambiguous. The fuzzy comprehensive evaluation method was therefore used in the current study to determine the indicator weights.

The factor set of the evaluation target was defined as U={X1, X2, X3}, where X1 represented the importance of the indicator, X2represented sensitivity, and X3 represented feasibility, and the weights of the three factors were defined as U=(0.40, 0.30, 0.30).

The evaluation grade was defined as V={V1,V2,V3,V4,V5}; V1 indicated a score of ＞9 but ≤10, V2 indicated a score was＞8 but ≤9, V3 indicated a score of ＞7 but ≤8, V4 indicated a score of ＞6 but ≤7, and V5 indicated a score of ≤6. Thus, V=(10,9,8,7,6).

For the qualitative evaluation indicator, u, the experts evaluated over one scheme and established a fuzzy subordination degree, r to V, where ri=di/d, d was the number of experts participating in the evaluation, and di was the number of experts who gave Vi to the evaluation indicator, u. The fuzzy matrix of the indicator was thus developed.

The comprehensive evaluation model was established as Yi=U×Rij×VT, where T was the transpose of V.

Calculation of indicator weights

A comprehensive score, Y, and weight, P, were calculated for each indicator using the above process (Table 2).

Discussion

The experts selected were representative and the consultation results were valid. The choice of experts was crucial for the Delphi method. The experts selected in the first round of consultation were those with rich experience in research and teaching, and having a professional title of deputy senior or above, who were renowned in the fields of community health services and the essential drug system. In the second and the third rounds of consultation, in addition to the Directors of the Ministry of Health and Provincial (Municipal) Health Bureau, who were responsible for community health services and the essential drug system, and Directors of the Community Health Care Service Center, specialized experts were also included. The performance evaluation indicators were selected by those experts who had theories and practical experience, such as researchers and teachers, health care administrators, and practitioners of community health service, to ensure that the indicators were scientific, and at the same time, practical.

The enthusiasm, authority, and concordance coefficients were the key factors determining the validity of the consultation results. In the three rounds of consultation, 100%, 90%, and 85% of the consultation questionnaires were retrieved, respectively. The authority coefficients for the first-, second-, and third-level indicators were 0.75, 0.76, and 0.76, respectively. The concordance coefficients in the second and third rounds of consultation were 0.489 and 0.487 (P＜0.001), respectively. It can thus be seen that the experts were enthusiastic about this study. The authority coefficient indicates that the consultation results were valid; the experts shared a high degree of agreement regarding the indicators, so that the performance evaluation indicators were very practical.

The structure of the indicator system was reasonable and the weights were scientific. The weights of the supporting,implementation, and effect indicators were 0.3215, 0.3379, and 0.3407, respectively. Thus, all the indicators were indispensable for evaluation of the community health service institutions’ implementation of the essential drug system. The fact that the effect indicator had the highest percentage suggests that effect will be the key target for performance evaluation. The second-level indicators were the results of decomposition of the first-level indicators, and the third-level indicators were more specific than the second-level indicators. Thus, the performance evaluation indicator system developed comprised three first-level indicators, nine second-level indicators, and 21 third-level indicators. The indicator hierarchy is reasonableand the indicator system reflects the implementation status and effect of the essential drug system in the community health service institutions. The indicator hierarchy can serve as an important tool for reviewing and evaluating the implementation of the essential drug system.

Table 2. Comprehensive scores and weights of performance evaluation indicators for implementation of the essential drug system in community health service institutions

The fuzzy comprehensive evaluation method is a comprehensive review method based on fuzzy mathematics. This comprehensive evaluation method transforms the qualitative evaluation into a quantitative evaluation according to the subordination degree theory of fuzzy mathematics. Thus, fuzzy mathematics can be used to provide a comprehensive evaluation of targets that are restricted by multiple factors. This method is clear and systematic and can solve fuzzy problems. The fuzzy comprehensive evaluation method can be used to solve various non-deterministic problems [6]. In the current study, some qualitative indicators were difficult to quantify and the fuzzy comprehensive evaluation method was used to determine the indicator weights.

China is a large country and development gaps inevitably exist among community health service institutions in different regions. Implementation of the essential drug system also differs significantly. The performance evaluation indicator system developed in this study had some limitations. First, it did not distinguish between community health service centers and community health service stations. If community health service centers and stations were managed under a single unified system, the center’s implementation of the essential drug system should be included in performance evaluation; if not, the evaluation indicator systems used for centers and stations will be different. Second, the study did not take into consideration the differences between governmental and non-governmental community health service institutions. Currently, most community health service institutions implementing the essential drug system are governmental. However, with the progress of the reform, more and more non-governmental community health service institutions will implement the system, and the differences between the two types of community health service institutions will thus need to be considered in the performance evaluation. The performance evaluation indicator system should be improved along with the adjustment of policies and practices.

Conflict of interest

The authors declare no conflict of interests.

1. Albabi, translated by Li Y. Social research methods. Sichuan, China; Sichuan Renmin Publishing House; 1987.

2. Zeng G. Modern epidemiological methods and application. Beijing, China; Beijing Medical College and Xiehe Union Medical College Publishing House; 1996.

3. Peng Y, Wang Y, Chang W, Liang W. Study of evaluating index system on general treatment quality. Chin Gen Pract 2004;7: 158–60.

4. Su H. Research on performance evaluation framework of public health service system. Wuhan City, Hubei Province, China; Tongji Medical College, Huazhong University of Science and Technology; 2010.

5. Li J, Liu M, Zhou L. Establishment of evaluation indicator system for medical research programs. Chin J Med Sci Res Manag 2005;18:33–6.

6. Chen X. Research on fuzzy comprehensive evaluation method applied in health administration. Soft Sci Health 2011;2:597–9.

1. National Health Development Research Center, The Ministry of Health, Beijing, China

2. Beijing Shijitan Hospital, Capital Medical University, Beijing, China

Lifang Zhang

National Health Development Research Center, The Ministry of Health, Beijing, China, 100191

E-mail: zlf0635@163.com

Funded by the “Research on Performance Evaluation of Chinese Community Health Service Institutions’ Implementation of Essential Medicine System”supported by National Natural Science Foundation (71103130)

5 November 2013;

Accepted 27 December 2013