Evaluation of Leadership Training Program Using Kirkpatrick Model Case Study in Telkom Corporate University

This study aims to examine the evaluation of the leadership training program conducted by TCU. The training program is namely the Great People Managerial Program (GPMP) where the program is designed for managerial positions of employees in terms of developing leadership competencies for the position when the program is run. The evaluation carried out by the Leadership Academy department for the GPMP training program was only to assess the learning impact based on the results of the pre-test and post-test. However, they have not carried out further evaluations, regarding the performance of participants who have completed GPMP training, such as improving employee performance toward changes in individual behavior. The findings obtained from this research are derived from questions adapted from opm.gov (U.S. Personnel Management Office). Utilizing primary data based on Kirkpatrick’s level 3, it reveals the answers to their behavior changes after participating in the GPMP training. Then, the researcher is also used 360-degree evaluation feedback in the form of a questionnaire distributed to supervisors, co-workers, and also subordinates from participants who were evaluated by the researchers. Finally, evaluation of individual behavior after training is considered as Good and must be implemented. Yet, there are still many improvements that need to be conducted, for further improvement in individual behavior obtained from 360-feedback evaluation.


INTRODUCTION
Leadership is a process that can influence others to achieve certain goals (Kruse, 2013). A leader is someone who has influence, power, authority, followers and has a concept in interpreting all work becomes easier. Leadership skills are needed to inspire others to act according to plan in order to achieve the stated goals (Starks, 2015). True leader is a leader who can direct their subordinates and provide a way for them to become new leaders in the future (Schmitt, 2013). A leader who can create new leaders is a successful leader. This is the essence of leadership skills. Even though it is not easy, leadership skills can be learned by anyone, including those who don't have leadership talent. Telkom decided to develop and transform the company. Telkom management chose a strategy that places human capital as the node of the entire process. Consequently, the emergence of the need to develop people who are highly qualified people and great leaders. The true function of a company is to create a leader (Rudito, 2014). When an organization is able to produce leaders in large numbers, these great leaders will become the backbone of the organization's long-term success. Human resources are obtained from three choices. First, finding qualified human resource on the market. Second, developing talent from current Telkom employee and third, a combination of the two.
Telkom Corporate University (TCU) is a strategic vehicle that was designed to support in achieving Vision and Mission of Group of PT Telekomunikasi Indonesia, Tbk. (Telkom Group) by carrying out various activities to strengthening individual and organizational learning skills in order to actualize the Centre of Excellence. TCU requires units that are able to accommodate the needs of Customer Facing Unit (CFU) and Functional Unit (FU). It is called Academy. Referring to the explanation of the theme, namely "Evaluation of Leadership Training Program Using Kirkpatrick Model". This research will explore about the evaluation leadership training of GPMP 4 2018 in Telkom Corporate University. At the end of this discussion, we will find out whether there has been a comprehensive evaluation or not based on the improvement of employee performance towards individuals, groups & departments.

Figure 2. Evaluation Process GPMP
The problem identification will be discussed based on the following time line Figure 2. TCU has spent a large amount of funds to provide employee leadership training from various levels. But, there has been no comprehensive evaluation regarding the effectiveness of leadership training on the improvement toward change behavior. The evaluation that has been carried out by the Leadership Academy for the GPMP training program is only to assess the learning impact (LIM) based on the results of the pre test and post-test. (1) Telkom employees who are participating in the training will be given a pre-test form before the GPMP leadership training program start and will be assessed by the Leadership Academy department. The pre-test covers what will be learned during the GPMP leadership training. (2) The participants took part in the GPMP program for 10 to 15 days of training. They will learn various things that can support their careers and improve their leadership skills.(3) After GPMP training finished, post test is held again with the exact same questions or may resemble a pre test form question. The participants are expected to get a better score than the results of the pre-test. (4)The Leadership Academy department compared the results of the pre-test and post-test whether there was a significant increase in final score or not, after the participants took part in the GPMP training. The evaluation that has been carried out by the Leadership Academy is still not comprehensive yet. They evaluate only by comparing the results of the pre-test and post-test. Nevertheless, they have not carried out a further evaluation, regarding the performance of the participants who have completed the GPMP training, such as the improvement of employee performance towards behavior change after the participants took part in the GPMP training.
the field of HRD (Stewart and McGoldrick, 1996). One strategy approach for HRD is the corporate university concept. Learning academy always strives to support and respond to organizational-based demands for management study and circumstance (Prince and Stewart, 2000). Lester (1999) has difficulty describing the meaning of a corporate university. To study the conception of a corporate university, it is compulsory to have an observation at the basis of the roles, purpose and action applied by the company. Based on Meister (1998) a corporate university is defined as a strategic umbrella for growing and educating worker, buyers, and distributor to meet the organization's business strategy. Further, astudy made by Fresina (1997), Dansford (1998) andMeister (1998), showed that the principal thing in determining a corporate university is focusing on achieving institutional goals and priority interests, compared to traditional approaches where training is based on individual needs. If the statement can be accepted, then the definition of a corporate university in the modern world will be more able to be developed, such as by forming a superior leadership team that can compete with other firms (Global Learning, 1998). Today many companies are trying to become organizations that continue to study and evaluate their employees. According to Walton (1999), an organization that responds thoughtfully to the development of learning can be seen from their struggle in learning activities of individuals and organizations. The company becomes a facilitator in this matter. Fresina (1997) tells the role and purpose played by a corporate university. Consists of three parts, namely: Reinforcing and perpetuating behavior. The corporate university has the primary responsibility in providing curriculum and teaching that can bring the culture and values that will be implemented to support the development of the organization. Managing change. Another task undertaken by a corporate university is to construct and introduce organizational changes. Programs designed for employees to design activities for the successful implementation of strategic change. Driving and shaping the organization. The last role might be said to be very ambitious with results that are not very visible. Here the company leaders make corporate university as forming the direction and future of the organization. The change program is related to the search for new contexts and alternatives for the company's future. This involves the manager and all employees taking part in training to look for things that currently do not feel unsatisfactory. Nevertheless, this can be faulty for the future. Martel (2001) argues that for now there are still a small number of companies that evaluate the influence of corporate university training effectiveness. So far they have viewed corporate universities as a fine tool to maintain employee talents and build more competitive organizations. Based on the phenomenon above, it is belived that Kirkpatrick's (1959Kirkpatrick's ( , 1976Kirkpatrick's ( , 1996, the four-level model of training evaluation, is the most influential model to date. This evaluation system has four main categories, namely: reactions, learning, behavior and results. Reactions criteria refer to emotions during a teaching agenda. Learning measures understanding and comprehension acquisition. Behavior measured by changes in behavior when after training will be measured again. And for the results measure the impact after training on established organizational indicators (increased productivity, savings in expenditure, and employee morale). In the 2000s, the principle of the Kirkpatrick model was remaining the same as in 1959 when this model was first used for evaluation purpose. Level 1: Reaction At this level, it is measured concerning customer satisfaction and also the response of trainees participating in the program. In general, trainees who emphatically serious, will not mind if they have to spend the cost of an in-house training program, they only need to prepare time and readiness to take part in the leadership training. In the circumstance they had to pay to take apart in training, the participants' reactions were measured. Other responses such as their opinion on training, whether they will attend the next training, and also whether they will recommend this training to their coworkers. For university partners, professional HRD companies, and also the related consultants must identify the reactions of the trainees during the training program. An in-house situation, the opinions of other people related to the training participants, have a considerable impact on the next training program and also the training budget. Training programs expect reactions from the trainees, due to two reasons: the trainer highly appreciates the opinions of the participants, and for measuring the responses of participants to get advice for better training in the future.

Level 2: Learning
To find out the extent of learning from training. Three things can be achieved in a training program: 1. Understand the notion, foundation area and techniques provided during training. 2. Develop and improve the ability of trainees. 3. Change symptom and manner. The training program has the same goal, specifically to increase the knowledge of the trainees. There are also several training programs that have specific objectives to improve technical skills as well as sales to build better individuals in the corporate environment. For programs such as "Leadership Training" it aims to develop leaders' talents. Learning assessment must be specific to establish leaders who are certified and have competitiveness in the worldwide work environment. Guidelines for Evaluating Learning 1. Make a pre-test (before training) and post-test (after training) that aims to measure knowing related to the material, skills and attitudes. 2. Use written tests with paper and pencils/pens for learning and attitude. 3. Use performance tests for skills. 4. Get a 100 per cent response. 5. If possible, compare groups that did not attend training and those who attended training. Level 3: Behavior To measure the influence of trainee behavior in the workplace as an outcome of attending a training program. This level is the most complicated to evaluate and most necessary compared to other levels.If the training participants do not practice what they get from the training, starting from what they have learned, it can be said that the training program was unsuccessful. Hence, it is essential to measure changes in participant behavior, the aim is not just to see whether the behavior has changed but to determine what elements cause changes in behavior that do not occur. This would be impossible to evaluate the entire program regarding changes in behavior. Nevertheless, an individual approach must be taken to get the results you want to know.

Guidelines for Evaluating Behavior
1. If possible, conduct assessment before and after training. But in reality, it is not possible to do that. There has to be a requirement to evaluate the training program is completed and identify any factors that individuals do before participating in the training, then compare with individual habits after take a part in the training. 2. Behavior frequently changes over time, so it takes time for participants to apply to their work. 3. Conduct a survey and interview as follows: a. Training participants. b. The leader of the training participants. c. The subordinates of the training participants. d. Colleagues who monitor the behavior of trainees. 4. Get responses from 100 per cent sampling. 5. Repeat in the right period. 6. If feasible use a control group. 7. Examine the benefits of evaluating against costs incurred.

Level 4: Results
To find out how much the results of the training program. The impact that can be determined by several factors such as less turnover, an increased amount of work, quality improvement, budget reduction, reduced time wasted, the boost in sales, company profits, and a return on investment (ROI). Just as in evaluating behavior, assessment is carried out on programs that are considered important and have substantial expenditure funds. It is suggested that ROI should only be made around 5 per cent of the firm's programs.

Guidelines for Evaluating Results
1. Take measurements before and after the training program. 2. Give space and time for results that might occur. 3. Repeat the right period. 4. If decent use a control group. 5. Take into consideration the possible benefits versus assessment costs Current applications of the four levels Four-Level is a perfect model for developing training programs that are needed by the company. For the company business model is very relevant to this evaluation model. Just as many companies offer specialized training ranging from Time Management to Sales Effectiveness, this model is very aligned with business needs that continue to emerge over time. To have an effective approach, it requires the intelligence of a training leader, because each individual has a different approach tendency. In the development of training programs must be based on what the company needs and evaluation are usually based on the requirements, the problems being faced, and the opportunities they have. Of course, to hold a new training must be adapted to the business needs that are truly needed by the company.

Evaluating Training Program (Kirkpatrick)
Basically, the reason for evaluating is to be a benchmark for the triumph of a training program. What is expected from an assessment is a positive and pleasant result, this outcome is needed by managers and people in the company who have an interest in the training program. Nevertheless, plentiful things must be given to the trainees to be convinced that the program being followed is very beneficial. The following factors must be considered carefully when interested parties implement effective training programs: 1. Specify necessity 2. Arrange purpose 3. Set the fundamental things 4. Pick entrant 5. Specify the optimum schedule 6. Select the exact facility 7. Choose the appropriate lecture 8. Pick and set up audiovisual aids 9. Coordinate the agenda 10. Assess the project The most powerful reason for evaluating is to determine whether a training program is effective or not and how the training program can be expanding better than the previous training. The following are eight principal factors of the decision to continue training programs: 1. Has the content presented fulfilled the needs of the attendees? 2. Have the participants fulfilled the requirements for training? 3. Have program leaders already used the most effective teaching methods? 4. Are the facilities obtained to satisfy the participants? 5. Is the training schedule suitable for the participants? 6. Are the tools needed by participants to help communicate during training? 7. Is the program arrangement satisfying? 8. Are there other things that can be done to provide a better program?
Analyzing the answers to these questions can help improve better programs in the future. Almost all organizations have pressure from company leaders, top management always hopes that there will be greater training benefits compared to their expenditure costs. It is not surprising that many managers place more importance on profits or share prices of their companies. Nevertheless, training expenses usually have little financial compared to other costs. If during the training participants did not complain and the trainers felt comfortable, then they were reluctant to spend more on training. Even so, if the trainee generates a negative reaction to the training program, then the coach feels worried. Highlevel managers will make decisions based on the information they complain about.
Top managers often put pressure on trainers, things that are required are usually such as increased sales, the productivity of work, the excellence quality of employees, and company profits related to the training conducted. The Four-Levels Kirkpatrick evaluation model provides a guide for dealing with situations like this, and they must evaluate comprehensively at the end of the training program, expected more from the reaction sheet.

RESEARCH METHOD
This study utilizes qualitative methods where the qualitative approach perceived by researchers is more suitable in this study. These results are obtained from a comparison of the qualitative approach and also the quantitative approach. The data were collected through interviews. The target respondent, needed to get the criteria desired by the researcher. Observation is conducted to gather information about changes in respondent's behavior. From the data that has been obtained, the next step is to process and analyze the data that will be done in the next chapter. The last stage of this research is to provide conclusions and recommendation as the main goal in evaluating leadership training based on Kirkpatrick models.

Figure 3. Research Design
This research began by gathering theories as for the basis of this research; the researcher focused on analyzing secondary data sources provided by leadership academy department. Secondary data presented in the form of the results of evaluation of GPMP 4 2018 training that applies the four-level evaluation of Kirkpatrick models. Based on the problems obtained by Telkom Corporate University (TCU), researchers felt the need for further research such as collecting data is needed by conducting interviews to determine indicators that support the variables tested. Qualitative data can be obtained through direct interviews, telephone and via email. Then from the results of the interview, the researcher processed the data, whether there was a change in the behavior of the trainees or not. In the end, the researcher will provide conclusions from this evaluation research based on the results of data analysis and also recommend further studies.

FINDING AND DISCUSSION
In this chapter, the researcher will discuss what was obtained in this study. Several stages will be addressed in this chapter.

Acquiring Data
In this stage, the researcher must obtain secondary data from the department of leadership academy TCU to get GPMP 4 training participants a year ago. This study uses GPMP 4 2018 as training that will be evaluated because it is based on Kirkpatrick's theory to analyze level 3 where the behavior of participants that we will measure. The training takes place about six months ago. As explained in the previous chapter, the researcher will identify participants before conducting an interview, the criteria of the participants are as follows: 1. Telkom employees participating in the GPMP 4 2018 2. Have good pre-test and post-test results.
3. Domiciled in greater Bandung. 4. Respondents are expected 7-10 people. From secondary data obtained from the department of leadership academy TCU. Total GPMP 4 training participants in Bandung and having good pre-test and post-test results were 14 people.From the 14 participants, the researchers had received their contacts and tried to contact all the participants. Only 9 people were willing and successful in the interview, the reason could not be contacted such as handling a task outside the city, off to work due to illness, off to work due to giving birth, or not being able to be contacted at all possible due to reasons of busyness. This research was conducted for approximately two months because the availability of time from the participants varied.

Interview
There were 9 people who participated in this study. From the 9 participants 4 of them can be interviewed directly, and 5 of them can be interviewed via WhatsApp.The researcher did not discriminate between trainees, because they all attended the same training. All participants are also expected to have the same knowledge seen from the results of a good pre test and post-test. The semi-structured interview was conducted for all interview participants, this proved to be very efficient in collecting data where the researcher also needed guidance from key behavior that had been established by the leadership academy department.Although some information is considered not important, it can help in finding and connecting problems. To ensure the interview remains on track, the researcher does the interview into two sections. In the first part the questions given to the participants are open answers. Questions related to behavior change towards participants who have been adjusted for GPMP training starting from day 1 to day 4.In the next section, the researcher begins asking questions about the participants that are related to key behavior. Participants are given the opportunity to answer with a scale (less agree to strongly agree). At the beginning of the interview, the researcher asked permission to record the entire conversation during the interview process. This is needed to facilitate data processing and also to be translated into interview transcripts. From the interview results obtained from 9 participants, the researcher feltthat there needs to be more measurement regarding changes in participants' behavior after the training program. Is there an actual change in behavior that occurs or not. Therefore, the researcher feels the need to do other methods to ensure behavioral changes to the participants.360-degree evaluation is a beneficial method for taking measurements after training. Because the respondent can provide a judgment to the training participants related to their behavior change.

360-Degree Feedback Questionnaire
As explained in the previous chapter, the purpose of this method is to clarify and confirm the truth of the interview results. Respondents who filled in the questionnaire were supervisors, coworkers & subordinates from each participant. This method is usually used for evaluating Kirkpatrick level 3. The respondents are people who can observe the behavior of participants directly in the work environment. In obtaining data on who is the right respondent to fill the questionnaire, the researcher asks the department of leadership academy who are the supervisors, coworkers, and subordinates of the participants. Another thing that can be done is by asking the participants who are their boss, coworkers & subordinates. But in the interview also the researcher asked whether the participants knew the other participants and whether they were directly involved in the work. This can be a new finding to make participant number 1 become the respondents for other participants, namely participant number 2. Because all the participants GPMP 4 have the same position. From a total of 9 participants interviewed, it was hoped that each participant would have their feedback from superiors, colleagues, and subordinates. If each participant has three respondents, a total of about 27 respondents must be obtained. This study can collect 28 respondents. The respondent gave an assessment for the participant.  Table 1., 3 out of 9 participants who received more than three respondents. Then, three other participants get each expected respondent (boss, coworkers & subordinates). Later, the last 3 participants received only two respondents, just from their colleagues and subordinates.However, not all trainees get feedback from superiors. On the other hand, some participants get feedback from more than one boss. The researcher can conclude that to approach superiors it is more difficulties than getting feedback from coworkers or subordinates from each participant.

Figure 4. 360-Degree Evaluation Pie Chart
From the 28 respondents, their classification is shown in the Figure 4.. From the pie chart there were 6 respondents as superiors (21.4%), 12 respondents as co-workers (42.9%), and 10 respondents as subordinates (35.7%).From the six response who were received by the researcher, there were only three supervisors, they evaluate one until three respondents. From the 12 responses of coworkers received by the researcher, participants who are being evaluated could also assess other participants if they worked in the same department. This is considered very useful by the researcher in saving time, and they also play a direct role as observers of other participants. For the assessment from the subordinates, the researcher can gather two respondents. But these two respondents evaluate more than one participants.

Data Analysis
After conducting semi-structured interviews and also getting a response from 360-degree evaluation feedback.All data obtained will be analyzed bythe researcher usingthe qualitative descriptive analysis method. The researcher began to look for relationships between one participant and the other participants, more precisely looking for correlations with one another.Then look for the similarities that are obtained from the results of the interview, and also look for things that are equally changing to the behavior of participants after attending the training. Another thing the researcher did was find something interesting from each participant. This analysis process has the main point from analyzing all the participants, the researcher can determine whether there have been changes to the behavior of the participants after participating in the GPMP training or not. The researcher gets more support from the evidence of 360-degree evaluation feedback. Because the respondents who have positions as superiors, coworkers, and subordinates act as direct observers. They feel whether there is a change in the behavior of participants in their work environment or not. This analytical method has references from Kirkpatrick level 3 regarding changes in participant behavior and key behavior that has been designed by department of leadership academy.

Kirkpatrick Training Evaluation Design
In this section the researcher connects between the level-3 Kirkpatrick theory and several interview questions.In this case, the measurement that occurs on the knowledge and skills obtained from the training is then applied by the participants in their daily work. Is there something felt by participants in behavior change? And how long can they maintain the change? To help the analysis process,  Did participants apply the learning gained from GPMP training to their work?
Whether the knowledge and skills obtained from GPMP training can be useful in completing their work?
Are there behavioral changes that can be measured when the participants return to their workplace?
Department of leadership academy then applied Kirkpatrick's theory to their key behavior that they had designed for GPMP training starting from day 1 to day 4.

Key Behavior Telkom GPMP
The main purpose of this key behavior is to become an assessment tool every single day from the module given by the instructor to the training participants. Starting from day 1 to day 4 all training activities consist of 2 to 4 modules. All modules that will be given to participants have different key behaviors. Leadership academy department has distinguished the key behavior of each band. In this study more focus on the key behavior band 4. In-band 4 the key behavior is more operational. To make it easier to understand the key behavior, it is shown in several tables. From the key behavior table, the researcher can make one to two questions for each module, these questions are used as interview material and also for 360-degree evaluation questionnaire. • Use certain methods to identify operational problems • Recognize data on current and past situations • Recognize the relationship of symptoms to the problem at hand • Use general theory to examine problems • Implement common sense to identify problems in the work unit  Participants are more sensitive to the problems at hand  Attitudes of participants in giving opinions to superiors about the problems being faced • Implement business concepts in general • Follow up on short-term business opportunities • Identify the basic principles of minimizing risk • Utilizing IT for operational tasks • Identify alternative ways to achieve results  Contribution of participants in improving markets, products and services  The willingness of participants to follow up on short-term business opportunities  Get support / influence from others • Look confident in dealing with familiar situations • Get the trust of someone • Demonstrate self-confidence when facing difficulties • Attempting to convince other parties with data • Provide explanations to colleagues for approval  Participants appear confident in all situations  Participants provide explanations to supervisors / co-workers / subordinates for approval

CONCLUSION AND RECOMMENDATION 5.1 Conclusion
In conclusion, evaluation in change behavior after training is a good thing and must be conducted.But this study is far from perfect, there are still many improvements that must be conducted in evaluating changes in participants' behavior after taking GPMP training. Behavior changes may be felt directly by the participants, but this assessment must be supported by substantial evidence. 360-degree evaluation is an effective method of evaluating behavior changes. Because the respondents also act as direct observers in the workplace.From the results of the 360-degree feedback, we can see what kind of behavior changes are felt by superiors, colleagues, and subordinates toward the training participants. For further improvement in individual behavior change, there are still many improvements that must be conducted and also there are still other methods that can be applied by the department of Leadership Academy. To get better 360-degree feedback results there are various things that can be done starting from evaluating the teaching process while in class, and then the participants might also do apply what they got from training, because of the work that makes them have to change.
So far the Leadership Academy department has conducted an assessment through the Assessment Center Indonesia (ACI). This is only for evaluation, but there are no supporting programs or tasks related to behavior change.ACI conducted evaluation a year after they attended the training. ACI evaluates, measures, and evaluates each participant. But there are no programs or tasks that support participants to implement what they get from GPMP training. The results of this study, the researcher can conclude that there is a change in participant behavior as measured by the interviews result and also with 360-degree feedback for each participant.

Recommendation
The researcher will give some suggestions to the Leadership Academy department. The suggestions from the researcher are optional, whether they will apply or not. First, the researcher suggests the need for programs and tasks that can support a change in participant behavior. This can be adjusted to the training material that the participants received during the training. Second, support from superiors are needed such as providing direct coaching or mentoring. From the results of the 360-degree feedback, there were respondents who felt there was no impact of changes behavior from several factors.For example, there is a participant who works in human resource development but the participant is measured by key behavior, which is improving services and products. This might happen because there are several key behaviors that are not really related to their work such as the attitude of participants giving advice to superiors. The thing to improve is to increase the duration of the training class, which usually takes 2 hours to be 4 hours. Or maybe there needs to be a special coaching for these people.