The team contacted the federal government, Ford Motor Company, Intel Corporation, and all fifty (50) state governments for information on their training measurement strategies. The following questions were asked each organization.
1. How do you approach measuring your training programs?
2. What do you measure?
3. How do you determine what you are going to measure?
4. Do you use a standard assessment or combination of assessments to determine the type of intervention needed (is it really training)?
5. What type of evaluation form(s) do you use? Do you have samples you can send me?
6. What do you do with the data collected? Do you have report samples you can send me?
7. Do you have written guidelines or a process that is used to measure training?
8. Is a cost benefit analysis done?
A comprehensive review of responses revealed that the majority of these organizations make an investment in employee training and use Kirkpatrick?s Four (4) Levels of Evaluation to measure the effectiveness of their training programs. The first three elements of Kirkpatrick?s Levels of Evaluation center on the employees/trainees, and include their reactions, their learning and any changes in trainee behavior, whereas the fourth element shifts to organizational payoffs/business results. Following is a synopsis of the team?s data collection.
Data Collection Synopsis
Arizona - Level 1 (levels 2, 3 and 4 in specific instances)
Connecticut - Level 1
Georgia - Level 1 (levels 2 and 3 in specific instances)
Federal Government ? Tools provided
Illinois - Level 1 (levels 2, 3 and 4 in specific instances)
Iowa - Level 1
Kansas - Level 1 (levels 2 and 3 in specific instances)
Maryland - Level 1 (levels 2, 3 and 4 in specific instances)
Michigan - Level 1 (levels 2 and 3 in specific instances)
Nebraska - Level 1 (levels 2 and 3 in specific instances)
New Jersey - Level 1 (levels 2 and 3 in specific instances)
North Carolina - Level 1 (levels 2, 3 and 4 in specific instances)
North Dakota - Level 1
Ohio - Level 1 (levels 2 and 3 in specific instances)
Rhode Island - Level 1 (level 2 in specific instances)
South Carolina - Level 1 (levels 2, 3 and 4 in specific instances)
South Dakota - Level 1
Utah - Level 1 (levels 2, 3 and 4 in specific instances)
Arizona, Georgia, South Carolina and Utah have a comprehensive approach to measuring training programs. However, Kirkpatrick?s Levels 3 (i.e., changes in trainee?s behavior because of training) and 4 (i.e., organizational payoffs/business results) are not always addressed as it is difficult to predict when a change in employee behavior occurred, and determining results in financial terms is difficult to measure. These states use self-assessments, and end-of-course evaluations to measure presentation of the training, knowledge and presentation skills of the facilitator, overall quality of the training, and whether the trainee has advanced their attitude, knowledge, or skills.
Most state of Michigan agencies focus on the Kirkpatrick's Level 1 assessment. For several agencies, select programs have higher levels of measurement. The Educational Services Institute International Project Management courses offered through Civil Service use testing and the Human Resource Academy training was an applied cohort project. The Department of Corrections uses a pre and post test for new employee orientation. The Family Independence Agency does pre and post testing for some of their policy programs and evaluation forms are customized for each program. They also use the American Society for Training and Development performance consultation model to assess the type of intervention needed. The Department of State is using participant surveys 60/90 days out for usefulness and impact on performance for new management training with a six month follow-up to the Supervisor to assess impact on participant performance. New recruit training at the Michigan State Police has very specific measures and specific training requirements for promoted staff.
The Federal Government responded to the inquiry with information on the approach it requires for training. The law and an executive order require agencies to review annually organizational, occupational, and individual needs for training. Executive Order 13111 specifically calls on agencies to include a set of goals to provide effective training opportunities and allied performance measures as part of the annual budget process. A systematic and continuing review of current and foreseeable organizational training needs provides a realistic basis upon which to plan, budget, direct, and evaluate an effective training program. The training needs of individual employees should be assessed within the context of the organization?s strategic goals to ensure employees? performance competency and development. The Guide to Strategically Planning Training and Measuring Results and the Training Needs Assessment: A Guide for Conducting a Multi-Level Needs Assessment are included as tools in this toolkit.
