Don Vandergriff IV: Assessing Students in Courses that Use the Adaptive Leadership Methodology
A section of the blog
post by Don Vandergriff and Fred Leland that I cited
yesterday is devoted to explaining how students in classes using the Adaptive Leadership Methodology (ALM) are assessed and graded.
Vandergriff and Leland describe a couple of assessment methods:
- Short-answer tests — "These examinations place [the students] in a specific tactical scenario and require them to make decisions. They must then explain the reasons behind their decision in writing. For example, student might be told that he is the commander for a convoy of vehicles that [must] travel to an assigned destination within the next several hours. After being presented with information about the composition of the convoy, a map, the nature of the enemy threat, and the specifics of the mission and Commander’s Intent,1 the student would have to determine which route he will take and then explain why he selected that route. The instructor then grades him on how he approached solving the problem using the information at hand and how well he communicates his reasoning. This gets to the point of examining 'how' the student is thinking, not 'what' he is thinking."
- Graded Tactical Decision-Making Exercises (TDEs) — "Just like the short-answer tests, these are scenario-based and require students to make decisions. This technique is virtually identical to a standard in-class TDE with the exception that students are required to write out or brief their solutions to their instructors who, in turn, grade those solutions. In many cases, these examinations require students to produce a concept sketch with short hand-written notes concerning exact guidance for individuals, their teams, the sequence of events, and most importantly the purpose behind various actions."
Vandergriff and Leland note:
Regardless of the technique or format of the assessment, the tactical scenario must allow for multiple “correct” ways to solve the problem. For the assessment to be truly effective, students must have the freedom to actually make a decision on their own and formulate a plan rather than being forced to regurgitate a pre-determined “template.” If tests fail to allow room for creativity, students become focused on identifying the “approved solution” rather than thinking for themselves. In order to permit freedom of thought, scenarios must have a significant amount of ambiguity. The situation must be such that one could reasonably interpret the available information in multiple ways. Of course, this does not preclude the existence of “wrong” answers. Violations of the Commander’s Intent, unethical conduct, poor communication, or an unrealistic course of action all constitute an automatic failure. Additionally, if the student is unable to make a decision in the face of the time and information constraints of the test (the worst of all possibilities), he is assigned a failing grade.
To ensure that different instructors are applying comparable standards in grading students,
... all instructors must participate in a free exchange of ideas regarding the key concepts that are the focus of the upcoming assessment. ... [T]hese group discussions ... begin with the instructors actually taking the test followed by an open discussion regarding the content of the exam and how to approach grading. At the end of this exchange, the Course Director compiles the applicable notes from the session into a short set of general guidelines.
According to Vandergriff and Leland, this approach has produced satisfactory consistency in grading.
__________
1 An August 14, 2009 post on the
Combined Arms Center blog quotes the following definition of "commander's intent" from paragraph 5-55 of the Army Field Manual 3-0,
Operations (February 2008): "The commander’s intent is a clear, concise statement of what the force must do and the conditions the force must establish with respect to the enemy, terrain, and civil considerations that represent the desired end state. The commander’s intent succinctly describes what constitutes success in an operation. It includes the operation’s purpose and the conditions that define the end state. It links the mission, concept of operations, and tasks to subordinate units. A clear commander’s intent facilitates a shared understanding and focus on the overall conditions that represent mission accomplishment. During execution, the commander’s intent spurs individual initiative."
The following paragraph adds, "The commander’s intent must be easy to remember and clearly understood two echelons down. The shorter the commander’s intent, the better it serves these purposes. Typically, the commander’s intent statement is three to five sentences long."
###Labels: Evaluation of training, Military training
The Kirkpatrick Model at 50
It was in 1959 that Donald Kirkpatrick published the first of four articles in the
Journal of the ASTD elucidating his four-level model for evaluation of training.
1 Krikpatrick had initially defined the model — though not with the levels' current names — Reaction, Learning, Behavior, Results — in his doctoral dissertation, "Evaluating a Human Relations Training Program for Supervisors."
Fifty years have elapsed, and the four-level model has held up well. The anniversary is being marked by various
review articles and
interviews with Kirkpatrick himself and with his son, Jim, who has been carrying on in his father's line of work for some years.
Jim Kirkpatrick, along with his wife and co-author Wendy, has provided a fresh overview of the model in a
white paper (reg req), the highlight of which is the diagram below. The diagram shows the Kirkpatricks' take on the flow of activities involved in developing and evaluating a training program.
(click to enlarge)
(from a white paper by Jim & Wendy Kirkpatrick)ROE = Return on Expectations,i.e., the degree to which the training's results meet management's expectations concerning what the training should accomplish.
KSAs = Knowledge, Skills, Attitudes,i.e., competencies.
The bulk of the ten-page white paper is devoted to explaining how the Kirkpatrick Model embeds the four levels into the planning of the training, as well as its evaluation. There is considerable attention to the practical details of the process, which makes reading the white paper in its entirety worthwhile.
As a quick summary, here is an example of the process in practice that Jim Kirkpatrick lays out in the interview with him and his father that
Training Magazine published in October:
Let's say an executive in your organization asks for a new program on leadership. Instead of conducting a cursory needs assessment, and putting together and delivering a program, start with the concept we call "the end is the beginning." You must begin with a clear picture of what a successful leadership program ultimately will bring to the organization in terms of Level 4. This means you first negotiate your stakeholders' expectations so they are satisfying to them and achievable for you, then you convert those expectations to observable, measurable success outcomes by using various forms of the question, "What will success look like?" (Level 4, Results). Then you determine the critical new leadership behaviors training grads will have to perform on the job (Level 3, Behavior) to bring about those success outcomes.
Also crucial at this stage is to determine, implement, and monitor the required drivers of those behaviors — the processes and systems that will serve to get grads to perform those critical behaviors on the job. Next, it is important to identify and reinforce pre-training necessities for success, for example, a culture of support and accountability, coaching guides, and job aids. Then, and only then, you develop and deliver the training that will produce competent employees (Level 2, Learning, and Level 1, Reaction) who will be able to perform the targeted, critical behaviors. In other words, start with Level 4 and work backward.
It is also important to monitor not only Levels 1 and 2, but the required drivers and critical behaviors at Level 3, and the outcomes at Level 4. This will allow for a good system to make sure learning to behavior transfer occurs, and also provide you with a compelling chain of evidence at the end to demonstrate the impact of the initiative.
As you can see, there is nothing obscure about the process. Using the four levels effectively is a matter, as the Kirkpatricks emphasize, of strong partnership between training professionals and organization management, accompanied by excellence in implementation.
__________
1 Donald L. Kirkpatrick, "Techniques for Evaluating Training Programs," Parts 1-4,
Journal of ASTD, Vol. 13, No. 11 (1959), pp. 3-9; Vol. 13, No. 12 (1959), pp. 21-26; Vol. 14, No. 12 (1960), pp. 13-18; Vol. 14, No. 13 (1960), pp. 28-32.
###Labels: Evaluation of training
Simplified ROI Measurement for Training
The November 2009 issue of
Chief Learning Officer has an
article by
Howard Prager and
Susan Vece, Directors of Corporate Education at Lake Forest Graduate School of Business, that offers a practical method for calculating the ROI for a training program.
The process has seven steps, details of which you will find in the article. The steps, in edited form, are:
- Identify the one or two key skills or competencies that most directly relate to the business driver you need to impact in order to achieve desired improvement in business performance.
- Tie the key skills or competencies to one or two metrics that are reliable indicators of success. For example, a training program intended to help increase sales to current customers might focus on customer satisfaction and sales per account.
- Create an ROI survey to get needed data from training participants, their managers, and clients quickly and simply. Prager and Vece emphasize the importance of keeping the survey short in order to encourage response and to shorten the time required to analyze the results.
- Conduct the survey.
- Calculate the costs of the training program. Measure costs as fully as possible in order to strengthen the validity of your reported results.
- Analyze the quantitative and qualitative information on training impacts produced by the survey. Measure impacts net of cost.
- Calculate the ROI for the training program by dividing net quantified program impacts by program costs.
Note that the final step involves
quantified impacts. It is important to take due account of all impacts, including any that do not particularly lend themselves to quantification.
For example, the training program might encourage more interaction amongst employees in communities of practice. It's hard to put a dollar value on this outcome, but bolstering of such communities is likely to be a significant plus. The key is to identify in some detail how the more robust communities of practice are contributing to business performance, so that when management are briefed on what their training investments are producing, they find the reported qualitative impacts credible.
###Labels: Evaluation of training
Training Law Enforcement Professionals at Art Galleries
The October issue of
Smithsonian has an
article by
Neal Hirschfeld about training New York City police that I find encouraging in a number of ways, but two in particular I'd like to mention here.
The first point I'll discuss is what you can learn from Hirschfeld's article concerning use of training exercises that aim to simulate key on-the-job responsibilities without duplicating any but the most immediately relevant features of participants' day-to-day working environment.
Specifically, Hirschfeld describes how
Amy Herman, Director of Educational Development at Thirteen/WNET, uses art to help policemen and other law enforcement, security, and military professionals hone their observation and communication skills by using art as practice material to be scrutinized and analyzed.
.jpg)
Madame X
John Singer Sargent, 1884
(www.johnsingersargent.org)Looking at art, such as the painting of
Madame X by John Singer Sargent featured in the
Smithsonian article, is not something members of the NYPD do on-the-job, but the process of concentrating on the details that are there to be seen if you're paying enough attention, and then articulating a precise description of what you've noted, emulates in an essential way what police do when they assess a crime scene or potential crime scene.
As Herman explains in a 2007
Museum News article, she asks participants to look closely at works of art she has selected for their richness of subject matter and formal elements, and to try to reach consensus on answers to the basic investigative questions, Who, What, Where, Why, and When.
Only after practicing on paintings and sculpture, do the participants go back to working on material close to their actual day-to-day experience:
Following their session in the Frick's galleries, participants return to the conference room, where they view a series of photographs of crime scenes, urban landscapes and portraits of individuals. Asked to articulate in descriptive language similar to that utilized when looking at paintings, the officers respond by describing, in detail, what it is they see before them.
The second point I think particularly worth highlighting is how well Herman's approach has stood the test of time. Her first art-based training program actually dates back to 2000, when she was director of education at the
Frick Collection in New York. She worked with the Weill Medical College of Cornell University to help medical students strengthen their observational skills. (You can read a
New York Times article about the program
here. The article also describes precedents for Herman's program dating back at least to 1998.)
Participant evaluations, officer feedback, and repeat business from participating organizations all indicate that Herman's program is successful in meeting its objectives of improving observational and communication skills.
For example, in Herman's
Museum News report on the Frick/NYPD program she ran for a number of years, she explained that all participants
complete a written evaluation ... which specifically asks them to rate, on a numeric scale, the relevancy of the program, its applicability to their daily responsibilities and the efficacy of correlating between paintings and photographs of crime scenes. ... The vast majority of evaluators wax enthusiastic about the concept of enhancing observation skills in a setting removed from the daily work environment and praise the methodology used by museum educators to refresh their perspective on acute observation.
Specific performance improvements cited in the
Smithsonian article include:
- the head of the New York Police Academy now giving more specific orders to investigators.
- the officer who coordinates the NYPD Grand Larceny Task Force of plainclothes officers now providing more detailed descriptions of suspected pickpockets, handbag snatchers, shoplifters, and other burglars.
- an undercover agent on a task force investigating organized crime's involvement in garbage collection in Connecticut reporting that Herman's program helped him "sharpen his observations of office layouts, storage lockers, desks and file cabinets containing incriminating evidence," information that "led to detailed search warrants and ultimately resulted in 34 convictions and government seizure and sale of 26 trash-hauling companies worth $60 to $100 million."
As for repeat business, Hirschfeld reports that the program has been picked up by a range of agencies involved in law enforcement and security, including the US Secret Service, the Department of Homeland Security, the Transportation Security Administration, the Strategic Studies Group of the Naval War College, the National Guard, and the Metropolitan Police of Scotland Yard.
###Labels: Arts, Evaluation of training, Military training, Simulation
A Case Study of Training Evaluation
For the story of how one company sought in a practical way to assess the success of a learning program, you can read the
write-up available at
corporatetraining.ie, an Irish education and training portal.
The company in question, identified in the narrative as "M Ltd.," is a manufacturing concern with 20 employees. M Ltd. undertook a
leadership training program that combined
classroom sessions with
one-on-one coaching. The individuals trained were a production manager, a supervisor and a senior employee.
In assessing the impact of the training, M Ltd. considered outcomes at Kirkpatrick's Level Two — amount of content learned — and Level Three — amount of on-the-job application.
- Level 2 — At an evaluation meeting, the training participants presented "individual 30-minute summaries of key ideas or skills acquired during training." The presentations were interactive, in the sense that the firm's general manager, who was the key audience member, asked the employees questions while they were speaking. (E.g., What was the most important skill you learned? What are the most common conflict-resolution styles that you see us using here at M Ltd.? How do you use the four basic communication styles with the people you supervise?)
- Level 3 — "The employees reported that the training had enhanced their ability to communicate and listen. Evidence of improved communication at work was noticed in less time spent by the manager and the employees in locating materials, messages and equipment. Better communication also improved message flow and reduced lost time. For example, after training [the general manager] instituted 30-minute overlap meetings between day- and night-shift supervisors. Previously, shift overlap lasted no more than five minutes, which resulted in confusing or missed communication. With increased overlap, day- and night-shift problems related to scheduling, machine operation and personnel issues were minimised resulting in shortened lines of communication and a smoother turnover of work shifts.
"Other evidence that some classroom learning was being applied to workplace situations was the institution of Monday morning meetings among the general manager, the production manager, the supervisor and the senior employee to review issues and anticipate problems for the upcoming week. Prior to training, these meetings were sporadic and did not always include all key personnel."
The narrative closes with recommendations for enhancements to the training and for improving the Level 2 evaluation meeting. The most important of the latter recommendations was to explicitly assess what the one-on-one coaching contributed to the learning process.
###Labels: Classroom training, Evaluation of training
Failing to Evaluate Training
Anybody in the training field is familiar with the issue of deciding how best to evaluate a particular training program. Most training specialists take it for granted that evaluation is necessary, and then go on to investigate the question of what level of evaluation is feasible and appropriate in a given case.
What the typical training specialist doesn't do is entertain the idea that the evaluation step can be skipped entirely. But skipping evaluation seems to be all too common at the Department of Defense. The particular instance that got me going on this subject today is a
column by Senior Chief Jim Murphy (USN, Retired) in the September issue of the
Proceedings of the US Naval Institute.
According to Murphy,
For at least the past two years, every uniformed, civilian, and contract member of the Department of the Navy has been required to complete training on protecting PII [Personally Identifiable Information]. ... The new fiscal year likely signals another round of this so-called training. These "courses" are simple PowerPoint presentations that allow learners to click through the slides until reaching the end, signaled by a course certificate with blanks to fill in one's name and completion date.
Computer security instruction is not much better. DOD [Department of Defense] Information Assurance Awareness training, a separate annual requirement for all personnel, also misses the mark. In this product, information is presented in a lectured e-learning format with scenario-based questions on various topics. The training is problematic because it also allows clicking through the lecture, and worse, clicking through dialogue boxes, which explain incorrect responses to assessment questions. Also there is no retesting of missed topics. In fact, an e-learner can advance through the entire curriculum, miss every question, ignore every explanation, and still successfully complete the training.
Certificate of Completion awarded to personnel who page through the Department of Defense course on "Information Assurance Awareness." No proof of actual learning required.
(US Naval Institute Proceedings)Not surprisingly, Murphy thinks, at a minimum, the training in question should be evaluated through "formative assessments to certify comprehension" —
Kirkpatrick Level 2.
Murphy sums up in his conclusion: "Only through effective training development will the Defense and Navy Departments stop wasting training funds, prevent losses of PII, and secure our systems. It's time we get serious about these issues."
###Labels: Evaluation of training, Military training
An Online Course on the Great Depression
The
Federal Reserve Bank of St. Louis has published a six-lesson
curriculum (pdf) on the Great Depression that is geared to high school students, but is suitable for anyone interested in brushing up on the era.
The curriculum begins with an
overview essay, which is followed by six lessons (which are designed to stand alone, so a teacher delivering the curriculum can freely choose which lessons to cover):
- Measuring the Great Depression — "This lesson introduces tools — such as Gross Domestic Product (GDP), the unemployment rate and the Consumer Price Index (CPI) — that are used to measure the economy’s health, through an analysis of simple bar charts and graphs. Developing an understanding of these concepts is critical to understanding the magnitude of the economic problems that took place during the Great Depression."
- What Do People Say? — "There are many suggested causes for the Great Depression. It is important for students to understand that occurrences such as the stock market crash — and other events that affected particular sectors of the economy — were important, but not significant enough to cause the Great Depression. By reading fictitious letters that reflect actual problems and people’s concerns during the Great Depression, students begin to identify with the people of that era and to uncover the problems that people experienced during the Great Depression."
- What Really Caused the Great Depression? — "Through participation in two simulations, students determine that bank panics and a shrinking money supply were the primary causes of the Great Depression. Through an additional activity, they see how the many other factors they have discussed, such as problems in the agricultural sector and the stock market crash, exacerbated the situation."
- Dealing with the Great Depression — "Students learn about programs initiated through the New Deal. By comparing and categorizing New Deal programs, they recognize that the value of most of these programs was their effects on the confidence that U.S. citizens had in the economy. Students also identify the impact that these programs had on the role of the U.S. government in the economy."
- Turn Your Radio On — "Students use excerpts from Franklin Delano Roosevelt’s 'fireside chats' to identify his plans for restoring the economy. They determine that using available technology to communicate was important to FDR’s effort to restore consumer confidence."
- Could It Happen Again? — "Students learn about the roles and functions of the Federal Reserve System. Through a simulation, they learn how the Fed manages the money supply through open market operations. They identify what central bankers have learned about implementing monetary policy as a result of the Great Depression. Furthermore, they recognize the steps the central bank has taken to respond effectively to financial crises since that time."
To test your learning, you can take a 33-question
test provided in the appendix. (You might want to try the test before you start working through the lessons, and then again after you've finished.)
There is also a
glossary, a
bibliography, a
list of resources (e.g., newsreels, photos, books, websites), and an
evalution form that teachers can use to provide feedback concerning their experience with the curriculum.
###Labels: Business acumen, Classroom training, Evaluation of training, Learning resources
Using an Audience Response System
At www.audienceresponselearning.com, you can get a helpful summary of the wide
variety of questions that can be incorporated into training sessions in which participants are equipped with wireless audience response keypads.
The website is the handiwork of
Will Thalheimer, a PhD researcher and practitioner who makes it his business to publicize research-based recommendations for training design and implementation, a subject close to my heart.
Thalheimer's audience response site also exhaustively lists the
benefits of using questions in a training session.
For a more complete compilation of knowledge concerning effective audience response questioning strategies, you can download Thalheimer's 124-page
report on the subject.
Finally, you can read an illuminating interview with Thalheimer
here. (The article on misuse of interactivity that Thalheimer mentions in the interview is
here.)
###Labels: Classroom training, eLearning, Evaluation of training, Learning resources, Teaching
The Relationship Between Study Time and GPA
It would seem we're in "dog bites man" territory, when we read that the father/son team of
Ralph Stinebrickner (professor of mathematics and computer science at
Berea College) and
Todd Stinebrickner (professor of economics at the
University of Western Ontario), has established a strong statistical relationship between the amount of time students study and the grades they receive.
This research result is newsworthy because, however obvious the connection between studying and grades may seem, it has been hard to pin down statistically. The problem the Stinebrickners (S&S) solved was a lack of data enabling researchers to control for the other factors that could conceivably affect grades (e.g. class attendance, time spent sleeping, etc.). In previous statistical studies, the link between study time and grades turned out to be weak.
S&S got their data from the
Berea Panel Study (pdf), their ongoing longitudinal survey of college entrants from low income backgrounds. As explained in their paper, "
The Causal Effect of Study on Academic Performance," (pdf) S&S were able to create two groups of students alike in all respects except one, namely whether or not a particular student's randomly assigned roommate brought a video game to school.
S&S established that students whose roommates had video games studied less than those whose roommates were sans video game; the former studied an average of 0.79 hours per week, while the corresponding figure for the latter was 4.06 hours per week.
The Stinebrickners then went on to estimate the impact of this difference in study effort on the GPAs of the students in the two groups. That estimate turned out to be 0.356 points on a four-point GPA scale, a result significant at the 92% level.
In addition to their basic conclusion that study effort has a significant effect on GPA (significant in both statistical and practical terms), S&S also concluded that:
- Programs which encourage struggling students to work harder should pay off, at least for students who respond by increasing their study time.
- One of the ways peers can affect how well fellow students perform academically is by influencing each other's time allocation.
- Video games can detract from learning. (This conclusion does not refer to classes in which video games are part of the learning methodology.)
- It makes sense to include some measure of applicants' academic work ethic as one of the factors considered in reaching college admission decisions.
I would just wrap up by noting that most students have conducted their own personal controlled experiments whose outcomes support the S&S result, i.e., students find that they do better or worse on tests for a particular class, depending on how much time they put into preparatory studying.
###Labels: Evaluation of training
Training Supervisors in How to Support Work/Life Balance
The November 2008 issue of the
Harvard Business Review offers another instance of how intelligent flexibility on the part of managers produces better results for a business than intensive enforcement of rigid rules.
In a brief
article,
Ellen Ernst Kossek, professor of human resource management at Michigan State University's
School of Labor & Industrial Relations, and
Leslie B. Hammer, professor of psychology at Portland State University, report the results of a multi-year study of supervisors and employees in American supermarkets.
1 In sum:
Teaching managers to be more supportive of their direct reports' work/life issues can be a simple and effective route to improving employee health and satisfaction ...
The core of Kossek and Hammer's research program involved developing and evaluating training for supermarket supervisors designed to help the supervisors more effectively
plan coverage in their stores and
deal with employees' scheduling conflicts.
The training had three components:
- A self-paced computer tutorial that took about 30-45 minutes for a supervisor to complete.
- A 75-minute discussion session with a few fellow supervisors and a faciltator.
- Having supervisors set individual goals for what they had learned and, for a few weeks following the discussion session, record their supportive behaviors on index cards.
In their training, the supervisors learned about good practice in four areas:
- Providing emotional support — "acknowledging employees' sometimes extensive responsibilities outside work."
- Providing structural support — "working with employees ahead of time to resolve scheduling conflicts."
- Modeling healthful behavior — "for example, showing that it is acceptable to occasionally attend important family functions during work hours."
- Partnering with other managers — i.e., working with fellow managers "to strategically address work/life issues through initiatives like interdepartmental cross-training, which increases coverage options."
Kossek and Hammer evaluated the impact of this training and found that after the supervisors were trained:
- Employees' perceptions that their supervisors were supportive on work/like issues were significantly improved.
- Employees "reported improvements in their overall health as measured by such factors as pain and psychological problems. This effect was most pronounced among employees who previously had the highest levels of work/life conflict — for example, a frequent need to change their hours to accommodate children's schedules."
- Employees were more satisfied in their jobs.
- Employees were less inclined to seek jobs elsewhere.
- Employees reported greater willingness to comply with safety programs.
- There was generally close agreement between employees' ratings of their managers' supportivesness and the managers' self-ratings. Prior to the training, the employees' ratings were markedly lower than the ratings managers gave themselves.
It is important to note that implementation of the practices Kossek and Hammer taught the supervisors required sanction from upper management. Short-sighted executives will balk at funding such training and at allowing the sort of flexibility Kossek and Hammer encourage. More far-sighted managers will recognize the business benefits of making reasonable accommodation of workers so that the latter feel valued and are not worn down by stress.
__________
1 Kossek and Hammer codirect the
Center for Work-Family Stress, Safety and Health in Portland OR.
###Labels: Evaluation of training, Management practices, Motivation, Organizational culture, Respect
An Auditing Simulation
For an interesting example of how an unflashy simulation can be put together and used in the classroom, I'd suggest having a look at what
Faye Borthick (Georgia State University)
1 and
Mary B. Curtis (University of North Texas) have created for helping students develop their skills for conducting audits that involve analyzing company financial information electronically.
In a 2005
paper (pdf), Borthick and Curtis present in detail a simulation of a due diligence engagement
2 for the inventory account of Threadchic, a clothing retailer.
3 The simulation's objectives are to help students learn to:
- Prepare a business process representation (diagram).
- Develop, by financial statement assertion (e.g., "Inventory is $X"), audit objectives for the inventory account.
- Design audit procedures to implement the audit objectives.
- Execute the audit procedures by querying the data.
- Communicate audit objectives, audit procedures, results from execution of queries, matters warranting follow-up, and lessons learned.
As Borthick and Curtis
explain,
The learning objectives represent essential skills for audit expertise in situations where auditors are expected to analyze data electronically to verify the internal consistency of accounting records and to detect conditions warranting further investigation.
. . .
Completing the simulation requires learners to wrestle with ambiguities embedded in a realistic audit setting as they make the inferences necessary to develop an approach to performing the audit.4
Borthick and Curtis note that "students have been surprised at the depth of thought required to design the audit program, execute it with audit or query software, refine their audit programs, make sense of the results, and imagine procedures for resolving unusual conditions."
Borthick and Curtis investigated the ability of the Threadchic simulation to accelerate students' acquisition of auditing expertise. They looked specifically at the impact on learning of using business process modeling (one of the learning objectives for the simulation, as listed above), and
report that "students with business process modeling experience outperformed students without that experience in every audit task" (i.e., developing audit objectives, designing an audit procedure, executing the audit procedure, and interpreting the results).
5__________
1 Clicking on the
Borthick link takes you to a page that provides links to summaries of other work Borthick has done in the areas of instructional design and evaluation of learning outcomes.
2 In a
2004 version of their paper (pdf), Borthick and Curtis provide an answer key for the simulation.
3 A due diligence audit is one carried out as part of the process through which a company decides whether or not to move ahead with a merger or acquisition it is considering. In the case of the Borthick-Curtis simulation, the inventory audit is part of the investigation a potential acquirer is undertaking to determine whether the financial statements Threadchic has provided are accurate.
4 You can look at the simulation as students experienced it
here. To gain access, enter user name "ac863" and password "Qd0319."
5 Here (2004 - pdf) and
here (2007 - pdf) Bortnick reports the results of research conducted with Carol. W. Springer, a colleague at Georgia State University, on developing accounting students' critical thinking skills. A working paper by Springer showing how critical thinking skills developed in an accounting course can carry over into other academic areas is
here (2004 - MSWord).
###Labels: Business acumen, Classroom training, Cognition, Critical thinking, Documentation, Evaluation of training, Expertise, Learning resources, Simulation
Just Read the Sidebar
In its October issue,
Chief Learning Officer has an
article on the subject of evaluating learning that tells you nothing new. However, embedded in the article is a helpful sidebar by training specialist Tom Kelly that offers a half dozen practical tips for making sure your measurement of learning impacts is aligned with business goals:
- Track macro trends, not individual results "In sales and other functions, too many variables exist to measure positive or negative impacts of training on an individual success/failure level. However, comparisons of regional teams, city teams or other groups can help identify a correlation between training and success."
- Measure and report the right metrics — You need to identify impacts on revenue, employee productivity, and/or customer satisfaction and loyalty.
- Look at the training strategies used by successful people or teams — Replicate what works, making appropriate adjustments to fit each group of employees being trained.
- Create a teaching culture — Take steps to ensure experts share their expertise.
- Decentralize content creation — "Peer-to-peer teaching tools mean giving up some control of the content creation process but increase the accessibility and speed that content can be updated or corrected."
- Centralize content deployment — Establish a portal that aggregates content (and allows control of redundancy), and that makes accessing the content easy for users.
As Kelly points out, there is a dual benefit of adopting the above practices. You both acquire information needed to make the business case for investing in your training programs, and you bolster the degree to which your company is a learning organization with "an evolving, organic culture that’s open to growth and change."
###Labels: Evaluation of training
Training in Support of Microlending Clients
Microlending has been much in the news over the last several years. In fact,
Muhammad Yunus, the best known proponent of microlending — through the
Grameen Bank which he founded in Bangladesh — was awarded the
Nobel Peace Prize in 2006.
2006 was also the year that
Dean Karlan, an economics professor at Yale, and
Martin Valdivia, research director a
Grupo de Análisis para el Desarrollo, issued a
paper (pdf) reporting their research indicating that "microfinance institutions can improve client outcomes cost effectively by providing entrepreneurial training along with the credit." Specifically, clients who received training showed improved business knowledge, practices, and revenues.
Karlan and Valdivia also found that the client training has a positive impact on the microfinance institution itself. Both repayment and client retention improved, which meant higher profitability.
Another key finding was a negative correlation between the interest a client expressed in receiving training and the magnitude of the impact of the training on the client's business practices and results. Karlan and Valdivia note that this result suggests that microfinance institutions should be wary of charging a fee for training, since a fee may especially deter the clients who would benefit most.
The training Karlan and Valdivia evaluated was delivered to clients of
FINCA Perú, a small non-profit microfinance institution. Some of the clients were in three particularly poor districts of Lima; others were in the area of Ayacucho, the capital of the Andean province of Huamanga.
In Lima 98% of FINCA's clients are literate; in Ayacucho the client literacy rate is approximately 85%. Most of the Lima clients have finished secondary school, and 40% have some post-secondary schooling; in Ayacucho only about 30% of clients had finished secondary school. This difference in the level of formal schooling led the researchers to use somewhat different training programs in Lima and Ayacucho:
The training materials in Lima were organized in two modules. The first module introduced attendees to what a business is, how a business works, and the marketplace. Clients were taught to identify their customers, competitors, and the position of the business in the marketplace and then learned about product, promotional strategies and commercial planning. The second module explained how to separate business and home finances by establishing the differences between income, costs, and profit, teaching how to calculate production costs, and product pricing.
. . .
In Ayacucho, the training program was grouped into 3 modules with topics less advanced than those taught in Lima. ... Module 1, “Manage Your Business Money,” begins by defining the differences between money for personal expenses and for the business. Women are taught how to calculate profits and about the use of profits for the household and business. Sessions cover how to handle selling to customers on credit, how to record business expenses, how to prevent losses, and the importance of investing in the business. ...
Module 2, “Increase Your Sales” begins by providing an overview of five key elements in sales: 1) customers, 2) business product or service, 3) product placement, 4) pricing, and 5) marketing. Many of the following sessions are dedicated to providing women with practical means of applying these concepts. ...
The third module, “Plan for a Better Business,” teaches members how to incorporate planning into their business. Sessions begin by presenting why planning is beneficial and what traits characterize a successful business. Attendees are instructed on how to solve business problems and how to introduce new products or changes. Later sessions teach the tools needed to prepare a sales plan, calculate business and loan costs, search for new resources, and handle unexpected problems and opportunities.
The research was carried out by assigning clients randomly to treatment and control groups.
The treatment groups received thirty to sixty minute entrepreneurship training sessions during their normal weekly or monthly banking meeting over a period of one to two years. Control groups remained as they were before, meeting at the same frequency but solely for making loan and savings payments.
Karlan and Valdivia note that follow-on research is needed to evaluate the degree to which the positive impacts of training are sustained.
###Labels: Business acumen, Evaluation of training, Latin America
"Impact of Learning" as an Evaluation Metric
I'm generally not particularly inspired by the material in
Chief Learning Officer magazine, but there were a couple of articles in the September issue that I read because they offer useful advice on evaluating training in a practical, meaningful way.
The first
article, by
Jim Naughton, explains use of "impact of learning" (IOL) as one's measure of the degree of success achieved by training initiatives.
1 The IOL approach has three steps:
- Create a map of the linkage from business goals to training initiatives and identify success metrics, i.e., measures reflecting the degree to which the intended outcomes of the training have been achieved.
- Gather data on the success metrics and individual success stories.
- Develop an impact report with compelling stories of individual and overall success, so executives clearly see the impact of training on organizational issues they care about.
The problem with the IOL approach is that executives are wont to ask whether the impacts detailed in the impact report were obtained at reasonable cost. This is where the second
CLO article proves helpful. In the article,
Michael Echols addresses the question of how to measure the value training initiatives create. He points out that measuring value added by training requires:
- getting agreement from top management on the value of changes in the specific outcomes the training is intended to affect (e.g., growth in revenue, employee engagement, retention, productivity, innovation, customer satisfaction, cash flow).
- using sound statistical analysis to estimate the impact of training, while controlling for other significant factors that influence the outcomes being tracked.
Echols provides a case study of how Chrysler used the methodology he describes to evaluate a proposed program of dealer training.
__________
1 This previous
post makes similar points concerning a practical approach to evaluating training. For a discussion of how to elicit the trainee's evaluation of training, see this
post.
2 This previous
post makes similar points concerning how to measure the value that training initiatives create.
###Labels: Evaluation of training
Getting an Employee's Manager on Board with a Training Program
My latest find, as I keep a lookout for practical and credible advice that fellow training practitioners are offering, is an
article by
Jeffrey Berk, COO of
KnowledgeAdvisors.
In the article, Berk concisely lays out ten best practices for managers whose employees are attending training programs. His list fits my own views, and I encourage
taking a look.
To help with improving the results obtained from training, Berk also suggests a half dozen questions to ask when a learner comes back from a program:
- What percent of learning actually was applied to the job?
- When did the learner apply the learning (e.g., time-to-job impact)?
- What are the major barriers to applying the learning on the job?
- Did you set expectations with your manager before the learning event?
- Were you provided adequate resources to optimally apply the learning?
- Did you determine specific uses for the training after it took place?
This combination of best practices and evaluation questions from Berk both embodies useful guidance and illustrates how straightforwardly one can talk about training issues (i.e., no jargon is needed).
###Labels: Classroom training, Coaching, Employee development, Employee performance management, Evaluation of training, Learning organization, Management practices, Productivity
Using Abstraction to Make it Easier to Apply Learning
"It is very difficult to extract mathematical principles from story problems. Story problems could be an incredible instrument for testing what was learned. But they are bad instruments for teaching."
That's what
Vladimir Sloutsky, a professor of psychology at Ohio State University, had to say in
talking about the relative merits of abstract examples vs. concrete examples when teaching math principles.
In a study led by
Jennifer Kaminski, a research scientist at Ohio State University's
Center for Cognitive Science, of which Sloutsky is the director, experiments showed that students were better able to apply principles to a variety of situations if they were taught the principlpes in the abstract, rather than via scenarios built from concrete details.
1The example cited in OSU's press release about the research is "the classic problem of two trains that leave different cities heading toward each other at different speeds. Students are asked to figure out when the trwo trains will meet." Even if they are able to get the correct answer, it turns out that solving the train problem does not much help when they encounter an equivalent problem with different details, such as levels of water rising over time.
The problem seems to be that "extraneous information about marbles or containers [typical props in concrete examples] divert attention from the real mathematics behind it all."
Even while recognizing that people have different
learning styles, there is a lesson here for business trainers who, all too often, overdo cutesy examples in their courses.
On the other hand, it is important to note that the above is not an argument against training that uses examples taken directly from the on-the-job context in which learning is to be applied. On the contrary, such an approach is advisable because it reflects important
principles of adult learning.
__________
1 Jennifer Kaminski, Vladimir M. Sloutsky, and
Andrew Heckler, "
The Advantage of Abstract Examples in Learning Math,"
Science, Vol. 320, No. 5875 (April 25, 2008), pp. 454-455. You can read a
New York Times report on this research, including a few caveats,
here.
###Labels: Classroom training, Evaluation of training, Learning resources
Standardized Patients in Medical Training
The Associated Press published an
article on June 8 that provides a look at how "standardized patients" are used to help medical students hone and demonstrate their clinical skills. As enumerated by Cafe Nicholas, the director of the
Standardized Patient Program at the University of Vermont's medical school, these clinical skills include how "you present yourself to a patient — communication-interpersonal skills, history taking skills, physical exam skills, clinical reasoning."
The
website of the Association of Standardized Patient Educators defines a standardized patient as
a person who has been carefully coached to simulate an actual patient so accurately that the simulation cannot be detected by a skilled clinician. In performing the simulation, the SP presents the gestalt of the patient being simulated; not just the history, but the body language, the physical findings, and the emotional and personality characteristics as well.
What the AP article doesn't cover is the four decades of effort that lie behind today's widespread use of standardized patient methodology for medical training and evaluation. You can read that informative history (through the mid-'90s)
here (pdf). A key point is that the methodology was and, in some quarters, still is, met with skepticism. It took its advocates many years to build support through hands-on demonstrations across North America and documentation of training and evaluation outcomes.
To pick up an additional dimension of the story, you can read a savvy account of the standardized patient experience from the point of view of the person playing the patient
here.
###Labels: Evaluation of training, Simulation
Tips for Training Evaluation
The Mine Safety and Health Administration of the US Department of Labor has a
section of their
website that provides practical tips for carrying out training evaluation.
There are five tips:
- Organizing your thinking about evaluation — reviews Kirkpatrick's four levels.
- Planning with a training evaluation worksheet — along with the worksheet itself, this tip provides straightforward instructions for using the worksheet.
- Ways to gather data — offers guidance for choosing the best evaluation method to fit your particular situation.
- Recognizing mistakes in survey questions — needs to be read in conjunction with Tip 5 (see below).
- Writing better survey questions — an excellent summary, with clear examples.
The credit for this succinct coverage of principles of training evaluation goes to Launa Mallett of the
Centers for Disease Control and Prevention.
###Labels: Evaluation of training
Game Plan for Learning
As a follow-up to yesterday's
post on how testing helps cement learning, I'd like to call attention to an
article that appeared in the
New York Times on January 28. In the article, Winnie Hu reports on use of a technology in the classroom that represents a practical application of the principle that people retain more of what they learn when they are regularly and repeatedly tested on the material they're studying.
Source: Southeast Missouri State UniversityThe technology in question is an
audience response system. Each student has a handheld keypad that s/he uses to register answers to questions the teacher asks. The students' responses can be displayed in real time so that both teacher and students know how everyone is doing on physics, or math, or history, or whatever the subject happens to be.
And, according to the Karpicke/Roediger
research outlined in yesterday's post, this interactive testing is also helping students remember more completely the material the teacher is covering.
Last but not least, there seems to be a strong motivational aspect to the system. The quiz show atmosphere entices most of the students to actively "play along" with the teacher's plan for the day's class.
Companies that use quiz show formats as part of their training are following a similar path to reinforcing and cementing learning. You can read an about a few representative examples
here.
###Labels: Classroom training, Evaluation of training, Motivation, Teaching
Tests are Your Friend
When I was in graduate school, my main study technique for exams was to create lists of questions and answers that I then used to test myself. I was quite satisfied with the results this approach produced.
Now I've learned that there is a name for the process: the
testing effect — "enhanced memory resulting from the act of retrieving information, as compared to simply reading or hearing the information."
Last week, Jeffrey Karpicke and Henry Roediger, psychology professors at Purdue University and Washington University, respectively, published a paper in
Science, reporting the results of recent research they conducted on the testing effect.
1 You can read details in a
press release posted by Purdue.
The opening sentence of the press release — "Students learn more from taking test than they do by studying" — is misleading. The point is that
how you study makes a big difference in how much you retain. You need to make information retrieval — in this context, self-testing — the main way in which you do your studying (once you've read through whatever it is you need to learn). Indeed, at the end of the press release, Karpicke is quoted as saying, "Testing is really most effective when students implement it on their own."
As Karpicke explains,
... during a test there are cognitive processes happening that actually promote learning. Testing is not just an assessment of what you studied. The act of retrieving information actually improves memory because you are practicing a skill. And that's the exact same skill you are going to need to retrieve that information again and again."
The Karpicke and Roediger research also clarifies that you should keep testing your recall even after you have succeeded once or twice in recalling particular concepts.
For instance, just because you are able to recall once or twice that the German word "Wissenschaft" means "science, knowledge," you should not drop "Wissenschaft" from the list of German words you test yourself on from time to time. To promote long-term recall, you should continue asking yourself periodically what "Wissenschaft" means.
__________
1 "
The Critical Importance of Retrieval for Learning," by Jeffrey D. Karpicke and Henry L. Roediger, III,
Science, Feb 2008, Vol. 319, no. 5865, pp. 966-968.
###Labels: Cognition, Evaluation of training, Teaching
