Increasing Discussion of Learning Evaluation and ROI, Part 1

With the current economy's impact on Learning and Development budgets, it makes sense that learning leaders would have an increased desire to prove their worth and the worth of the learning and training programs they provide to the organization. At the same time, standards such as the Kirkpatrick model, cost-benefit analysis (CBA), return on investment (ROI), and so on are often criticized by many in the industry. In fact, during a recent Twitter LearnChat, several learning leaders included "ROI for learning" as one of the most overhyped aspects of our industry (alongside the use of virtual worlds, the use of reusable learning objects, and other topics).

At the same time, I've noticed a flurry of recent magazine articles and other resources on this topic, ranging from introductory basics, to good overviews, to provocative, forward-looking challenges. This posting to Element K Blog brings these recent resources together for you in one place.

For starters, if you need a quick primer on the well-established Donald Kirkpatrick’s Four Levels of Evaluation, check out this website's useful slide presentation.

Another good place where newcomers to the subject can start might be the eLearning Guild's newly published Guild Research report with the title "Building the Business Case for e-Learning." They note that this report is the "first in a new series of Getting Started Guides to help ramp up your organization quickly and successfully in e-Learning."  Written by Temple Smolen, the abstract is as follows: "E-Learning can feel like a bit of a minefield when first starting. It seems like there are countless decisions to be made, and mistakes can be costly. Overall, however, most organizations do meet their financial goals with e-Learning, and the majority yield positive returns. In an April 2009 survey of eLearning Guild members, 48% reported a positive return on investment (ROI) from e-Learning, while only 5% reported no return or a negative ROI. In fact, not only do most members generate a positive return, but 50% report a ROI greater than 15%. This report ... will focus on the evaluation phase of investing in e-Learning to determine whether e-Learning is the right delivery method for your organization or project. For most organizations, these decisions are subject to business considerations, so this report will tackle that evaluation from a business management and financial perspective. It will illustrate the early decisions needed, and demonstrate a high-level financial analysis with a detailed ROI calculation."
 
The May 2009 issue of Chief Learning Officer magazine included "Overcoming Analysis Paralysis," by IDC's program director of learning services Cushing Anderson. Cushing provides several results from the January IDC survey of Chief Learning Officer magazine's Business Intelligence Board (BIB) on the topic of learning measurement. Amongst the numerical survey results, he notes: "There is little disagreement among learning professionals about the value of measurement. When done properly, measurement can demonstrate training’s impact on the company’s top and bottom lines. Key metrics may include employee performance, speed to proficiency, customer satisfaction, improved sales numbers, and more. As the BIB survey shows, the challenge lies in gaining access to these key metrics and finding the time and resources to conduct measurement." He concludes by describing three steps that CLOs can take to "help break the chicken-and-egg tension between measurement and relevance": define success early, establish metrics at the project or business-unit level, and set expectations up front.

The June 2009 issue of Chief Learning Officer magazine included two articles on this general topic. The first was "The Real Reasons We Don't Evaluate," by Jack Phillips and Patti Phillips. The authors describe the following as being perceived barriers to "demonstrating the business contribution of learning and development:"

• It can't be done.
• It's too complicated.
• It's too costly.
• Evaluation takes too much time.
• It's not appropriate to evaluate learning and development.
• No one is asking for it.

In each case, they describe what they see as the reality as against these perceived barriers.

The second article in that issue is "CBA or ROI: That Is the Question," by Deanna Hartley. The author notes that while CBA and ROI are often used interchangeably, "people often denote benefit-to-cost as a ratio, while others in the HR and learning industries refer to ROI analysis as a percentage." Another difference is the perspective: is the endeavor being seen as a cost to be contained, or primarily as an investment? She then discusses the importance of aligning learning with business goals, and several key steps in conducting a cost-benefit analysis for a learning program.

In part two of this two-part blog posting, I'll describe and link to the two most provocative articles I've seen on this general topic in the past few months.

— Thomas Stone (Tom_Stone@elementk.com)

Comments

[Sam]

Great post. We recently created an interactive talent strategy benchmarking assessment that discusses some of these very concepts (avail here: http://budurl.com/3zzq).

One stat revealed by our collaboration partner on the project (noted analysts Bersin & Associates), is that 59% or organizations are now integrating employee development with career & succession planning to align organizational competencies with individual employee career aspirations.

This is an encouraging trend, but hopefully we will see even more adoption in the future. Anyone interested in taking the 19 question interactive assessment we co-developed with Bersin may access it here:

http://budurl.com/3zzq

Thanks.