 |
« June 2009 |
Main
| August 2009 »
I will now continue where I left off in the previous Element K posting, "Increasing Discussion of Learning Evaluation and ROI, Part 1." In this part two of two, I will describe and link to the two most provocative articles I've seen on this general topic in the past few months.
In the June 2009 issue of Training magazine, industry book author Saul Carliner gets a bit more provocative with his article "Maybe ROI Really Is a Waste of Time." He notes that one hurdle with ROI for training is that often "senior management is not interested in the evaluation because training isn't a significant part of their portfolio." That is, the total cost of training might be so small, relatively speaking, that cutting its budget will not have a significant cost-reduction impact on the organization. But Carliner then notes "That doesn't mean organizations lack interest in seeing a return on their investment in training. But the effort of showing how each course returned ROI provides little ROI—and that's why senior organizational managers don't actively seek it as much as some of the trade press might have us believe. "
He then raises the importance of non-quantitative approaches to training evaluation: "As suggested by the C-level study mentioned earlier, perceptions play a significant role in assessing the training function. Indeed, the survey by my research team found that word of mouth was the most common measure of effectiveness that training managers felt was important to their sponsor, ranking higher than ROI and course evaluations. "
Carliner concludes his "soapbox" column in this issue with the following advice: "In short, perhaps trainers should stop trying to collect ROI on individual courses because information about the ROI of individual courses requires resources that are rarely available to collect; does not provide a credible measure; and, most significantly, senior managers are not asking for it. Furthermore, because perceptions of training seem to have more credibility—and are more challenged among senior managers—and because other financial data is both more credible and provides more established evidence of the actual return on training, perhaps trainers instead should focus efforts on tracking and managing perceptions and following the broader measures that provide insights into the effectiveness of an organization's total investment in training rather than investments in individual courses."
And finally, in the July 2009 issue of Chief Learning Officer magazine, Jay Cross and Jon Husband take the discussion of ROI in a truly path-breaking direction with their article "Productivity in a Networked Era: Not Your Father's ROI." The authors begin by setting up the discussion: "The network era now replacing the industrial age holds great promise. Networked organizations are reaping rewards for connecting people, know-how, and ideas at an ever-faster pace. Value creation has migrated from what we can see (physical assets) to intangibles (ideas). …Understandably, seasoned executives, chief learning officers among them, are having a devil of a time shifting from the industrial age mindset of logic, certainty, and bounded constraints to the network gestalt of interaction, self-organization, unpredictability, and fewer limits to potential. …We are shifting into an era in which knowledge work and learning occur where re-engineered business processes collide with a participative and interactive ecology of information flows."
Cross and Husband then introduce a new concept as follows: "In an environment of continuous flow and interaction, there’s a need to consider an emerging metric: return on investment in interaction (ROII). The working definition of ROII is the observable development of capacity and capability to create economic values out of intangibles."
In the rest of the article, the authors describe what traditional ROI was, persuasively convince the reader of the increasing value of intangibles, and note the inherent clash between this trend and the demand for ROI metrics. They describe the essential characteristics of business networks, and argue that ROI is simply not up to the task. They then give some assumptions for their analysis of ROII and describe some of the potential components that would be quantifiable. In doing so they include illuminating examples from such companies as Cisco, Ford, and Capgemini. In closing, they frame the challenge as being: "Network returns are asymmetric, so simplistic count-’em-up approaches are no longer viable. But how can one make a solid network-era case to an executive who is still playing by yesterday’s rules?" Read this entire article to learn their initial advice on how to answer that question, and to understand this ground-breaking conceptual proposal for the L&D industry.
— Thomas Stone (Tom_Stone@elementk.com)
With the current economy's impact on Learning and Development budgets, it makes sense that learning leaders would have an increased desire to prove their worth and the worth of the learning and training programs they provide to the organization. At the same time, standards such as the Kirkpatrick model, cost-benefit analysis (CBA), return on investment (ROI), and so on are often criticized by many in the industry. In fact, during a recent Twitter LearnChat, several learning leaders included "ROI for learning" as one of the most overhyped aspects of our industry (alongside the use of virtual worlds, the use of reusable learning objects, and other topics).
At the same time, I've noticed a flurry of recent magazine articles and other resources on this topic, ranging from introductory basics, to good overviews, to provocative, forward-looking challenges. This posting to Element K Blog brings these recent resources together for you in one place.
For starters, if you need a quick primer on the well-established Donald Kirkpatrick’s Four Levels of Evaluation, check out this website's useful slide presentation.
Another good place where newcomers to the subject can start might be the eLearning Guild's newly published Guild Research report with the title "Building the Business Case for e-Learning." They note that this report is the "first in a new series of Getting Started Guides to help ramp up your organization quickly and successfully in e-Learning." Written by Temple Smolen, the abstract is as follows: "E-Learning can feel like a bit of a minefield when first starting. It seems like there are countless decisions to be made, and mistakes can be costly. Overall, however, most organizations do meet their financial goals with e-Learning, and the majority yield positive returns. In an April 2009 survey of eLearning Guild members, 48% reported a positive return on investment (ROI) from e-Learning, while only 5% reported no return or a negative ROI. In fact, not only do most members generate a positive return, but 50% report a ROI greater than 15%. This report ... will focus on the evaluation phase of investing in e-Learning to determine whether e-Learning is the right delivery method for your organization or project. For most organizations, these decisions are subject to business considerations, so this report will tackle that evaluation from a business management and financial perspective. It will illustrate the early decisions needed, and demonstrate a high-level financial analysis with a detailed ROI calculation."
The May 2009 issue of Chief Learning Officer magazine included "Overcoming Analysis Paralysis," by IDC's program director of learning services Cushing Anderson. Cushing provides several results from the January IDC survey of Chief Learning Officer magazine's Business Intelligence Board (BIB) on the topic of learning measurement. Amongst the numerical survey results, he notes: "There is little disagreement among learning professionals about the value of measurement. When done properly, measurement can demonstrate training’s impact on the company’s top and bottom lines. Key metrics may include employee performance, speed to proficiency, customer satisfaction, improved sales numbers, and more. As the BIB survey shows, the challenge lies in gaining access to these key metrics and finding the time and resources to conduct measurement." He concludes by describing three steps that CLOs can take to "help break the chicken-and-egg tension between measurement and relevance": define success early, establish metrics at the project or business-unit level, and set expectations up front.
The June 2009 issue of Chief Learning Officer magazine included two articles on this general topic. The first was "The Real Reasons We Don't Evaluate," by Jack Phillips and Patti Phillips. The authors describe the following as being perceived barriers to "demonstrating the business contribution of learning and development:"
- It can't be done.
- It's too complicated.
- It's too costly.
- Evaluation takes too much time.
- It's not appropriate to evaluate learning and development.
- No one is asking for it.
In each case, they describe what they see as the reality as against these perceived barriers.
The second article in that issue is "CBA or ROI: That Is the Question," by Deanna Hartley. The author notes that while CBA and ROI are often used interchangeably, "people often denote benefit-to-cost as a ratio, while others in the HR and learning industries refer to ROI analysis as a percentage." Another difference is the perspective: is the endeavor being seen as a cost to be contained, or primarily as an investment? She then discusses the importance of aligning learning with business goals, and several key steps in conducting a cost-benefit analysis for a learning program.
In part two of this two-part blog posting, I'll describe and link to the two most provocative articles I've seen on this general topic in the past few months.
— Thomas Stone (Tom_Stone@elementk.com)
Sometimes the best learning experiences have surprisingly simple media treatments. There are several media treatments that have become quite popular in recent years. One is the whiteboard-sketching model used in several television commercials, most notably by UPS. But another approach is the innovative, and now frequently imitated, "In Plain English" style used by Common Craft. See their website, and see also their home on YouTube.
In their numerous short videos, the folks from Common Craft, led by Lee LeFever, use cut-up bits of paper and move them around to convey the basic concepts and processes they are explaining. This is similar to what an animation would do, but with a distinctively low-tech aesthetic style. To be sure, much of the benefit from the videos comes from the outstandingly clear narration. But much of the value also comes from the very simple media usage, a clear counter-example to anyone who might argue that the use of robust Flash or the highest-quality video are necessary for great learning content.
You've probably seen at least a few of the more popular Common Craft educational videos already. They have created them in a range of topic areas, but most of them are on Internet-related concepts -- including several that are quite famous that explain social media / Web 2.0 concepts such as Twitter and wikis. Here is the current list of videos -- each under 4 minutes in length -- in this area:
My advice: take a few minutes each day to watch one of these videos to solidify your knowledge of these important concepts, and help drill home the point that when it comes to multimedia treatment in learning content, fancy and advanced uses are not always required.
— Thomas Stone (Tom_Stone@elementk.com)
It’s time for another edition of our new feature, the Learning and Development Roundup! See also volume one which was posted in mid-April, and volume two posted at the end of May.
ASTD International Conference and Exposition 2009
The first week of June included the 2009 version of the always impressive ASTD International Conference and Exposition, this year held in Washington, DC. Element K was an exhibitor of course, and we'd like to thank everyone who stopped at our booth to discuss their learning challenges and the learning solutions we can provide. We'd also like to thank everyone who attended the two sessions that Element K gave: "Serious Gaming Frameworks: Architecting Reusable, Customizable Games" "Generating Business Impact: A Four-Phase Approach to Learning Design and Implementation."
Many industry bloggers who attended the conference have been writing their perspectives and key take-aways. For now I'll provide links to two of them:
About Instructional Designers and the Names Used
In early June, Jonathan Atleson wrote a provocative blog posting titled "The Various Roles of Instructional Design." Jonathan describes the many overlapping job roles that people variously mean when they speak of "instructional designers." On a similar topic, Ellen Wagner wrote two good postings this past month: "On eLearning, ID, and Learning Technologies" and "Why It Matters What They Call Us."
New Roles in Learning and Development
And speaking of job roles in our industry, Jay Cross continued sharing his provocative thoughts in his June column in Chief Learning Officer magazine: "More Human Than Human." He notes:
"When my colleagues and I advocate cutting back on workshops and classes in favor of building “learnscapes,” we aren’t suggesting firing the instructors. Rather, we recommend redeploying them in new capacities, serving as connectors, wiki gardeners, internal publicists, news anchors, and performance consultants."
Then, after describing some of the tasks and skills that will be increasingly needed in organizations, he concludes with:
"These tasks won’t happen by themselves. Furthermore, people throughout the organization will need to share the burden of helping everyone learn. Distributing learning throughout the social fabric of an organization requires storytellers, mentors, bloggers, community elders, schedulers, and editors. We’re all in this together.
Some instructors will continue to instruct, but they will increasingly do so with network support and in smaller bursts. It’s a better use of their time. Face-to-face instruction packs a punch but is difficult to scale. Economics dictate that traditional instruction will play a diminishing role in corporate learning.
Traditional instructors and instructional designers are ideally suited to excel in these roles. They understand how adults learn and how to transform information into learning. It’s important for corporations to benefit from their learning people, not give them pink slips."
Supporting the Increase in Social Learning
Along similar lines, Jane Hart concluded a three-part series of presentations on the subject of Social Learning. This last installment, "Supporting Formal and Informal Social Learning", focuses on "some of the people issues involved in implementing a social approach to learning." In this presentation, Jane goes over some of the key ways that learning professionals will need to adjust what they are doing to help their learners to better self-manage their learning, use new learning tools effectively, and share, communicate, and collaborate with their fellow learners. Watch Jane's entire slideshare presentation to get all of her key insights.
The MASIE Center Provides a New Learning Leaders Fieldbook
The MASIE Center recently announced their latest free e-book, the Learning Leaders Fieldbook. In addition to the text content, there is a series of podcasts about how the book's contributors became learning leaders and the "day-to-day realities of their unique roles."
On Brain Rules
For the past several months, Clive Shepherd has been reading and commenting at his blog on the book Brain Rules, by John Medina (see the Brain Rules website). He has now concluded this undertaking with a good summary posting, with not only his conclusions but also links to each of his 12 chapter-by-chapter postings. Clive's journey makes for good reading, either in conjunction with reading the book or as a teaser.
— Thomas Stone (Tom_Stone@elementk.com)
|
|
