Judgement in a Vacuum

I’ve been in this field for a while now. I’ve been to a number of conferences over the years. I read lots of magazines and blogs. Over the years, I’ve seen lots of industry professionals commenting on the current state of our industry. From the quality of design, to the tools we use, to the data we collect… there are lots of experts sharing opinions with the community about things we as an industry are doing wrong. While expressing their concerns, they often will share examples of best practices, providing an example of the type of work we should be doing. If you’re one of the people who are participating in conversations like this – and there are a growing number of you out there – I have a simple request.

Please stop.

This may seem like a strange request. After all, I’m a huge advocate of sharing and connecting with peers to learn. We need these examples of high-quality practices to give us a benchmark against which to judge our performance. I applaud anyone who shares their work with peers, and I think our industry needs more of this.

gavel1For me, the issue starts to shift when there are judgments being made about the work of others. I’ve been involved in countless conversations that consist of experienced professionals looking at a learning program and deciding, essentially, “This is crap”.

To clarify, I don’t have a problem with judging someone else’s work in and of itself. I have a problem when a judgment is made without enough information upon which to make an informed assessment. Let’s explore a few reasons why I think judging without context needs to stop.

The Context of Reality

You simply can not judge a learning program – and by extension the people who designed and developed it – without understanding the context under which it was developed. For example, let’s say I present you with two compliance elearning programs to evaluate.

  • The first is a story-driven program that tells the story of Bob, a new employee. It uses rich visuals and animations as it follows Bob along his workday and presents scenarios with compliance concerns. It explores the ramifications of decisions and actions Bob takes for each scenario. It’s also contextual, knowing the role of the person taking the course, so that the learner is only presented with information related to their role.
  • The second is mostly text-based, with a few interactions. It covers the policies of the company, common mistakes that are made, and ends in a multiple choice test.

Which program is better? Which was built by the stronger instructional designer? Which one better met the organization’s compliance needs?

Here’s the thing. You don’t have enough information to answer those questions. Let’s add some context reexamine these questions.

  • Both of these programs were developed to address the same organizational need. There was a major change in governmental regulation that changed a number of the policies and procedures for front-line employees. There was little notice of these changes; they became effective less than two weeks after they were announced. Senior Management contacted the L&D group and explained that these changes needed to be communicated to affected employees in advance of the regulation taking effect.
  • The story-driven program was a much deeper learning experience. It was delivered three weeks after the regulation took effect, and during that time there were two regulatory violations resulting in fines. The project also booked unexpected budgetary expenses for external graphic design development.
  • The text-based “page turner” was a much weaker learning experience. It addressed the need of making employees aware of the regulatory changes in advance of them taking effect. It did not result in unexpected budgetary expenses.

Now, let’s look at those questions again.

Which program is better? Which was built by the stronger instructional designer? Which one better met the organization’s compliance needs?

Let me throw in a fourth question – Which scenario do you think pissed off the L&D Manager’s boss?

The Differences Between External and Internal Points of View

from_my_point_of_view_king_681795I’ve worked both internally and as a consultant/freelance developer. These are two VERY different worlds. In fact, I’ve served both roles for the same organization. There are always constraints to work with on a project. The simple fact is that there are usually fewer constraints in place when a project is outsourced then when it is developed in-house.

When I worked internally for the organization I mentioned, I needed to work within the constraints of daily business. I would always negotiate as best I could to create quality learning experiences, but ultimately I was addressing a business need, not a learning need. Many instructional design projects are not everything they could be because of the constraints of what can be based on limited budget, time, or skill set.

These constraints still exist for external development houses and consultants, but they are almost always more relaxed. I still provide occasional design and development support to the organization I worked for. Any time they call me for help, they have already made a few important decisions about the project scope:

  • The project requires a development skill set beyond what is available internally
  • The project has a higher budget than the norm
  • The project is on some level “more important”, and therefore quality is getting more attention than speed or cost-savings

Any time I am contacted for assistance from this organization, those decisions are already in place. As such, it raises the ceiling I have to work with in terms of the level of quality I can deliver while still operating within the project scope. Much like a Subject-Matter-Expert that has forgotten what it was like to be a novice, many external experts have forgotten what it is like to work internally, and are therefore judging projects using criteria that are not applicable in the internal practitioner’s world.

“Best Practices” Vs “Right and Wrong”

As I said earlier, I love when people share great examples of the possibilities that exist for learning programs. There are a number of experts that share their work as examples of industry best practices, and we should thank them for that. Best Practices give professionals a standard to strive for.

Let’s just stop saying that not meeting the standards of “best practice” is somehow “wrong”. Part of applying best practices is the ability to adapt best practices to current conditions. That’s not lowering the bar, that’s adjusting the bar to the current reality.

Picture 5Be Your Own Yardstick, and Adjust it Accordingly

If there was a scale that measured the quality of a learning experience, the examples and best practices shared by experts would be all the way to the right, near the top of the quality scale. When they share their work, they provide examples that truly define the ceiling of quality we should all be aware of.  It provides an excellent margin for comparison as we all try to improve the quality of our work, but it does not provide an all-encompassing standard for judging the work of others.

There are too many variables (level of design/development skill, company culture, available resources, etc) in play to say that any one standard should be applied to all. What IS important is understanding the various levels of quality that can exist, and where the programs you are developing land on that scale.

Being aware of best practices helps us understand what is possible, but it’s not the yardstick you should use to judge your work. Instead, judge yourself on the work you did last week, last month, and over the last few years (if applicable). What’s really important is that you are continually moving forward, building your skills and striving to make each program you build better than the last.


, , ,

28 Responses to Judgement in a Vacuum

  1. Justin McDonough April 11, 2014 at 10:15 pm #

    Interesting article David.
    It reminds me of the term “Fundamental Attribution Error” found in
    social psychology. Basically, it is the
    tendency people have to emphasize internal factors rather than external factors
    when considering another person’s behavior.
    When people see work that they consider sub-par, they tend to attribute
    the failure to the designer being incompetent, lazy, etc. instead of
    considering external factors like time and budget constraints. It’s good to remember that there is always a
    story behind the story.

    • LnDDave April 13, 2014 at 11:59 am #

      Thanks Justin. Thanks for adding the social psychology aspect to the dicussion.

  2. Timothy Hartwell April 12, 2014 at 3:43 pm #

    Good article David. I have also been on both sides of the fence. Both presentations have merit; albeit the story driven one is likely to get the best traction; especially in today’s world where learning must have small, interesting bites to best be effective. But you’re right. Don’t knock a method until you can step into all the details of “why.”
    There are a lot of reasons why organizations and individuals do what they do. You can attempt to design the same exact program for two completely different businesses, and end up with two animals that look nowhere relative to each other.

    • LnDDave April 13, 2014 at 12:02 pm #

      Thanks Timothy. I think you can judge the quality of a learning experience in isolation; The challenge is that most designers are addressing needs and working within constraints that have little to do with “learning”.

      • Timothy Hartwell April 13, 2014 at 12:33 pm #

        I have seen entire programs designed as time being the guideline for the design, rather than learning being the initiative

        • LnDDave April 13, 2014 at 12:49 pm #

          Unfortunately that’s a common balance in the time/cost/quality paradigm.

  3. Luke Kempski April 12, 2014 at 6:37 pm #

    I understand your view David. I’m a believer in the Serious eLearning Manifesto in that it provides inspiration and direction to develop learning experiences that produce better performance results. I do have a concern when people use it to make negative generalizations about the state of the industry or to as you put it “judge without context.” Thanks for sharing.

    • LnDDave April 13, 2014 at 12:07 pm #

      Thank Luke. I would agree that the manifesto provides inspiration. I don’t see it as providing direction though. Direction implies a path to follow to get somewhere, and the manifesto doesn’t provide that.

  4. Nick Shackleton-Jones April 13, 2014 at 6:22 am #

    I’m just not sure that stumbling around in the dark can be described as “continually moving forward” David. Nor that we should kid ourselves that it is so. I like the distinction you draw between best practice and a scale and agree that falling short of best practice does not mean you aren’t somewhere on the scale. It’s just that we don’t have a scale. And for that reason I’m not even persuaded by many of the examples of ‘best practice’. As an example, I monitor users’ reactions to eLearning daily (using sentiment140); they don’t like it. So I quite like Patti Schank’s willingly-unwillingly distinction: short of a scale,I know, but if our consumers actually liked our products and used them willingly, that would be a persuasive piece of context.

    • LnDDave April 13, 2014 at 12:14 pm #

      Thanks Nick. I completely agree that people should not be stumbling in the dark. “Continually moving forward” requires conscious effort to seek out what is possible and to always be developing our skills. A large percentage of professionals lack that inner drive to do more. At the same tie though, that’s not a “learning professional” issue; that problems exists in all fields.

      I think there’s an implied scale of elearning quality, if only based on the volume of work available for comparison. The willing/unwilling discussion is equally powerful though.

  5. purist April 13, 2014 at 7:41 pm #

    Interesting article, Dave. While rich, story-driven, branched learning may appear ideal, one definitely has to consider the bigger picture of budget, time, and other constraints. Use judgment based on the situation and perhaps incorporate elements as resources allow. When the time is ripe to outsource, at least know what is possible given the appropriate support.

    • LnDDave April 14, 2014 at 11:32 am #

      Knowing what is possible is key. You need to understand something in order to adapt it to the constraints of the current situation.

  6. Roman Ferrer April 14, 2014 at 10:24 am #

    Much of my frustration with many past projects has been that I approached them as training needs rather than business needs because I am an instructional designer, therefore I typically apply my learning-focused decision-making model to the situation and dismiss viable options because they are not “real” training. Until recently, I complained a lot that I wasn’t able to use my ID skills and was doing all of this other “unrelated” stuff. I am presently in this bizarre transition state from being the ID guy to being a business partner, and I think it’s solidifying. Thanks for the insight!

    • LnDDave April 14, 2014 at 11:31 am #

      I think you’re describing the importance of the balanced skill set. It’s not learning OR business; it’s both. We need to understand and be able to speak to the needs of the organization so that we can adapt our learning programs accordingly so that we can build the best learning and skill-building experience that is possible given the needs of the business.

  7. Marcella Simon April 14, 2014 at 2:20 pm #

    That’s the problem with showing work samples- they are without context and also almost never completely the work of a single designer since SMEs, stakeholders, and other ISDs may have had a hand in their production.

    • LnDDave April 14, 2014 at 3:09 pm #

      I don’t think there’s any problem with showing your work on a project; or at least there shouldn’t be any problems with it.

      The problem is people who judge the work out of context. If I’m looking at your work, I can partially judge it as a learning experience out-of-context. But I can not judge you as a designer or how successful the project was in meeting objectives without context. It just can’t be done, and yet it happens all the time.

      You comment touches on why this paradigm bothers me. Why would someone share their work and ask for genuine feedback on an environment where they know people will judge the output without understanding of the context?

  8. Diane Elkins April 14, 2014 at 4:58 pm #

    Love this article! We were recently the “victims” of a drive-by critique at a showcase event. I was just flabbergasted how someone could decide in just a few seconds that we were doing something wrong. There is a fine line between discourse that raises the professionalism of the industry and elitism: “I’m good, and everyone else is lousy.”

    I also think that if we are not careful, we start designing for each other instead of for our audience. A friend of mine had to take an information security course where he had to search through a building and find all the problems. It drove him and his co-workers nuts. They had to jump through a lot of hoops to get very little information, and they would have just rather gotten the information in a straightforward way and gotten on with their day. The course was overdesigned, but it’s the type of course that would get praise in the e-learning industry. Super Bowl ads are a perfect example of designing for your peers instead of for your audience. Ad agencies design Super Bowl ads as showcases for the advertising industry. The next day, everyone is talking about the doberhuahua ad for “I don’t know…some car company.” Well if you don’t know what car company, then it didn’t do it’s job–I don’t care how clever or creative it was.
    Thanks for the discussion

    • Kathe Lynom April 15, 2014 at 7:38 am #

      Excellent response Diane! I love your drive-by critique—spot on!!

    • LnDDave April 15, 2014 at 9:49 am #

      “I also think that if we are not careful, we start designing for each other instead of for our audience.”

      Yes. Exactly that. Thanks for sharing Diane.

  9. Ray Jimenez April 14, 2014 at 10:01 pm #

    David, Thanks. You remind me of the failings of consultants. Let me illustrate. My wife is an executive. She leads over 200 people. Often she asks
    for my advice. Of course, I offer my insights. After my advice, I give her a warning.
    I am good at giving advices, but I never really have followed my own advice.
    Our work should be judged within the confines of our realities. Best practices
    and judgments, as promoted by others, can be misleading or worse deceiving. We
    only succeed in our projects one step, one inch at a time. And we should be
    proud and happy of our small, “but big”, successes. Ray

    • LnDDave April 15, 2014 at 9:47 am #

      Best practices can be great as an example, but not as a standard for judgement. Judgement requires context.

      Thanks for adding your context to the discussion.

  10. Helen Blunden April 15, 2014 at 6:02 pm #

    Thanks Dave, great article and I agree it’s a tough call. My personal opinion is that it’s going to get worse. Like yourself, I have worked both as an internal consultant, an external consultant for a content development company, had all roles from a facilitator, through to now virtual learning consultant. I’ve also worked in the business and also in the centralised HR function – I like to believe that I’ve gathered knowledge and experience in my work and I’m thankful for the opportunity to have done this. Many others don’t. In fact, they won’t be able to – or choose not to. It’s just the future of work – people will come and go into our field…

    I have worked with many people in my field with a variety of backgrounds. I have seen some fantastic examples of online learning – and I’ve seen some shockers. (Actually some cringe worthy examples that take my breath away that are so bad that I’m mortified that they’re even accepted by Learning and Development management teams). The former were designed by people who who had experience or skills in adult learning principles; the others were because the business did not invest in the right solutions – whether it was investing in training their people to use the content authoring tools or have basic instructional design skills.

    In fact, I’m currently coaching/guiding/supporting a few people now who are in learning who have NO experience or skills in it but who have (for whatever reason), expressed a desire to move into the field and by sheer resource pressure, the business is eager for anyone to help and have given them the project. Do they want to stay in learning? No. It’s only a stepping stone to another role in change management, comms or marketing or some other role.

    The reality is that there will be people who have the motivation to be in learning and stick with it as a profession; there will be many others who see it as a stepping stone to something else but will not invest the time to develop nor seek out ways to learn more about what is happening in our field and apply it and also there is the organisation will not support their development, or seek external resources or spend any money.

    The result is a mish mash of different ideas, programs, courses, tools, methodologies.

    Personally, I’ve quietly come to the realisation that all I can do is act as a peer guide or coach to others new to the field and guide them along but their instrinsic motivation is all theirs to do a good job with learning. Also, at the same time I like to use Yammer and enterprise social networking to promote good learning design so that the business managers out there can see the difference and not just get it done as opposed to get it done right.

  11. Kenneth Fung April 21, 2014 at 5:57 pm #

    Great piece David! Can I add one rider: let’s please stop using the term “best practices”. Context is everything. Best practice in one place may not fit, or work elsewhere, or just be plain wrong.

    • David May 4, 2014 at 7:40 pm #

      Good point. Context affects what may be “best”.

  12. Tim Tyler May 7, 2014 at 8:46 am #

    If I may say a word or two in defense of judgement it would be this: the judgements directed at our industry from outside are far harsher than any coming out of our own conferences. This is especially true of the internal learning departments referenced in the article.

    I’ve seen far too many learning departments, slashed, disbanded and “embedded” in the business function, or simply eliminated altogether not to be conscious of the judgements of our work and our profession from outside. If we, and here I especially mean internal learning departments and practitioners don’t identify our competition, especially the disrupters in our field, and adapt by receiving and accepting those judgements of our work, especially from outside, and adapt, we will become irrelevant, and ultimately replaced.

    Embrace the judgement, especially that which threatens your existence. It will help you to survive.


  1. Judging My Own Work: A Response - April 11, 2014

    […] just read through David Kelly’s blog posting “Judgement in a Vacuum” and I’m very glad he has brought this […]

  2. Judgement in a Vacuum — e-Learning Feeds - April 11, 2014

    […] Read the full story by David Kelly Subscribe tο get quality news from the industry's Top e-Learning Blogs, delivered by email Spam Free. SUBSCRIBE googletag.cmd.push(function() { googletag.display("ndfpm-eLearningFeeds-Post-Footer-1392381359"); }); […]

  3. Sunshine Posts (weekly) | Sunshine on the Gulf - April 12, 2014

    […] Judgement in a Vacuum | David KellyDavid Kelly […]

Leave a Reply