Assessments Today

Creating and writing your own assessment or survey - DON"T

Here is my editorial article from my October ezine in case you missed it:

Besides what is the best assessment to use, the next most popular question is how do I go about writing my own assessment.  My advice is DON'T!  It has been quite a few years since I designed, developed and implemented a climate survey and I remember how challenging it can be.  Working on the new 1-Focus Leadership 360 has taken about one hour per question so far and we have just finished the draft mode. Since we are basing this on competencies and behaviors based on established research, we do not intend to do any significant scientific rigor on the tool.  This would be another massive expenditure in time and money.

Why is it easier to let someone else create the questions for you?  Let's use a simple example.  Perhaps I want to "measure" how well my step-son is "doing" in comparison to other children his age.  So I use this question amongst others that I develop:

Is this child able to count to 10?

Simple, right?  Oh contraire my friends.  To begin, this is a question and statements are better.  However even if we write it as Can count to 10, it remains an either / or statement - YES or NO. This isn't that helpful either.  Hence we encourage a range such as from "not at all" to "all the time".  Some designers will recommend a range of 10; others 5 or 7.  In addition, some questionnaires may ask the additional question of whether it is relevant in this situation or environment. (i.e. How important is it for my step son to be able to count to 10?)

We shall assume that the we have a range, but next, how important is it to ascertain speed, hesitancy, accuracy, willingness and/or comprehension of the concept?  For example on comprehension, we might train a parrot to quickly and accurately repeat the numbers from 1 to 10 but the parrot have no understanding of what it is repeating.  My step son is very fast and quite accurate - except that he usually misses 8 (both in French and in English).   However, he rarely will count for us upon our request for him to so.

What are we really trying to measure?  That is always the challenge.  The problem is that the more we refine it, we may end up adding adverbs, adjectives and qualifiers that instead of clarifying - end up making it more complex.  Take this typical statement for communication:  Communicates clearly and concisely in speech and written.

This is actually not a bad statement but would be "neater and tighter" if presented as:

Clear and concise in oral communications. It removes the confusion if the person is good at one form of communication and not the other.  It is written in the neutral tense and it is OK to use two descriptors that are tied together in general usage - clear and concise.  Two words that are not usually put together may cause greater uncertainty.

Finally, there are often inherent assumptions in these statements that is often missed.  Here is a great example in the following humor:

There are 10 kinds of people in the world:
-Those who understand binary
-Those who don't

Online Versus Paper Assessments: What's the Best Approach?

By Roberta Hill

I am always surprised when others, including coaches, fail to share my excitement about online assessments. However, there are good reasons to be skeptical. My mother's childhood expression, "What you lose on the swings, you gain on the roundabouts," is, I think, an apt summary of the tradeoffs between paper and online assessments. For purposes of this article, I will consider online and computer-based instruments together, since the same principles apply to both.

Despite the hype and excitement about online assessments, paper versions continue to remain more popular. Here are a few reasons why:

• Getting everyone to complete an assessment online can be an administrative challenge. Working in groups requires a strong and well-structured communications strategy, so this issue must be sorted out on an individual basis before offering the tool to your client.
  
• Despite my obsession with technology, I have discovered that my enthusiasm is far from being shared by the general public. Even in the business world, I find that about one third of the people I deal with neither like nor trust assessments—particularly online ones. They doubt both the confidentiality and the credibility of the process.
  
• Although technology seems great, emails and reminders are often forgotten, or they can end up in 'Junk' folders. Company firewalls can block emails and/or prevent client access to the internet site that holds the results of their assessments. And, believe it or not, not everyone has a personal email account or internet access at home.

In my view, there are only two 'real' but significant drawbacks to paper versions: Inventory and time. When purchasing paper assessments, a good unit price usually depends on buying a minimum of ten instruments. When I cleared out my inventory of various hard copy assessments last year, I discovered that I had close to a thousand dollars worth of retail material for the Myers Briggs Type Indicator that I will never use. Some of it was dated—I had various versions or editions of material that I could only administer in one-on-one situations.  In addition, I had approximately another thousand dollars invested in other types of assessments. 

While it takes about the same amount of time for a client to take an online or paper assessment, the administrative process is more cumbersome with a paper version. It takes time to deliver the assessment to the client, collect it back, score it (which can be quite labor-intensive and time-consuming if the instrument is complex), and return the output for client review. Conversely, when using an online assessment, distribution to the client, data collection, scoring and response to the client are automatic and seamless.

That said, I have encountered the following drawbacks to online tools:

• 25% of clients fail to complete them by the due date;
• The instruments are often limited to the English language;
• As with paper versions, virtual 'inventory' or credits can be a problem. Purchasing a distributorship or even opening an account often requires the purchase of materials that you don't want and never use;
• While the unit cost appears to be much lower than unit cost for paper, there can be hidden charges, such as annual access fees, which result in equivalent cash outlay over time;
• Not all of the online systems are terribly user-friendly—either to the client or to the administrator. Lengthy logon systems, difficulty in accessing the reports, and other technical issues can be problematic, and not all systems have reliable Help Desks; and
• Finally, there is a tendency for coaches/administrators to rely too heavily on the report, letting it take the place of proper introduction of the model behind the tool, adequate debriefing, and resultant dialogue between client and coach. 

It is far too easy to send out the information for an online assessment and then let the process take care of itself. If you are administering a paper instrument, you need to 'know your stuff.' You must be there to answer questions, explain the tool, review the material and discuss the results with your client. There is no fancy 30-page color report to fall back on. Instead, you personally discuss what the information means to the individual. An automated assessment often takes on a life of its own!

The level of training and certification required to administer online assessments seems to be somewhat inconsistent. While this may or may not be true, it often appears that anyone can purchase an instrument or a 'knock off' online. This tends to be less common with paper versions.   

On the other hand, instruments that can be taken online can be perceived as more 'valid' than paper versions. This is simply not the case. Psychometric rigor is even more of an issue with automated systems. Just do a Google search on 'personality tests' and you will find thousands of free quizzes available. There are many companies that provide survey and checklist creation services. Unfortunately, anyone—and I do mean anyone—can add a personally created 'assessment' to their website. While these informal inventories can be great tools when used properly, they are not legitimate assessments. Consequently, clients have good reason to be somewhat wary.

When it comes to using multi-rater assessments, online instruments win hands down. Computer programs can instantaneously perform the calculations that take hours to do by hand. Graphs of all sorts can easily be created to depict the results. However, I often think that the growth of the 360-degree market, and its prominence in the business world, is due more to technology than to good Human Resource Management strategies.

In summary, I continue to use paper in many team settings. However, with most of my business now being conducted virtually, I find that my needs are best met by online assessments. Besides, I am a techie assessment junkie! The ultimate question really is, "Will this particular online assessment meet the needs of my clients?" The answer to this question will always determine my recommendation.

Copyright 2006 Roberta F. Hill, AssessmentsNow.com Used with permission.   This article first appeared in Business Coaching Worldwide (Issue 3, 2006).

First Impressions

Here is a pretty good summary article about using assessments in hiring but the author goes into some solid background and statstics on assessments in general. 

By Dan Kiurski | Feb 1, 2001  Excerpt from Multichannel Merchant

Standardized assessments add objective information to the highly subjective hiring process. They help generate reliable feedback. Assessments are not designed to make the decision for you, but they do give you more useful information - information that is often below the surface - about a candidate's strengths and weaknesses. Basically, assessments help to identify candidates with the highest probability of success and eliminate those who are less likely to work at the level you require.

Take one A USA Today study of human resource executives shows that 63% of all hiring decisions are made in the first 4.3 minutes of an interview. The rest of the interview time is usually spent justifying this initial decision. Obviously, such a quick choice is based on the personal experiences and likes or dislikes of the interviewer rather than job fit. We tend to hire people we like whether they are the most competent applicants or not. Gut feel does count for something, but as an employer, be careful not to rely too heavily on your instinctive reactions.

An estimated 40% of companies use formal assessments for hiring and development today, compared to less than 16% only about fifteen years ago. Most assessments designed for business use are easy to use, affordable, and readily available to companies of all sizes. Assessments typically take anywhere from 15 to 60 minutes to complete and can be scored and processed immediately with a computer program or on the Internet.

Some assessment tools are general in nature, others specific to one sort of position (customer service or sales, for example), and some even have a feature that allows you to customize them by job classification to suit your company's needs. Some measures are simple aptitude tests that evaluate basic math or verbal skills or proficiency on specific software programs. Many are personality tests.

Tool types Tests vary greatly from provider to provider. Some assessments are current and geared for today's workforce; others have not been changed in over 20 years. Some tests have no validation, others are validated with only a small, controlled test group, and still others with a large, diverse group of working adults. Most assessments are geared toward hiring, but some provide excellent coaching, training, and developmental tools as well. Diagnostic self-assessments help individuals increase their self-awareness and acceptance of others and perhaps understand the need to change their own behavior. For current employees, 360-degree feedback instruments provide multi-rater feedback from a variety of sources such as self, boss, peers, and direct reports.

Personality tests are probably the oldest type of assessment, and they still offer the most variety. Most such tests rely on the honesty of the individual who completes the assessment, as they contain no check for distortion. A person could easily answer the questions as he thinks you would like them to be answered. Because of this, personality tests are usually not recommended for hiring. They can, however, serve as excellent coaching and training instruments.

Once you pick the correct assessment tool(s) for the position you are trying to fill, you should be sure to use it on a consistent basis as part of your hiring process. The best assessments for hiring contain some type of distortion score that measures the frankness and honesty of the candidate in answering the questions and gives the interviewer confidence in the results. They also provide appropriate follow-up questions based on the applicant's answers to help the interviewer better determine if this individual is a good fit for the job and a match with the corporate culture.

Honesty and integrity tests are very popular with many employers today, although they can be used only as a pre-employment tool. These low-cost, easy-to-use assessments are most commonly used for entry-level positions through first-line supervisory jobs. They are given early in the hiring process, often when the applicant comes in for the first interview. Usually administered in paper-and-pencil format and scored on a computer, these tests typically are written to an eighth-grade reading level, take about 20 minutes to complete, and contain from 75 to 125 multiple-choice questions. Although a few companies might still worry about the legalities of testing during the hiring process, in reality, using a job-related and validated assessment tool in a consistent manner can actually help protect against some negligent hiring situations. According to a recent U.S. Department of Labor publication, employment tests can be used to gather accurate information about job-relevant characteristics and to predict applicant and employee job performance. This information helps assess the fit or match between people and jobs. The appropriate use of professionally developed assessment tools usually enables organizations to make more effective employment-related decisions.

Dan Kiurski is the president and owner of Partners for Success, Inc., a company that specializes in helping businesses screen, hire, and retain top performers. Kiurski can be reached at by phone at (248) 828-1490 or by e-mail at dank@partnersforsuccess.net.