Newsletter Archives

Validating Competencies For 360-degree Feedback

The Survey Company newsletter has a new name! "Action Surveys" signifies our commitment to the use of surveys as practical tools for planning and decision making. To receive your very own copy of Action Surveys each month, subscribe today.

TODAY'S TOPIC

We periodically receive inquiries similar to this recent request:

I'm interested in learning more about what you have to offer in terms of surveys/survey items related to leadership competencies. I'm currently working with a client whose senior management team has identified several leadership competencies that they would like a
360 systems designed around. I'm interested in questions that specifically relate to these competencies, that have been well researched, tested, and validated. I'm also interested in reliability estimates for questions related to a single competency.

Can you provide this type of service/information?

To get a fresh perspective on this question, I asked Jerry Talley of Organizational Diagnostics to respond to Sylvia's inquiry. I asked his permission to include his informative and practical reply in our newsletter this month.

Dear Sylvia:

You asked about valid and reliable questions around particular leadership competencies. I imagine you are looking at building an instrument for your client and want to make sure the questions will provide a solid basis for decision making.

While your concern for validity and reliability are the traditional questions to pursue, my experience suggests another area is more critical for securing the best outcomes: the decision process at the end of the 360 feedback. Most executives are quite comfortable with questions that have face validity alone. Where they are most concerned is with how the data will be used...and they're right to worry about it.

Feedback data can be easily misused. And even when used appropriately, it is easy to distort the data collection process. Agreements up front on appropriate use and how
to ensure the integrity of the data will have a much stronger impact on the final result than time spent on crafting the most sophisticated set of questions.

I find that using scales of several questions is the best way to ensure validity. So a dimension such as " Exemplifying teamwork" is made up 3-4 questions that come at the same topic from slightly different perspectives. For example, I might ask...

1. This person demonstrates by his or her own behavior that they are able and willing to collaborate with their peers.

2. This person requires that their direct reports work together collaboratively.

3. This person makes sure that people are rewarded as much for their teamwork as for their individual achievements.

4. This person actively solicits a range of opinion, even when there is an emerging consensus of thought. Statistical analysis of the data (Cronbach's alpha) will tell us if these questions are behaving similarly and can be justifiably taken as an indicator of a single
concept. Prior to administration, those being evaluated can be queried if these questions capture their notion of supporting teamwork.

But the real question is how the data will be used. Will it be public? Or only given to the target? Will part of their bonus be contingent on their scores? Or on their improvement in scores? Or will their improvement plan become part of their performance contract (independent of the data)? What resources are available to the target to improve their scores? The real goal here is not just measurement, but improvement. What's the mechanism to support change?

If people will be compared to their peers, how will those comparisons be interpreted? For example, someone overseeing a sales group has a much lower need for teamwork than someone overseeing a design department. Demanding equally high scores would seem foolish.

In summary, I'd focus on negotiating the understandings for use of the data. Once that structure is in place, you will probably find that the questions are continually tweaked and adjusted over the years as new themes become relevant. For example, right now
supporting efficiency is more important than supporting innovation; in a year or two it could be the reverse.

Jerry Talley of Organizational Diagnostics
(650) 967-1444
2/20/2003

BEYOND 360-FEEDBACK

Foster talent development and organizational alignment throughout your company by linking 360-feedback to organizational goals, job profiles, performance management, career development and succession planning. Learn more here.

Call us toll-free, 877-666-2486, to discuss how we deliver results that matter for your business objectives. Or complete our contact information form to let us know how we can reach you.

TELL A FRIEND about ACTION SURVEYS™

If you find Action Surveys helpful, please pass it on. New readers can subscribe at
http://www.surveycompany.com/subscribe.html.

Archives: http://www.gainensurvey.com/resources_archives.html

Joanne Gainen, Ph.D.
The Survey Company
Online survey management and analysis
http://www.gainensurvey.com

You are welcome to quote this article or any of our newsletters, or to create a link on your web site to the archived version. Just include the following information:

Reprinted (or excerpted) from Action Surveys(tm), published by The Survey Company,
http://www.surveycompany.com. We would also appreciate receiving a copy or the URL where a link may be found.