Needing Experts
Last Saturday down in Washington DC at redUXdc, Dave Cooksey gave a great presentation and part of his talk centered on cardsorting using Delphi techniques. Dave emphasized that the technique should be used with participants who are experts.
In his abbreviated talk Dave didn’t go into how to ensure that you have an expert participant for cardsorting. If the methods that you are using presume using expert users (or at least users with domain knowledge) then to increase the validity of your results you should try to qualify your participants.
Finding Experts
User Centered Design, UCD, is centered on users. But that shouldn’t mean any users, as much as possible it should be the right users, experts users in the domain you are investigating. UCD methods without users means almost nothing, with the wrong users you could think you are right, but with the wrong users.
We need to be better about finding or qualifying people who we test or research. Here are 3 simple tricks you might use to help select better participants.
- Give them a short test of three or five questions about the domain. It might be knowledge of certain financial terms for a 401K website, of certain garden pests for a seed website, or of Fast & Furious movie facts for a auto enthusiasts website. Only accept those with a certain number of right answers. (It is even possible to do this without pre-determined “right” answers using cultural consensus modeling.)
- Ask them to rate themselves on a scale of one to ten. If they rate themselves below a certain number they don’t recruit them. (A issue is confounding their expertise with their ability to self-evaluate.)
- Ask them to recommend other people who they think are experts. Contact those people. (A very sophisticated method of doing this uses network analysis in which you contact those experts who are identified by numerous other people.)
In much of the work I have done I have always found the best investment of time and resources was in recruiting and screening (sometimes coming under the general method of sampling.) If you have really knowledgeable participants you can sometimes get by with less than stellar questions or inexperienced interviewers/researchers. But the reverse doesn’t seem to be true, even with the best questions and top-notch interviewers/researchers they can’t get the answers you need out of someone who doesn’t know them. It would be like trying to squeeze blood from a stone. So spend your time (and your clients money) more wisely and put a little more effort into your selecting participants for your next cardsort or usability test.
Acknowledgements:
I want to thank Chris Farnum and Margaret Hanley for their help and encouragement. Any errors in this post are, of course, my own.