Eliciting knowledge from experts: A methodological analysis

The psychological study of expertise has a rich background and has recently gained impetus in part because of the advent of expert systems and related technologies for preserving knowledge. In the study of expertise, whether in the context of applications or the context of psychological research, knowledge elicitation is a crucial step. Research in a number of traditions - judgment and decision making, human factors, cognitive science, expert systems - has utilized a variety of knowledge elicitation methods. Given the diversity of disciplines, topics, paradigms, and goals, it is difficult to make the literature cohere around a methodological theme. For discussion purposes, we place knowledge elicitation techniques into three categories: (1) analysis of the tasks that experts usually perform, (2) various types of interviews, and (3) contrived tasks which reveal an expert′s reasoning processes without necessarily asking about these processes. We illustrate types and subtypes of techniques, culminating in a discussion of research that has empirically evaluated and compared techniques. The article includes some recommendations about “how to do” knowledge elicitation, some cautionary tales, and a discussion of the prospects.