The efficacy of knowledge elicitation techniques: a comparison across domains and levels of expertise

Despite an increased interest in knowledge elicitation, there is still very little formal evidence evaluating the relative efficiency of the techniques available. In this paper we compare four KE techniques: structured interview, protocol analysis, card sort and laddered grid. Studies are reported across two classification domains, using eight experts in each. Despite its common usage, protocol analysis is shown to be the least efficient technique. The implications of this finding are reviewed. Finally, a study is reported in which non-experts are subjected to "knowledge elicitation". Subjects entirely ignorant of a domain are able to construct plausible knowledge bases from common sense alone. The ramifications of these findings for knowledge engineers is discussed.