Comparing knowledge elicitation techniques: a case study

Three knowledge elicitation techniques were used to extract knowledge bases from experts on lighting for industrial inspection tasks. The techniques were: I a structured interview; II 'twenty questions' -imputing rules from information requests; and III a card sort. The first two techniques generate protocols, and in these cases two knowledge engineers independently extracted production rules from the protocols. In the third technique rules were derived from the classification of lighting solutions. The first two techniques led to about the same number of rules, the card sort to less. There was a slightly higher percentage of agreement between the rules extracted separately by the knowledge engineers for the normal interview than for the 'twenty questions' technique. Disagreements between the knowledge engineers were resolved by discussion, and an expert system to select special lights for inspection tasks was implemented. Of the rules finally agreed on, the percentage which could be implemented in the expert system was less for the twenty questions technique than for the others. The agreed rules were mailed to the expert, who indicated whether she agreed, disagreed, or wanted to make a modification. This secondary stage of elicitation revealed no evident difference between the three techniques in terms of the proportion of rules validated by the expert.