Good teaching is more a giving of good questions than a giving of good answers. -Josef Albers, artist and educator

The art and science of being a good teacher often comes with the ability to ask good questions. That is the take-home message from a recent study in the journal CBE-Life Sciences Education; the type of question — how a question is framed — can tell educators a lot about student thinking on the concepts being taught. 

Exam questions are either open-ended or closed-ended. Closed-ended questions such as multiple-choice, multiple-true-false, etc. are easy to grade, do not depend on student writing skills or suffer from grading inconsistencies (where different graders assess answers differently). However, open-ended questions like free-response (which require essay-type answers), short answers, diagramming, etc. are particularly good at capturing student thinking since the students must construct answers rather than simply choosing them. 

This study by Hubbard et al. was carried out during an introductory biology course at the University of Nebraska – Lincoln, on a total of 405 students. The students were exposed to multiple-choice, multiple-true-false- and free-response type questions throughout the course to familiarize them with the formats, before being asked to take four unit tests for assessments. Each multiple-choice and multiple-true-false question stem was followed by four statements, one of which had to be chosen for the former, while each statement had to be marked as ​‘true’ or ​‘false’ for the latter. In each of the tests, students were asked eight multiple-choices and 10 multiple-true-false questions that were common for all students. Each test also contained four experimental questions that were presented either as multiple-true-false or free-response questions, with almost identical question stems, except for minor adjustments in syntax or additional prompts for free-response questions. 

For experimental questions, each of the four statements in the multiple-true-false format represented a part of the expected answer to the corresponding free-response question. Based on whether students addressed the concepts represented by these statements in their answers, assessors could score parts of free-response answers as ​‘correct’, ​‘incorrect’ or ​‘unclear’ for each corresponding multiple-true-false statement. An ​‘unclear’ code was used when a student either gave an ambiguous answer, or because the concept corresponding to that multiple-true-false statement was not addressed in their answer. 

When student responses to the experimental questions were analyzed, the researchers found that although more students provide higher numbers of correct answers for multiple-true-false questions, correct response rates across the two question formats were highly correlated. 

“Our main findings were that each format has particular strengths and limitations and that instructors should keep these question tendencies in mind when designing tests and interpreting results. By asking students to evaluate specific statements, multiple-true-false questions can uncover incorrect ideas that may go undetected in the free-response format,” say authors Brian Couch and Joanna Hubbard. ​“However, multiple-true-false questions tend to overestimate the frequency of particular conceptions, while free-response questions require students to put their thoughts in their own words and thus provide a more unfiltered portrait of student thinking,” they add. 

When presented with multiple-true-false questions, 33% of students could classify all true/​false statements correctly, while 66% gave a combination of incorrect and correct statements, revealing that they had mixed conceptions. Free-response questions produced a wide array of patterns based on combinations of correct, incorrect and unclear answers — only 9% gave fully correct answers, addressing all four conceptions. The majority of free-response answers — 58%— were a mix of correct, incorrect and unclear conceptions and a fairly large subset of students (29%) also provided answers with no correct elements, consisting of wholly unclear or unclear and incorrect answers. 

In all, though multiple-true-false questions are useful in judging how students tackle specific concepts, they can also obscure nuances in student thinking. Since multiple-true-false formats require students to simply state if a statement is correct or not, and not construct their own answers, unclear concepts are simply not detected.

Free-response questions on the other hand, may provide a more reliable estimate of student thinking, but are limited in their ability to identify the specifics of incorrect conceptions. For example, if a student is unsure of a particular conception, he or she may not write about it, or else might provide unclear answers that make it difficult for an assessor to identify specific problems in understanding. Furthermore, the study also indicates that the free-response format has extremely low diagnostic value for lower-performing students, since most of their answers tended to be unclear. 

Considering these tradeoffs, teachers must choose the appropriate questions types or mix question types to assess students. Narmada Khare, who teaches biology to undergraduates at the Indian Institute of Science, Bangalore, agrees with the mixed-question type solution. ​“I think that question papers should have a combination of free-response and multiple-true-false questions,” she says. Khare believes that although multiple-choice-questions and multiple-true-false formats are very practical choices, especially for evaluating large numbers of students, free-response questions are important too. ​“Multiple-true-false and multiple-choice-question formats do not improve a student’s ability to express himself or herself, a skill that is essential in a scientist,” she adds.

The study therefore leads one to conclude that when evaluating students, it isn’t a matter of ​‘asking the right questions’; instead, the challenge lies in asking questions the right way.