Bad Qs, The Bar Where Your Choice Always Matters!
Summary:
After four years of opening the Bad Qs Choice Bar, I still stumble upon this corporate learning assessment issue with multiple-choice questions. Let's close the bar!
Select The Correct Statement!
What Do You Think The Correct Answer Is?
Welcome to Bad Qs, the bar where your choice always matters! You may have seen this funny multiple-choice question as an ice-breaker within learning geek meetings. I'll come back to the solution at the end of this article. But for now, let's look at a real example.
Prepping for the AWS Certified Cloud Practitioner certification, I ran across this example.
Which of the following options is TRUE for AWS Database Migration Services (AWS DSM)?
A) AWS DMS can migrate databases from on-premise to AWS
B) AWS DMS can migrate databases from AWS to on-premise
C) AWS DMS can migrate databases from EC2 to Amazon RDS
D) AWS DMS can have Amazon Redshift and Amazon DynamoDB as target databases.
E) All the above
You may not know the answer to the question, so let's simplify this to a general dilemma:
Which of the following options is TRUE?
A) TRUE
B) TRUE
C) TRUE
D) TRUE
E) All the above
What would be your choice? A, B, C, D, or E?
The Problems With "All The Above"
If a multiple-choice question has 5 options to choose from, and you're not using "all the above," the theoretical odds to make the right choice is 1 in 5 (20% chance). In practice, psychometrics has various science-based approaches to determine a more realistic value because not all options are equal. 🙂
The moment you replace the last option with "all the above," you make this a much easier game. As long as you can identify two TRUE answers, you know that the expected answer is E. As long as you identify one single answer that you know is FALSE for sure, you eliminate option E as well.
But, there's an even bigger issue you should be aware of. If this test is a high-stakes test in your organization (certificate, legal, etc.), and you allow participants to contest questions, you will be likely to see this argument:
Selecting a TRUE answer is not the same as NOT selecting a FALSE answer.
What does that mean? As long as the selected option satisfies the question stem, it doesn't matter whether there are better options. It all depends on how you word the question stem or how explicitly you instructed users to pick the best option!!
Which of the following options is TRUE?
A) TRUE
B) TRUE
C) TRUE
D) TRUE
E) All the above
By selecting A, for example, I can argue that the stem ("Which of the following options is TRUE?") will be perfectly satisfied with the option A. Is option A TRUE? Yes. Correct. I'm not saying other options are FALSE, I'm saying option A is a correct answer.
Lessons learned: avoid using "all the above" unless it's absolutely necessary. But if you do, you must word the question stem to make only one of the options correct, either by ranking the correct answers or eliminating some of them.
For example, you may see question stems including words like "most appropriate" answer, or "simplest," or "quickest," or "least expensive," etc. The proper wording suggests multiple options may satisfy the question stem; however, out of those, you're looking for one in particular. Context matters!
If you really have to have multiple correct answers, then just use multiple select. Speaking of multiple select, there are two ways to indicate the number of options to select: explicitly and implicitly. You can explicitly say "Select two." Or just imply that there are multiple correct answers by saying "Select all that apply."
There is a huge difference between these two approaches!
Let's say there are 5 options and 2 correct answers to select. If you explicitly say to select 2 options, the challenge comes down to 2 out of 5. There 10 different ways to pick 2 out of 5.
If you say "select all that apply": 0 out of 5 + 1 out of 5 + 2 out of 5 + 3 out of 5 + 4 out of 5 + 5 out of 5. That is 32 different ways. And then you have to think about whether you are awarding partial points or not.
Back To Our Bad Qs Question!
What was your answer? A,B,C,D,E,F, or none of them?
A) Incorrect – It would make all options below correct. It conflicts with option E or F, for example. Also, if A is true, then B is true. And C-F would need to be both true and false at the same time…
B) Incorrect – It would make all options below incorrect. It conflicts with option D, for example.
C) Incorrect – This is more interesting. It claims that either A or B is correct but not both. That contradicts A because option A states both A and B are correct. And it contradicts B, because option B states that C is incorrect.
D) Incorrect – Option D can't be correct because, again, it would make both A and B correct at once. (Just a note: if you ever use "all the above" or "none of the above," don't shuffle them. The order does matter!)
E) Correct – Check this out for yourself!
F) Incorrect – If option F is correct, then answers A-D are all incorrect. However, if that is true, option E is true as well. Oops. And if option E is incorrect, as stated by option F, then A-D can't be all incorrect.
Conclusion
A) Don't use "all the above" or "none of the above" unless you really need to.
B) If you do use "none of the above" and it is correct, make sure you tell the user the correct answer in the feedback.
C) All of above
- Update
Since the original article, you can now use AI to assist you with your questions, or even to write them for you. So, let's make sure we close down the Bad Qs Bar for good!
Originally published at www.linkedin.com.Originally published at eLearning Industry.com
