The letter of introduction and instruction begins
In an effort to improve high school economics education in the state, we are asking you and other experts from colleges and universities across North Carolina to assess the quality of selected questions from the state's high school civics and economics and U.S. history tests. To test these questions, we added question evaluation options for you to consider.This paragraph taints everything that follows. The first sentence frames the survey as "an effort to improve high school economics education." The wording implies improvements are needed and that the respondent should help by finding things that need improvement.
Additional non-neutral framing appears later in the letter.
Over 100,000 public high school students took the North Carolina end-of-course civics and economics test last year. According to the NC Department of Public Instruction, seven out of ten students passed the test. Unfortunately, pass rates say little about what high school students know and the testing instruments used to assess that knowledge.So again, the letter is framing the survey negatively and, worse, signalling the respondent that the test is unreliable ("Unfortunately, pass rates say little about what high school students know").
The survey form itself asks about six test questions: four drawn from the civics and economics test and two drawn from the U.S. history test. The questions appear to be very selectively chosen.
Each test actually contains 80 questions. From the civics and economics test, there are at least two dozen questions that bear directly on economics. From the U.S. history test, there are another dozen that ask about economic knowledge or reasoning and several more that address political economy issues.
Thus, from the 160 questions on the two tests and from 3-4 dozen questions that address economic issues in some way, the Locke Foundation has selected six.
Anybody who has ever developed a test knows that different questions are better at discriminating students' knowledge than others. This why tests include many questions and why pass criteria are based on answers on the full test or test section, not a subset of answers.
Education researchers further know that there are formal statistical procedures for testing the validity and reliability of questions. Those procedures would also involve evaluating the whole set of questions in a domain.
There are additional problems besides the survey's framing and selectiveness. The formatting of the survey form pushes respondents toward indicating that the questions have problems.
Consider the first item on the survey asks about question 14 from the civics and economics test:
1. A person opened a booth at a flea market to sell paintings this is an example of which factor of production?You can see how the format of the question prompts the respondent to identify a problem. In particular, the check boxes, which are set apart from the test question, prompt particular negative responses.A. Capital
B. Entrepreneurship
C. Natural Resources
D. Machinery
☐ None of the above
☐ Two or more of the above
☐ Defective or misleading question
☐ Comment (optional) _____________
Respondents are also pushed toward these responses from the letter of instruction which told them that they are to "assess the quality" of the questions and that they have "question evaluation options ... to consider." The check boxes are the only entries on the survey form that explicitly allow for assessment.
It's only if the respondent has carefully read a shaded box on the survey sheet that the respondent would see buried in the middle of a paragraph directions that say that he or she can also circle a letter if that appears to represent the answer. The survey is putting a lot of faith in people's willingness to read directions.
I called Terry Stoops, the Director of Education Studies at the John Locke Foundation and the letter author, to relate these concerns. Mr. Stoops was kind enough to take the call and patiently listen. Throughout the call, he expressed confidence in the appropriateness of all of the aspects of the survey instrument.
I'm, in turn, confident that the survey is an exercise in providing a social-science veneer to a pre-ordained conclusion. I strongly recommend that my economics colleagues not participate.
This is not to say that test questions shouldn't be reviewed or that they can't be improved. However, there are much better and less shady ways of going about it.
