Friday, March 5, 2010

Shady economics survey from the John Locke Foundation

This week John Locke Foundation mailed surveys to academic economists in North Carolina asking them to assess economic portions of the state's 2009 high school civics and economics test and U.S. history test. The survey is a good example of the shady methods that are used by some partisans to influence policy.

The letter of introduction and instruction begins

In an effort to improve high school economics education in the state, we are asking you and other experts from colleges and universities across North Carolina to assess the quality of selected questions from the state's high school civics and economics and U.S. history tests. To test these questions, we added question evaluation options for you to consider.
This paragraph taints everything that follows. The first sentence frames the survey as "an effort to improve high school economics education." The wording implies improvements are needed and that the respondent should help by finding things that need improvement.

Additional non-neutral framing appears later in the letter.

Over 100,000 public high school students took the North Carolina end-of-course civics and economics test last year. According to the NC Department of Public Instruction, seven out of ten students passed the test. Unfortunately, pass rates say little about what high school students know and the testing instruments used to assess that knowledge.
So again, the letter is framing the survey negatively and, worse, signalling the respondent that the test is unreliable ("Unfortunately, pass rates say little about what high school students know").

The survey form itself asks about six test questions: four drawn from the civics and economics test and two drawn from the U.S. history test. The questions appear to be very selectively chosen.

Each test actually contains 80 questions. From the civics and economics test, there are at least two dozen questions that bear directly on economics. From the U.S. history test, there are another dozen that ask about economic knowledge or reasoning and several more that address political economy issues.

Thus, from the 160 questions on the two tests and from 3-4 dozen questions that address economic issues in some way, the Locke Foundation has selected six.

Anybody who has ever developed a test knows that different questions are better at discriminating students' knowledge than others. This why tests include many questions and why pass criteria are based on answers on the full test or test section, not a subset of answers.

Education researchers further know that there are formal statistical procedures for testing the validity and reliability of questions. Those procedures would also involve evaluating the whole set of questions in a domain.

There are additional problems besides the survey's framing and selectiveness. The formatting of the survey form pushes respondents toward indicating that the questions have problems.

Consider the first item on the survey asks about question 14 from the civics and economics test:

1. A person opened a booth at a flea market to sell paintings this is an example of which factor of production?

A. Capital
B. Entrepreneurship
C. Natural Resources
D. Machinery

☐ None of the above
☐ Two or more of the above
☐ Defective or misleading question
☐ Comment (optional) _____________
You can see how the format of the question prompts the respondent to identify a problem. In particular, the check boxes, which are set apart from the test question, prompt particular negative responses.

Respondents are also pushed toward these responses from the letter of instruction which told them that they are to "assess the quality" of the questions and that they have "question evaluation options ... to consider." The check boxes are the only entries on the survey form that explicitly allow for assessment.

It's only if the respondent has carefully read a shaded box on the survey sheet that the respondent would see buried in the middle of a paragraph directions that say that he or she can also circle a letter if that appears to represent the answer. The survey is putting a lot of faith in people's willingness to read directions.

I called Terry Stoops, the Director of Education Studies at the John Locke Foundation and the letter author, to relate these concerns. Mr. Stoops was kind enough to take the call and patiently listen. Throughout the call, he expressed confidence in the appropriateness of all of the aspects of the survey instrument.

I'm, in turn, confident that the survey is an exercise in providing a social-science veneer to a pre-ordained conclusion. I strongly recommend that my economics colleagues not participate.

This is not to say that test questions shouldn't be reviewed or that they can't be improved. However, there are much better and less shady ways of going about it.


jimcaserta said...

Would you take the whole test if they gave you that, or even the whole econ section?

That question seems sub-optimal to me - is the answer B? Maybe classically A? Either way, I don't see much relevance to being able to answer it correctly.

Having professionals take tests that will eventually be used by students is useful. If a majority of professionals can't answer a question because it is obscure or misleading, the question should be re-written. The AP program (or LSAT) seems to be one of the best test houses at being rigorous about the value of each test question.

Dave Ribar said...


I also think that the question is poorly framed ("entrepreneurship," the correct answer) isn't really a standard "factor" of production.

However, one or two marginal questions don't invalidate a test. Also, the Locke Foundation could have indicated (but didn't) that it selected the most problematic questions.

Finally, the Foundation could have made it clearer (but again didn't) that you could indicate that the question was okay.

When I went through the entire test, I was actually pretty impressed with what it covered and how.