Questions to test students’ understanding of research methods

Research methods are a compulsory component of many undergraduate programmes. But it is challenging to design good questions for assessing students’ understanding of research methods in closed book exams. There is a fine balance between ensuring the often broad and ambitious course objectives and content are fully reflected in the questions while avoiding asking students to replicate all the research procedures.

In the final exam for our research methods class, we include eight to 10 varied questions. Below are some common types:

Defining key concepts and terminologies

While research methods are not purely technical and procedural, there is a need for students to understand the foundational theories before they proceed further. So, we set basic and factual questions to check students’ fundamental understanding of various research concepts and terminologies. This is not aimed at forcing students to memorise definitions or exact wording. We want students to understand and explain concepts in their own words.

• What is a critical case study?
• What is the difference between a panel survey and a longitudinal survey?

Giving real-world examples

The best way for students to demonstrate that they understand the research concepts is by giving real-world examples as concrete illustrations or applications. We go further by asking students to use their own daily experiences and observations to illustrate research concepts in their own words. This process helps students to understand and reflect on how the concepts fit into a greater societal context that is far more interesting.

• Explain what is a measurement reliability, with an example.
• Use an example to illustrate the Belmont Report ethical principle of justice in research involving human subjects.

Application of real-life contexts

Students should be capable of making sense of complex research concepts by applying them in real-life research contexts. But rather than asking them to draft a full research proposal which would involve framing research questions, identifying variables, stating measurement tools, framing research instruments and so on, we can ask questions that focus on just one or two of these components. In this way, we can check whether students understand the crux of various research items.

• Construct an example of 2x2 factorial design in a survey experiment that investigates public opinion towards foreign immigration.
• Write a research question and explain how you can personally use participant observation to answer the question.

Doing simple mathematical calculations

Although students often feel anxious about the quantitative part, it is essential for them to have a basic understanding of the mathematical or statistical steps and procedures relating to research calculations. To check their understanding of fundamental statistical concepts, we ask them to do or describe simple calculations that are manageable under the time and space constraints of the exam.

• Chelsea conducted a survey to study turnout patterns in the last Student Council election. She collected the following data. Consider the respondent’s Grade Point Average, calculate the mean and compare it with the median.
• Describe without using any numbers or mathematical symbols the steps for conducting a t-test for the difference in means.

Offering evidence-based commentaries

To facilitate deeper thinking on a variety of research concepts, we ask students to assess and respond to a number of hypothetical research claims and scenarios, using sound reasoning and concrete evidence.

• Ren Shen suspects that taking Ginseng Tonic will improve academic performance. He surveys his classmates and finds that those who take it have an average GPA score significantly higher than those who do not. He concludes that taking it will improve GPA scores. Is Ren Shen’s causal conclusion a credible one? Explain.

• Suppose you have invented a time machine that allows you to travel between the past and the present. Explain how you may potentially use your machine to solve the fundamental problem of causal inference.

Aligning with real research in the field

To align students’ learning with authentic research, some questions can be rooted in real research. These might include asking students to identify the relevant information in an abstract, summarising the research design, or interpreting the relevant data output and making resulting conclusions. The following research experiment conducted by the team has a number of ethical problems. Identify one of these problems.

• Consider the following regression output. Identify the dependent and independent variables. Which variables are statistically significant? How do you know?

Adrian Man-Ho Lam is course tutor in the department of politics and public administration at the University of Hong Kong.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.