Research Methods

Shamus Khan, Princeton University

Gwen Sharp, Nevada State College

Research Methods (Fall 2021 Edition)

2

Research Methods

S H A M U S K H A N , P R I N C E T O N U N I V E R S I T Y

G W E N S H A R P , N E V A D A S T A T E C O L L E G E

INTRODUCTION

The importance of being wrong

Research ethics

TYPES OF RESEARCH METHODS

Five common sociological methods

Choosing a method

DESIGNING A RESEARCH METHOD

From topic to question

Variables

Independent and dependent variables

From research question to hypothesis

Selecting a sample

CORRELATION & CAUSATION

Validity & Reliability

CONCLUSION

Research Methods (Fall 2021 Edition)

3

INTRODUCTION

 How do we “know” things about the social world?

 What principles guide ethical research on people?

For decades, scholars knew that people who had served time in prison are much less

likely to have a job than other people are, but we didn’t exactly know why. The answer may

seem obvious, but as it turns out, there are lots of possible answers. One is discrimination:

perhaps employers just don’t trust people who were incarcerated and don’t hire them. Or

maybe people with criminal records are somehow different than other job applicants—

perhaps they aren’t very interested in working, so they don’t search very hard for jobs or quit

more quickly if they don’t like their coworkers. Maybe they missed out on getting important

training and skills while they were in prison, so they aren’t as qualified as other job applicants.

Or they might have trouble following rules, so they get fired.

Which explanation is correct? Are several of them accurate? How would we know?

Devah Pager studied this question as

a graduate student. She conducted an audit

study to look for an answer.1 She sent young

people to apply for jobs to see who was

most likely to get an interview; two people

applied for each position. She created fake

resumés for them to use with fake

qualifications that were similar, with one

exception: whether or not they had a (fake)

criminal record for a non-violent drug offense

(she also used Black and White applicants, to

see whether race mattered; you’ll learn

more about that in another chapter).

The advantage of an audit study is that if everything about the applicants is carefully

matched except one characteristic, then any differences you see must be explained by the

one thing that was different—in this case, whether applicants said they had a criminal history.

And Pager found that it mattered: having a criminal record affected the applicants’ chances

of getting an interview. Even though their qualifications were the same, applicants who

revealed their criminal record were less likely to be called back for an interview.

When Pager decided to use an audit study, she was following a particular method—a

study design that allows us to systematically investigate the world and be relatively certain

that we arrive at accurate conclusions. Sociology is a social science, and a critical aspect of

any science is that there are agreed-upon ways to generate knowledge. This sets science

apart from other ways of explaining the world, such as common sense or religious faith. At the

(Source)

Research Methods (Fall 2021 Edition)

4

core of scientific methods is a particular research attitude: skepticism. No matter who makes a

claim, and even if it seems to make sense, the job of scientists is to be skeptical of the claim

and to try to find problems with it.

All scientific studies of the social world share a key feature: scholars work hard to find

evidence that our conclusions are wrong. This may seem confusing – don’t we want to show

that our conclusions are right? But this is how scientific knowledge advances: it’s not enough

to provide evidence that a claim is right; you must search for evidence that it’s wrong. We’re

never absolutely certain that our claims about the social world are correct, but the more times

we try to show that our claim is wrong and can’t do it, the more comfortable we can be that

our explanation is correct. Whether we’re testing subjects in a lab or wandering the hallways

of a school observing how students and teachers interact, the basic approach is the same: we

look for other potential explanations for what we observe, or any evidence that our claim isn’t

accurate.

Remaining skeptical and considering other explanations can help us avoid confirmation

bias, the tendency we all have to look for and accept information that reinforces what we

already believe.2 Confirmation bias is a basic part of our psychology. We don’t do it on

purpose, and usually we aren’t aware it’s happening. But confirmation bias can lead us to

quickly accept information that matches our existing theories or beliefs, while we remain

doubtful about, or fail to notice, evidence that contradicts what we already think. The

scientific emphasis on searching for evidence that a claim is wrong can help us address this

bias in our thinking as we try to explain the social world around us.

Research ethics

The most essential consideration of any research project should be ensuring the project

is done safely and ethically. Research ethics are important for all research, but they are

especially crucial when you are conducting research on people, or human subjects.3

Unfortunately, scientists haven’t always agreed on what makes research ethical, and

they don’t always design ethical research projects. The most infamous cases involve medical

research. For instance, during World War II, German researchers (mostly doctors) conducted

painful and often deadly experiments on people imprisoned in Nazi concentration camps;4

the prisoners were forced to take part, and the experiments left them with burns, wounds, and

other injuries. Aside from the horrific suffering and death they caused, many of these

experiments had little or no scientific value; they didn’t help scientists cure diseases or

otherwise benefit humanity.

After the war ended, many of these researchers were criminally charged and

convicted. The international outrage at what the Nazi experimenters had done led to the

establishment of the Nuremberg Code in 1948, which outlined basic ethical principles for

research on people.5 The first, and perhaps most important, principle is that people who take

part in research must voluntarily consent to do so; they cannot be forced. The Code also

established other key ethical rules, including the following:

Research Methods (Fall 2021 Edition)

5

 Researchers should avoid all unnecessary physical and mental suffering and injury to

subjects;

 The degree of risk to subjects has to be justified by the likely benefit to humanity of the

knowledge gained from the research;

 Subjects must be free to stop participating at any time;

 If researchers discover their project poses serious risks to human subjects, they must end

the project immediately.

Despite these clear

principles, researchers

sometimes ignored the

guidelines. The Tuskegee Syphilis

Experiment, conducted in

Alabama from 1932 to 1972,

looked at how the symptoms of

syphilis developed over time if

left untreated.6 Researchers from

the U.S. Public Health Service

used hundreds of poor Black

men in rural Alabama as their

subjects. They never told the

men that they had syphilis—they

said they had “bad blood.”

Worst of all, after 1947 there was

a treatment for syphilis: penicillin

could completely cure it in the early stages. Even after the establishment of the Nuremberg

Code in 1948 and its acceptance by the U.S. scientific community, the Tuskegee study

researchers didn’t tell their subjects about the cure or offer them penicillin; they let the men’s

syphilis progress so they could see what happened. Many of the men died when they could

have been cured. Others gave the disease to their female partners, who transferred syphilis to

their children during pregnancy, leading to lifelong complications including seizures and

blindness. The study finally ended in 1972 when a whistleblower reported the project.

The Tuskegee experiment’s lingering impacts came up as a major concern during the

COVID-19 outbreak as public health experts tried to convince people to get tested and, later,

vaccinated. Doctors and others working in Black communities worried that the legacy of the

Tuskegee experiment would make it harder to convince Black Americans to now trust the

medical establishment on the best way to address COVID-19.7 The harm of unethical research,

they argued, isn’t just in the suffering of those directly affected by the study, but in the anger

at and lack of trust in scientists and medical experts that may last for decades. The

understandable mistrust Black communities may feel as a result of past unethical research

could make it harder to effectively treat health issues today. Public health officials worried that

Doctor drawing blood from a patient as part of the Tuskegee Syphilis

Study. (Source: National Archives, Atlanta, GA.)

Research Methods (Fall 2021 Edition)

6

this would lead to more outbreaks of COVID-19 among African Americans, which could then

lead to higher numbers of hospitalizations and deaths among them than in other racial

groups.

However, other researchers found that African Americans’ concerns about the vaccine

were driven by many of the same factors causing other groups to be hesitant—a concern

about its safety or a broader mistrust of how it had been so quickly developed under President

Trump’s administration—and that we should be careful about assuming that African

Americans’ mistrust or hesitancy about medical issues is only rooted in unethical research that

happened in the past.8 Doing so can allow us to see research ethics as part of history, rather

than confronting more recent problematic research as well as unequal treatment in the

medical system that may affect how different racial groups feel about, and how much they

trust, doctors and other healthcare providers today.

There are many other examples of unethical research.9 As a result of such ethical

failures, today federal guidelines attempt to protect research subjects.10 Though most of these

guidelines were established primarily to cover medical research, regulations also cover social

science research. A key requirement is informed consent. This means that all human subjects

must be informed about the research project, including any likely risks, before they agree to

participate. For a participant to give informed consent, they have to fully understand the risks

(and possible benefits) of the research.

While the problems with unethical medical research can appear obvious, it can be

harder to imagine how social scientists could hurt participants. But social scientists often

collect sensitive information about people, and it could be harmful if that information is

released. For instance, imagine you were interviewing married subjects about whether they

had ever had an affair. That information could be very harmful if you released it in a way that

allowed readers to figure out the identities of your participants. It could potentially affect their

reputations in the community or end their marriage, and could also be very embarrassing and

upsetting for their spouse, who wasn’t even a participant in your study. For sociologists,

protecting the privacy and identities of participants is essential; we must make sure that the

research findings we publish do not put participants at risk by releasing private information

that could hurt them.

Research Methods (Fall 2021 Edition)

7

TYPES OF RESEARCH METHODS

 What are the benefits of experiments, surveys, participant observation, historical analysis,

and content analysis?

 What are the weaknesses of each of these methods?

 How do we choose a particular method?

As you plan your research project, you will decide how to collect your data and what

types of data you’ll collect. Data generally fall into two categories: quantitative and

qualitative. Quantitative data come in the form of numbers and reflect quantities or amounts.

Qualitative data aren’t numbers; they usually reflect general themes and might include

transcripts from interviews, survey questions that ask people to explain something in their own

words, or detailed notes from visiting a particular place to observe it. Each of the methods we

review below can produce both quantitative and qualitative data. While some researchers

prefer one or the other, in reality many use a mixture of both.

Five common sociological methods

At the beginning of this chapter, we described Devah Pager’s audit study. Audit studies

are one type of experiment, a research method in which characteristics or behaviors are

carefully controlled. By controlling the environment, researchers can isolate the impacts of the

one characteristic that changes. Perhaps we want to know whether people feel more anxious

after looking at their friends’ social media accounts. We might bring people into a lab and

give them a short survey to measure how anxious they are. We could then have them scroll

through their friends’ social media accounts for 15 minutes and give them the anxiety survey

again afterward. Since nothing else happened during the study, if we find they’re more

anxious after looking at social media than they were before, we can presume that viewing

their friends’ posts increased their anxiety.

Experiments can be extremely useful because they allow us to carefully study the

impact of one thing at a time. Because we can control what happens to subjects, we can

make sure that the only thing that changes is the item we’re interested in. But there are

downsides to experiments, too. Especially for those that take place in a laboratory

environment, researchers may wonder whether the situation was realistic. Would we see the

same effect in the “real world” outside of the carefully-controlled lab? It’s possible that a

relationship that appears in an experimental setting wouldn’t work the same way in our

everyday lives, where we’re never affected by just one factor at a time. Finally, because

experiments give researchers so much control over subjects, it’s especially important to think

about ethical issues when designing them.

Research Methods (Fall 2021 Edition)

8

You may never have taken part in an experiment. But there’s a very good chance

you’ve participated in surveys, or sets of questions that subjects answer. They may be

conducted in person or sent through the mail, but increasingly surveys are completed over the

phone or online. During the 2020 U.S. presidential campaign, you may have received phone

calls asking you to rate how concerned you were about different issues or how likely you were

to vote for a particular candidate. Or maybe you’ve been asked to complete a satisfaction

survey after contacting a customer service office, rating your feelings from “very satisfied” to

“very unsatisfied.” Because so many groups use surveys today—including social scientists,

marketers, political campaigns, companies, and more—you’re likely to encounter them

frequently.

Surveys are a very common

method because they’re a relatively

cheap and quick way to get lots of

information from large groups of

people. That can give us a good

idea of widespread patterns, as well

as differences between groups (for

instance, we might get different

survey responses from men and

women). But surveys can have

problems, too. A common issue is

low response rates; that is, only a

small proportion of people you try to

contact complete the survey

(perhaps because they’re frustrated

from receiving so many requests to complete surveys!). Another problem is wording issues.11

The way you write questions can affect the answers you get. For instance, one group of

political scientists found that people responded differently when asked about “gay or lesbian”

rights than when asked about “homosexual” rights;12 because people tend to feel more

negatively about the word “homosexual,” using it can change how they respond on surveys.

As you read other chapters in this text, you’ll encounter several descriptions of

participant observation.13 In this method, the researcher spends time among a group, directly

observing and participating in that social world. This can mean moving to another country to

live among a different culture, but you can also do participant observation closer to home. For

instance, as she describes in the book Class Acts, sociologist Rachel Sherman worked at the

front desk of two expensive hotels in the U.S. to study how the hotels ensure that their wealthy

guests feel pampered.14

The benefit of participant observation is that it allows researchers to collect a lot of

extremely detailed information about social life in a particular group; we can learn what

people do, how they interact, and what they think about those interactions. Sherman learned

Researchers may visit public places and collect survey responses on

the spot. (Source)

Research Methods (Fall 2021 Edition)

9

about the tactics hotel employees engaged in to create a “luxury” experience. For instance,

room service waiters took notes on how hotel guests like their food served and gift store clerks

kept track of any special requests from guests. This information was entered into a computer

database, allowing one guest to receive her papaya cut exactly the way she wanted without

having to ask each time and another to have his favorite cigarettes waiting in his room on

future visits, though the hotel didn’t normally stock that brand. Observing and actively

participating in life at the hotel allowed Sherman to understand the intricate ways hotel

employees attended to the needs and preferences of their wealthy guests, making the guests

feel valued and effortlessly pampered.

However, participant observation can be

time-consuming and expensive (especially if you

have to move somewhere specifically to do your

research). It may take years to earn the trust of a

group and feel confident that you truly

understand the social world you’re studying

(especially if there are language barriers). And

you’ll only gather data on a small number of

people; you can’t realistically get to know and

talk to thousands of people. This can lead to

questions about whether your findings apply

outside of that small group.15 Finally, two related

methods are historical analysis and content

analysis.16 These methods involve analyzing

existing sources (such as historical records, media stories, or episodes of TV shows) to find key

themes. Sociologists Erin Hatton and Mary Nell Trautner completed a content analysis of Rolling

Stone cover photos, looking at how men and women were sexually objectified by the

magazine.17 Analyzing nudity, poses, and the focus of the photography, they found that

sexualization of both men and women has increased over time, but that women are still

sexualized more often, and to a greater degree, than men. In his study of suicide, Émile

Durkheim used historical death records from towns across France to see how frequently suicide