Making the Sausage: The Recipe Matters
Our Journey So Far…
Defining BS
Detecting BS (logically)
Describing where and why BS thrives
Deciphering BS (statistically)
Detecting BS (methodologically)
Constructs
Hypothetical construct is a concept that: § Does not have a single observable referent § Cannot be directly observed § Has multiple referents, but none are all-inclusive
Cronbach & Meehl (1955)
Examples: § Center of mass § Company performance § Service quality § Intelligence § Team mental models
Observations
Observations are: § Collected information about a phenomenon § Can be sensed or measured with instruments § Can be qualitative or quantitative § Often only referred to as ‘variables’ (imprecisely)
Examples: § Height, weight, physical attributes § Sales volume, productivity, scrap rate § Direct/indirect costs § Employee or consumer behavior § Survey responses, test scores, etc.
From Definitions to Positions…
For our purposes, a variable is a measurement that we use as input into a model to be analyzed or tested…
IV (independent
variable)
DV (dependent
variable)
X Y
Predictor, Antecedent, Intervention
Criterion, Consequent,
Outcome
From Definitions to Positions…
Moderator – partitions the IV into subgroups that establish domains of maximal effectiveness in regard to a given DV
IV (independent
variable)
DV (dependent
variable)
X Y
A moderator changes the relationship between variables (amplifies or attenuates it). It reveals “for whom” or “under
what conditions” a relationship may change
Mod (moderator)
From Definitions to Positions…
Mediator – a generative mechanism through which the IV is able to influence the DV (i.e., it explains the relationship)
IV (independent
variable)
DV (dependent
variable)
b
A mediator conveys the influence of the IV onto the DV. It is an intervening force. It reveals the “how” and “why” a
relationship exists
Med (mediator)
a
c
Research Settings and Strategies
Directly manipulate one or more IVs
Randomly assign units to conditions
Test effects on DVs & mod/med variables
Hold constant confounds by design
True experiment ✓ ✓ ✓ ✓
Quasi- experiment ✓ ✗ ✓ ✓
Non- experiment ✗ ✗ ✓ ✗
Generally conducted to determine & establish cause-and- effect relationships among variables
Generally conducted to ascertain & describe relationships among specified variables of interest in a given situation
Generally conducted to learn about some phenomenon, especially when there is very little prior research
General Research Purposes
Exploratory
Descriptive
Causal
It All Starts with Design… “Though there are numerous techniques of data analysis, no technique, regardless of its elegance, sophistication, and power can save the research when the design is poor, improper, confounded, or misguided. As we have stated, and will state again, sound inferences and generalizations from a piece of research are a function of design and not statistical analysis.”
~Keppel & Zedeck (1989, p. 12)
General Research Designs
Types of Strategies
Researcher manipulates 1 or more variables to determine a causal relationship
Experiment
Observational
Longitudinal
Qualitative
Mixed method
Researcher observes (does not intervene) to find correlations among the collected data
Observational study that involves repeated measures over long periods of time
Researcher explores beliefs, experiences, & perceptions through non-numerical data
Researcher blends numerical and non- numerical data to “triangulate” findings
General Research Designs
Types of Studies
§ Randomized controlled study § Controlled study § Before-after study § Cohort/panel study § Cross-sectional study § Case study
Controlled Studies
Intervention
Outcome OutcomeOutcome Outcome
Intervention
Baseline BaselineBaseline Baseline
T im
e
Controlled Studies (posttest only)
Intervention
Outcome
Intervention
Baseline BaselineBaseline Baseline
T im
e
OutcomeOutcome Outcome
Before-after Studies
InterventionBaseline Outcome
Time
Cohort/panel Studies
Baseline
Time 1
Time 1
Time 2
Time 2
Time 3
Time 3
Time 4
Time 4
Time
Measurement
Cross-sectional Studies
Population Sample
Variable 1
Variable 2
Variable 3
Variable 4
Variable 5
Variable n
One Time Point
Methods differ with respect to Control & Fidelity
Computer Simulation Laboratory Experiment
Field Experiment Interview/Survey
Observation Archival Study
Methods are Always Imperfect D
e g
re e
o f
C o
n tr
o l
D e
g re
e o
f F id
e lity
Methods are Always Imperfect
Potential Benefits Potential Costs
Computer Simulation Very precise manipulations; Model Dangerous/harmful situations Results only as good as the model; Cannot model all relevant variables
Laboratory Experiment Provides causal evidence; Randomassignment removes confounds Contrived setting; May lack the complexities of the “real world”
Field Experiment Provides casual evidence; Takes place in a “real” context Differential treatments may be prohibited; Confounds can occur
Interview/Survey Captures in-depth data; Provides insight on experiences/attitudes Difficult to show causal effects; Subject to biases in poor designs
Observation Within the real or natural context;Allows researcher participation Researcher interference; Can have misinterpretations; Time-consuming
Archival Study Often large scale and/or broad scope; Cost effective Typically cannot explain “why”; Has omissions, “unmeasured” variables
Measures are Always Imperfect…
“True Score”
What should be measured
“Actual Score”
What is really measured
Contamination DeficiencyRelevance
Examined “Epistemic Stroop Effect,” which refers to the fact that people involuntarily reject factual propositions that conflict with one’s knowledge of the world. These authors asked whether opinions have a similar effect. Conducted four separate experiments.
§ Showed 88 opinion statements on politics, social issues, personal tastes, etc. § E.g., “The Internet has made people more sociable [or isolated]” § For each statement, they made a grammatically incorrect version § We’re faster to verify grammatically correct statements (vs. non-grammatical)
§ Assessed extent that individuals agreed with statements
Key Findings:
§ Participants were quicker to identify statements as grammatically correct when they agreed with the opinion in the statement, compared with when they disagreed
§ There was no difference in time for identifying ungrammatical statements as ungrammatical
§ Results held even though agreement with the opinion was irrelevant to the grammatical task
“The results demonstrate that agreement with a stated opinion can have a rapid and involuntary effect on its cognitive processing”
Break into small groups (3-4 people) and address the following question: In chapter 4, Seethaler discusses 10 specific “context connections.” 1. Compare technologies to other technologies 2. Put findings in a geographical context 3. Consider the historical context 4. Express figure on a comprehensible scale 5. Qualify figures by circumstances where they hold true 6. Ask how the numbers being cited compare to ”normal” 7. Be careful not to be misled by averages 8. For percentages, ask “percentage of what?” 9. Reframe losses as gain or gains as losses 10. Determine if there is a context that explains an observation
Describe 2 connections and her examples Think of one other example of each (from life, business, prior lectures, etc.)
Lies, Damned Lies, & Science
Four “Types” of Validity
Can we infer a relationship between study variables based on statistical results?
Can we infer an observed relationship is a causal connection?
Can we infer measures effectively reflect the underlying constructs and relations among these constructs
Can we infer the observed effects will generalize to other persons, places, measures, or times?
Conclusion
Internal
Construct
External
Confounds
Generalizations
Statistical Conclusion Validity § Low statistical power § Individual heterogeneity (i.e., subject differences) § Context effects (i.e., extraneous environment) § Range restriction (i.e., artificially truncated data)
Threats to Valid Inferences
Threats to Statistical Conclusion Validity
A workforce analyst is interested in examining the key predictors of employee turnover. Using data collected during 2007-2009 from a large sample of employees at over 50 firms, spanning multiple industries, he finds several significant effects for variables not found in previous turnover research and is excited to write-up and publish the findings.
Context effects Individual heterogeneity Range restriction Low power
Threats to Statistical Conclusion Validity An educational researcher is testing the effects of using a new technology platform on learning a foreign language. To maximize the scope of the study, she recruits students from middle school, high school, and college. She is disappointed in the results, which fail to show any consistent significant effects associated with using the technology.
Context effects Individual heterogeneity Range restriction Low power
Threats to Statistical Conclusion Validity
A researcher designs a study to examine the effects of using a homeopathic drug to reduce cholesterol among high-risk individuals. He therefore specifically recruits individuals who have above average cholesterol levels to participate in the study. Unfortunately, he does not find evidence for the efficacy of the treatment.
Context effects Individual heterogeneity Range restriction Low power
Internal Validity § Regression to the mean § Maturation (i.e., natural change over time) § Mortality (i.e., subject attrition) § Instrumentation (i.e., measurement issues) § Subject selection (i.e., who/what is chosen)
Threats to Valid Inferences
Threats to Internal Validity A best-selling business book focuses on what makes companies great. The research on which the book is based includes in-depth analysis of 11 companies that went from so-so performance to top-notch performance, as defined by a sustained period of stock value dramatically beating market and competitor values. One reason the book is popular is because it offers straightforward company characteristics for leaders to imitate in their own firms.
Regression to mean Maturation Mortality Instrumentation Subject selection
Threats to Internal Validity An educational researcher wants to examine the effectiveness of Massively Open Online Courses (MOOCs). She conducts an experiment where students are randomly assigned to either a MOOC course or a traditional course. The content and duration for both courses are identical. The sample included 100 students (50 condition). At the conclusion of the study, 37 students complete the MOOC and 46 students complete the traditional course. She compares scores on a knowledge test, finding that students in the MOOC format, on average, scored 36% higher.
Regression to mean Maturation Mortality Instrumentation Subject selection
Threats to Internal Validity A firm discovers that direct reports of new managers (less than 1 yr. in position) have substantially lower engagement levels than the company average. To remedy these issues, an onboarding initiative is launched, requiring attendance in the first month of becoming a manager. A follow-up study 18 months after the initiative shows engagement for participating managers’ direct reports are at the level of the company’s average.
Regression to mean Maturation Mortality Instrumentation Subject selection
Threats to Internal Validity An energy engineer wants to assess the effectiveness of an energy conservation program. This program included a conservation campaign as well as an improved method for monitoring the firm’s energy usage. The amount of energy used was based on archival sources for the 2 years prior to the program and 2 years following the end of the program. The engineer found a significant decrease in energy use at about the time when the program was initiated.
Regression to mean Maturation Mortality Instrumentation Subject selection
Construct Validity § Reactivity/expectancies (i.e., ‘guessing’ the importance) § Novelty/disruption effects (i.e., too new or dramatic) § Compensatory rivalry (i.e., competition, not intervention) § Treatment diffusion (i.e., intervention ‘leaks’)
Threats to Valid Inferences
Threats to Construct Validity A Fortune 10 firm places “high potential” leaders in an intensive two-week, off-site program aimed at increasing self-awareness, learning agility, and leadership. The firm regularly collects data on the program’s effectiveness (e.g., simulations, 360 data, etc.). The VP of HR recently integrated an assessment that brings to bear the “latest brain science” purported to underlie effective leadership. The follow-up results show a significant gain in effectiveness of the program after just the first use of the assessment. Subsequent programs show much lower gains.
Reactivity and expectancies Novelty and disruption Compensatory rivalry Treatment diffusion
Threats to Construct Validity A finance professor embarks on an investigation of the effects of providing up-front information describing the common decision-making errors people make when investing. His hope is that exposing people to such information will lessen the likelihood of poor decision making (i.e., avoid the errors). He recruits financial advisors from 6 top investment firms to participate. He tracks participants’ views of the online information module and the effectiveness of their investment decisions for a 2-week duration after the module.
Reactivity and expectancies Novelty and disruption Compensatory rivalry Treatment diffusion
Threats to Construct Validity A commercial construction firm decides to run a test to examine if it’s worth it to purchase new robotic bricklaying machines. The lead engineer chooses two projects that involve the same type of building (i.e., size, shape, materials, etc.). One project uses the bricklaying robot and the other uses only human bricklayers. After two weeks, the data show that the robot is outperforming the human bricklayers by only about 5%. The firm decides the robotic bricklaying machine is not a worthy investment.
Reactivity and expectancies Novelty and disruption Compensatory rivalry Treatment diffusion
Threats to Construct Validity The Chief Research Officer at a large software firm wants to investigate the effects of “open office” layouts to see if this design facilitates cooperation among employees in their development teams. Timing is perfect as 6 teams are about to move into a new building. She randomly assigns 3 of the 6 six teams to floor with the “open office” and the other teams to regular layouts (i.e., cubicles). She tracks levels cooperation for several months in all 6 teams. The results show that cooperation has significantly increased for all 6 teams compared to historical benchmarks.
Reactivity and expectancies Novelty and disruption Compensatory rivalry Treatment diffusion
External Validity § Setting specificity § Outcome specificity § Respondent specificity § Meditation dependency (i.e., missing ‘mechanisms’)
Threats to Valid Inferences
Threats to External Validity A professor specializing in technology and innovation research has found strong support over multiple studies for the positive effects of using “design thinking” principles on the effectiveness and efficiency of software development teams. He decides to apply these principles in other types of teams, including marketing teams, production teams, and sales teams. Unlike his original research, his latest findings are quite “mixed” in their support of the benefits of design thinking.
Setting specificity Outcome specificity Respondent specificity Mediation specificity
Threats to External Validity A national retail store wants to understand how satisfied customers are with in-store experiences. They hire a retail consulting firm that posts a link to a customer satisfaction survey on the store’s website and shares the link across social media platforms. The results show that the vast majority of customers have negative in-store experiences. The store is now contemplating several potential interventions, all of which require substantial resources.
Setting specificity Outcome specificity Respondent specificity Mediation specificity
Threats to External Validity A substantial amount of evidence shows a strong relationship between scores from standardized tests (SAT, ACT, GMAT, etc.) and first-year GPA. A new large-scale study examines the impact of “test optional” policies on graduation rates and cumulative GPA. A major finding of the study is that being “test optional” does not have negative effects on these outcomes. The study also finds that high school GPA is a good predictor of college GPA. The authors claim their study suggests that standardized tests have very little value in higher education admissions.
Setting specificity Outcome specificity Respondent specificity Mediation specificity
Threats to External Validity A public health researcher is testing the effects of a community-based crime prevention program in economically depressed areas. The program uses neighborhood associations to solicit interest in organizing blocks of “neighborhood watches.” Association members tend to be the first “block captains,” which greatly reduces program start-up times. So far, the results have been very promising with quick increases in neighborhood watch participation and subsequent reductions in overall crime incidents.
Setting specificity Outcome specificity Respondent specificity Mediation specificity
