It is often the case that researchers attempt to generalize findings from a single convenience sample to the population. They may also wish to make the claim that the sample effect sizes they discover are reasonably similar to the population parameters. The current study attempts to show that they can be mistaken in this assumption, and that different samples can vary dramatically in effect sizes due to myriad discrepancies, such as demographics, sample size, and random error, among other aspects of the samples. Seven hundred and eighty-one participants were recruited from Amazon’s Mechanical Turk, Florida Institute of Technology, Embry-Riddle Aeronautical University, and San Jose State University. The context of the study centered on participants’ willingness to fly on board completely autonomous (pilotless) commercial airliners. Participants were presented with one of two scenarios: a traditional configuration with two human pilots in control, and a flight piloted by an entirely autonomous autopilot system, with no human pilots in the cockpit. Not surprisingly, respondents in every sample were less inclined to fly without a human pilot. However, the differences in demographics and effect sizes between samples were significant; Cohen’s d effect sizes for the four samples were 2.10, 1.54, 1.51, and 1.76 for MTurk, FIT, ERAU and SJSU, respectively. These results indicate that, while a convenience sample may, to some degree, reflect the direction of the results one should expect to find in the general population, one such sample is not entirely sufficient to make claims of generalizability; rather, replications are necessary in order to produce a more precise population effect size.



To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.