Clarification: Please do Part A before class on Wed 10/20 & Part B before the end of the week by 10/25.

Part A: The Replicability Crisis

Read at least ONE of the following papers for discussion: Add an entry in your notebook reflecting your thoughts on replication in the context of the paper that you read. Skim through any of the others for discussion

Nosek, B. A., & Errington, T. M. (2020). What is replication?. PLoS biology, 18(3). https://doi.org/10.1371/journal.pbio.3000691
Omar S. Gómez, Natalia Juristo, and Sira Vegas. 2010. Replications types in experimental disciplines. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '10). Association for Computing Machinery, New York, NY, USA, Article 3, 1–10. DOI: https://doi.org/10.1145/1852786.1852790
Hendrick, C. (1990). Replications, strict replications, and conceptual replications: Are they important? Journal of Social Behavior & Personality, 5(4), 41–49. Replications, Strict Replications, and Conceptual Replications: Are They Important? - ProQuest
Kasper Hornbæk, Søren S. Sander, Javier Andrés Bargas-Avila, and Jakob Grue Simonsen. 2014. Is once enough? on the extent and content of replications in human-computer interaction. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI '14). Association for Computing Machinery, New York, NY, USA, 3523–3532. DOI: https://doi.org/10.1145/2556288.2557004
Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). Association for Computing Machinery, New York, NY, USA, 111–120. DOI: https://doi.org/10.1145/1357054.1357074
Jones, K. S., Derby, P. L., & Schmidlin, E. A. (2010). An investigation of the prevalence of replication research in human factors. Human factors, 52(5), 586–595. https://doi.org/10.1177/0018720810384394.
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584-585. https://www.science.org/doi/full/10.1126/science.aal3618
Anderson, S. F., & Maxwell, S. E. (2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305-324. https://www.tandfonline.com/doi/pdf/10.1080/00273171.2017.1289361?needAccess=true
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM, 63(8), 70-79. https://dl.acm.org/doi/fullHtml/10.1145/3360311

Other interesting but optional reads:

Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70(3, Pt.1), 151–159. https://doi.org/10.1037/h0026141
Great resources for other references:
Christian Greiffenhagen, Stuart Reeves. 2013. Is replication important for HCI? In workshop on replication in HCI (RepliCHI), SIGCHI Conference on Human Factors in Computing Systems (CHI), Paris, France. (PDF) Is Replication important for HCI? (researchgate.net)
Max L. Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby. 2011. RepliCHI - CHI should be replicating and validating results more: discuss. In CHI '11 Extended Abstracts on Human Factors in Computing Systems (CHI EA '11). Association for Computing Machinery, New York, NY, USA, 463–466. DOI: https://doi.org/10.1145/1979742.1979491

Part B: Revisit Your Papers: (for your notebook entry)

Reflecting on your paper that you are working on this semester, or it can be another project of yours....

- Design a quantitative evaluation for your work (this may not completely match your current paper, ie if you are writing a lit review or if using qualitative methods) but think of it as an exercise to apply what we have learned the past 3 weeks.

- Given your quantitative evaluation, what are your research questions, independent & dependent variables, hypotheses, and study design? Reflect on your choices.

-Also make some choices as to how you might analyze your data and present your data. If you have data already, try exploring that data through frequencies, stem&leaf plot, or box & whiskers plot across your groups.

- Find (using IEEE Xplore or ACM portal or Google Scholar) at least 2 papers that have similar study designs and compare/contrast your design choices with what the researchers completed in the papers. Revise your design accordingly- what changes, if any, did you make and why?