Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Richard Klein, Michelangelo Vianello, Fred Hasselman, Bryon Adams, Reginald Adams, Sinan Alper, Mark Aveyard, Jordan Axt, Mayowa T. Babalola, Stepan Bahnik, Rishtee Batra, Mihaly Berkics, Michael Bernstein, Daniel Berry, Olga Bialobrzeska, Evans Binan, Konrad Bocian, Mark Brandt, Robert Busching, Anna Redei & 170 others Huajian Cai, Fanny Cambier, Katarzyna Cantarero, Cheryl Carmichael, Francisco Ceric, Jesse Chandler, Jen-Ho Chang, Armand Chatard, Eva Chen, Winnee Cheong, David C. Cicero, Sharon Coen, Jennifer Coleman, Brian Collisson, Morgan Conway, Katherine Corker, Paul Curran, Fiery Cushman, Zubairu Dagona, Ilker Dalgar, Anna Rosa, William Davis, Maaike de Bruijn, Leander De Schutter, Thierry Devos, Marieke de Vries, Canay Dogulu, Nerisa Dozo, Kristin Dukes, Yarrow Dunham, Kevin Durrheim, Charles Ebersole, John Edlund, Anja Eller, Alexander English, Carolyn Finck, Natalia Frankowska, Miguel-Angel Freyre, Mike Friedman, Elisa Galliani, Joshua Gandi, Tanuka Ghoshal, Steffen Giessner, Tripat Gill, Timo Gnambs, Angel Gomez, Roberto Gonzalez, Jesse Graham, Jon Grahe, Ivan Grahek, Eva Green, Kakul Hai, Matthew Haigh, Elizabeth Haines, Michael Hall, Marie Heffernan, Joshua Hicks, Petr Houdek, Jeffrey Huntsinger, Ho Phi Huyhn, Hans IJzerman, Yoel Inbar, Ase Innes-Ker, William Jimenez-Leal, Melissa-Sue John, Jennifer Joy-Gaba, Roza Kamiloglu, Heather Kappes, Serdar Karabati, Haruna Karick, Victor Keller, Anna Kende, Nicolas Kervyn, Goran Knezevic, Carrie Kovacs, Lacy Krueger, German Kurapov, Jamie Kurtz, Daniel Lakens, Ljiljana Lazarevic, Carmel Levitan, Neil.A. Lewis, Samuel Lins, Nikolette Lipsey, Joy Losee, Esther Maassen, Angela Maitner, Winfrida Malingumu, Robyn Mallett, Satia Marotta, Janko Mededovic, Fernando Pacheco, Taciano Milfont, Wendy Morris, Sean Murphy, Andriy Myachykov, Nick Neave, Koen Neijenhuijs, Anthony Nelson, Felix Neto, Austin Nichols, Aaron Ocampo, Susan O'Donnell, Haruka Oikawa, Masanori Oikawa, Elsie Ong, Gabor Orosz, Malgorzata Osowiecka, Grant Packard, Rolando Perez-Sanchez, Boban Petrovic, Ronaldo Pilati, Brad Pinter, Lysandra Podesta, Gabrielle Pogge, Monique Pollmann, Abraham Rutchick, Patricio Saavedra, Alexander Saeri, Erika Salomon, Kathleen Schmidt, Felix Schonbrodt, Maciej Sekerdej, Davd Sirlopu, Jeanine Skorinko, Michael Smith, Vanessa Smith-Castro, Karin Smolders, Agata Sobkow, Walter Sowden, Philipp Spachtholtz, Manini Srivastava, Troy Steiner, Jeroen Stouten, Chris N. H. Street, Oskar Sundfelt, Stephanie Szeto, Ewa Szumowska, Andrew Tang, Norbert Tanzer, Morgan Tear, Jordan Theriault, Manuela Thomae, David Torres, Jakub Traczyk, Joshua Tybur, Adrienn Ujhelyi, Robbie van Aert, Marcel van Assen, Marije van der Hulst, Paul van Lange, Anna van 't Veer, Alejandro Echeverria, Leigh Vaughn, Alexandra Vazquez, Luis Vega, Catherine Verniers, Mark Verschoor, Ingrid Voermans, Marek Vranka, Cheryl Welch, Aaron Wichman, Lisa Williams, Michael Wood, Julie Woodzicka, Marta Wronska, Liane Young, John Zelenski, Zeng Zhijia, Brian Nosek

Research output: Contribution to journalArticle

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (​p​ < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (​p​ < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s ​d​ effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.
LanguageEnglish
Pages443-490
Number of pages48
JournalAdvances in Methods and Practices in Psychological Science
Volume1
Issue number4
DOIs
Publication statusPublished - 1 Dec 2018

Cite this

Klein, Richard ; Vianello, Michelangelo ; Hasselman, Fred ; Adams, Bryon ; Adams, Reginald ; Alper, Sinan ; Aveyard, Mark ; Axt, Jordan ; Babalola, Mayowa T. ; Bahnik, Stepan ; Batra, Rishtee ; Berkics, Mihaly ; Bernstein, Michael ; Berry, Daniel ; Bialobrzeska, Olga ; Binan, Evans ; Bocian, Konrad ; Brandt, Mark ; Busching, Robert ; Redei, Anna ; Cai, Huajian ; Cambier, Fanny ; Cantarero, Katarzyna ; Carmichael, Cheryl ; Ceric, Francisco ; Chandler, Jesse ; Chang, Jen-Ho ; Chatard, Armand ; Chen, Eva ; Cheong, Winnee ; Cicero, David C. ; Coen, Sharon ; Coleman, Jennifer ; Collisson, Brian ; Conway, Morgan ; Corker, Katherine ; Curran, Paul ; Cushman, Fiery ; Dagona, Zubairu ; Dalgar, Ilker ; Rosa, Anna ; Davis, William ; de Bruijn, Maaike ; De Schutter, Leander ; Devos, Thierry ; de Vries, Marieke ; Dogulu, Canay ; Dozo, Nerisa ; Dukes, Kristin ; Dunham, Yarrow ; Durrheim, Kevin ; Ebersole, Charles ; Edlund, John ; Eller, Anja ; English, Alexander ; Finck, Carolyn ; Frankowska, Natalia ; Freyre, Miguel-Angel ; Friedman, Mike ; Galliani, Elisa ; Gandi, Joshua ; Ghoshal, Tanuka ; Giessner, Steffen ; Gill, Tripat ; Gnambs, Timo ; Gomez, Angel ; Gonzalez, Roberto ; Graham, Jesse ; Grahe, Jon ; Grahek, Ivan ; Green, Eva ; Hai, Kakul ; Haigh, Matthew ; Haines, Elizabeth ; Hall, Michael ; Heffernan, Marie ; Hicks, Joshua ; Houdek, Petr ; Huntsinger, Jeffrey ; Huyhn, Ho Phi ; IJzerman, Hans ; Inbar, Yoel ; Innes-Ker, Ase ; Jimenez-Leal, William ; John, Melissa-Sue ; Joy-Gaba, Jennifer ; Kamiloglu, Roza ; Kappes, Heather ; Karabati, Serdar ; Karick, Haruna ; Keller, Victor ; Kende, Anna ; Kervyn, Nicolas ; Knezevic, Goran ; Kovacs, Carrie ; Krueger, Lacy ; Kurapov, German ; Kurtz, Jamie ; Lakens, Daniel ; Lazarevic, Ljiljana ; Levitan, Carmel ; Lewis, Neil.A. ; Lins, Samuel ; Lipsey, Nikolette ; Losee, Joy ; Maassen, Esther ; Maitner, Angela ; Malingumu, Winfrida ; Mallett, Robyn ; Marotta, Satia ; Mededovic, Janko ; Pacheco, Fernando ; Milfont, Taciano ; Morris, Wendy ; Murphy, Sean ; Myachykov, Andriy ; Neave, Nick ; Neijenhuijs, Koen ; Nelson, Anthony ; Neto, Felix ; Nichols, Austin ; Ocampo, Aaron ; O'Donnell, Susan ; Oikawa, Haruka ; Oikawa, Masanori ; Ong, Elsie ; Orosz, Gabor ; Osowiecka, Malgorzata ; Packard, Grant ; Perez-Sanchez, Rolando ; Petrovic, Boban ; Pilati, Ronaldo ; Pinter, Brad ; Podesta, Lysandra ; Pogge, Gabrielle ; Pollmann, Monique ; Rutchick, Abraham ; Saavedra, Patricio ; Saeri, Alexander ; Salomon, Erika ; Schmidt, Kathleen ; Schonbrodt, Felix ; Sekerdej, Maciej ; Sirlopu, Davd ; Skorinko, Jeanine ; Smith, Michael ; Smith-Castro, Vanessa ; Smolders, Karin ; Sobkow, Agata ; Sowden, Walter ; Spachtholtz, Philipp ; Srivastava, Manini ; Steiner, Troy ; Stouten, Jeroen ; Street, Chris N. H. ; Sundfelt, Oskar ; Szeto, Stephanie ; Szumowska, Ewa ; Tang, Andrew ; Tanzer, Norbert ; Tear, Morgan ; Theriault, Jordan ; Thomae, Manuela ; Torres, David ; Traczyk, Jakub ; Tybur, Joshua ; Ujhelyi, Adrienn ; van Aert, Robbie ; van Assen, Marcel ; van der Hulst, Marije ; van Lange, Paul ; van 't Veer, Anna ; Echeverria, Alejandro ; Vaughn, Leigh ; Vazquez, Alexandra ; Vega, Luis ; Verniers, Catherine ; Verschoor, Mark ; Voermans, Ingrid ; Vranka, Marek ; Welch, Cheryl ; Wichman, Aaron ; Williams, Lisa ; Wood, Michael ; Woodzicka, Julie ; Wronska, Marta ; Young, Liane ; Zelenski, John ; Zhijia, Zeng ; Nosek, Brian. / Many Labs 2 : Investigating Variation in Replicability Across Samples and Settings. In: Advances in Methods and Practices in Psychological Science. 2018 ; Vol. 1, No. 4. pp. 443-490.
@article{a796ed0d910d43229889d82163751ba8,
title = "Many Labs 2: Investigating Variation in Replicability Across Samples and Settings",
abstract = "We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (​p​ < .05), fifteen (54{\%}) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (​p​ < .0001), fourteen (50{\%}) provide such evidence reflecting the extremely high powered design. Seven (25{\%}) of the replications had effect sizes larger than the original finding and 21 (75{\%}) had effect sizes smaller than the original finding. The median comparable Cohen’s ​d​ effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57{\%}) had small effect sizes (< .20) and 9 (32{\%}) were in the opposite direction from the original finding. Across settings, 11 (39{\%}) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.",
keywords = "Replication, Many Labs, Context, Psychology, Culture, Social psychology, Cognitive psychology, Individual differences, Sampling effects, Situational effects, Meta-analysis, Registered Report, Open data, Open materials, Preregistered",
author = "Richard Klein and Michelangelo Vianello and Fred Hasselman and Bryon Adams and Reginald Adams and Sinan Alper and Mark Aveyard and Jordan Axt and Babalola, {Mayowa T.} and Stepan Bahnik and Rishtee Batra and Mihaly Berkics and Michael Bernstein and Daniel Berry and Olga Bialobrzeska and Evans Binan and Konrad Bocian and Mark Brandt and Robert Busching and Anna Redei and Huajian Cai and Fanny Cambier and Katarzyna Cantarero and Cheryl Carmichael and Francisco Ceric and Jesse Chandler and Jen-Ho Chang and Armand Chatard and Eva Chen and Winnee Cheong and Cicero, {David C.} and Sharon Coen and Jennifer Coleman and Brian Collisson and Morgan Conway and Katherine Corker and Paul Curran and Fiery Cushman and Zubairu Dagona and Ilker Dalgar and Anna Rosa and William Davis and {de Bruijn}, Maaike and {De Schutter}, Leander and Thierry Devos and {de Vries}, Marieke and Canay Dogulu and Nerisa Dozo and Kristin Dukes and Yarrow Dunham and Kevin Durrheim and Charles Ebersole and John Edlund and Anja Eller and Alexander English and Carolyn Finck and Natalia Frankowska and Miguel-Angel Freyre and Mike Friedman and Elisa Galliani and Joshua Gandi and Tanuka Ghoshal and Steffen Giessner and Tripat Gill and Timo Gnambs and Angel Gomez and Roberto Gonzalez and Jesse Graham and Jon Grahe and Ivan Grahek and Eva Green and Kakul Hai and Matthew Haigh and Elizabeth Haines and Michael Hall and Marie Heffernan and Joshua Hicks and Petr Houdek and Jeffrey Huntsinger and Huyhn, {Ho Phi} and Hans IJzerman and Yoel Inbar and Ase Innes-Ker and William Jimenez-Leal and Melissa-Sue John and Jennifer Joy-Gaba and Roza Kamiloglu and Heather Kappes and Serdar Karabati and Haruna Karick and Victor Keller and Anna Kende and Nicolas Kervyn and Goran Knezevic and Carrie Kovacs and Lacy Krueger and German Kurapov and Jamie Kurtz and Daniel Lakens and Ljiljana Lazarevic and Carmel Levitan and Neil.A. Lewis and Samuel Lins and Nikolette Lipsey and Joy Losee and Esther Maassen and Angela Maitner and Winfrida Malingumu and Robyn Mallett and Satia Marotta and Janko Mededovic and Fernando Pacheco and Taciano Milfont and Wendy Morris and Sean Murphy and Andriy Myachykov and Nick Neave and Koen Neijenhuijs and Anthony Nelson and Felix Neto and Austin Nichols and Aaron Ocampo and Susan O'Donnell and Haruka Oikawa and Masanori Oikawa and Elsie Ong and Gabor Orosz and Malgorzata Osowiecka and Grant Packard and Rolando Perez-Sanchez and Boban Petrovic and Ronaldo Pilati and Brad Pinter and Lysandra Podesta and Gabrielle Pogge and Monique Pollmann and Abraham Rutchick and Patricio Saavedra and Alexander Saeri and Erika Salomon and Kathleen Schmidt and Felix Schonbrodt and Maciej Sekerdej and Davd Sirlopu and Jeanine Skorinko and Michael Smith and Vanessa Smith-Castro and Karin Smolders and Agata Sobkow and Walter Sowden and Philipp Spachtholtz and Manini Srivastava and Troy Steiner and Jeroen Stouten and Street, {Chris N. H.} and Oskar Sundfelt and Stephanie Szeto and Ewa Szumowska and Andrew Tang and Norbert Tanzer and Morgan Tear and Jordan Theriault and Manuela Thomae and David Torres and Jakub Traczyk and Joshua Tybur and Adrienn Ujhelyi and {van Aert}, Robbie and {van Assen}, Marcel and {van der Hulst}, Marije and {van Lange}, Paul and {van 't Veer}, Anna and Alejandro Echeverria and Leigh Vaughn and Alexandra Vazquez and Luis Vega and Catherine Verniers and Mark Verschoor and Ingrid Voermans and Marek Vranka and Cheryl Welch and Aaron Wichman and Lisa Williams and Michael Wood and Julie Woodzicka and Marta Wronska and Liane Young and John Zelenski and Zeng Zhijia and Brian Nosek",
year = "2018",
month = "12",
day = "1",
doi = "10.1177/2515245918810225",
language = "English",
volume = "1",
pages = "443--490",
journal = "Advances in Methods and Practices in Psychological Science",
number = "4",

}

Klein, R, Vianello, M, Hasselman, F, Adams, B, Adams, R, Alper, S, Aveyard, M, Axt, J, Babalola, MT, Bahnik, S, Batra, R, Berkics, M, Bernstein, M, Berry, D, Bialobrzeska, O, Binan, E, Bocian, K, Brandt, M, Busching, R, Redei, A, Cai, H, Cambier, F, Cantarero, K, Carmichael, C, Ceric, F, Chandler, J, Chang, J-H, Chatard, A, Chen, E, Cheong, W, Cicero, DC, Coen, S, Coleman, J, Collisson, B, Conway, M, Corker, K, Curran, P, Cushman, F, Dagona, Z, Dalgar, I, Rosa, A, Davis, W, de Bruijn, M, De Schutter, L, Devos, T, de Vries, M, Dogulu, C, Dozo, N, Dukes, K, Dunham, Y, Durrheim, K, Ebersole, C, Edlund, J, Eller, A, English, A, Finck, C, Frankowska, N, Freyre, M-A, Friedman, M, Galliani, E, Gandi, J, Ghoshal, T, Giessner, S, Gill, T, Gnambs, T, Gomez, A, Gonzalez, R, Graham, J, Grahe, J, Grahek, I, Green, E, Hai, K, Haigh, M, Haines, E, Hall, M, Heffernan, M, Hicks, J, Houdek, P, Huntsinger, J, Huyhn, HP, IJzerman, H, Inbar, Y, Innes-Ker, A, Jimenez-Leal, W, John, M-S, Joy-Gaba, J, Kamiloglu, R, Kappes, H, Karabati, S, Karick, H, Keller, V, Kende, A, Kervyn, N, Knezevic, G, Kovacs, C, Krueger, L, Kurapov, G, Kurtz, J, Lakens, D, Lazarevic, L, Levitan, C, Lewis, NA, Lins, S, Lipsey, N, Losee, J, Maassen, E, Maitner, A, Malingumu, W, Mallett, R, Marotta, S, Mededovic, J, Pacheco, F, Milfont, T, Morris, W, Murphy, S, Myachykov, A, Neave, N, Neijenhuijs, K, Nelson, A, Neto, F, Nichols, A, Ocampo, A, O'Donnell, S, Oikawa, H, Oikawa, M, Ong, E, Orosz, G, Osowiecka, M, Packard, G, Perez-Sanchez, R, Petrovic, B, Pilati, R, Pinter, B, Podesta, L, Pogge, G, Pollmann, M, Rutchick, A, Saavedra, P, Saeri, A, Salomon, E, Schmidt, K, Schonbrodt, F, Sekerdej, M, Sirlopu, D, Skorinko, J, Smith, M, Smith-Castro, V, Smolders, K, Sobkow, A, Sowden, W, Spachtholtz, P, Srivastava, M, Steiner, T, Stouten, J, Street, CNH, Sundfelt, O, Szeto, S, Szumowska, E, Tang, A, Tanzer, N, Tear, M, Theriault, J, Thomae, M, Torres, D, Traczyk, J, Tybur, J, Ujhelyi, A, van Aert, R, van Assen, M, van der Hulst, M, van Lange, P, van 't Veer, A, Echeverria, A, Vaughn, L, Vazquez, A, Vega, L, Verniers, C, Verschoor, M, Voermans, I, Vranka, M, Welch, C, Wichman, A, Williams, L, Wood, M, Woodzicka, J, Wronska, M, Young, L, Zelenski, J, Zhijia, Z & Nosek, B 2018, 'Many Labs 2: Investigating Variation in Replicability Across Samples and Settings', Advances in Methods and Practices in Psychological Science, vol. 1, no. 4, pp. 443-490. https://doi.org/10.1177/2515245918810225

Many Labs 2 : Investigating Variation in Replicability Across Samples and Settings. / Klein, Richard; Vianello, Michelangelo; Hasselman, Fred; Adams, Bryon; Adams, Reginald; Alper, Sinan; Aveyard, Mark; Axt, Jordan; Babalola, Mayowa T.; Bahnik, Stepan; Batra, Rishtee; Berkics, Mihaly; Bernstein, Michael; Berry, Daniel; Bialobrzeska, Olga; Binan, Evans; Bocian, Konrad; Brandt, Mark; Busching, Robert; Redei, Anna; Cai, Huajian; Cambier, Fanny; Cantarero, Katarzyna; Carmichael, Cheryl; Ceric, Francisco; Chandler, Jesse; Chang, Jen-Ho; Chatard, Armand; Chen, Eva; Cheong, Winnee; Cicero, David C.; Coen, Sharon; Coleman, Jennifer; Collisson, Brian; Conway, Morgan; Corker, Katherine; Curran, Paul; Cushman, Fiery; Dagona, Zubairu; Dalgar, Ilker; Rosa, Anna; Davis, William; de Bruijn, Maaike; De Schutter, Leander; Devos, Thierry; de Vries, Marieke; Dogulu, Canay; Dozo, Nerisa; Dukes, Kristin; Dunham, Yarrow; Durrheim, Kevin; Ebersole, Charles; Edlund, John; Eller, Anja; English, Alexander; Finck, Carolyn; Frankowska, Natalia; Freyre, Miguel-Angel; Friedman, Mike; Galliani, Elisa; Gandi, Joshua; Ghoshal, Tanuka; Giessner, Steffen; Gill, Tripat; Gnambs, Timo; Gomez, Angel; Gonzalez, Roberto; Graham, Jesse; Grahe, Jon; Grahek, Ivan; Green, Eva; Hai, Kakul; Haigh, Matthew; Haines, Elizabeth; Hall, Michael; Heffernan, Marie; Hicks, Joshua; Houdek, Petr; Huntsinger, Jeffrey; Huyhn, Ho Phi; IJzerman, Hans; Inbar, Yoel; Innes-Ker, Ase; Jimenez-Leal, William; John, Melissa-Sue; Joy-Gaba, Jennifer; Kamiloglu, Roza; Kappes, Heather; Karabati, Serdar; Karick, Haruna; Keller, Victor; Kende, Anna; Kervyn, Nicolas; Knezevic, Goran; Kovacs, Carrie; Krueger, Lacy; Kurapov, German; Kurtz, Jamie; Lakens, Daniel; Lazarevic, Ljiljana; Levitan, Carmel; Lewis, Neil.A.; Lins, Samuel; Lipsey, Nikolette; Losee, Joy; Maassen, Esther; Maitner, Angela; Malingumu, Winfrida; Mallett, Robyn; Marotta, Satia; Mededovic, Janko; Pacheco, Fernando; Milfont, Taciano; Morris, Wendy; Murphy, Sean; Myachykov, Andriy; Neave, Nick; Neijenhuijs, Koen; Nelson, Anthony; Neto, Felix; Nichols, Austin; Ocampo, Aaron; O'Donnell, Susan; Oikawa, Haruka; Oikawa, Masanori; Ong, Elsie; Orosz, Gabor; Osowiecka, Malgorzata; Packard, Grant; Perez-Sanchez, Rolando; Petrovic, Boban; Pilati, Ronaldo; Pinter, Brad; Podesta, Lysandra; Pogge, Gabrielle; Pollmann, Monique; Rutchick, Abraham; Saavedra, Patricio; Saeri, Alexander; Salomon, Erika; Schmidt, Kathleen; Schonbrodt, Felix; Sekerdej, Maciej; Sirlopu, Davd; Skorinko, Jeanine; Smith, Michael; Smith-Castro, Vanessa; Smolders, Karin; Sobkow, Agata; Sowden, Walter; Spachtholtz, Philipp; Srivastava, Manini; Steiner, Troy; Stouten, Jeroen; Street, Chris N. H.; Sundfelt, Oskar; Szeto, Stephanie; Szumowska, Ewa; Tang, Andrew; Tanzer, Norbert; Tear, Morgan; Theriault, Jordan; Thomae, Manuela; Torres, David; Traczyk, Jakub; Tybur, Joshua; Ujhelyi, Adrienn; van Aert, Robbie; van Assen, Marcel; van der Hulst, Marije; van Lange, Paul; van 't Veer, Anna; Echeverria, Alejandro; Vaughn, Leigh; Vazquez, Alexandra; Vega, Luis; Verniers, Catherine; Verschoor, Mark; Voermans, Ingrid; Vranka, Marek; Welch, Cheryl; Wichman, Aaron; Williams, Lisa; Wood, Michael; Woodzicka, Julie; Wronska, Marta; Young, Liane; Zelenski, John; Zhijia, Zeng; Nosek, Brian.

In: Advances in Methods and Practices in Psychological Science, Vol. 1, No. 4, 01.12.2018, p. 443-490.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Many Labs 2

T2 - Advances in Methods and Practices in Psychological Science

AU - Klein, Richard

AU - Vianello, Michelangelo

AU - Hasselman, Fred

AU - Adams, Bryon

AU - Adams, Reginald

AU - Alper, Sinan

AU - Aveyard, Mark

AU - Axt, Jordan

AU - Babalola, Mayowa T.

AU - Bahnik, Stepan

AU - Batra, Rishtee

AU - Berkics, Mihaly

AU - Bernstein, Michael

AU - Berry, Daniel

AU - Bialobrzeska, Olga

AU - Binan, Evans

AU - Bocian, Konrad

AU - Brandt, Mark

AU - Busching, Robert

AU - Redei, Anna

AU - Cai, Huajian

AU - Cambier, Fanny

AU - Cantarero, Katarzyna

AU - Carmichael, Cheryl

AU - Ceric, Francisco

AU - Chandler, Jesse

AU - Chang, Jen-Ho

AU - Chatard, Armand

AU - Chen, Eva

AU - Cheong, Winnee

AU - Cicero, David C.

AU - Coen, Sharon

AU - Coleman, Jennifer

AU - Collisson, Brian

AU - Conway, Morgan

AU - Corker, Katherine

AU - Curran, Paul

AU - Cushman, Fiery

AU - Dagona, Zubairu

AU - Dalgar, Ilker

AU - Rosa, Anna

AU - Davis, William

AU - de Bruijn, Maaike

AU - De Schutter, Leander

AU - Devos, Thierry

AU - de Vries, Marieke

AU - Dogulu, Canay

AU - Dozo, Nerisa

AU - Dukes, Kristin

AU - Dunham, Yarrow

AU - Durrheim, Kevin

AU - Ebersole, Charles

AU - Edlund, John

AU - Eller, Anja

AU - English, Alexander

AU - Finck, Carolyn

AU - Frankowska, Natalia

AU - Freyre, Miguel-Angel

AU - Friedman, Mike

AU - Galliani, Elisa

AU - Gandi, Joshua

AU - Ghoshal, Tanuka

AU - Giessner, Steffen

AU - Gill, Tripat

AU - Gnambs, Timo

AU - Gomez, Angel

AU - Gonzalez, Roberto

AU - Graham, Jesse

AU - Grahe, Jon

AU - Grahek, Ivan

AU - Green, Eva

AU - Hai, Kakul

AU - Haigh, Matthew

AU - Haines, Elizabeth

AU - Hall, Michael

AU - Heffernan, Marie

AU - Hicks, Joshua

AU - Houdek, Petr

AU - Huntsinger, Jeffrey

AU - Huyhn, Ho Phi

AU - IJzerman, Hans

AU - Inbar, Yoel

AU - Innes-Ker, Ase

AU - Jimenez-Leal, William

AU - John, Melissa-Sue

AU - Joy-Gaba, Jennifer

AU - Kamiloglu, Roza

AU - Kappes, Heather

AU - Karabati, Serdar

AU - Karick, Haruna

AU - Keller, Victor

AU - Kende, Anna

AU - Kervyn, Nicolas

AU - Knezevic, Goran

AU - Kovacs, Carrie

AU - Krueger, Lacy

AU - Kurapov, German

AU - Kurtz, Jamie

AU - Lakens, Daniel

AU - Lazarevic, Ljiljana

AU - Levitan, Carmel

AU - Lewis, Neil.A.

AU - Lins, Samuel

AU - Lipsey, Nikolette

AU - Losee, Joy

AU - Maassen, Esther

AU - Maitner, Angela

AU - Malingumu, Winfrida

AU - Mallett, Robyn

AU - Marotta, Satia

AU - Mededovic, Janko

AU - Pacheco, Fernando

AU - Milfont, Taciano

AU - Morris, Wendy

AU - Murphy, Sean

AU - Myachykov, Andriy

AU - Neave, Nick

AU - Neijenhuijs, Koen

AU - Nelson, Anthony

AU - Neto, Felix

AU - Nichols, Austin

AU - Ocampo, Aaron

AU - O'Donnell, Susan

AU - Oikawa, Haruka

AU - Oikawa, Masanori

AU - Ong, Elsie

AU - Orosz, Gabor

AU - Osowiecka, Malgorzata

AU - Packard, Grant

AU - Perez-Sanchez, Rolando

AU - Petrovic, Boban

AU - Pilati, Ronaldo

AU - Pinter, Brad

AU - Podesta, Lysandra

AU - Pogge, Gabrielle

AU - Pollmann, Monique

AU - Rutchick, Abraham

AU - Saavedra, Patricio

AU - Saeri, Alexander

AU - Salomon, Erika

AU - Schmidt, Kathleen

AU - Schonbrodt, Felix

AU - Sekerdej, Maciej

AU - Sirlopu, Davd

AU - Skorinko, Jeanine

AU - Smith, Michael

AU - Smith-Castro, Vanessa

AU - Smolders, Karin

AU - Sobkow, Agata

AU - Sowden, Walter

AU - Spachtholtz, Philipp

AU - Srivastava, Manini

AU - Steiner, Troy

AU - Stouten, Jeroen

AU - Street, Chris N. H.

AU - Sundfelt, Oskar

AU - Szeto, Stephanie

AU - Szumowska, Ewa

AU - Tang, Andrew

AU - Tanzer, Norbert

AU - Tear, Morgan

AU - Theriault, Jordan

AU - Thomae, Manuela

AU - Torres, David

AU - Traczyk, Jakub

AU - Tybur, Joshua

AU - Ujhelyi, Adrienn

AU - van Aert, Robbie

AU - van Assen, Marcel

AU - van der Hulst, Marije

AU - van Lange, Paul

AU - van 't Veer, Anna

AU - Echeverria, Alejandro

AU - Vaughn, Leigh

AU - Vazquez, Alexandra

AU - Vega, Luis

AU - Verniers, Catherine

AU - Verschoor, Mark

AU - Voermans, Ingrid

AU - Vranka, Marek

AU - Welch, Cheryl

AU - Wichman, Aaron

AU - Williams, Lisa

AU - Wood, Michael

AU - Woodzicka, Julie

AU - Wronska, Marta

AU - Young, Liane

AU - Zelenski, John

AU - Zhijia, Zeng

AU - Nosek, Brian

PY - 2018/12/1

Y1 - 2018/12/1

N2 - We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (​p​ < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (​p​ < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s ​d​ effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

AB - We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (​p​ < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (​p​ < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s ​d​ effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

KW - Replication

KW - Many Labs

KW - Context

KW - Psychology

KW - Culture

KW - Social psychology

KW - Cognitive psychology

KW - Individual differences

KW - Sampling effects

KW - Situational effects

KW - Meta-analysis

KW - Registered Report

KW - Open data

KW - Open materials

KW - Preregistered

U2 - 10.1177/2515245918810225

DO - 10.1177/2515245918810225

M3 - Article

VL - 1

SP - 443

EP - 490

JO - Advances in Methods and Practices in Psychological Science

JF - Advances in Methods and Practices in Psychological Science

IS - 4

ER -