Articles
-
October 23,听2024
Relentlessly headed for another defeat?
After having her defamation claims against the DataColada and Harvard defendants dismissed rather decisively (and very pointedly), Gino amends her 25 million US dollar suit against the University to include Title VII and discrimination claims.听听
October 5,听2024
Don't Stop Believing Though.
Another day another retraction involving some of the usual suspects (Alison Wood Brooks, Juliana Schroeder, Jane L. Risen, Francesca Gino, Adam D. Galinsky, Michael I. Norton, Maurice E. Schweitzer).
See Schroeder鈥檚 on this matter.
September 29,听2024
Scores of papers by Eliezer Masliah, prominent neuroscientist and top NIH official, fall under suspicion
This promises to be massive. Read the that Mu Yang links to.
September 28,听2024
A curious reading of the memorandum with which Judge Joun dismissed Gino鈥檚 defamation claims against the DataColada and Harvard defendants.
A curious reading of Judge Joun's Memorandum at best. I guess there is a reason why Gino turned off comments on her post. 听
听
What Gino omits is this:听
"For the reasons below, the Harvard Defendants鈥 Motion to Dismiss is GRANTED in part and DENIED in part, and the Data Colada Defendants鈥 Motion to Dismiss is GRANTED."听
听
Judge Joun's Memorandum makes for fascinating reading. I am not a lawyer but ... (here is AO鈥檚 summary):听
All defamation claims have been dismissed. Often in damning and no uncertain terms. What's left for the "Harvard Defendants" is the claims around contract violation and procedural irregularities. I doubt they will go anywhere. Gino and her lawyers have made much of the alleged switch in policies from the 2013 Research Integrity Policy to the 2021 Interim Policy and Procedures for Responding to Allegations of Research Conduct, this latter based apparently on federal guidelines. Alas, the 2013 policy gives the Dean considerable leeway in the way s/he deals with such allegations (see p. 6 of the Decision), which includes arguably the creation of such Interim Policy. Time will tell what the good judge makes of it ...听听
September 24,鈥2024
Retracted by Nature Human Behavior: A Study that Was Hailed as a Win for Science Reform听
This retraction had been a long time coming. Bak-Coleman & Devezer asked important questions about it, published now as . On 11 December 2023 the NHB editors alerted their readers that "this paper is subject to criticisms that are being considered by the editors. 鈥" Jessica Hullman, referring to the then circulating Bak-Coleman & Devezer critique, contributed on 27 March 2024
-
" ... one of the questions raised by Bak-Coleman and Devezer about the published version was about their claim that all of the confirmatory analyses they present were preregistered. There was no such preregistration in sight if you checked the provided OSF link. I remarked back in November that even in the best case scenario where the missing preregistration was found, it was still depressing and ironic that a paper whose message is about the value of preregistration could make claims about its own preregistration that it couldn鈥檛 back up at publication time. 听
...听
It seems clear that the dishonesty here was in service of telling a compelling story about something. I鈥檝e seen things like this transpire plenty of times: the goal of getting published leads to attempts to find a good story in whatever results you got. Combined with the appearance of rigor and a good reputation, a researcher can be rewarded for work that on closer inspection involves so much post-hoc interpretation that the preregistration seems mostly irrelevant. It鈥檚 not surprising that the story here ends up being one that we would expect some of the authors to have faith in a priori. 听
...听
What do I care? Why should you? 听
...听
There are many lessons to be drawn here. When someone says all the analyses are preregistered, don鈥檛 just accept them at their word, regardless of their reputation. Another lesson that I think Andrew previously highlighted is that researchers sometimes form alliances with others that may have different views for the sake of impact but this can lead to compromised standards. Big collaborative papers where you can鈥檛 be sure what your co-authors are up to should make all of us nervous. Dishonesty is not worth the citations."听
听
(Jessica Hullman on the Gelman blog, March 27, 2024)On 24 September 2024, in which she identified four reviewers of the Protzko et al. manuscript (Elson, Yarkoni, Lakens, herself). You can sense her disappointment and frustration clearly.
That very same day Stefanie Lee also posted a must-read on The Chronicle of Higher Education that she titled
On 26 September 2024, Andrew Gelman himself posted a long summary of the events titled ""
In conclusion: "The 2023 paper that claimed, 鈥渢his high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries,鈥 was a disaster. The 2024 retraction of the paper makes it less of a disaster. As is often the case, what appears to be bad news is actually the revelation of earlier bad news; it鈥檚 good news that it got reported.
Confusion remains regarding the different purposes of replication, along with the role of procedural interventions such as preregistration that are designed to improve science.
We should all be thankful to Bak-Coleman and Devezer for the work they put into this project. I can see how this can feel frustrating for them: in an ideal world, none of this effort would have been necessary, because the original paper would never have been published!
The tensions within the science reform movement鈥攁s evidenced by the prominent publication of a research article that was originally designed to study a supernatural phenomenon, then was retooled to represent evidence in favor of certain procedural reforms, and finally was shot down by science reformers from the outside鈥攃an be seen as symbolic of, or representative of, a more general tension that is inherent in science. I鈥檓 speaking here of the tension between hypothesizing and criticism, between modeling and model checking, between normal science and scientific revolutions (here鈥檚 on that). I think scientific theories and scientific measurement need to be added to this mix."
On 14 October 2024, Holly Else provides and the questions it prompts (e.g., about the challenges pre-registration faces.)
Meanwhile, NHB has invited the authors to revise and resubmit their study 鈥 how strange is that? A subset of the authors has posted a piece titled on OSF.
July 30,听2024
Heterogeneity 鈥 it鈥檚 a thing. A thing that limits the generalizability of published scientific findings.
A PNAS publication by Holzmeister, Johannesson, Boehm, Dreber, Huber, & Kirchler.
"In conducting empirical research in the social sciences, the results of testing the same hypothesis can vary depending on the population sampled, the study design, and the analysis. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings."
January 15,听2024
What are pre-registrations good for? (Absolutely nothing!?)听
This is a review of some relevant references and results. It has since its publication repeatedly updated, the last time on 25 September 2024.鈥
-
-
To be updated soon.
-
To be updated soon.
-
April 23,听2021
Unconscious thought theory is dead. As dead as can be.
鈥淭he main article that contains evidential value was published in 2016. Based on these results, I argue that 47 of the 48 articles do not contain credible empirical information that supports the claims in these articles. These articles should not be cited as if they contain empirical evidence.鈥
April 21,听2021
A discussion and meta-discussion of statistical听modeling, causal inference, and social science
Andrew Gelman of Columbia has 听that should make you suspicious about the claims of any paper:
Here are some quick reasons to distrust a paper (with examples):
- The claimed effect is implausibly large (beauty and sex ratio, ovulation and voting)
- The results look too good compared to the noise level (various papers criticized by Gregory Francis)
- The paper makes claims that are not addressed by the data at hand (power pose)
- The published numbers just don鈥檛 add up (pizzagate)
- The claim would seem to violate some physical laws (ESP)
- The fitted model makes no sense (air pollution in China)
- Garbled data (gremlins)
- Misleading citation of the literature (sleep dude)
And lots more.
-
January听4,听2020
Warne on Mindsets Research
Growth mindsets can improve academic performance鈥. Savage.
-
December 31, 2019
A Toast To Error Detectors
The tide is .
December 30, 2019
Wansink on What He Learned
鈥淚n 2017-19, about 18 of my research articles were retracted. These , and they also offer some useful next steps to those who want to publish in the social sciences. Two of these steps include 1) Choose a publishable topic, and 2) have a rough mental roadmap of what the finished paper might look. That is, what鈥檚 the positioning, the study, and the possible contribution.鈥
Very strange.
December 26, 2019
Comparing Meta-Analyses (the gold-standard for many) and Multi-Lab Replications
鈥淲e find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15鈥塵eta-replication pairs. These differences are systematic and, on average, .鈥 Wow. There goes the gold-standard.
December 23, 2019
What we can learn from five naturalistic field experiments that failed to shift commuter behavior?
That ? The evidence accumulates (see 听and )
December 11, 2019
Do Elevated Viewpoints Increase Risk Taking?
听Yes, we are as shocked as you are. This (failed) replication was the inaugural one by our friends from , introducing a new feature called .
December 10, 2019
Data Replicada
Two of our heroes at Data Colada 鈥 they of the 2011 鈥 have started a new feature called in which they will report the results of 鈥渋nformative鈥 (i.e., properly powered-up) replication attempts of recent and well-designed studies recently published in two top behavioral marketing journals: the听Journal of Consumer Research听and the听Journal of Marketing Research. All replication attempts (i.e., there will be no file drawer).
November 15,听2019
Do PAPs (Pre-Analysis Plans) work?
鈥淪o, even if improvements in research credibility do not come from every PAP, the growing in these fields.鈥 Dah.
September 1,听2019
The Title Says It All
Null results of oxytocin and vasopressin administration across a range of social cognitive and behavioral paradigms: .
July 21,听2019
Warne On A Big, And In His View Well-Designed UK Mindset Training Study
Clearly he is not impressed:听:
- Reading achievement
- Writing mechanics
- Math achievement
- Self-worth
- Self-efficacy
- Test anxiety
- Self-regulation
June 30,听2019
How Effective is Nudging? Well, It Depends.
The authors report the results of a with 100 primary publications including 317 effects sizes from different research areas. After having sliced and diced their data in various ways, they find for example huge differences for relative effect sizes in environment, finance, health, and privacy on the one hand and energy and policy making on the other. (Take a guess where you find the large effects.) Note that primary publications are neither meta-analyses nor pre-registered multi-lab studies which suggest that all effect sizes reported here are over-estimated. See the .
-
November 19,听2018
Too Harsh An Assessment
on the .听We think it鈥檚 too harsh an assessment. Certainly, this massive replication attempt provides a roadmap for identifying robust effects. Even in psychology. Another good write-up can be found .
November 14,听2018
Good Advice on How to Conduct a Replication Study
Annette L. Brown provides useful and sensible advice .
For example: Don鈥檛 present, post or publish replication results without first sharing them with the original authors. Word.
November 8,听2018
Replicability Ranking of Eminent Social Psychologists
Uli Schimmack, of fame (and also a featured speaker at the听, has computed a replicability ranking of eminent social psychologists. And why not? Given the prevalence of questionable research practices in social psychology in particular, some such ranking seems more useful than citation indices. Some surprises (positive and negative) there but .
November 8,听2018
Pre-Registration is Not For the Birds
In , identifies what he calls 鈥 sensationalistically 鈥 .
It is actually a study that鈥檚 three years old. The authors collected every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012.
Prior to 2000, questionable research practices were fair game. 13 out of 22 studies (59 percent) showed significant benefits. Once pre-registration was required, only 2 out of 21 did. Surprise! Not!
October 9,听2018
James Heathers On The Wansink Story 鈥 How It Began And How It Has Ended, For Now
After the JAMA family of journals , and after , there was a and reflections. One of the most interesting ones, was . James Heathers was one of the four musketeers to investigate selected pieces of Wansink鈥檚 oeuvre. Here James describes how he got involved initially. Shockful of interesting links. A very readable background story about the motives of the people who engineered .
September 12, 2018
, himself a very prolific author, and his co-authors (the equivalent of one paper every 5 days) in any one calendar year between 2000 and 2016, a figure that many would consider implausibly prolific. Lots of interesting details about where these authors live and what disciplines they work in. A fabulous illustration.
September 7, 2018
Talking about Spaceship Earth and the Dinosaurs鈥 Extinction Can Get Heated
A very long but very interesting . Lots of interesting facts about spaceship earth and the apparently five major extinctions it has gone through. The latest one has been cause for a scientific controversy (about whether the last extinction was rapid and caused by an asteroid or a series of collossal volcanic eruptions) that has gone on unabated for decades, with a viciousness against which even current debates in psychology are nothing. Originally published August 2018.
August 27, 2018
Another Day, More Replication Failures
An international team of researchers, mostly from economics and psychology and听known as the Social Sciences Replication Project, published the results of their听. The results were in our view somewhat sobering: The researchers succeeded in only 13 of 21 cases and where they succeeded the effect size was about half. Arguably the most interesting finding was that the prediction market the researchers conducted in parallel predicted amazingly well which studies would fail and which would succeed. Good write-ups on this latter finding can be found听and听
You might want听to play this rather interesting 听yourself.
June 14, 2018
Retracted and Replaced for No Good Reason?
The Washington Post reported on .听Was there a good reason for it?
The lead author on the study told The Washington Post that the causal link is just as strong as the original report. Which poses the interesting question: If the original study was so problematic that the authors chose to withdraw it entirely, could the new one be trusted?
June 7, 2018
Deconstruction and Re-evaluation of (In)famous Experiments
Both, Zimbardo鈥檚 (in)famous and the equally (in)famous have recently been deconstructed and re-evaluated. A good write-up about the developments pertaining to the former can be found ; a good write-up about the developments pertaining to the latter can be found .
May 1, 2018
Thinking Clearly About Correlations and Causations
who was one of our featured speakers at has two excellent recent papers worth a read. The first is and the second illustrates through a study of birth-order effects. Recommended.
April 17, 2018
An Upbeat Mood May Boost Your Paper鈥檚 Publicity
Well, .
April 7, 2018
Four Misconceptions About Statistical Power
Explained by 听 鈥
- Misconception 1:听Statistical power can only be increased with larger sample sizes
- Misconception 2:听You鈥檝e听reached enough statistical power, or your study is underpowered.
- Misconception 3:听The problem with low power is that you鈥檒l miss true effects.
- Misconception 4:听Effect sizes and standard deviations from pilot studies should be used to听calculate sample sizes for larger studies.
A related good read:
March 30, 2018
Keeping Science Honest, One Whistle Blown at a Time
Wherein听whistleblowers听Josefin Sundin听听and听Fredrik Jutfelt on the authors听of a widely publicised听sensationalist 鈥 (Fish prefer microplastics to live prey!) 鈥撎齜ut听fraudulent听research article听published in Science in June 2016 led in their听case ultimately to vindication and retraction of the paper, at听huge听private and professional consequences to them.
They conclude: 鈥淚deally, whistle-blowing should not be necessary. The scientific community must听enforce a culture of honesty. Sometimes that takes courage.鈥
True fact. But it should not have to.
March 28, 2018
How to Publish Statistically Insignificant Results in Economics
MIT economist Alberto Abadie that statistically insignificant results are at least as interesting as significant ones. One of Abadie鈥檚 key points (in a deeply reductive nutshell) is that results are interesting if they change what we believe (or 鈥渦pdate our priors鈥). With most public policy interventions, there is no reason that the expected impact would be zero. So there is no reason that the only finding that should change our beliefs is a non-zero finding. 听is a very readable听write-up about the Abadie paper (less readable)听by the听 at the World Bank.
March 20, 2018
How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science
鈥淲e developed and validated a 1-hr lecture communicating issues surrounding the replication crisis and current recommendations to increase reproducibility. Pre- and post-lecture surveys suggest that the lecture serves as an excellent pedagogical tool. Following the lecture, students trusted psychological studies slightly less but saw greater similarities between psychology and natural science fields. We discuss challenges for instructors taking the initiative to communicate these issues to undergraduates in an evenhanded way.鈥 ().
March 19, 2018
Big Booze, like Big Banks and Other Big Business, is Relentless in Its Pursuit of Profit
(now also featured on our useful links page) that suggests that lead researchers on a $100 million NIH听study of the effects of moderate alcohol consumption had extensive discussions with the alcohol industry prior to securing the sponsorship and related reports. Apparently the NIH鈥檚 standards are slipping.听The alcohol industry has also tried to buy favourable reporting by journalists. Which makes sense: 鈥淎fter all, if you鈥檙e going to invest $100 million in a study, wouldn鈥檛 it make sense to cultivate journalists to help put a nice shine on the results?鈥
March 9, 2018
How to Make Replication The Norm
The enquiring minds of Paul Gertler, Sebastian Galiani, and Mauricio Romero wanted to know.听Focusing on economics, political science, sociology and psychology, in which ready access to raw data and software code are crucial to replication efforts, they surveyed deficiencies in the current system and听propose reforms that can both encourage and reinforce better behaviour 鈥 a system in which authors feel that replication of software code is both probable and fair, and in which less time and effort is required for replication.
Here you can find the World Development Report 2015: .
And is a critical review of it by one of our own.
March 8, 2018
Is There or Is There Not (A Reproducibility Crisis)?
Somewhat surprisingly (given his own contributions over the last ten years or so), Daniele Fanelli suggests in a recent 听that the 鈥渘arrative of crisis鈥 is mistaken, and that 鈥渁 narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.鈥
We have our doubts. So do many others as the 听shows.
February 24, 2018
The New Lancet Study about Antidepressants is Not Exactly News
A new 听about antidepressants got lots of play in the press. One of our favorite curmudgeons, , 听that it tells us very little that we didn鈥檛 already know, and it has a number of limitations: 鈥淭he media reaction to the paper is frankly bananas.鈥
February 20, 2018
Brembs on Prestigious Science Journals: Do They Live up to Their Reputations?
Brembs, well-known to the readers of this site through his , has just published 听that speaks to the issue of the reliability (trustworthiness) of prestigious science journals. He comes to the somewhat distressing conclusion that these journals can鈥檛 be trusted any more than lower-ranked journals. In fact, the evidence seems to adjust that science published in lower level journals is more reliable. You read that right.听听
February 19, 2018
Introducing听
Daniel Simons introduces new journal which is designed to foster discussions of, and advances in, practices, research design, and statistical methods.
February 18, 2018
Gelman on the Replication Crisis and Underpowered Studies: What Have We Learned Since 2004 (or 1984, or 1964)?
The man behind the 听website, Andrew Gelman, takes 听to consider what we have learned from critiques way back when that sound very modern. Apparently a little bit.听
February 10, 2018
RCTs are Not Always the Gold Standard: How We Figured Out that Smoking Causes Cancer
A very good discussion of the limits of RCTs and what it takes to establish causality .
(We have added the source of this article to our Useful Links page because it has lots to offer.)
February 1, 2018
Replication听is Not Enough: The Case for 鈥淭riangulation鈥
on why replication is not enough and why a problem has to be attacked in several ways.
January 31, 2018
More on the Cornell Food and Brand Lab Situation
Wansink鈥檚 former students听. And who can blame them?
Meanwhile鈥 Cornell is still belaboring its internal investigation 鈥渋n compliance with our internal policies and any external regulations that may apply鈥. Some scholars .
January 18, 2018
Replication Studies: A Report from the Royal Netherlands Academy of Arts and Sciences
The Dutch Academy of Arts and Sciences just published 听that was charged to identify, and address, the problems that seem to beset in particular Dutch psychologists (cue Stapel and others). Eric-Jan Wagenmakers was a member of that committee and provides .
January 15, 2018
The Ultimate Reading List for a Graduate Course on Reproducibility and Replicability, For Now
听and 听have compiled, from the hard labors of many other people, an , focusing on readings that 1) identified the reasons for the current crisis, and 2) provide ways to fix the problems.
Subsections include:
- Definitions of reproducibility & replicability
- Houston, we have a problem (and by 鈥渨e鈥 we mean everyone, not just psychologists or social psychologists)
- The problems that plague us have been plaguing us for a very long time
- The problems that plague us: low power
- The problems that plague us: selective publication; bias against the null
- The problems that plague us: procedural overfitting
- The problems that plague us: quality control 听
- NHST, P-values, and the like
- Preregistration
- Power and power analysis
- On Replication
- Open Science
- Complexities in data availability
- Informational value of existing research
- Solutions
January 5, 2018
Progress Assessed
The editor of Science, Jeremy Berg, assesses 听and is hopeful:
鈥淥ver the past year, we have retracted three papers previously published in听Science. The circumstances of these retractions highlight some of the challenges connected to reproducibility policies. In one case, the authors failed to comply with an agreement to post the data underlying their study. Subsequent investigations concluded that one of the authors did not conduct the experiments as described and fabricated data. Here, the lack of compliance with the data-posting policy was associated with a much deeper issue and highlights one of the benefits of policies regarding data transparency. In a second case, some of the authors of a paper requested retraction after they could not reproduce the previously published results. Because all authors of the original paper did not agree with this conclusion, they decided to attempt additional experiments to try to resolve the issues. These reproducibility experiments did not conclusively confirm the original results, and the editors agreed that the paper should be retracted. This case again reveals some of the subtlety associated with reproducibility. In the final case, the authors retracted a paper over extensive and incompletely described variations in image processing. This emphasizes the importance of accurately presented primary data.鈥
January 5, 2018
Should the Bem Feeling-the-Future Article (JPSP 2011) be Retracted?
听has done 听and, based on it, makes a fairly persuasive case for this article (which in a sense involuntarily started the replicability revolution in psychological science) to be retracted.
A very good read that will teach you, by example, more about questionable research practices than pretty much anything else.听That someone like Bem would make the comments attributed to him is hard to believe.
-
December 18, 2017
The Next Stapel?
Nick Brown, one of the researchers who听was behind the questions raised about the Cornell Brand and Food Lab (see and ), finally has raised questions about another matter. After having for two years tried to get answers from听Nicolas听Gu茅guen, of the Universit茅 Bretagne-Sud in France, on numerous papers of that researcher听that had obvious problems, he .听Good questions all in听all we can see.
One also wonders how any respectable journal would publish some of these papers?:
听鈥淎s well as the articles we have blogged about, he has published research on such vital topics as whether women with larger breasts get more听invitations to dance in nightclubs (they do), whether women are more likely to give their phone number to a man if asked while walking near a flower shop (they are), and whether a male bus driver is more likely to let a woman (but not a man) ride the bus for free if she touches him (we鈥檒l let you guess the answer to this one).鈥
Why the next Stapel? .
December 1, 2017
The Health Benefits of Volunteering May Not Be What They Have Been Made Out to Be
Leif Nelson, one of the three musketeers behind Data听Colada has , other than the warm听glow. He concludes:
鈥淚 see three findings, all of which are intriguing to consider, but none of which are particularly persuasive. The journalist, who presumably has been unable to read all of the original sources, is reduced to reporting their claims. The readers, who are even more removed, take the journalist鈥檚 claims at face value: 鈥榠f I volunteer then I will walk around better, lower my blood pressure, and live longer. Sweet.'鈥
November 30, 2017
Failed
Uli Schimmack has produced an听听of the 鈥渆vidence鈥 in听Bargh鈥檚 book Before you know it: The unconscious reasons we do what we do.
Concludes he: 鈥淭here is no clear criterion for inadequate replicability, but Tversky and Kahneman (1971) suggested a minimum of 50%.听 Professors are also used to give students who scored below 50% on a test an F.听 So, I decided to use the grading scheme at my university as a grading scheme for replicability scores.听 So, the overall score for the replicability of studies cited by Bargh to support the ideas in his book is F.鈥
November 29, 2017
Five (Out of a Zillion) Ways to Fix Statistics
Nature asked influential statisticians to recommend .
Relatedly, one of the authors (Gelman) has听听with all that talk about false positives, false negatives, false discoveries, etc.
Update: December 5, 2017
Ed Hagen comments on these five comments and stresses that, before better statistics can be effective, it is imperative to change the incentives for researchers to use them. Specifically, 鈥渃hanging the incentives to reward high quality studies rather than sexy results would have enormously positive effects for science.鈥 He makes the .
November 28, 2017
d=2.44? Too Good to be True?
.. so men might not be less likely 听after all. Which right there restores our faith in mankind.
November 23, 2017
The听Cornell Food and Brand Lab Saga Continues
No, the magic number is not , apparently it is .
November 12, 2017
Changing the Default p-Value? Why Stop There?
In the wake of听recent proposals to (not)听change the p-value threshold,听McShane, Gal, Gelman,听Robert, and Tackett听听entirely, leaving p-values as just one of many pieces of information with no privileged role in scientific publication and decision making.
That鈥檚 of course not a particularly new idea: Gigerenzer, Leamer, Ziliak & McCloskey, and Hubbard have promoted it for decades (something the present authors seem to either not know or prefer not to acknowledge) but it is worth being recalled.
November 10, 2017
The Cuddy Saga Continues
Following developments , , , , , and , Cuddy听now claims it was all a great misunderstanding and that power posing effects听are about felt power (i.e. mere thinking and feeling) rather than subsequent choices. An interesting refocusing of the narrative鈥
Update: December 6, 2017
In , 听Joe Simmons, Leif Nelson, and听Uri听Simonsohn take on this analysis. You will not be surprised that they come to a different .
October 25, 2017
Replicability and Reproducibility Discussion in Economics
The Economic Journal (one of the leading economics journals) has a ( a collection of articles) on the replicability of economic science. Of particular noteworthiness is the article by Ioannidis, Doucouliagos, and Stanley titled .
If you have any illusions about the sorry听state of the economics research, the abstract will abuse you of it:
鈥淲e investigate two critical dimensions of the credibility of empirical economics research: statistical power and bias. We survey 159 empirical economics literatures that draw upon 64,076 estimates of economic parameters reported in more than 6,700 empirical studies. Half of the research areas have nearly 90% of their results under-powered. The median statistical power is 18%, or less. A simple weighted average of those reported results that are adequately powered (power听鈮ヌ80%) reveals that nearly 80% of the reported effects in these empirical economics literatures are exaggerated; typically, by a factor of two and with one-third inflated by a factor of four or more.鈥
Chris Doucouliagos is one of the key speakers at our .
October 25, 2017
Making Replication Mainstream
Behavioural听and Brain Sciences (the journal) has accepted by 听and colleagues on an important issue. Comments are currently being solicited.
Rolf was one of the speakers at our .
October 24, 2017
Criticising a Scientist鈥檚 Work Isn鈥檛 (Sexist) Bullying. It鈥檚 Science.
听. Says she 鈥
鈥淐uddy鈥檚 story is an important story to tell: It is a story of a woman living in a misogynistic society, having to put up with internet bullies 鈥μ. But it is also a story of a woman experiencing completely appropriate scientific criticism of a finding she published. Conflating those issues, and the people delivering the 鈥渁ttacks,鈥 does a disservice to the fight for gender equality, and it does a disservice to science.鈥
October 19, 2017
Did Power-Posing Guru Amy Cuddy Deserve her Public Shaming?
In the wake of The New York Times piece on Amy Cuddy, Daniel Engber 听that she was dragged through the mud听because she is a woman.
October 19, 2017
Wansink鈥檚 Cornell Food and听Brand Lab Saga, Continued
Last month yet another flawed study by听 Wansink鈥檚 lab was retracted (one of about 50 currently questioned, under review, or already retracted for good), . Unfortunately, this corrected study itself apparently needs also correction. How many rounds will it go? We wonder, too.
Relatedly,听 BuzzFeed has provided email excerpts from Wansink鈥檚 attempts to save his reputation; predictably they do not really help his case. You can read about it .
October 18, 2017
Power Poses, Revisited
The New York Times鈥檚 Susan Dominus has written a long piece on . The piece was actually as much about her, as it was about her collaborator (Carney) on the original power poses paper, Simmons and Simonsohn (who critiqued it), and the culture that led to it and which听is currently swept away. A good, if overly sympathetic look at the travails in which Cuddy now finds herself.
Relatedly, Gelman (who previously has commented repeatedly on the power poses controversy) published on the same day a 听which is worth a read, as is the extensive听commentary that follows it (as of October 21 more than 100 pages if you were to print it out).
October 9, 2017
You Always Wanted to do This, So Here is Your Chance
听now has a course on offer on Coursera on
Recommended.
October 8, 2017
(Actually an article from last year which we have become aware of only now)
Lots of eminently citable quotes in there which听capture the essence of what is becoming, or arguably has become, a serious problem: 鈥淎lbert Einstein remarked that 鈥渁n academic career, in which a person is forced to produce scientific writings in great amounts, creates a danger of intellectual superficiality鈥. True fact.
See also: Simine Vazire鈥檚 blog entry below.
October 1, 2017
Can We Really Not Avoid Being Blinded By Shiny Results?
Simine Vazire has some 听for thought.
September 28, 2017
Cornell Food and Brand Lab Scandal: Update
BuzzFeed News has a very useful update on the Wansink food lab story; no words are minced. See .
September 27, 2017
Gelman Has Sympathy for Big-Shot Psychologist That He Skewered Earlier
The beginning of a beautiful new friendship? Find out .
September 26, 2017
James Coyne Doubles Down on His Two Earlier Power Pose Blog Entries,听Taking No Prisoners in Typical Coyne Style
The titles of his two blog entries are, as usually, descriptive. Part 1 is titled听听and Part 2 is titled .
See whether you agree.
July 31, 2017
Changing the Default p-Value?
A group of mostly fairly well known economists, psychologists, and others interested in the replication crisis of the social sciences have proposed .听Much hilarity ensued.
See, for example, ,听听and .
July 22, 2017
Should Journals Be Responsible for Reproducibility?
, one of the top journals in the field, wants to know. Very seriously. A good write-up from 听.
July 21, 2017
Another Day, Another Challenge.
Our friends from the have issued a :
鈥淧reregistration increases the credibility of hypothesis testing by confirming in advance what will be analyzed and reported. For the Preregistration Challenge, one thousand researchers will win $1,000 each for publishing results of preregistered research.鈥
How tempting is that? We will soon find out.
The Leaderboard for the Preregistration Challenge can be found here: .
The project is closely related to the which we recommend on our .
July 15, 2017
听and vicariously reporting鈥
July 8, 2017
Psychologist Disputes His Findings Won鈥檛 Replicate
Andrew Gelman recently posted a comment on a new controversy involving findings by German psychologist Fritz Strack who found quite a while ago that holding a pen in your mouth forces you to smile. This very famous result has recently repeatedly failed to replicate. A comment by Strack on the ensuing controversy led to Gelman鈥檚 comment and extended commentary as well as on various other discussion groups such as the Facebook .
Well known philosopher of Science has on this controversy.
June 28, 2017
Power Poses Have No Lasting Power
The relatively new journal Comprehensive Results in Social Psychology just published .
Going back to work done by Dana Carney, Amy Cuddy, and Andy Yap a few years back, an emergent literature on power poses originally suggested that nonverbal expressions of power affect people鈥檚 feelings, behaviors, and hormone levels.听In particular, they claimed that adopting body postures associated with dominance and power (鈥減ower posing鈥) can increase testosterone, decrease cortisol, increase appetite for risk, and cause better performance in job interviews. Subsequent attempts to replicate the effects failed and last year Dana Carney听made it clear听that and that the original study that started it all was methodologically flawed.
Carney was one of the special-issue editors of Comprehensive Results in Social Psychology鈥檚 just published issue on power poses. This issue is of interest both because of its results (no power poses effects) and its methodology (pre-registered replication studies).
Quite some discussion of this issue did also occur on the Facebook .
On the PLOS Blogs听听has, in a two part series (see听听and听), also usefully reflected on this controversy.
1 June 2017
Quantity Uncertainty Erodes Trust in Science
Of interest to those interested in听the recent replicability and credibility crisis in the social sciences is 听outlining the inability of consumers of science to adequately evaluate the strength of scientific studies without appropriate transparency.
31 May 2017
Cornell Food and Brand Lab Scandal
A recent scandal that has played out in the blog-sphere (and even the popular press) has been the questionable on-goings in the Cornell Food and Brand Lab. 听is a recent summary of the controversy and the important questions about questionable research practices it poses.
听and 听are more relevant discussions.
31 May 2017
Priming Research Train Wreck Debate
In a dated 2 February 2017, Ulrich Schimmack, Moritz Heene, and Kamini Kesavan wrote that the 鈥減riming research鈥 of Bargh and others featured in Kahneman鈥檚 book Thinking: Fast and Slow 鈥渋s a train wreck鈥 and should not be considered 鈥渁s scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness鈥 鈥 had conceded听that he placed too much faith in under-powered studies. 听 is a good start to read up on the debate.
28听May 2017
The Social Cost of Junk Science
In this , Andrew Gelman discusses听what needs to be done to avoid it.
7 May 2017
On Eminence, Junk Science, and Blind Reviewing
In this , Andrew Gelman discusses these issues听with and .
30 April 2017
Ujjval听VYas on Evidence-Based Design and Ensuing Discussion
Andrew Gelman recently posted a comment by Ujjval Vyas as separate entry on his blog and has drawn, not surprisingly, huge reactions. Yyas pointed out that evidence-based design abounds with the kind of questionable research practices that makes Wansink look scrupulous! A fascinating read which you can find .