Always thought this was problematical. How many experiments in papers are ultimately repeated, validated, with the same data? Or ultimately are they just 'experientially' validated (or not) over time?.
Furious AI Researcher Creates a List of Non-reproducible Machine Learning Papers, By The Next Web, March 10, 2021
On February 14, a researcher who was frustrated with reproducing the results of a machine learning research paper opened up a Reddit account under the username ContributionSecure14 and posted the r/MachineLearning subreddit: "I just spent a week implementing a paper as a baseline and failed to reproduce the results. I realized today after googling for a bit that a few others were also unable to reproduce the results. Is there a list of such papers? It will save people a lot of time and effort."
The post struck a nerve with other users on r/MachineLearning, which is the largest Reddit community for machine learning.
"Easier to compile a list of reproducible ones…," one user responded.
"Probably 50%-75% of all papers are unreproducible. It's sad, but it's true," another user wrote. "Think about it, most papers are 'optimized' to get into a conference. More often than not the authors know that a paper they're trying to get into a conference isn't very good! So they don't have to worry about reproducibility because nobody will try to reproduce them."
A few other users posted links to machine learning papers they had failed to implement and voiced their frustration with code implementation not being a requirement in ML conferences.
The next day, ContributionSecure14 created "Papers Without Code," a website that aims to create a centralized list of machine learning papers that are not implementable.
"I'm not sure if this is the best or worst idea ever but I figured it would be useful to collect a list of papers which people have tried to reproduce and failed," ContributionSecure14 wrote on r/MachineLearning. "This will give the authors a chance to either release their code, provide pointers or rescind the paper. My hope is that this incentivizes a healthier ML research culture around not publishing unreproducible work."
From The Next Web
No comments:
Post a Comment