Since the 17^th century, when the first scientific journals, Philosophical Transactions of the Royal Society and Journal des Sçavans, were published, until today, the primary aim of research publications has not changed. It is to share information and learn by building upon the works of one’s colleagues; however, this is no longer the sole apparent reason today. ​“The publishing system has become somewhat undermined as a sort of research assessment tool primarily, but the real function is to share information properly, and this function is still very relevant,” said Bernd Pulverer, Head of Scientific Publishing at the European Molecular Biology Organisation (EMBO) and Chief Editor at EMBO Reports, in a YI Huddle webinar.

IndiaBioscience organises a series of monthly webinars called YI Huddle, which serves as a discussion series for Young Investigators (YI) in India. In the third YI Huddle titled To be noted: Research Documentation and Reproducibility for YIs, they discussed the importance of documentation and reproducibility of research, highlighting challenges and solutions, especially for YIs. This discussion questioned the purpose of publishing research findings in scientific journals and how that shapes science.

Research documentation and reproducibility of research findings go hand in hand. However, a news feature based on a survey organised by Nature, where 1576 researchers participated, mentioned, ​“More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” Rather than research integrity, the problem lies primarily in the detailed reporting of highly complex methods and data, as there have been numerous instances of researchers exchanging information and reagents and spending lots of money and time. Yet, they were unable to replicate the work of their colleagues.

Enhancing research reproducibility

Divya P Kumar, Assistant Professor, JSS Medical College, Mysore, and the moderator of the YI Huddle, asked, ​“Do we have a framework that young investigators can keep in mind or define that in their lab setting to have research reproducibility?” Pulverer emphasised the existing gaps that undermine research reproducibility. ​“The funders are not incentivising attempts to reproduce science or even to report in a reproducible way in the first place. Then there’s no incentives in terms of your research career — the research assessment doesn’t encourage it either.” He added that what matters most for publications is still often the journal name and impact factor instead of any framework to follow up on the funded work and to assess whether it is reproducible. 

He further added, ​“At the journal level also, we’re really still publishing in the same way we did in the last century, which is focused on narrative-driven papers, where you basically describe in free text data that is shown typically in a flattened format such as a PDF, from where data cannot be extracted for further analysis. Figures are often seen more as pretty pictures that illustrate the textual interpretation of the data, rather than sharing proper ​‘raw’ or ​‘source’ data in a way that has associated metadata.” Such metadata could benefit the researchers in reproducing research done by their colleagues.

However, even before publishing research in a scientific journal, research documentation starts in the lab notebooks. Contemporary researchers adopt diverse approaches to maintaining these notebooks, ranging from handwritten records to digital documents equipped with hyperlinks and traceable features for accessibility. While electronic notebooks offer advantages such as legibility and prevention of misplacement, both electronic and handwritten formats are viable. Certain journals, like the Journal of Visualized Experiments (JOVE), allow scientists to submit their methods as videos. For complex experimental methodologies, besides keeping well-documented lab notebooks, scientists should also ensure filming their work. A simple ​‘trick’ often does the job when we scratch our heads over months due to the lack of such documentation.

Promoting Reusable and Open Methods and Protocols (PRO-MaP) is an initiative that encourages new standards for protocols and methods. As protocols and methods are ​‘living’ documents, editors, including Pulverer, recommend that they be updated as and when a change is incorporated into the process and versioned so that one can return to a version that worked for a specific set of experiments. Platforms such as pro​to​cols​.io allow such versioning.

“When I was doing my research, we were still really pulling out primary data in lab meetings and sharing the autorads and stuff, so you could see the primary data that came out of the machines,” recalled Pulverer. ​“Now this has changed dramatically: what typically is exchanged in lab meetings is already highly processed data, sometimes even ready-made PowerPoint slides that are, in essence, figures that are on the way into a paper.” In contrast, Pulverer believes sharing primary data with the lab members is crucial for constructive discussion and uncovering mistakes early on. Kumar also enlightened us with a lab practice that she follows in her lab and recommends others to do so.

She believes that even after following all protocols correctly if scientists cannot replicate their own data or their colleagues’, it’s essential to discuss that in the lab openly without feeling offended and taking things personally — stressing the need for an open lab environment.

Sharing research using preprints

As opposed to peer-reviewed papers, preprints in biology started being increasingly used only about a decade ago. Still, they became a game changer in sharing new research and early research documentation. A preprint is similar to a manuscript without peer review and can be submitted to preprint servers and updated with new versions based on feedback and new advances (versioning). Several preprint servers host preprints in Biological Sciences, including bioRxiv and medRxiv. As they cut down on the time required for sharing work in a peer-reviewed journal, they are often the go-to-choice for researchers before sharing their work for publication in a journal.

“Preprints empower authors to share ​“works-in-progress”; manuscripts that are updated as experimental work progresses towards more traditional publication. These works-in-progress are a novel feature of preprints that would be impossible to achieve with traditional journal publishing and are much more representative of developing work,” explained Jonny Coates, Associate Director, Accelerating Science and Publication in Biology (ASAPbio). At the same time, since sharing early work is often met with reluctance from authors because of the chance of ​‘scooping’, preprints can be the perfect platform to do so. Coates elaborated, 

“Preprints provide a citable, permanent record of research, and can be posted, or updated, whenever the authors feel is appropriate. This allows authors to document the entire research process from early sharing of work as a preprint to the final version of a project.”

However, just the formal documentation of research using preprints might not always be sufficient to protect from scooping. Pulverer mentioned, ​“At EMBO Press, we have, therefore, a ​‘scooping protection policy’: from the date of posting your preprint, our editors will not take other papers (or preprints) into account in their assessment of the conceptual advance of the manuscript once it is submitted to our journals.”

There’s a bag full of benefits in sharing work as a preprint. Not only does one have the chance to receive feedback through all different stages of the experimental process, but one can also publish negative data and replication studies, which are often not accepted in most peer-reviewed journals, although tremendous efforts go into such studies. ​“This also benefits the larger community who can replicate work, strengthening the claims made by authors, or correcting mistakes and false claims,” added Coates.

Role of institutes and individual labs

Institutes can play a crucial role in implementing proper research documentation. Pulverer presented one interesting example from the Center for Stroke Research at Charite, Berlin. ​“They have a unit of about 120 researchers, and they actually hired two professional data and resource managers that help the PIs to structure the information, at the level of operations, protocols and reagent tracking, and also they help them in writing the papers in a structured way including the protocols.” It would be beneficial if other institutes also professionalise their research management this way — it may even save money.

However, one cannot ensure research reproducibility only by documenting research in a better way. Pulverer mentioned that it’s best to think about reproducibility within the lab, ​“If you have an interesting finding, get another postdoc or student to try to reproduce it themselves.” He also mentioned that in a consortium, experiments should even be reproduced across collaborative labs, and they should be reported. ​“So, if I, as an editor, see that this level of work was done … [I would be] obviously very positive about a paper like that.”