Strategies for Effective Reading and Writing of ML Research Papers
Written on
Peer review is a fundamental aspect of the scientific process, occurring prior to the publication of any research in journals or conferences. This process aims to gather feedback from fellow researchers who assess the work based on their expertise, evaluating criteria such as originality, technical accuracy, and clarity.
In the realm of artificial intelligence—spanning areas like computer vision, natural language processing, and speech processing—this critical step is frequently postponed or overlooked due to platforms like arxiv.org. While arXiv serves as a preprint repository intended for the early sharing and discussion of new research, many submissions gain traction without undergoing peer review. Although numerous contributions found here are indeed significant, some ideas proliferate without receiving the necessary validation from the academic community, which can be concerning.
Is this a positive or negative development? On one hand, arXiv mitigates traditional gatekeeping that may hinder the careers of emerging researchers. Conversely, the overwhelming volume of papers can make it challenging for newcomers—such as data scientists or engineers—to distinguish between high- and low-quality work. Given this deluge, establishing a method to refine our focus is essential.
In this article, I wish to convey the insights I've gained regarding paper writing and reading, particularly through the lens of peer reviewing. When I refer to "reading," I mean it in the context of critically evaluating the material. Approaching all papers—even the most renowned ones—as if you were reviewing them is a beneficial practice. Research does not always present well-established knowledge, and it often takes years for the scientific community to replicate experiments and integrate new findings. Thus, fostering a mindset of skepticism and critical thinking can greatly enhance your learning experience, even if it extends the time spent on reading.
Writing Tips
In this section, I will outline various elements of a research paper to consider, which align with the criteria you would assess in a conference peer review. These recommendations stem from four years of experience in writing, reading, facing rejections, and refining my work based on reviewer feedback. While my four years may seem minimal compared to the extensive experience of many professors, the lessons I've learned through trial and error can potentially ease the journey for others.
Firstly, a strong command of English and scientific writing is crucial. While solid research is necessary for paper acceptance, it alone is insufficient. Readers must grasp your work and its significance. Continuously seek to improve your scientific writing skills, as complacency can hinder your progress. There are native speakers and linguists in your field who may articulate ideas more effectively. However, writing is a continuous journey, and with diligent effort, you may look back on past papers with a sense of embarrassment regarding your earlier writing. To accelerate improvement, don't limit yourself to drafting papers only at the end of research. Engage in writing proposals, blog posts, or drafts, and most importantly, seek constructive feedback. Growth occurs when errors are identified and corrected.
Secondly, always keep your audience in mind. Your paper should adhere to the writing conventions of the journal or conference for which you are submitting, and the results should align with their expectations. For example, a study on technology assisting translators may not fit well in a conference focused on machine learning. Similarly, a new interface for data collection, although useful, may not gain traction unless it offers novel deep learning models appropriate for conferences like NeurIPS, which demand a higher level of mathematical rigor.
Thirdly, continually consider your reader's perspective. Remember that your reader is not privy to your thoughts. They lack your knowledge, experience, and the context behind your writing. Therefore, clarity is paramount. Your paper should not serve as a platform for self-aggrandizement; rather, it should communicate new knowledge that captivates your readers. Always reflect on why someone would be interested in your work and how to present it compellingly. In the business realm, the most successful companies excel at interpreting customer needs and crafting products accordingly; writing demands a similar mindset.
Lastly, scientific writing need not be sterile. When composing, aim to weave a narrative for your audience. If each section of your paper stands alone without a cohesive story, it may serve as a reference but will likely fail to engage your reader. A research paper should unfold like a narrative: a formidable challenge threatens our world (the problem), a compelling reason to address it (motivation), the emergence of a solution (proposed method), acknowledgment of prior attempts and their failures (related works), the ensuing conflict (experiments and results), and finally, an analysis of the outcome. Conclude by inviting the community to join in the ongoing discourse, promising to revisit the topic soon. By crafting an engaging story, you offer your readers far more than mere information.
Critical Aspects of a Research Paper
Novelty
Novelty is a complex issue in scientific research. While research must generate new knowledge, the interpretation of "new" varies among individuals and communities, often shaped by prevailing trends or perceived challenges of the time.
A notable instance arose in NLP conferences, where the field shifted towards deep learning. It seemed that the only worthwhile research was the development of new deep learning architectures for various tasks, while other vital areas—such as linguistic resource creation and system evaluation—were overshadowed.
This trend likely stemmed from inexperienced reviewers (myself included) who focused more on the potential of deep learning rather than on gaining deeper insights into problems and their solutions. Consequently, many papers were accepted that have since faded from memory, as their "new" methods did not significantly differ from simpler, earlier models.
Fortunately, conferences are now encouraging reviewers to adopt a broader perspective regarding acceptable papers. Novelty can manifest in unexpected connections between different research areas, the creation of essential corpora, improved evaluation methods, or even the introduction of new tasks.
Novelty also influences what constitutes a research paper. A few years back, demonstrating the feasibility of deep learning applications (like RNNs or CNNs) was a valid research question, but today, such inquiries are often considered uninteresting unless they tackle unexplored tasks.
When reading a paper, connect it to existing literature and assess its novelty. Has this issue been previously addressed? Does it offer a new perspective? Are the resources and methods presented valuable for replication?
When writing, emphasize your paper's novelty in every possible way. If you don't highlight it, no one else will, and doing so can help shape the narrative you wish to convey.
Clarity
While the quality of content is crucial in a research paper, clarity in writing is equally important. A well-structured paper facilitates reading and comprehension. The abstract and introduction should broadly convey the main message and encourage continued reading. The body of the paper must clearly outline your hypothesis, experimental setup, background, and positioning within the existing literature. Conclusions should succinctly summarize the findings and underscore their significance.
> "The abstract, introduction, and conclusions must be crafted with particular care, as their purpose is to convince the audience that the paper merits their attention."
For novice writers, it's wise to adhere to the common structure of similar papers in your chosen venue. Over time, you'll learn to adapt it to better fit your work without rushing the process. Maintain a clear structure and prioritize your writing.
Strong English proficiency is the second, more challenging aspect of clarity. Strive for grammatical accuracy and concise, clear sentences. Avoid vague or general statements, and ensure you explain any figures and tables thoroughly, without making assumptions. Always remember:
The reader is not inside your mind.
To ensure your paper is understandable to peers, seek feedback from individuals with different backgrounds to identify any unclear sections. You may be surprised by their insights regarding readability.
When reading, appreciate well-written papers, and disregard those that are poorly constructed. If the content is unclear or ambiguous, it's likely that many others will struggle to understand it, rendering it ineffective in contributing new knowledge.
Motivation
The introduction of a paper should provide background information along with the specific scientific problem it addresses, detailing the chosen approach and the motivation behind it. This section can often be undervalued during the writing process, possibly due to the assumption that its importance is self-evident or because one thinks only results matter. However, unless your research is in an oversaturated niche, it's unlikely that your readers will grasp its significance without clear motivation. I've encountered papers that were primarily rejected due to insufficient motivation.
You may wonder why motivation is critical if the research is sound. The reason is that if you cannot articulate the importance of addressing a problem, how can others recognize it as an issue? Furthermore, every publication venue has limited capacity for papers, so a well-motivated paper will generally hold more value than one with uncertain relevance.
I recommend taking the time to reflect on why your research would engage your audience and articulating it explicitly. If your motivation resonates, readers will champion your work and advocate for its acceptance. Conversely, a lack of motivation may lead to responses like, "The method is intriguing, and the results are solid, but why should I care?"
When reviewing, exercise caution. If the motivation fails to resonate, it may be due to your lack of understanding of the problem being addressed. However, if the paper does not clearly frame the general problem and the specific aspects it investigates, this is a valid critique.
Hypothesis
A foundational aspect of scientific inquiry is the falsifiable hypothesis: after observing a phenomenon, one formulates a hypothesis and designs experiments to validate its assumptions. While hoping that experiments do not disprove the hypothesis, they may reveal its inadequacies in the future.
A hypothesis should explain a phenomenon or justify an engineering enhancement. In a research paper, it must be clearly articulated from the abstract onward and reiterated throughout. It's no coincidence that many successful papers feature hypotheses that revolutionize our understanding of problems. Titles like “Attention is All You Need” posited that self-attention surpasses recurrent neural networks for sequence modeling; “BERT” suggested that vast amounts of unlabeled text data could enhance systems for various NLP tasks; “Distilling the Knowledge in a Neural Network” argued that a neural network could uncover latent relationships among target classes more beneficial than the original data for training new models. These groundbreaking ideas became common knowledge due to their powerful and succinct presentation, rather than merely through numerical data in tables. Conversely, papers with weak hypotheses are often dismissed as "incremental work."
When writing, explicitly state your hypothesis, ensuring that your proposed methods align with it. Although this should be established during the method design phase, the writing process is the time to make it clear for your audience.
During reading or reviewing, assess whether the hypothesis is articulate, interesting, and if the experiments align with it. A disconnection between the method and hypothesis leads to confusion rather than knowledge. Clicking "reject" is disheartening, especially when the research seems sound, but a method justified by an unsupported hypothesis results in misleading claims. The same applies if a paper's title does not align with its content or if the analysis draws unfounded conclusions. In science, while errors are acceptable, outright falsehoods cannot be tolerated.
Datasets/Evaluation
The dataset utilized in experiments is not a trivial component of experimental design. A dataset embodies a range of domains (or languages, in the context of speech and language processing), assumptions, and biases that must be acknowledged during study. It should accurately reflect the phenomenon under investigation. Some research employs "proxy" datasets, which is typically unwise.
A common dataset issue in machine translation research involves studies on low-resource language pairs. I've made this mistake myself; many papers propose methods for low-data conditions yet experiment on a small (and sometimes not so small) subset of a larger dataset. This approach can give the illusion of effectiveness, as small data may yield weak baselines. Moreover, it assumes that low-resource and high-resource languages are alike, merely differing in vocabulary familiarity—an overly simplistic view given the vast diversity of languages. Concepts like "to be" and "to have" may be expressed differently, or even not at all, across languages; varying levels of formality can result in distinct vocabularies; idiomatic expressions are often culturally specific; and multiple writing systems can coexist.
Evaluation also presents challenges. Researchers may rush to propose new solutions without establishing suitable evaluation methods first. In such cases, developing an effective evaluation approach can significantly advance the field more than introducing a "novel method." For instance, “Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus” introduced a reproducible evaluation method for assessing speech translation quality concerning gender-related issues, necessitating a meticulously annotated test set. Improvements are always possible, but our understanding remains limited.
If you aim to address a task but find existing datasets or evaluation methods lacking, consider developing them. Such contributions are often more valuable than employing complex neural networks to tackle trivial problems. Remember that newcomers to any research area will require software, datasets, and automated evaluation methods. By facilitating access to these resources, you can attract attention (and citations) within your field. In my case, I gained recognition in speech translation due to the code and dataset I developed.
When writing, ensure your datasets and evaluation methods align with the problem you intend to solve. When reading, verify that the claims made in the introduction and conclusion accurately reflect the experimental findings.
Baseline
Your baselines serve as a critical indicator of the diligence invested in your work. A strong baseline—ideally stronger than those in other published studies—indicates that the improvements introduced by your methods are credible. Occasionally, papers present baselines comparable to those from five years ago, neglecting advancements made since then. While improvements over such baselines may seem significant, the final results may not be trustworthy. This is because many methods or inductive biases contribute less when applied to stronger baselines. For example, the impact of embedding pre-training in a neural machine translation model is challenging to gauge with ample training data, whereas it becomes more evident with smaller datasets.
When writing, be attentive to your baselines, and they will return the favor. Stronger baselines can offer fresh insights into a problem and reveal flaws in previous approaches. Weak baselines may only serve as a means to publish a paper, relying on reviewers unfamiliar with your field. Avoid this practice.
When reading a paper, before being swayed by impressive results, check if recent studies in the same area are cited and ensure the baselines are consistent with those findings. Tables illustrating the state of the art can add value, but always verify their accuracy. I've seen papers that omitted relevant results, leading to inflated conclusions. This practice is unethical and should be addressed promptly.
Results/Analysis
Results are vital in empirical fields like NLP, but the rise of deep learning necessitates a cautious interpretation. After confirming the credibility of baselines, it's important to evaluate the magnitude of improvements. Even statistically significant advancements may stem from randomness (such as different random seeds or varied method implementations) or simply a better choice of hyperparameters. Understanding the reasons behind improvements can be challenging, prompting a focus on the analysis. If the analysis demonstrates a coherence between the hypothesis and results, I tend to view the paper positively. However, if the analysis seems obligatory rather than insightful, it raises concerns about the overall quality of the work.
Producing a compelling analysis can often be difficult, especially as it may require delving into the inner workings of network weights, which remain largely opaque. Consequently, we often rely on proxies to glean insights about our methodologies. These proxies should still relate to the phenomenon and the models under examination. If you encounter a paper with unsatisfactory analysis, refrain from being overly critical unless you can offer a more valuable perspective.
When writing, ensure that your results are persuasive and that the methodology aligns with previous studies for easy comparison. When reading, assess whether the results genuinely stem from the applied methods or from unrelated factors. Scrutinize the analysis to determine if it convincingly relates to the studied phenomenon.
Conclusions
Writing, reading, or reviewing scientific papers can be daunting, but with practice and experience, it becomes more manageable over time. Many papers adhere to established structures that simplify reading; when they deviate, they often stand out as either exceptionally good or notably poor. Both cases are usually easy to identify. If you are just starting to write scientific papers, consider enhancing your scientific writing skills. The internet offers a wealth of resources, and those I have linked are merely a small sample. Personally, I've gleaned many valuable insights and recommendations through this process. If you're new to reviewing, gaining experience by reviewing published papers and seeking feedback from seasoned colleagues can be beneficial. I hope this article helps you focus on the critical aspects that will enhance your understanding of a paper's value and help you avoid pitfalls that I encountered early in my journey.