Archive
The promise of AI’s efficiency and speed in sifting through vast amounts of data and producing content has led many to speculate that it can replace parts of the journalistic process. (This despite a lukewarm reception from news consumers and ambivalence among journalists themselves.) However, Axios’ recent move to ask employees to argue why a human should do a job that AI might be able to do seems to signal a false binary. In this either-or situation, we’d replace expertise with expedience, possibly at the risk of accuracy – which could have grave consequences.
Axios is far from alone in experimenting with AI to augment journalism. However, as is the case in many industries, there is a rush to employ AI in a way that supplants rather than supplements. And that, in turn, could create a gap in the entry level employee pipeline. In the case of journalism, this may mean that the established process of developing expertise breaks down. It is certainly a challenge to build deep subject domain knowledge. But it is possible that this lengthy process could be restructured to be both more supportive and efficient.
A recent study, Generalization bias in large language model summarization of scientific research published in Royal Society Open Science, highlights a critical issue: media coverage of scientific research often fails to capture the nuance and complexity inherent in the scientific process. However, the research finds that AI also fails at the task of summarizing scientific papers and abstracts accurately. As more and more people – journalists, medical professionals, scientists, educators and students – rely on AI-generated summaries and content, the results of this study beg the question as to whether we’re integrating AI to effectively support work processes.
Media coverage of scientific research has issues
As the public’s primary source of information on the latest scientific discoveries and advancements, the media holds the power to shape public understanding, perception and opinion. Accurately presenting findings from scientific and medical studies is paramount. As this study and others show, AI agents tested weren’t able to do the job. This seemingly automatable task would appear to be not-so-automatable after all.
Reporting on scientific research isn’t simple, as we’ve seen in the past. The temptation to prioritize sensationalism and oversimplification over nuanced reporting looms large, in part because of media business models driven by engagement metrics and fast news cycles. Scientific work does not produce definitive outcomes (inferences) so much as it accumulates evidence to suggest a truth. Even in the case of studying phenomena that are reasonably and reliably measured, such as in physics, the theories that are the outcome of science are, in fact, theories! For example, if a study says a certain treatment was found to reduce incidences of a disease in a very specific group of people during a 10-week trial, the exciting news might seem to be that a given disease is now curable. Sadly, it isn’t. This would be a misinterpretation of the study results.
Moreover, science itself is rarely straightforward. Study results can be contradictory, because of subtle, or not so subtle, differences in hypotheses, cohorts, measurements and methodology. Particularly because journalists often work on tight deadlines, statistics may be misinterpreted or decontextualized thus potentially misleading the public.
This new research found that, when summarizing scientific texts, chatbots may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. (Mind you, they compared the chatbot-derived summaries to those published in The New England Journal of Medicine Journal Watch, and did not check other types of summaries written by individuals with lesser domain expertise.) Press releases written to “hype” new research, may succumb to overgeneralization, as well. And, like all research, there are likely other teams putting AI tools to similar tests. So, even as I anchor my point to the article published in RSOS, I remain keenly aware that new research that reinforces, expands, or contradicts these findings may be published imminently.
What’s at stake when media fails to get science right
It is critical not only that we best employ emerging tools such as AI to support the production of quality journalism, but also that we continue to support and foster the development of science reporters. There’s much at stake if we fail to get scientific coverage right.
At its worst, the media’s coverage of scientific research can contribute to misinformation and the erosion of public trust in science.
However, at its best, science-based journalism has the power to inform the public and help set the agenda in two ways: First, by circulating what the scientific community considers the most important discoveries and applications. And secondly, by generating public interest in specific ideas, discoveries, or technologies that have previously been rejected or overlooked, as noted by Maxwell E. McCombs and Donald L. Shaw over 50 years ago.
Science communication acts as a bridge, shaping perceptions and priorities on both sides. As new breakthroughs emerge, the media plays a crucial role in determining which ones capture the public’s attention. This symbiotic relationship between science and society is an important dynamic for media professionals to understand. Journalists must embrace the nuance and complexity of scientific research to effectively inform and educate the public, even as AI threatens to displace journalists and media organizations move beyond delegation towards automation.
Using the tools properly to create the best science journalism
If science reporting is indeed in crisis, reliance on AI will not improve the situation. Instead, we should give novice journalists the opportunity to summarize abstracts and papers, thereby learning to translate scientific jargon into more accessible language without losing context and nuance. Perhaps, this could be achieved through collaboration with a “dumb” AI chatbot and a human mentor, fostering a learning environment that emphasizes the importance of nuanced understanding.
If the future of white collar work is to rely more heavily on experts and cutting entry level positions, as suggested by both Anthropic’s CEO and Axios’ chief executive, then the future does indeed look grim. Experts aren’t hatched fully formed. Expertise is cultivated through experience and engagement with complex tasks. It starts, however, by slogging through the boring, automatable tasks now passed on to AI. As a doctoral student, I would have been tempted to use AI to summarize academic papers, find relevant quotes. But I wouldn’t be the expert I am today if I had taken that shortcut. A person becomes an expert. No one is born an expert, and neither is an AI assistant.
I’m not arguing against the use of AI. It’s a brilliant tool with valuable applications. However, as with most major shifts – and scientific discovery and reporting itself – it is often necessary to invest time and effort to get things right. Media leaders may want to take a step back, evaluate the hype and explore ways to integrate AI that supports the delicate task of reporting on science in a nuanced, thoughtful way. Perhaps then, we can ensure that reporting on science informs rather than misleads, ultimately fostering a more scientifically literate society.
