No single human scientist is expected to be an expert in hydrology, aerospace, materials science, psychology, and programming all at once. Yet some AIs are.
Sophisticated AI models that can autonomously interact with external tools and conduct tasks to serve a programmed goal, known as agentic AIs, are increasingly being used in scientific workflows. Some research software engineers (RSEs) are using agentic AIs to create programs from scratch, and even to test and validate those programs—but the significantly reduced development time comes at a cost.
“Existing research shows that an AI can generate a test that looks sophisticated, runs successfully, and passes—while testing the wrong scientific assumption,” said Assistant Professor Nasir Eisty, a research software engineering and quality assurance expert in the Min H. Kao Department of Electrical Engineering and Computer Science. “A climate model and a biological simulation may have very different scientific assumptions and definitions of correctness. Failing to account for those differences can ultimately affect the reproducibility and credibility of scientific results.”
The Alfred P. Sloan Foundation awarded Eisty a $430,000 grant to make AI-based development of research software transparent, traceable, and reproducible—ensuring that the final products and their outputs are both trustworthy and scientifically relevant. The funding includes support for two PhD students who will work with Eisty every step of the way.
“I am very grateful to the Sloan Foundation for supporting this work,” Eisty said. “AI is changing software engineering very quickly, and this project gives us an opportunity not only to study that transformation but also to train researchers who can help shape AI’s responsible use in scientific software development.”
Convenience with a Cost
Two years ago, if an RSE wanted to verify their simulation of a physical system, they would need to manually identify important scenarios, encode the scenarios as tests, run the software, and evaluate whether the results of the simulation matched known outcomes or scientific properties.
With generative AI tools, the scientist could instead ask an AI system to analyze the scenarios and generate those tests. An agentic AI could go even further, generating and running tests, analyzing the results, and modifying the tests accordingly.
“This could be particularly valuable in science because many researchers are domain experts first and software developers second,” said Eisty. “AI could make good software engineering and testing practices accessible to research teams that may not have dedicated software engineering resources.”
However, AI-generated tests might fail to capture important edge cases or domain-specific theoretical constraints, leading to incorrect assumptions and irreproducible results.
Eisty’s Software Analytics and Intelligence Lab (SAIL) investigates how to leverage AI in the scientific software pipeline while maintaining reproducibility, correctness, and long-term utility. Thanks to the Sloan Foundation’s support, he and his PhD students will evaluate how well AI agents ‘understand’ scientific constraints and design tests that catch real defects in scientific programs.
“Our goal is to determine where AI is useful, where it is unreliable, and how we can keep humans appropriately involved,” Eisty said.
Balancing Utility and Reliability
Eisty and his students will study current RSEs and scientists to understand how they are using generative agentic AIs to design and test their software. Next, they will deliberately introduce multiple types of defects into existing research software. Working with domain scientists, the team will evaluate how well AI-generated tests detect the injected issues and whether they reflect the scientific assumptions and questions underlying the software.
“I think one of the hardest questions will be determining what it actually means for an AI-generated test to be scientifically correct,” Eisty said. “We plan to address this by studying 30 to 40 research software projects across at least five scientific domains, including computational biology, astronomy, climate modeling, machine learning, and computational social science.”
Once the experiments are complete, Eisty and his team will create a curated benchmark suite, open source tools, and guidelines for responsible AI-assisted software testing across diverse research domains—such as a tool to help researchers give AIs appropriate scientific context during prompting and a test-development framework that records important information about the AI model and other factors that go into creating each test.
Their framework will be model-agnostic, allowing RSEs to continue creating reliable and reproducible research software on new AI systems as they emerge.
“Our project is ultimately about finding the right balance: taking advantage of AI’s ability to accelerate software development while preserving the human judgment, transparency, and rigorous validation that science requires,” Eisty said.
Contact
Izzie Gall (egall4@utk.edu)