New AI scientist conducts its own research

From generating a project idea and conducting experiments, to writing a scientific paper and evaluating its own work, an AI scientist can now carry out a research experiment from beginning to end — all without the assistance of humans.

In a new study published in Nature, researchers from UBC Computer Science, Sakana AI, the Vector Institute, and the University of Oxford found a way to automate the entire research process, potentially speeding up scientific discovery.  

“While AI has been used by scientists to help them with specific tasks such as predicting the structure of proteins or analyzing medical images, this is the first time that AI has been shown to go through the entire scientific research process on its own,” says UBC Computer Science Professor Jeff Clune, a lead author of the paper.

“It’s amazing to see what it’s been able to do so far, but even more incredible to consider what lies ahead in the very near future.”

The AI scientist can generate new ideas, check the literature to see if they are truly new, write code and fix its own bugs for its experiments, analyze data and make graphs, write the manuscript, and perform its own review. The researchers developed the AI scientist using foundational models (like ChatGPT), which are trained on large datasets to perform a wide variety of tasks.  

To evaluate the quality of work, the researchers submitted the entirely AI-generated scientific paper to a workshop at a major machine learning conference—the International Conference on Learning Representations. The paper passed the peer-review process.  

Moreover, the researchers built an automated reviewer to evaluate AI-generated papers and found that it can accurately predict conference acceptance decisions, producing review scores similar to scores provided by human evaluators. Using the automated reviewer, the researchers showed that they can improve the quality of the generated papers by either improving the models used under the hood by the AI scientist or simply giving it more compute resources.

"One of the most exciting directions this work points toward is that the AI scientist could improve itself,” says Shengran Hu, a PhD student in UBC’s Computer Science department and co-author of the paper.

“The AI Scientist opens doors to recursive self-improvement in which the AI system doesn't just discover new scientific knowledge, but uses those discoveries to become better at making further discoveries. That's a qualitatively different kind of scientific progress than anything we've seen before."

While the researchers demonstrated that the AI scientist is capable of completing all the steps of scientific discovery, they documented some limitations. The AI scientist sometimes produced underdeveloped ideas or generated inaccurate citations. And, currently at least, the AI scientist can only conduct research in computer science, but the researchers believe the technology could independently conduct research in other fields as well.

“With additional research, this system could be used to create entire scientific communities of AI agents,” says Dr. Clune.

“Each new discovery could build on the system’s own prior discoveries. That could create an open-ended process of endless scientific discovery, just as what happens with communities of human scientists. That is when we’ll see the next major scientific revolution.”