Artificial intelligence recognizes patterns in behaviour

Artificial intelligence recognizes patterns in behavior: The open-source platform A-SOiD developed by Researchers from Carnegie Mellon University, the University Hospital Bonn and the University of Bonn can learn and predict behaviors, just from video.
(c) Matthew Salyers / Carnegie Mellon University

Researchers from Carnegie Mellon University, the University Hospital Bonn and the University of Bonn have created an open-source platform known as A-SOiD that can learn and predict user-defined behaviors, just from video. The results of the study have now been published in the journal “Nature Methods”.

“This technique works great at learning classifications for a variety of animal and human behaviors,” said Eric Yttri, Eberly Family Associate Professor of Biological Sciences at Carnegie Mellon. “This would not only work on behavior but also the behavior of anything if there are identifiable patterns: stock markets, earthquakes, proteomics. It’s a powerful pattern recognition machine.”

Unlike many artificial intelligence (AI) programs, A-SOiD is not a black box. Instead, the researchers allowed the program to re-learn what it did wrong. They first trained the program with a fraction of the dataset, with a focus on the program’s weaker beliefs. If the program was not certain, the algorithm would reinforce the belief of that training data.

Because A-SOiD was taught to focus on the algorithm’s uncertainty rather than treating all data the same, Alex Hsu, a recent Ph.D. alumnus from Carnegie Mellon, said that it avoids common biases found in other AI models.

AI tool does justice to every class in a data set

“It’s a different way of feeding data in,” Hsu said. “Usually, people go in with the entire data set of whatever behaviors they’re looking for. They rarely understand that the data can be imbalanced, meaning there could be a well-represented behavior in their set and a poorly represented behavior in their set. This bias could then propagate from the prediction process to the experimental findings. Our algorithm takes care of data balancing by only learning from weaker. Our method is better at fairly representing every class in a data set.”

Because A-SOiD is trained in a supervised fashion, it can be very precise. If given a dataset, it can determine the difference between a person’s normal shiver and the tremors of a patient with Parkinson’s disease. It also serves as a complementary method to their unsupervised behavior segmentation platform, B-SOiD, released two years ago.

Besides being an effective program, A-SOiD is highly accessible, capable of running on a normal computer and is available as open source on GitHub.

A-SOiD is accessible for everyone in science

Jens Tillmann, a postdoctoral researcher from the University of Bonn at the University Hospital Bonn, said that the idea of having this program open to all researchers was part of its impact.

“This project wouldn’t have been possible without the open science mindset that both of our labs, but also the entire community of neuroethology have shown in recent years,” Tillmann said. “I am excited to be part of this community and look forward to future collaborative projects with other experts in the field.”

Yttri and Martin K. Schwarz, principal investigator at the University Hospital Bonn and member of the Transdisciplinary Research Areas (TRA) “Life & Health” at the University of Bonn, plan on using A-SOiD in their own labs to further investigate the relationship between the brain and behavior. Yttri plans to use A-SOiD in conjunction with other tools to investigate the neural mechanisms underlying spontaneous behaviors. Schwartz will use A-SOiD in conjunction with other behavioral modalities for a fine-grained analysis of known behaviors in social interactions.

Both Yttri and Schwarz said they hope that A-SOiD will be used by other researchers across disciplines and countries.

“A-SOiD is an important development allowing an AI-based entry into behavioral classification and thus an excellent unique opportunity to better understand the causal relationship between brain activity and behavior,” Schwarz said. “We also hope that the development of A-SOiD will serve as an efficient trigger for forthcoming collaborative research projects focusing on behavioral research in Europe but also across the Atlantic.”

The research was funded by the German Research Foundation (DFG); Project ID 227953431 – SFB 1089 of the University of Bonn.

Jens F. Tillmann, Alexander I. Hsu, Martin K. Schwarz and Eric A. Yttri; A-SOiD, an active learning platform for expert-guided, data efficient discovery of behavior; Nature Methods;

Press contact:
Dr. Inka Väth
Deputy Press Officer at the University Hospital Bonn (UKB)
Communications and Media Office at Bonn University Hospital
Phone: (+49) 228 287-10596

Heidi Opdyke
Interim Director of Communications
Mellon College of Science at Carnegie Mellon University; USA
Phone: +1 412-268-9982

About Bonn University Hospital: The UKB treats around 500,000 patients per year, employs around 9,000 staff and has total assets of 1.6 billion euros. In addition to the 3,500 medical and dental students, 550 people are trained in numerous healthcare professions each year. The UKB is ranked first among university hospitals (UK) in NRW in the science ranking and in the Focus clinic list and has the third highest case mix index (case severity) in Germany. In 2022 and 2023, the F.A.Z. Institute recognized the UKB as the most desirable employer and training champion among public hospitals in Germany.

Wissenschaftliche Ansprechpartner:

Dr. Martin K. Schwarz
Research Group Leader “Functional Neuroconnectomics”
Institute for Experimental Epileptology and Cognitive Research
University Hospital Bonn
TRA “Life & Health” of the University of Bonn
Phone: (+49) 228 287-19347

Dr. Eric A. Yttri
Family Associate Professor of Biological Sciences
Carnegie Mellon University, Pittsburgh, USA


Jens F. Tillmann, Alexander I. Hsu, Martin K. Schwarz and Eric A. Yttri; A-SOiD, an active learning platform for expert-guided, data efficient discovery of behavior; Nature Methods;

Media Contact

Dr. Inka Väth Kommunikation und Medien
Universitätsklinikum Bonn

All latest news from the category: Health and Medicine

This subject area encompasses research and studies in the field of human medicine.

Among the wide-ranging list of topics covered here are anesthesiology, anatomy, surgery, human genetics, hygiene and environmental medicine, internal medicine, neurology, pharmacology, physiology, urology and dental medicine.

Back to home

Comments (0)

Write a comment

Newest articles

“Nanostitches” enable lighter and tougher composite materials

In research that may lead to next-generation airplanes and spacecraft, MIT engineers used carbon nanotubes to prevent cracking in multilayered composites. To save on fuel and reduce aircraft emissions, engineers…

Trash to treasure

Researchers turn metal waste into catalyst for hydrogen. Scientists have found a way to transform metal waste into a highly efficient catalyst to make hydrogen from water, a discovery that…

Real-time detection of infectious disease viruses

… by searching for molecular fingerprinting. A research team consisting of Professor Kyoung-Duck Park and Taeyoung Moon and Huitae Joo, PhD candidates, from the Department of Physics at Pohang University…

Partners & Sponsors