Here's how scientists are using machine learning to listen to fish

This is an Inside Science story.

As the sun rises over the island of American Samoa, a chorus of animal voices drifts upward. They're not the calls of birds, though -- the purrs, clicks and groans are coming from under the water. New research shows how automation can make it increasingly easy to eavesdrop on the fish making the sounds and uncover how their environment impacts them.

Jill Munger first heard about fish that make sounds while she was an undergraduate student. A veteran researcher told her about marine acoustics.

"She described it in this really cool way: I get to spy on critters in the ocean, without disturbing them," said Munger, currently a marine ecology researcher at Oregon State University. "When you're a diver you disturb the wildlife as you swim through, so you don't get to witness what they are doing when you're not there." But passive acoustic monitoring offers an unadulterated soundscape.

Munger was given a hard drive with 18,000 hours of sound from a 12-station hydrophone area maintained by NOAA and the National Park Service in American Samoa -- an amount of data that seemed impossible to sort through, she said. "It's like if someone handed you a weeklong mixtape, you're never going to listen to it all."

She used software that created a spectrogram -- a visual readout of the noise. Some of the sounds were easily identified, such as whale calls. But there were other sounds that were totally foreign. "It was really almost like a puzzle to figure out what different sounds were," she said. Some sections were noisy, like wind, and others like an old scratchy record that was dominated by the sound of snapping shrimp. Eventually, she determined that they were coming from fish.

She was able to pick out the times that fish were calling together -- mornings and nights, like a chorus. There was one sound that excited Munger in particular. She is a cat lover, and the damselfish's call reminded her of purring.

"To me, [the fish calls] are very distinct and chock-full of personality," she said, adding that damselfish make this particular sound by snapping their pharyngeal teeth together and boosting the sound through their swim bladders. The technical term is a pulse train, and it's used in mating.

Despite all the intriguing sounds on the hard drive, Munger knew she would never be able to sift through all the data. So she turned to her brother, Daniel Herrera, a machine learning engineer, for help. He told her he'd partner with her on the project.

Herrera wrote the code and together they trained the model. The pair's findings were published in the journal Marine Ecology Progress Series. The machine learning sample or training data included 400 to 500 damselfish calls. With that start, Herrera, a co-author of the paper, built a machine learning model that accurately identified 94% of damselfish calls.

Research bottlenecks can arise when it's difficult to efficiently analyze vast amounts of data -- but systems like this can be game changers, said Carrie Bell, a researcher at the Cooperative Institute for Research in Environmental Sciences at the University of Colorado Boulder, who was not involved in the new paper. "When you start to introduce algorithms and approaches like machine learning and artificial intelligence and this deep learning, you're able to build something that is smarter and way more efficient to get through large amounts of data."

Beyond simple identification, the method can uncover clues about an ecosystem's health, Bell said. Since fish calls change with environmental conditions like wind speed, water temperature, tidal amplitude and sound pressure level, the noises themselves can be an indicator of how the ecosystem is faring -- especially in oceans that are rapidly experiencing climate change. These machine learning approaches have been used in analyzing humpback whale calls, but not in the fish world yet.

In addition to the purrs and clicks that Munger likes, Bell said in her research she's heard fish -- she's still not sure which species -- make mooing sounds, and other mysterious sounds that remind her of tubas and Jet Skis.

"The unknowns are the most fascinating to me, because it's amazing that here we are, in 2022, and we've been able to understand environments for decades but there are still so many things we haven't even described yet."

Inside Science is an editorially independent nonprofit print, electronic and video journalism news service owned and operated by the American Institute of Physics.