When AI converts news to audio, tone and truth can shift
A new tool from Google aims to help students learn across a range of modalities. Some may learn best through reading, while others respond to seeing work visually. Still others prefer learning via listening.
But what happens when AI turns written works into one of these other mediums? Earlier this month, Straight Arrow asked journalists to assess how well Google turned their written stories into TikTok-style videos. Now, Straight Arrow spoke with a new set of journalists to evaluate how well Google’s NotebookLM turned their reporting into podcasts. The results, they said, were “surprising,” “unfunny” and “explanatory.”
How do AI models decide what goes in their podcasts?
Audio can be a useful tool, said Parsa Hejabi, a computer science PhD student in the morality and language lab at the University of Southern California. But turning an article into a different format using artificial intelligence could potentially introduce bias, emotion or miscommunication, he told Straight Arrow.
In general, the kind of AI that turns written words into a different format usually contains multiple steps, Hejabi told Straight Arrow. And, “since this is a pipeline with different steps, each of these steps have their own failure mode,” he said.
Typically, the AI begins by extracting the main information from the source material, then sifting through the data and prioritizing the most important threads.
The third step adds the narrative, converting selected segments into a script and adding transitions, explanations and conclusions. In the final step, the AI uses text-to-speech to produce the host or narrator’s voice.
But each stage can create distortions. The extraction stage might miss some information. Then, the prioritizing stage might overemphasize a particular perspective or highlight the wrong details. And the stage that creates the narrative might invent new connective details or irrelevant metaphors.
“The voice, music, pacing or visuals might introduce an emotional framing that is really absent from the source,” Hejabi said. “ You’re adding this dimensionality that might introduce an emotional framing.”
Emotion can introduce bias, or make the listener believe or prioritize details that weren’t as important in the original content.
And attribution can disappear when the information is compressed into its new format.
Straight Arrow asked journalists to evaluate podcasts generated by NotebookLM. They did report that several of the tendencies Hejabi mentioned, including adding emotion and re-prioritizing perspectives, were evident in the AI’s podcast outputs.
Rockaway Shows Up At City Hall
Outlet: The Wave.
Reporter: Gabriela Flores.
Gabriela Flores, a freelance reporter at The Wave, had to keep reminding herself that what she was hearing wasn’t coming from a human.
“I was freaked out in the beginning because of how eerily human it sounded,” she told Straight Arrow. “It would take pauses and chuckles and here and there, but then I think after minute three, that’s when I realized: Oh, OK, this is AI.”
Overall, she said, the podcast did well at explaining the first half of her story, which was about the New York mayor’s approach to two big local infrastructure projects. But unlike her objective news story, the podcast focused on just one of the projects, and “sort of took the perspective of the protesters,” she said.
Then came the inaccuracies. As her story covered a proposed new train line, the podcast reported that the train would connect an “isolated” geographic area.
Not true, she said. Queens, where the story is set, is actually quite connected.
“I guess this is a common talking point about AI, that it hallucinates after a certain time,” she said.
“There were some phrases, hyperbolic language that it used,” she said, like “pouring concrete over a time capsule” when it was describing why the New York mayor would choose one project over another.
“That kind of took me aback,” she said. She said that the podcast was trying to come up with reasons one project might be prioritized over another — “time consuming, noise complaints,” or maybe that the chosen project is “a quick fix,” Flores said. But “literally nothing in the article mentioned that.”
“I think that’s like a dangerous, slippery situation to be in, for someone who’s trying to grasp what actually is going on on the ground — to have AI give you their own opinionated input of what someone’s intentions are,” Flores told Straight Arrow.
The podcast also “kind of magnifies the negative,” she added, by describing the proposed area of the new trainline as totally severed from the city’s transit grid, which also isn’t true. Flores thinks this underscores the importance of human reporters.
Why North Forest residents, HCC leaders fought a controversial flood relief plan — and won
Outlet: Open Campus.
Reporter: Miranda Dunlap.
Miranda Dunlap expected robotic narration before her first listen.
“It sounded like a real human so I was pretty surprised,” said Dunlap, an education reporter.
It was, perhaps, too human-like in parts, Dunlap told Straight Arrow. The narration included “a ton of shock, and how ‘crazy’ this is” throughout its 18-minute run time. “There is some shocking element of this story, but not every element of it is, like, super crazy.”
Dunlap’s story, about a community affected by flooding, mentioned homeowners paying out of pocket for flood repairs on their homes. The podcast “went on a tangent about how disaster insurance is a scam,” she said.
“There were several things that the speakers expounded on at length that were not really unpacked,” in her writing, she added. Yet the podcast “went on a tangent about how disaster insurance is a scam.”
The podcast also included information about flood mitigation strategies and flooding mechanics, which she hadn’t included.
“It got more into the weeds of urban development than I definitely included,” she said.
Including information from outside sources made her question whether parts of the podcast were even true.
“So, that was a definite red flag for me,” she said.
“My problem with these sorts of tools is that I find the investment of time doesn’t end up being that much less because I still feel the need to go back through the primary source material and make sure that everything it said is accurate,” she said. “The hang ups about things that it adds in still remain a little bit baffling to me.”
Dunlap found the podcast cool. But had it been on a topic she was less invested in, she said, the interactions between the hosts might become “a little bit exhausting.”
If she made her own podcast, she said, she would slow it down and change the format.
“Both of the speakers were clearly supposed to be people who had read the story and knew what they were talking about,” she said. But she’d rather listen to a classic two-way interview, where one person knows the story and the other asks questions.
The Vatican’s Anime Mascot Is Now an AI Porn Sensation
Outlet: 404 Media.
Reporter: Emanuel Maiburg.
“My first impression is that it takes a shockingly long time for the podcast to get to the point,” said Maiburg, who covers technology and AI at 404 Media. “There’s this concept of the inverted pyramid where you start with the most pertinent information at the top, and then kind of go into more details as you get deeper into the story.”
With written reporting, the audience can gain an understanding of the story very quickly, often just from the headline, Maiburg said. But the NotebookLM podcast didn’t reach the main point of the story until about four minutes in, ”which is such a long time to get a piece of information,” Maiburg told Straight Arrow. “I think that is a huge waste of time.”
Maiburg didn’t like the editorial and connecting choices that the platform made. The majority of the piece is “unfunny quips, meandering explanations, and circling the point rather than just getting to the point,” he told Straight Arrow. “It’s like zero value, fluff, empty words, is 90% of the podcast.”
He thinks that might be because the platform is trained on popular, banter-filled podcasts, which are not necessarily “useful from a news perspective — or a media consumption efficiency perspective.”
The podcast also misread the humor of the article, which was about AI users creating adult-themed content from the Vatican’s new mascot, Luce.
“It kinds of turns into ridiculing the church,” he said. “That’s not what the article does. So it kind of changes the tone.”
Maiburg said that the podcast could help people learn something about the topic, but that he found it “a super arbitrary, inefficient way to consume this information.
“Like, if I were to make like, a three act, interpretive dance,” he said, “and you went from zero to seeing my ballet about it, will the ballet teach you more? It’s like, yes, but it’s like, why the hell are we doing it this way?”
He said he would prefer to hear the article read aloud, rather than a “15-minute banter-filled podcast that’s pretending to be an affable human host.”
How the search for vaccine safety led parents away from the experts
Outlet: Straight Arrow.
Reporter: Jess Craig.
Jessica Craig is an epidemiologist and the health reporter for Straight Arrow. She was broadly impressed by the way the podcast summarized a complex scientific topic in its rendition of her weeks-long reporting project about how parents navigate vaccine decisions for their children.
“The podcast did a good job of not oversimplifying or swaying the audience one way or another,” she said.
“I purposefully picked this article because it was very nuanced,” Craig said. She had deliberately included a range of scientific perspectives in her article, and she felt that the podcast “really hit on each of those areas without trying to lead the listener to any sort of conclusion.”
The AI podcast did a good job of highlighting each of her sources, Craig said. “I think they really touched on each of the individual stories that I presented, because I included them for a specific reason — they each represented a different viewpoint and different area on the spectrum.”
Craig said she liked the different metaphors that the LLM introduced to connect the topics within the podcast piece.
“I didn’t include some of those metaphors in my piece,” she said, “so it came up with that all on its own and explained some scientific concepts in a good way.” She said that one of the metaphors — about how the immune system acts like a “sleepy security guard” that can be woken up by a “siren” in vaccines — helped explain a technical scientific concept in a way that would help people understand.
“There were times it felt a bit contrived,” Craig said. “It was just so strange that someone else knows your reporting so well and is talking about you and your reporting, and then it’s just hard to get your mind around how that’s actually not even a person.”
What should listeners keep in mind?
Ultimately, Hejabi said, the most important thing to bear in mind when listening to AI-generated content is the nature of that first step in the process, where the model selects what it considers to be important.
Those points “might not be as important as you think,” he said. LLMs are embedded with weights, values and priorities that are individual to each model, and different across companies, subject domains and products.
He also emphasized that “the generated things might sound plausible in general but there might be inaccuracies.”
“Double checking is the most important phase that users should do, no matter how much they rely on this system, whether they have the time or not,” he said. “Blindly accepting the output is something that should not happen.”
Round out your reading
- From Obama to Trump to Biden: When networks said ‘no’ to presidential speeches.
- ‘The end of American science as we know it.’ White House funding plan sparks alarm.
- AI data centers are popping up everywhere. What exactly do they do?
- The Trump administration is accused of breaking a law built to target the KKK.
- One Republican senator is dead, another is still alive. Both are the center of conspiracy theories.
Ella Rae Greene, Editor In Chief
