AI can give you your voice back

Episode 3 transcript

Laura Sanders: Jules Rodriguez’s voice was disappearing. 

Jules Rodriguez: For me, it felt like I was fading away, like when Marty McFly was seeing his family fade away in the picture he had in “Back to the Future.” 

Sanders: Jules has ALS, and like many people with that disease, this fading felt inevitable. But there was something Jules could do about it. I’m Laura Sanders, and this is episode three of The Deep End: Talk To Me, a Science News podcast on the human voice and all it can hold.

[voices] Hey mom, can I tell you something? The voice is absolutely an integral part of your identity. Do you want to talk? That added layer of it being you. Hello? Your voice. Hi.

Sanders: On today’s episode, we’re going to hear about the return of Jules’ voice. 

[voice] Your voice is you. 

We touched on this in the last episode, but as a reminder, there are a few traditional ways people preserve their ways to communicate when they’re faced with losing a voice. One is for people to record themselves saying common or special phrases if they can still talk. That’s called message banking. And then there’s a technique called voice banking. It requires a person to record themselves saying very specific words. And then these sounds get fed into a computer program that can stitch them together to make any word a person wants to say. Sounds good in theory, but in practice, the output didn’t sound very human. Jules banked his voice soon after his diagnosis. MA describes what that was like. 

MA: I remember both of us looking at each other while he was recording, basically the old school way, and they still recommend to bank your voice this way, besides your cloned voice, but you’re sent all these phrases that you’re just recording, like 350 phrases, and that goes into, you know, a file, and then when you do need it, the company will pull it up and plug it into your device. 

Sanders: I spoke with Richard Cave. He’s a speech therapist at University College London who works with people with ALS. That’s also called motor neuron disease, or MND. 

Richard Cave: My role was to build out the voice banking initiative across the UK. This was back in 2017. And I remember… After the first, I don’t know, six months, nine months, being very proud that I managed to put through 100 people or so at an average duration to complete the voice banking, all 1,600 phrases in around three months each. Very proud of that. They weren’t awful, but they weren’t good. And they didn’t really represent the person. They represented kind of part, a little bit, of the person. But that was the best we had. 

Having decoded the signals from some input device, such as the switch, one then has to broadcast it in a form people can understand. This could be visually, with text or diagrams on the screen. But if you want to communicate like others, which is what people in my position would like, you need to speak. 

Sanders: The late physicist Stephen Hawking had one of perhaps the most iconic computerized voices. Here he is giving a talk in 1994 at MIT.

[Hawking audio] Computerized speech synthesizers have improved a great deal in recent years, and this is important. Not only does one want to be understood, but also one doesn’t want to sound like Mickey Mouse or a Dalek. 

Sanders: He’s describing his synthesized voice, which it turns out, he quite liked.

[Hawking audio] This voice of mine may be a bit tinny and Irish, but it is almost human. 

Sanders: Hawking was happy with that voice. Jules was not. Jules described his banked voice in a stand-up routine as sounding like if you played Daft Punk at quarter speed. 

Harder, better, faster, stronger.

[Sanders voice clone] Okay, fast forward a bit, artificial intelligence came onto the scene. Now, with just short snippets of already recorded sound, these powerful programs can create synthetic voices with appropriate umms, breaths, sounds, and cadence that are very close to the original. These voices are so realistic that sometimes you can’t even hear the difference.

Sanders: Maybe you just noticed something, but maybe you didn’t. Those previous three sentences weren’t spoken by me at all. They were said by a voice clone I made. Okay, one more time. This is me talking. 

[Sanders voice clone] And this is my voice clone.

Sanders: I used ElevenLabs to clone my voice. That’s a company that uses AI technology to create lifelike, digital replicas of voices. To do it, I used the professional option, which required 30 minutes of audio to make. So for input, I recorded myself reading a few podcast scripts. 

Jules Rodriguez’s voice was disappearing. 

Sanders: Some really long texts from my mom. 

I’m at Food Lion. I’m getting pickles, half and half. 

Sanders: And chapter nine of Anne of Green Gables. 

It’s a good thing Rachel Lynde got a calling down. She’s a meddlesome old gossip. 

Sanders: After a few minutes, poof, there it was, my voice clone. It was spooky how much it sounded like me. I spent a fun afternoon playing with it. You can control so many different parts.

[Sanders voice clone] It can speak quickly. And it can speak slowly. My voice clone can laugh. It can be utterly bored. It can also scoff when it thinks something is ridiculous. It can be menacing on Laura’s behalf.

Sanders: I could even speak fluent French.

[Sanders voice clone] Il peut même parler d’autres langues.

Sanders: The technology uses a few different types of artificial intelligence at different stages. It extracts features of my voice, such as my pitch and tone, my prosody, even my breathing patterns. This lets it build a model of my voice, and then that model can be let loose to say anything I type. I got the idea from speech pathologist Kate Nilson, who has tried it herself. 

Nilson: We’ve even played the game of like, is it really me talking or is it the computer? And it’s kind of scary how realistic the computerized version of my voice is. The really awesome part about the voice cloning process is that you take recordings and submit that. It’s not like you record it right now. You take old recordings. And so for people you know for instance people living with ALS, if their speech is the first thing that’s affected and their speech has already changed, it might be too late to do the recordings, but if they have old voicemails or a speech that they gave or a podcast that they were on, something like that, then we can take those recordings and submit them to make a voice. And so that’s really opened a lot of doors for a lot of people. 

Sanders: In June of 2024, Jules used recordings of his own voice to create an AI-generated voice clone, also through the company ElevenLabs. 

Jules: When I first heard my cloned voice, it felt like I reappeared. When my wife first heard it, she cried because of how close it sounded like me before I lost my voice. 

Sanders: After months of gradually hearing Jules’ voice less and less, his family had gotten used to the new normal. The voice clone was a big change. 

Jules: They were all shocked at first, but now it just feels natural. It’s pretty crazy. 

Sanders: When you first started using the voice clone, do you remember what the first thing you said was? Were there things you really wanted to say?

Jules: Well, it was probably the cheesiest thing. I remember telling my wife how much I love her. 

Sanders: Here’s how his wife MA remembers it. 

MA: The first thing he played for me, I was like chills all over. And it was, it was incredibly beautiful and incredible and like, wow. And also I cried because it felt like there was a part of him that I was getting back that I had lost a long time ago.

Sanders: It was surprisingly hard to get this one thing back in the middle of so much loss. 

MA: It felt confusing emotionally to get that part back and not the rest of him back. Like, I just I feel it right now in my body as I talk about it. Things with ALS change so quickly and the voice is something so intimate you’re sort of like, you’re training yourself little by little to adapt and then all of a sudden this like really vibrant part of him came back, but everything else was the same. But that vibrant voice part came back and it was like, yeah. It was a bizarre experience but ultimately really, really special for us and for his son too.

Sanders: Jules immediately took off the training wheels.

Jules: I just wanted to explore the limits of it.

Sanders: In fact, Jules’ voice clone sounded so good that he decided his dream of doing stand up comedy was possible, long after he had given up the idea. 

Jules: I started to write jokes with it pretty soon after hearing it. 

Sanders: So Jules started working on a set. 

MA: When he tested the AI voice, it’s like a little light bulb went off and he was like, I  think I could do comedy again. And I remember hearing that and going like, oh, my God, like that’s such a huge like thing to, to, it’s like you come alive in a new way, is what I saw through his eyes. He was like, I think I can do comedy again. I was like, wow. 

Jules: Oh, man, thank you. I wasn’t even sure if I was going to make it out here. So many things could stop me from being here. Hospital stays, tracheostomy, broken elevator. And the biggest thing would be any step larger than two inches. I’m so happy to be here with you all. 

Sanders: It’s tempting to think that this personalization shouldn’t matter much, that having a voice, any voice, would do the job. But it’s not true. Having a voice that sounds like your real voice does matter a tremendous amount.

Karina Nagin: You need to communicate what you’re thinking, your health directives. You need to tell your loved ones you love them. But I think we didn’t realize what a huge impact your voice sounding like you, what a big piece of identity that was.  

Sanders: Karina Nagin directs Bridging Voice, a nonprofit organization that helps people with ALS find and use communication technology. That includes voice clones from ElevenLabs. It’s the group that helped Jules clone his voice. She’s heard stories about how voice clones and the connections they bring back led people to reconsider their medical directives. 

Nagin: And people, once they received these voice clones, it was so life-altering that they were changing their health directives of what they wanted to do. Many people changed their health directives. They said, if I have this voice and I can sound like me, I can continue to communicate and feel like myself, I want to live longer. This is something that makes it worth living for me. I think our team didn’t realize how impactful it was going to be, but we’ve had people use these voice clones to tell their wedding vows. I got a testimonial from a grandmother the other day who just got her voice clone, and she said, I immediately called up my grandchildren, the youngest of who had never heard my real voice. And so this is the first time they were hearing my voice.

Sanders: Nagin says her team was surprised by the impact on not just the person whose voice is restored, but on their families too. 

Nagin: We were so focused on people and giving them the ability to communicate. The idea that it sounds more like them or not, we said, oh, that’s a nice add-on, but that’s not really the core of it. But as soon as we gave that to people, we realized that is the core of it, your identity and feeling like yourself. And it’s not just for the patients, but it’s their family members. 

Sanders: A personalized voice, one that sounds like the old one, is more than an add-on. 

Nagin: Endless videos of people. I mean, we have one of a college-aged boy who’s hearing his father’s voice for the first time in two years, and he just breaks down. Like, this is, you know, they’re going through this horrific experience of this diagnosis. And to be able to actually connect with a family member and hear their voice again, I think it equally brings joy and comfort to the families as much as it does to the person who’s been diagnosed with ALS.

Sanders: ALS forces families to confront tough medical decisions about life and death. Jules and MA had thought very carefully about how they’d approach the future. What kind of medical interventions would he be OK with, and what kind would he not want? Big picture, what sort of life was worth living? Jules had been getting by with just an external oxygen mask to help keep his body going. He had decided that he didn’t want to take a next big step of having a tracheostomy, a life-extending procedure that leaves a small tube in his throat connected to a ventilator. With a trache tube, Jules would need round-the-clock care, and he would lose his voice permanently.

MA: He was someone who didn’t want to do a tracheostomy. Like, he was opting for, once I’m done, like, that’s it. And that was our plan. 

Sanders: In the summer of 2024, not too long after he had cloned his voice, Jules had reached a very tough spot.

MA: September, we went to the hospital again and we had one evening where we really thought like that was it, like he was going to, he was going to pass away that night. I remember texting his family and saying, you know, I don’t know that it’s going to happen right now, but we’re, I feel like we’re really close. And he went through this like crisis moment. And then on the other side, when they were able to like stabilize him again, he looked at me and he was like, babe, well, if the tracheostomy can help me live, then I think I’d change my mind. I want to do it. Let’s talk to the doctors about it. 

Sanders: This decision to have a trache tube placed in his throat meant that Jules’ voice, diminished as it already was, would be gone. Remember, this is a couple who built their relationship on long, deep talks. Their intimate, funny, thoughtful conversations were so meaningful to them that they started an entire podcast based on them. But this decision, a proactive choice to fully and permanently lose the capacity to talk, represented a hard stop. A definitive end to talking. What you’re about to hear is a tough conversation. MA is telling me about a moment before the surgery. 

MA: So I remember the night, the day, yeah, he was, the operation was like in the evening when they did it, and me recording like a last little video of us talking because I was like, I just want to capture your voice one more time. And he was like, oh babe, you know, being all like, and I haven’t even looked at that video, but it’s there. And so that was a big, it was a big transition. 

Sandes: Yeah.

Sanders:  You can’t see it, of course, but maybe you can hear it. As we talk, we’re both crying.

MA: I think I mentioned when we spoke last time how of all the things from the very beginning that I was like, oh, he’s going to lose the ability to this and this and this and this and this. I was like, oh, my God, when he loses the ability to speak, like, I’m going to really, really, really, really, really have a hard time. And all of the abrupt changes and the way you have to adapt to them kind of prepare you for the next things. And so usually the fear of the thing happening is more than when it actually happens. And then you’re there and you’re like, okay, we’re here and we’ll adapt to it. That’s usually been the case for pretty much everything. But yeah, it was big.

Sanders: But there was something that made the final and complete loss of Jules’ voice easier. Jules’ new digital voice. This was a very realistic AI-generated recreation of his old voice, the one that his family knew and loved. The existence of this new voice factored into Jules’ decision to get a trache.

MA: I really do believe that the AI voice and the ability for him to actually communicate in that way is one of the things that helped him decide to keep living. 

Sanders: It wasn’t everything, of course. A decision like this depends on so many moving pieces. But it was big.

MA: I firmly believe that it had an enormous positive impact in him considering it, like, as a way of you know having certain goals and experiences that he thought, you know, wouldn’t have been there, had it not been for the cloned voice. So I’ve I’ve gone as far to say that if it wasn’t for the clone voice, I don’t think he, I don’t think he would have chosen the trache. Yeah, then that moment anything could happen, but he was pretty, a pretty like no, I don’t want the tracheostomy kind of guy from the beginning. And then all of that changed. And so I think it was a huge driver.

Sanders: At the time of Jules’ hospitalization and surgery, they talked to their son Skylar about what was going on. 

MA: The whole like communicating with him afterwards, because I said, you know, now he’s going to lose his voice completely. He’s like, we have the computer. That’s great. And I was like, okay, yeah, true. The things he’s told us over time are like bonkers. How he sees, the perspective he has on things, is incredible. So that was a really big, pivotal time for like what it means to like use your voice, to keep your voice, to fight for your voice and to express yourself in ways that you didn’t even think you would before. That takes a lot of courage and willingness. 

Sanders: And what is your voice? I mean, to Skylar, it didn’t matter because his voice was still there. 

Sanders: To Jules’ son, the voice clone was his dad’s voice. That’s really all there was to it. 

MA: He had his tracheostomy October, October 1st, and he was in the hospital for another two weeks. And then, mid-November, he was performing at, you know, an improv at one of our local comedy clubs.

Jules: In all seriousness, thanks for coming out. My name is Jules and because of the advancements in AI, I now have the ability to come up here and do this again with my original voice.

Sanders: Jules was back. Jules, with his voice clone, can laugh and sing and start trouble with his wife. 

Jules: One thing in particular is how it inserts tone and inflections on words. Sometimes my wife will get mad because my response can come off with some sass.

Sanders: I asked Jules if he can swear with it. 

Jules: [Beep] yeah, I can. With the latest version of this technology, I have more control of how I sound with prompts. That includes laughing, whispering, and even singing. I could never sing, even though my wife and I met at a karaoke bar. 

Sanders: I think you can sing. Anyone who wants to sing can sing. 

MA: He could totally sing.

Sanders: Cave has been a speech therapist for decades, but even he was won over by the capabilities of these newer AI-generated voice clones.

Cave: I was really pleasantly surprised. These voices were the most authentic sounding I’d ever heard from this technology. It actually sounded like the people I was working with. And so it really changed my world. And because I was known for voice banking, I was getting a lot of inquiries. And I ended up spending so much time on answering questions about ElevenLabs. What really, really excited me was that many people reported that their social networks was being strengthened again, that more people were coming to talk to them and to reflect on times past and times current. And that to me is gold. That’s what voice banking is or should be, to maintain social and work networks. 

Sanders: I’d say that we’ve now gone well beyond almost human, as Stephen Hawking put it. Today’s voice clones are fluid, individual, and in many ways, seamless. My synthetic voice says certain words the exact same way I would say them.

[Sanders voice clone] My real voice says certain words the exact same way I would say them. It’s spooky. 

Sanders: The stories I’ve heard of how people are using these voices back this up. I emailed with Peggy Merrill, who lost her voice from ALS. She uses her voice clone to read to her  grandkids, teach yoga, and lead guided meditation. 

Peggy: My name is Peggy and it is my honor to share this guided yoga nidra meditation with you. 

Sanders: Peggy wrote to me, “It renews my sense of self to use and hear my own voice.” And there’s Patrick Darling, who you heard from in an earlier episode. He’s a composer and musician, and he’s been writing new songs and performing them with his voice clone. 

Patrick Darling: The first time I performed with the AI voice, I was surprised how physically connected I felt with it, even though I was not actually singing. 

[music] 

Patrick: It’s hard to explain. There’s just this strange energy about it and you feel like a performer again.

Sanders: Patrick’s description of feeling physically connected to his voice, I think, might be a big hint about why this voice cloning technology is so powerful. These realistic voices are able to create connections both within people and between people. These connections were lost, and then they’re remade. And they matter because of not just the big heavy conversations, but the little everyday stuff too.

Jules still has to use eye-tracking technology to type out what he wants to say. He has a device that senses where he’s looking on a screen and uses his gaze to write. Only then does his voice clone say what he wants it to say. That takes a lot of time and focus. Scientists are working on ways to get the words out faster by using signals earlier in the speech process.

On the next episode…researchers want to pull words straight from the brain. 

Casey Harrell: And the more I read and I thought, why not me? And why not now? 

Sanders: What does it feel like to have your thoughts turned into words? 

Leigh Hochberg: The first thing that he said was something along the lines of, you know, is this thing on? And then the second thing that we decoded was, I’m looking for a cheetah. And it’s the second day. You know, there’s no way that the second sentence on the second day of using an implanted brain computer interface is going to say anything about a cheetah. You know,  that’s obviously a decoding error, and that’s fine. Except for the fact that his four-year-old daughter had just walked into the room dressed as a cheetah.

Sanders: This is The Deep End. Talk to me. I’m Laura Sanders. If you like this podcast, tell your friends and leave us a review. It helps the show a lot.

Send us your questions and your comments at podcasts@sciencenews.org.

The Deep End is a production of Science News. It’s based on original reporting by me, Laura Sanders. This episode was produced and mixed by Ella Rowen. Our project manager is Ashley Yeager. Nancy Schute is our editor in chief. Our music is by Blue Dot Sessions. The podcast is made possible in part by the Burroughs Wellcome Fund with support from PRX.