By the time Elton John retired from touring in 2023, everyone knew his story. It followed a memoir and a biopic, which cemented the Elton lore: how in 1967 jobbing musician Reg Dwight was given an envelope of Bernie Taupin’s lyrics, forging the greatest songwriting partnership since Lennon and McCartney, a baroque new identity, a rampant hit rate of era-defining albums and the commensurate cocaine addiction. Then came sobriety, an emphatic commitment to funding HIV and Aids treatment, finding love with David Furnish; The Lion King, Candle in the Wind, one of the UK’s first gay marriages, two sons. Last year, the documentary Never Too Late told his story again. Just last month, his greatest hits collection, Diamonds, finally hit UK No 1 after 374 weeks. No one could ask for a more enduring brand, a more deserved curtain call. John has different ideas.

“I’m 77. If I don’t push myself, Laura, what’s the point in carrying on?” he says, via video call in early January. “Just be ‘Elton John’ for the rest of my life? Which would have filled me with absolute fucking horror.”

John is sitting next to Grammy-winning Americana artist Brandi Carlile, 43, at his Los Angeles breakfast table. They can see the wildfires outside. “I’ve never seen anything like it,” says John. “It’s apocalyptic.” He wears a navy Gucci tracksuit and pointy blue glasses, more out of habit than for his sight – as right now John can hardly see at all. Last summer he contracted a severe eye infection, delaying the release of his and Carlile’s surprise album – only his second full-length collaboration, after 2010’s The Union with his hero Leon Russell.

John, an impatient workaholic, might disagree about the wait, but Who Believes in Angels? was worth the delay. A hinge-rattling Americana-tinged rock’n’roll album, it’s enough to make you pity John’s piano, presumably reduced to splinters from the ecstatic walloping it gets here. It’s easily the best album he has made this century, a more authentic embrace of the present than recent remixes of his classic hits with Dua Lipa and Britney Spears. “I could do an ‘Elton John’ record easily,” he says. “That’s what I didn’t want to do any more. I wanted a challenge.”

***

After headlining Glastonbury in 2023, says John, “I wanted to put all my old stuff to bed.” He propositioned Carlile, a lifelong fan of his turned confidante. “I was like, Elton, really, I don’t sell records,” she says. “And he’s like, I don’t care – you’re about to now, baby.”

John had a lot of ideas. Maybe they could be Robert Plant and Alison Krauss. Or a modern Eurythmics. Or Patsy Cline and Buddy Holly. “Then one day,” Carlile, a born storyteller, says with a grin, “he says: let’s not be any of that. Let’s just be ourselves – and we don’t know what that is. Don’t write, don’t do anything in advance. And that’s when it got really dangerous.”

Between Glastonbury and entering the studio with Carlile in Los Angeles, John had hit his head, was recovering from a second hip replacement and needed a new knee. But he didn’t want to cancel and let anyone down. “I was very grumpy, I was tired, I didn’t feel well and I was extremely nervous,” says John. “And that nervousness caused quite a few sparks to fly.”

It’s one for the annals of understatement. Carlile appears for a solo interview a few hours earlier, an uncanny experience in which disembodied hands do her makeup as she talks. “He smashed his iPad,” she says, wide-eyed. “He smashed headphones. There were really amazing, classic Elton John outbursts. My lyrics got torn up and thrown on the ground – he goes: ‘Fuck off, Brandi.’ He would yell: ‘Predictable! Cliche!’”

“It wasn’t me being anxious about other people,” says John, “it was me being anxious about me. Once we got through the first three songs, we knew we had something. And I was much more relaxed … but I was a bit of a nightmare.”

Co-writing the lyrics alongside Taupin, with John writing the music, Carlile was equally anxious. “It was unspeakable pressure, because it could be really, fundamentally invalidating of who I am musically,” she says. “When I got signed in my early 20s, I’d read artists trashing one another in the press, and I remember thinking: what if I ever got famous and Elton John didn’t like my music? It was odd, as a woman past 40, to have that feeling again as it was about to happen in real life.”

Taupin was in the studio; Andrew Watt, who led the Rolling Stones’ comeback Hackney Diamonds, was producing, a copy of John’s live album 17-11-70 positioned above the desk as a reminder of the energy he wanted. John wanted to do a “proper duet record”, he says. “Not one line here, one line there, I wanted to harmonise.” But not only was singing to Carlile’s phrasing tough, he reacted to her and Taupin’s lyrics as he first encountered them at the piano, prompting on-the-hoof rewrites.

“He would start singing and cutting shit or saying words the wrong way, and I would get in and fix the issue,” says Carlile, pulling an oh God face. “That’s what Bernie’s job has always been. You can call it momentum or extreme impatience – they’re the same. You don’t stop Elton, you frantically replace the lyric in front of him and then he’s singing that. It was really volatile, but really cool.”

When John blew up, Carlile never told him that he had crossed a line, “but there were definitely points that I felt that”, she says. To defuse things, Taupin – “poetic and cryptic, but also really paternal and helpful” – would take her for whisky sours and steak at his favourite diner. John’s outbursts, however, served a purpose, making a hero human-sized. “Elton doesn’t want to be put on a pedestal,” she says. “He doesn’t like listening to stories of my childhood love for him. There was so much riding on my perception of Elton versus who Elton really is.”

***

Carlile first learned about John when writing a school report about Ryan White, a teenager who died from Aids in 1990 after contracting HIV from a contaminated blood treatment. John supported his family, which inspired the newly sober musician to dedicate himself to HIV and Aids healthcare, feeling that he hadn’t helped enough while strung out at the height of the crisis.

Growing up in Washington state in the 90s, Carlile “lived in a mobile home with conservative parents”, she says. “I’d never met a gay person in my life. Didn’t even really know what a liberal thinker was. Also, Elton had got sober; my dad had attempted to get sober and he didn’t get through it at that time. He represented the exact opposite of what my life was.”

She had never even seen a picture of John until she borrowed a biography from the library. “I was like, what the fuck is going on with this guy! He’s amazing!” While her classmates had just about grasped John’s existence from The Lion King, the teenage Carlile, now out, dressed in a white satin suit to perform Honky Cat at a talent show. She wrote John and Taupin letters; aged 18, she went to LA to meet a producer who shared management with Taupin. “He had a desk at the manager’s office,” she says. “I had a little Polaroid camera, I took a picture and sat in the chair.”

Carlile did get signed, breaking through as a singular lyricist with a gale-force voice with her 2007 album The Story, produced by T Bone Burnett. Soon after, she sent John another letter, during his Vegas residency, which led to him appearing on her 2009 album Give Up the Ghost, and to a friendship. After she married charity co-ordinator Catherine Shepherd in 2012, “something changed”, she says. “He had just had kids and knew that we wanted to. He wasn’t just supportive of my music any more; suddenly he was really invested in my personal life.”

Their relationship – in which John might call several times a day to gossip, then hang up – inspired Carlile to relate to her parents anew. “Elton showed me how to have a relationship with people I really love when I don’t have time to labour over all of life. I call my parents now every chance I get, but I’ve only got two minutes – and that’s OK.”

To listen to John and Carlile talk on their joint call is to be caught in the crossfire of an ardent love-in. He calls her “phenomenal” and insists “she pushed me more than I pushed her”. They bicker lovingly over which one of them is really “saving” the other. In 2019, John watched Carlile perform Joni Mitchell’s Blue in Los Angeles, in front of Mitchell, and was “absolutely floored”.

“He called me a crazy bitch!” Carlile laughs.

“I thought, I’ve got to work with this person,” says John. “She wants to take risks. I’d have been shitting myself.”

Carlile, famously, had coaxed Mitchell to start performing again after a brain aneurysm in 2015 left her unable to play. John joined their backyard “Joni Jams”, which he likens to “an old postwar sing-song around the piano. If you don’t sing, you can’t come.” In October, they sang I’m Still Standing together (albeit sitting) at the Hollywood Bowl. “Music brought Joni back to life,” says John. “To a certain extent, and she may say no, but Brandi brought me back to life.”

“I do say no,” she says, “but I love that you say that.”

***

Their unvarnished connection inspired Brandi to write about the experience of being in the studio together, inspired by Captain Fantastic and the Brown Dirt Cowboy, lyricist Taupin’s 1975 portrayal of his and John’s friendship. The rapturous title track and stirring Someone to Belong To cherish the rewards of their brutal honesty; Never Too Late crystallises John’s capacity for survival. “Fuck off, heaven’s gate!” he sings, stealing the “fuck off!” mantle from Succession’s Logan Roy. “That was what I was thinking!” says Carlile. “Brian Cox should not have that crown. That’s Elton’s crown.”

A wider theme coalesced. The album’s baroque curtain-raiser, The Rose of Laura Nyro, pays homage to the sorely underrated US songwriter, one even less remembered as an LGBTQ+ iconoclast. As a struggling young musician, John pored over the lilac-scented lyric sheet to her 1968 album Eli and the Thirteenth Confession. “I was astonished by the variety of her songwriting and the fact that the songs didn’t follow the normal pattern,” he says. “They slowed down, they sped up. She never really got the dues that she should have.”

“Joni loves her too,” says Carlile, “and Joni’s a hard one.”

“Joni doesn’t like many people!” John stage-whispers, leaning in, and they crack up.

Then the ripping Little Richard’s Bible portrays John’s oft-cited hero swerving from outrageousness back to religion when the money ran out. It’s an opening salvo that celebrates LGBTQ+ trailblazers, but it’s not straightforward: highlighting forgotten histories and self-doubt, even self-loathing.

“Conceptually, I think that’s really important and I’m really proud that you noticed that because it is a huge part of the narrative to me,” says John. “You’ve got these really tortured and complicated LGBTQ icons, and then you’ve got the people that have reaped the benefits from that, hopefully sending a message to the people that can have an easier path because all of us have existed.”

Has the complexity of John’s radicalism been underestimated, made cuddly by his national treasure status? Many people will know of his Aids Foundation’s glamorous Oscars viewing party, for example, but less so the fact that the organisation has made HIV testing available in Walmart. “I think what Glastonbury did for me, and for my reputation in England, was something I’d yearned for all my life,” he says. In America, he feels better understood. “I know what you mean about the cuddly … That’s why I wanted to put the old Elton to bed.”

He gives himself credit. “When I was the gay chairman of a football club” – his beloved Watford FC – “we were very successful,” he says. “That was pretty radical. But I’m not Bowie, and I’m not Mick. And I’ve never been ‘cool’ like Nick Cave – who I adore. I’m just me and I have to accept that, but that gives me the inspiration to change a little bit with this record and say: come on, you don’t want to be cuddly Elton for the rest of your life.”

Curiously, his press team, present on mute, seem less keen. Last year, John remarked that Donald Trump calling North Korean leader Kim Jong-un “rocketman” was “brilliant”, adding that Trump was a fan: “I’ve always been friendly toward him and I thank him for his support,” he said. As he later clarified, clearly they have different values: does John have to be diplomatic to maintain presidential support for Aids treatment? The muted windows spring to life in distorted protest.

“I can answer this,” says John, as a voice says: “We’re at time now.” (We aren’t.) “I have the Aids thing to be very aware of, and Pepfar [the US president’s emergency plan for Aids relief globally] is very important for continuing to help people who need the money. And so far, it’s always been kept up by the American government, Trump included; Democrats, Republicans. It was started by a Republican. I have to be diplomatic and fight for those people. My job is not to take sides, my job is to try and keep the money flowing so that people can live a normal life.”

Three weeks after we speak, Trump freezes Pepfar and instructs foreign organisations to stop distributing HIV medication bought with US aid, with appointments being cancelled, treatment interrupted and patients turned away from clinics. In a statement, John’s Aids Foundation urged the US government “to fully reinstate all global HIV funding as soon as possible” and said the country’s “history-defining” gains in the fight “may be hard or impossible to recoup”.

During the interview, I paused to see whether the PRs were trying to say anything important. “It’s just that they wouldn’t want me to talk about politics, but it’s a good question and it’s what I have to do,” says John, jumping in. “Politics is always about tactics and playing your cards right. People rely on me – and a lot of other people – to try and keep the money coming and that’s what I have to do and I’m determined to do.”

***

John’s determination is not limitless. One bittersweet new song, Little Light, reflects on a moment he and Carlile shared at this very breakfast table. “She was staying next door, she came in and I was pretty down,” says John. “It was Israel- Gaza. I said, look at this, the world is fucked. I don’t know why we’re in the studio recording when this is going on. She wrote the song that day and we recorded it that day.”

“He was contemplating not doing the album,” says Carlile. “He was like, maybe this is not the time. My verse is what I wish I’d said.”

“It’s such a fine line between faith and apathy,” she sings, evoking the Italian philosopher Gramsci’s saying about having pessimism of the intellect but optimism of the will. “You could tattoo that quote on him,” Carlile says of John.

“I’m an optimist permanently,” he says. “I’ve got bad eyesight at the moment. There have been days where I’ve been miserable, but then I think about how lucky I am. I’ve got a new respect for sight-impaired people and people who are totally blind, some of whom I’m in touch with. But to hell with it, I’ve got a great life, and hopefully my sight will improve. What she’s saying is I’ve overcome a lot. This is not going to beat me. I’ve beaten addiction, I’ve beaten health issues, and I can pick myself up and dust myself off.”

Overall, Who Believes in Angels? looks resolutely forward. Carlile’s Swing for the Fences – a more riotous riff on the theme of her Grammy-winning 2018 anthem The Joke – is a Stonesy cry for queer youth to break the rules. Between that and You Without Me, her gorgeous, one-take solo song about watching the elder of her two daughters become her own person (“that’s the feeling every parent gets – you’re proud and devastated the first time they decide not to go your way”), the album suggests that a key part of legacy is creating the space for new generations to write fresh scripts.

As John did with Nyro and Little Richard, ardent admirers such as the pop star Chappell Roan have taken his influence somewhere new, beyond him. To that end, the album closes with Taupin’s valedictory ballad When This Old World Is Done With Me. As ever, John first tackled the brand new lyrics at the piano. Carlile typically sat behind him, so close her knees touched his bum, ready as the “butterfly catcher” to sing along with great melodies that John might otherwise let slip by – watching him come to life off the page, as she had as a kid. “He got to the line, ‘return me to – the tide’,” she says, adding a stumble, “and I go, ah shit, he’s gonna cry.”

John had realised the song was about the end of his life, something only a friend of six decades could have written. “The glasses came off, his whole body is crumpled over the piano, he wept,” says Carlile. “I put my arms around him and held him and he didn’t really talk about it. He’s a bit butch, a stoic, a man’s man. Andrew [Watt], he’s very dramatic, like a Jewish grandmother. He comes running in with tissues. We’re having this really profound moment where Elton has come face to face with his mortality without really being able to say it.

“When he looks up, we’re like, what’s he gonna say? And he goes: ‘Is it too Lion King?’

“I said, ‘No, no!’ Andrew burst out laughing. Elton burst out laughing. It was unbelievable. At first, it was the saddest thing I’ve ever seen, and then it got really funny, really fast.”

“It touched a nerve,” says John, “because I have children now and I have a husband, and it’s not just being ‘Elton’ on his own, it’s wanting to be with the people I love.” He asked Watt to put a brass band over the ending. “It’s beautiful, spiritual music. Something about a brass band moves you so much – colliery bands with the miners; watching the cup final as a kid it was always Abide With Me with a brass band, and I always used to cry.”

John recently had the chance to rehearse his own death, filming a scene for Spinal Tap 2 in which he’s killed off in concert. That was the fun bit, but driving back into a huge auditorium, having quit touring, gave him a panic attack. “I did kind of hyperventilate,” says John. “I said, ‘Jesus, I’m so glad I don’t have to do this any more.’”

Carlile differs. “I want to die on stage,” she says.

“I used to say that,” says John, “but I don’t want to. Well – I can’t. I’ve retired.”

The perks of being off the road, he says, are that he can “throw out the shit and just pick the good stuff”. As well as continuing to treat his sight, he’ll heartily promote this album and keep doing his Apple Music radio show, highlighting new artists. There’s talk of a biographical stage show, but he’s not sure.

“My husband deals with all of that,” he says. “I don’t want to do too much, but what I want to do should be interesting and different. I don’t want to repeat myself and I think this is what this album is all about. It’s energised. And I have that energy now. What I’m going to do with it, who knows? But it’s lucky at my age and with the career I’ve had to feel as if I have something more to do.”

• Who Believes in Angels? is released via Island/EMI on 4 April