In the 1960s and 70s, Billy Preston was the musician’s musician. A self-taught prodigy who grew up playing the organ in his Los Angeles church, he was accompanying Mahalia Jackson and appearing on The Nat King Cole Show before he was 11. By high school, he was travelling with Little Richard on his European tour, standing stageside every night to watch support band the Beatles.

By 1969 he had been dubbed “the fifth Beatle” (he was co-credited on the track Get Back), and went on to be one of George Harrison’s right-hand men after the guitarist went solo. “Billy never put his hands in the wrong place,” Ringo Starr says. “He was so great.”

Sam Cooke, Ray Charles, the Rolling Stones, Aretha Franklin, Barbara Streisand and Elton John all hired him; he put out solo records and wrote hits for others, including Joe Cocker’s You Are So Beautiful. But he struggled with addiction and was imprisoned after violent drug-related incidents. He died, aged 59, in 2006.

Now, in the documentary Billy Preston: That’s the Way God Planned It, directed by US TV veteran Paris Barclay, friends and colleagues including Eric Clapton, Olivia Harrison (George’s widow) and the late gospel singer Sandra Crouch tell his story.

The film opens with Preston’s talents on full display: a 24-year-old virtuoso performing at George Harrison’s 1971 Concert for Bangladesh. Harrison announces his band – Starr, Clapton, Leon Russell, Bob Dylan and Ravi Shankar, all playing Preston’s That’s the Way God Planned It. At one point Preston slips out from behind the keys to take centre stage in a joyful praise dance. “I just couldn’t help myself,” he says in a voiceover. “The band was jamming and they was pumping, the people were with us and I just had to rejoice.”

With his gap-toothed smile and infectious energy, Preston’s performance style was as informed by the gospel choirs he grew with as it was Little Richard and his later mentor, Ray Charles. The church gave him a place for his talents to flourish. But as a Black gay man, he struggled to square his sexuality with his Christianity – a fact complicated by a sexual assault he suffered in the church while young.

Like Preston, Barclay is Black, gay and also grew up in the church, playing organ and piano – “Not anywhere near the level of Billy Preston, but that’s what I wanted to do,” he says. Both Preston and Barclay lost brothers in fatal accidents, “and like Preston, I had some really untoward sexual experiences that shaped me and the way I saw the world”.

Preston was famously tight-lipped about subjects other than music, and his homosexuality was hidden in plain sight. Though he wouldn’t refer to himself as gay until later in life, he often brought partners on tour or to recording sessions, introducing them as friends or cousins. Barclay was able to locate some of Preston’s former lovers, though none wanted to appear in the film. “I wish we could’ve gotten them on camera,” Barclay said. “But they verified and allowed us to feel comfortable putting their stories out there.”

Preston attempted to escape his troubled private life by throwing himself into his work. He was a regular on the TV variety show Shindig! and the title character in the 1978 musical film flop Sgt Pepper’s Lonely Hearts Club Band. He had hits including 1972’s Will It Go Round in Circles and his 1979 duet with Motown singer Syreeta Wright, With You I’m Born Again, written by Carol Connors and David Shire. “He was a moving target,” Harrison says. “Billy just kept moving, playing, dancing, singing – and how could you get to that guy and sit him down and say, ‘Hey man, what’s going on?’”

Professionally, “I would say he had more good judgment than bad,” says friend and former Motown executive Tony Jones. “I think the bad judgment came when the career started falling apart.”

With a stalling career and time on his hands, Preston’s addiction to cocaine, alcohol and, later, crack grew too powerful for him to resist. He was plagued with legal troubles throughout the 90s, with charges of probation violation, insurance fraud and assault. Bernard J Kamins, the judge who presided over several of Preston’s criminal cases and eventually sentenced him to prison in 1997 gives his first-ever interview in the film.

“Although [the sentence] was for an assault, I put drug conditions on it because there was no way that this man would have assaulted anyone had he not been so severely addicted,” Kamins says in the film. Preston later wrote to thank him, though he would not manage to remain sober for long after serving 18 months of a four-year sentence. (While inside, he led religious services and a choir.) Despite some briefly successful stints in rehab, in 2005 Preston, who had a history of kidney disease, contracted pericarditis, went into a prolonged coma and died in June 2006 from kidney failure.

Documentaries sometimes omit grimier details, but what makes this film powerful and “human”, as Harrison says, is that it doesn’t shy from them – something she also experienced while working with Martin Scorsese on the 2011 documentary George Harrison: Living in the Material World. On that film, Harrison says, “there were things where I was like: ‘Oh, no, no, don’t – that wasn’t his best time! He was struggling!’ And it’s actually my son who said, ‘You can’t sanctify, Mom. You have to have the dark and the light.’”

But earlier this year, a lawsuit waylaid the Preston film just before its premiere at the South by Southwest film festival in Texas. Preston’s former manager Joyce Moore, her musician husband, Sam, and musician Keni Burke, claimed that the documentary was “about pushing the director’s and producers’ personal agendas and financial benefit rather than the homage to Billy Preston that was sold to plaintiffs”. The suit was eventually dropped after some additional edits were made, says producer Nigel Sinclair: “We did change about a couple of minutes – repetitive things that maybe overegged the pudding.

“There’s a lot of emotion involved in situations like this,” Sinclair says, adding that the discussions between plaintiffs and defendants made them realise “how much they had in common. How [Joyce Moore] worked so hard for Billy, and we’d worked hard to make this film. It was emotional resolution as well.”

Those involved with That’s the Way God Planned It hope that the film will raise important questions of how loved ones can support individuals with drug problems like Billy Preston – or indeed Liam Payne, Barclay says, speaking shortly after the former One Direction star’s death.

“Had I made this before, I probably would have dealt with Cory Monteith very differently,” Barclay says of the actor he directed in a brief stint on Glee, who died from an overdose of heroin and alcohol in 2013. “I knew that he was in trouble – I was a recovered alcoholic. I did have a couple of conversations with him, but I kind of thought, ‘I’m just a guest director here. This isn’t my business.’ If I’d done this movie before [I did Glee], I would have stepped across that line and I would have gone to [Monteith]. I would have tried to help him get the bridge to sobriety.”

As a portrait of an artist, That’s the Way God Planned It is a story about addiction, and the love and patience people like Preston need. Tony Jones says he thinks his friend would love being shown as he is, “to be acknowledged and to be appreciated and have heartfelt sentiments expressed about his plight in life. If he could see this, he would feel understood, and he would know that he really did matter.”

• Billy Preston: That’s the Way God Planned It can be streamed in the US via Doc NYC until 1 December

• This article was amended on 29 November 2024. Billy Preston was the title character, not the lead, in the 1978 musical Sgt Pepper’s Lonely Hearts Club Band.