‘It was the exact spot where pop art and pop life came together,” said Andy Warhol of Max’s Kansas City. “Everybody went there.” Indeed they did – from painters to poets, musicians to movie stars, and politicians to drag queens. A baby elephant was even photographed in there once.

Almost 60 years since it first opened its doors as a restaurant in New York in 1965 – “steak lobster chick peas,” read the sign – Max’s Kansas City has become legendary. “It was just where we hung out with friends,” says Peter Crowley, who booked bands for the watering hole and eatery. Crowley has now written his memoirs, Down at Max’s, with a focus on what many consider to have been New York’s wildest nightclub. “But looking back, it was responsible for the cultural future of America. It was a place where anything could happen.”

Or more specifically, it had a room where anything could happen. The coveted back room was a VIP hothouse with an anything goes policy. Danny Fields, the manager of Iggy Pop and the Ramones, was one of the earliest to frequent it, describing it as “the most desirable place to sit in New York City”. He was not alone in his praise. “A million ideas were launched in that back room,” said Alice Cooper, while Jimi Hendrix called it a place where “you could let your freak flag fly”. Fashion designer Halston dubbed it “a constant happening” while to William S Burroughs it was “the intersection of everything”. Patti Smith labelled the venue “a social hub of the subterranean universe”, while Lou Reed judged it “the most democratic meeting ground imaginable”. Reed also called it “the home of many a career-to-be and life-to-end, and drug casualties in the extreme”. The extras for the famous party scene in the 1969 film Midnight Cowboy were recruited from the hip yet debauched Max’s crowd.

There was huge cross pollination in terms of creativity, ideas and people. Jane Fonda might be nestled next to a drug-dealer, drag artist Jackie Curtis could be sat chatting with a member of the Kennedy family, while John Lennon might be chewing Alice Cooper’s ear off about politics. But it wasn’t always convivial. “Me, Iggy and Lou Reed were at one table with absolutely nothing to say to each other,” David Bowie once said. “[We were] just looking at each other’s makeup.”

It could also be something of a free for all, with naked performance art shows, people openly injecting speed, and Jim Morrison urinating into wine bottles. “There were enough drugs in the back room to cause genetic defects,” photographer Derek Callender once said, while Reed, no stranger to chemicals, made this observation: “Some of these drugs were so new they weren’t illegal yet.”

The Velvet Underground’s final live show before Reed left the band was at Max’s. When it was subsequently released as the Live at Max’s Kansas City album, it also captured the musician and poet Jim Carroll trying to score drugs, as he was holding the microphone for the recording. The writer Steven Gaines described the vibe of the back room in those days as: “Kinetic and rubbery, people bouncing off the walls, skittering table to table, drink to drink, drug to drug, ashtrays filled with endless smokes, an occasional handjob under a napkin, a blowjob under a red tablecloth.”

It was never intended to be such a place. When Mickey Ruskin, a Cornell-educated attorney, opened the nightspot on 213 Park Avenue South, he happened to have some artists follow him from a previous coffee bar he owned. “We called them the abstract expressionist heterosexual alcoholics,” Danny Fields recalls of the likes of Willem de Kooning, John Chamberlain, and Dan Flavin who would prop up the bar in the main restaurant and rack up staggering tabs that Fields claims could reach $70,000.

They would use their artworks to barter for payment, and so Ruskin’s collection became gallery-like. “Things worth $2,000 then would be $20m now,” says Fields. Some of Flavin’s work has since sold for millions, and an edition of the red neon light sculpture that lit up Max’s swinging yet sordid back room – which people stuck chewing gum on – later sold for $662,000.

The glamorous pop art superstars and entourage started frequenting Max’s due to its proximity to Warhol’s factory. Then the counterculture and rock’n’roll crowds followed, often encouraged by Ruskin’s renowned generosity. “Mickey was personally responsible for my survival for three years because he fed me every day,” Lou Reed later said. “It made it possible for me and a small army of other artists to exist just to the left of the line that defines more extreme modes of criminality – the difference between car theft and a stabbing. It’s scary to think what would have happened to me without Mickey in my life.” Alice Cooper also owes a similar debt: “I probably lived on chickpeas and black Russians”

By 1970 Cooper, the Stooges and the Velvet Underground were all playing there, with the Velvets doing an epic two-month residency. By 1972, the New York Dolls were pretty much the house band, and everyone from Tom Waits to Big Star via Emmylou Harris and Gram Parsons performed there. “There’s David Bowie and David Johansen [of the New York Dolls] putting their heads together in the back room, then upstairs there’s Iggy smashing a bottle and gouging at his chest to open up and bleed,” recalls Patti Smith Group guitarist Lenny Kaye, who also remembers catching Bob Marley and the Wailers opening for Bruce Springsteen. Not bad for the shabby upstairs room of a restaurant that held 50 or so people. “It was just another night out on the town,” Kaye recalls. “But in retrospect, it seems truly remarkable. You were at the centre of the universe.”

Tommy Dean Mills and Laura Dean took over in 1975 after Ruskin could no longer keep up with his huge debts. For some people this was the end of an era, but for others it was the start of a new one. Crowley was brought in as promoter and Max’s, along with CBGB, became a destination for the burgeoning punk movement. Bowie introduced Devo on stage as the band of the future, and the likes of Suicide, Television, the Cramps, Blondie and the Ramones all performed. “When Peter started booking, they had the weirdest shit you’ve ever heard,” says Lydia Lunch, who at the time was playing in the no wave band Teenage Jesus and the Jerks. “Stuff that was just so out there – it was pretty special.”

There was some overlap between Max’s and CBGB, although many held firm allegiances. “Max’s was a classier place than the dog shit-ridden stenching Bowery bar that CBGB was, with the most foul bathrooms you’ve ever seen,” says Lunch. Aside from being a party mecca, Max’s was also a place where artists could push their limits. “It allowed you to experiment with your music, take chances, screw up and figure it out,” says Kaye. “It was like a laboratory.”

Things got darker, heavier and rougher as the years went on. Mills started counterfeiting $100 bills in the basement of the club, which he was later jailed for. “It started getting sketchy in the early 80s,” recalls Kaye. “It was a haven for kids from out of town wanting to get wasted.”

The venue was in decline and debts were rocketing. “The last year in Max’s was tragic,” says Crowley. “It was dying of neglect – 1981 felt like a long illness, and then finally it died.” The final show was a clear indicator of where music and culture were heading next: the hardcore band Bad Brains, supported by the Beastie Boys, gave the last performance.

The original spirit of Ruskin’s era lives on though, through the Max’s Kansas City Project, established by his wife Yvonne Sewall-Ruskin to provide grants and funding for struggling artists. And Max’s clearly remains a deeply special place to those who were at the core of its scuzzy charm. “I still feel a sense of loss and nostalgia,” says Kaye. “I sometimes go into the deli that’s there now and walk into where the back room would have been to buy a beer and celebrate a holy site of New York’s artistic creativity. It was a beautiful run.”

Fields, too, is heavy with love and sentiment for the place and what it hosted and harnessed. “What an astonishing historic coalition of life, beauty, art, comedy, drama, glamour, sex, fame and fun,” he says. “My life would not have been remotely the same without it. I mean – it was my life.”

• Down at Max’s by Peter Crowley is published by Far West Press