I met Lou Reed in the cafeteria of Syracuse University in 1961. Lou was a sophomore. I was a freshman. No one introduced us; we were just in line for some lunch and kind of sniffed each other out as cool guys from the city. He was from Long Island and I was from Brooklyn and he was probably already way cooler than me, but nonetheless we started hanging out once we discovered that we both loved doo-wop. I was a Frankie Lymon and the Teenagers freak and he loved the Velours, the Moonglows, the Jesters and the Paragons. We only liked the groups that we considered to be the real thing.

There was some kind of standard. I don’t think either of us was thinking seriously about becoming songwriters or performers at the time, but we definitely both cared about pop music in the way that people do who know that music can save your life.

One day, not long after, he said, “Meet me at the bar,” and that meant the Orange Bar. I fell by and was surprised to find him sitting in a booth with an older guy, a kind of rumpled guy with dark hair. I didn’t get their relationship at first and was a bit suspicious since he was clearly not a student, but we started meeting there every day at 4pm and this guy was always there.

Turned out he was Delmore Schwartz, Lou’s English professor, and he was drinking there with Lou every day. I just came around and listened to them talk, not always getting the literary references, but happy to be a part of it. I learned about sophisticated drinks like Dubonnet on the rocks with a twist of lemon.

Looking back, it was almost as if Lou, by sitting there every day, was soaking up Delmore’s genius. He was learning how to be an intellectual. They developed a really strong bond that continued after college and I really do believe it when Lou writes in the song My House how “he was the first great man that I had ever met”.

After I graduated, I came back to New York City and enrolled in graduate school in art history at NYU’s Institute of Fine Arts. But when I saw Lou was starting to write and perform music it made me think: “Well if he can do it, maybe I can do it. And he can hardly sing!” I dropped out of grad school and never looked back.

One night about 13 years ago I was doing a benefit for the McBurney YMCA (where Lou had also been a member in the 80s) in Chelsea at Joe’s Pub. Lou had generously done a song with the band and me, and was getting ready to leave. I was still on stage and spotted him and Laurie Anderson near the exit, waving goodbye and I just started singing So Young by the Students, a cappella. Lou stopped, listened, and then got down on his knees and gave a bow with both arms.

Garland Jeffreys’ album 14 Steps to Harlem is out on 28 April

• This article was amended on 18 April 2017 to correct the name of the album to 14 Steps to Harlem – not Heaven.