"Very cruise ship, very cabaret, very pub singer," Louis Walsh said of one performance on last Saturday's X Factor. "Just that little bit too cheesy for me." The week before, Walsh's fellow judge Tulisa used "cabaret" as an insult; it's also a favourite put-down of chief judge Gary Barlow, as it was for Simon Cowell before him.

Viewers use it that way too. During showtime, Twitter heaves with comments such as "Thank God for the mute button #xfactor #cabaret", "That's just cabaret. Awful. #XFactor" and "Didn't realise xfactor was a cabaret show now #shitttttttt".

Which is curious, because outside the world of The X Factor the word is a badge of pride. And the UK's booming underground cabaret scene is starting to push back. At the Bethnal Green Working Men's Club, I watched 20-odd leading members of London's "cabarati" gathered last week to set The X Factor straight in a form Barlow et al would surely understand: a pointedly ironic music video.

This "cabariot" is the brainchild of satirical pop mashup duo Frisky and Mannish, aka Laura Corcoran and Matthew Floyd Jones, who have attracted a sizeable following with their witty riffs on contemporary pop; they've just enjoyed their fourth hit Edinburgh run and are currently on a national tour. The conceit is that Mannish has gone to ITV to remonstrate with Barlow but returns brainwashed, corralling the cabarati into a bland mainstream makeover and leading them in a ballad whose refrain runs: "Cabaret! / Don't give in to cabaret / Don't wear something that suggests you're gay / You want the public to like you …"

"We love The X Factor," Floyd Jones tells me between takes, "but whenever a contestant did something the judges felt was past it or too glitzy, they would be described as 'cabaret' and we thought that was unfair." He compares The X Factor's use of the word to playground use of 'gay' as an all-purpose pejorative. "They're using it in a warped sense. We wanted to point out in a gentle, tongue-in-cheek way that there are other things that go by the name 'cabaret' that you might like to know about."

Cabaret incorporates aspects of music, comedy, variety, drag, burlesque, dance, circus, live art, photography and film-making, taking place at dozens of venues in London alone each week; many cabaret performers, it might be pointed out, enjoy greater success than some X Factor winners. Nor is it hard to find "relevant" pop acts with cabaret pedigrees. Amanda Palmer began her career with punk cabaret band the Dresden Dolls, Scissor Sisters are closely tied to New York's underground performance world and Paloma Faith came up through London's vintage cabaret scene. Even Alan Titchmarsh has got in on the act. And when The X Factor does have some genuine contemporary cabaret on its stage, it doesn't know where to look – when notorious mess-maker Scottee popped up to help Rihanna have a cake fight, Dermot O'Leary nearly had an aneurysm.

"Weirdly, I think what Gary Barlow thinks cabaret is – mainstream, accessible, unchallenging, boring, bland – is actually what he does," says "gentleman juggler" Mat Ricardo, whose shows fuse dexterity with storytelling, as well as a rich knowledge of variety heritage and his signature tablecloth trick. "Today is a chance for us to tell him where to shove it – in a witty way."

The cabarati's finished video, titled Too Cabaret: A Message for Gary Barlow, was posted on YouTube yesterday morning, racking up several thousand hits by lunchtime. Things took an even more gratifying turn at 6.30pm when, to his great credit, Gary Barlow tweeted the link to his 2.2 million Twitter followers, calling it "amazing". By midnight, views had passed 15,000. By this morning, the figure was over 23,000.

While Barlow's apparent conversion is cause for celebration, it remains to be seen whether he will recant on TV, or take up the video's closing invitation to attend an actual cabaret show. Either way, don't expect to see the scene's stars on The X Factor stage any time soon. "If some of these people came to a cabaret show and have their minds opened by something they didn't know was going on, that would be amazing," Floyd Jones says. "But I don't think actual cabaret could fit on The X Factor. The very nature of the best cabaret is in-the-room, visceral experiences where you're communicating directly with the audience. You can't replicate that on TV."

• Ben Walters is Time Out London's cabaret editor. He tweets about cabaret at @not_television.