David Cameron has added his voice to criticism of Gary Barlow over claims that the singer invested in a tax avoidance scheme, but said he should not be stripped of his OBE.

The prime minister entered the row over Barlow's financial affairs after a judge ruled that music investment schemes used by the singer, two other members of Take That and their manager were primarily designed to avoid tax. Barlow, Howard Donald, Mark Owen and Jonathan Wild put around £66m into music partnerships but may have to pay tens of millions of pounds to the exchequer.

When asked about the case of the Take That singer – a Conservative party supporter who was appointed OBE in 2012 for his work for music and charity – Cameron told the Times: "I am opposed to all aggressive tax avoidance."

However, when asked on ITV's Good Morning Britain about whether Barlow should lose his OBE, the prime minister said: "I don't think that's necessary, frankly. Gary Barlow has done a huge amount for the country, he's raised money for charity, he's done very well for Children in Need so I'm not sure. The OBE was in respect of that work and what he's done. But clearly what this scheme was was wrong and it's right that they're going to have to pay back the money."

Cameron's criticism falls short of his stance on Jimmy Carr, the comedian who was exposed as an investor in a similar scheme in 2012. At the time, the prime minister said that was morally wrong and "particularly egregious".

Margaret Hodge, the chair of the Commons public accounts committee, has suggested that Barlow "might want to show a bit of contrition by giving back his OBE". Charlie Elphicke, a Conservative MP who has campaigned against tax avoidance, said: "People who have seriously abused the tax system should be stripped of their honours."