A lawyer acting for Britney Spears has threatened legal action against her sister Jamie Lynn, and accused their father of financial misconduct, including the hiring of a private security firm to keep Britney and people close to her under surveillance.

A letter written by Britney’s lawyer Mathew Rosengart instructs Jamie Lynn to “cease and desist from referencing Britney derogatorily during your promotional campaign” for her new memoir. “If you fail to do so or defame her, Britney will be forced to consider and take all appropriate legal action.”

The letter states that Jamie Lynn’s memoir contains “misleading or outrageous claims”, and that Britney believed Jamie Lynn had “exploited her for monetary gain”. Britney made the latter allegation previously in a recent Instagram post, saying: “She wants to sell a book at my expense.”

Jamie Lynn responded to that earlier statement, claiming: “My book is not about her. I can’t help that I was born a Spears too, and that some of my experiences involve my sister.”

She has not responded to the letter, which continues: “Publicly airing false or fantastical grievances is wrong, especially when designed to sell books. It is also potentially unlawful and defamatory.”

The letter continues the spat between the sisters, which began after Jamie Lynn was interviewed on Good Morning America and said Britney’s behaviour had historically been “erratic, paranoid and spiralling”.

In the book, Jamie Lynn describes an alleged incident at home with Britney during her childhood: “Britney said to me, ‘Baby I’m scared.’ And she took a large knife from the kitchen, pulled me along to my room and locked us both inside. She put the knife in the bedside table drawer and simply repeated, ‘I’m scared.’ ” Regarding this passage, Britney wrote on Twitter: “Jamie Lynn … you’ve stooped to a whole new level of low. I’ve never been around you ever with a knife.”

Jamie Lynn discussed the incident further in a podcast interview, saying: “I never felt like she was going to harm me or anything – it just felt weird.”

A later Twitter post saw Britney soften her stance, telling Jamie Lynn “just know I love you”, but also accused her of a lack of loyalty and help during the conservatorship that controlled many aspects of Britney’s life, which was terminated in November.

Britney has also newly accused Spears’s father Jamie of financial impropriety during his management of the conservatorship.

Jamie Spears has attempted to have legal bills generated by the conservatorship hearings paid by Britney. In response, Rosengart made a court filing that states: “The allegations of misconduct against him are specific, credible and serious, ranging from abuse to conflicts of interest, financial mismanagement and corruption of the conservatorship to implicating state and federal criminal law.”

The alleged incidents cited in the filing, reported by the New York Times, include the hiring of security firm Black Box for a total of $6m to monitor Britney’s phone and her mother’s, download notes from her therapy sessions despite being forbidden from seeing them, and track the movements of people associated with her via GPS.

Elsewhere in the court filing, Jamie Spears is also accused of selling Britney’s childhood home to himself, of charging high expenses for the maintenance of her property – including sending funds to a company owned by Jamie Lynn’s husband – and of approving unusually large legal fees involving dozens of law firms.

The allegations made in the court documents reflect findings by an investigator used by Britney’s legal team to examine Jamie Spears’s management of her affairs. Her investigation included an interview with a former Black Box employee.

Jamie Spears has not responded to the allegations. The Guardian has contacted his lawyer for comment. In a previous response to allegations of surveillance, his lawyer told the New York Times: “All of his actions were well within the parameters of the authority conferred upon him by the court. His actions were done with the knowledge and consent of Britney, her court-appointed attorney, and/or the court.”

Jamie Spears’s request regarding legal fees is expected to be addressed in a court hearing on Wednesday in Los Angeles.