Princess Beatrice and Princess Eugenie are grappling with the growing fallout from their father’s past, feeling they were “duped” by the former Prince Andrew as fresh revelations continue to emerge about the full extent of his links to convicted sex offender Jeffrey Epstein, sources tell PEOPLE.

The sisters — both mothers to young children — have had to watch aghast as a steady stream of damning emails has surfaced in recent weeks from the latest tranche of Epstein-related files released by the Department of Justice. The disclosures directly undercut the former Prince Andrew’s long-maintained account of events.

Andrew has claimed that he cut off all contact with Epstein in 2010. However, documents released since then indicate the relationship continued. Among the most damaging is a 2011 email, sent one day after the publication of the now-infamous photograph showing Andrew with his arm around Virginia Giuffre, in which he allegedly wrote to Epstein, “It would seem we are in this together.”

The revelations have also proved deeply damaging for their mother, Sarah Ferguson. Newly released emails appear to indicate that Ferguson took her daughters to visit Epstein just days after his release from prison in 2009.

“They are pretty torn because they believed [their father],” says renowned royal author Robert Jobson. “Just like the late Queen and Charles, Andrew told them all the same story — that he had done nothing wrong. My understanding is they feel pretty duped by the whole thing.”

Jobson, whose latest book, Windsor Legacy, is out now, says Beatrice, 37, and Eugenie, 35, are struggling with the fallout. “I do think they believed their father, and now it has all backfired,” he tells PEOPLE. “I know that Eugenie, especially, has found it very difficult.”

The situation has been further complicated by allegations now centering on their mother.

“I’m shocked by what’s come out,” Jobson adds. “They were only 19 or 20-year-olds at the time. If, at that age, your parents say we are going to see someone who’s a friend of theirs, you don’t think much about it beyond that.”

The sisters’ measured public approach has not gone unnoticed. Royal historian Robert Lacey tells PEOPLE: “The discretion with which they have behaved during their parents’ [controversies] also speaks volumes.”

PEOPLE has reached out to Andrew and Ferguson for comment, and has also contacted Princess Beatrice and Princess Eugenie.

Jobson does not believe the sisters will fully sever ties with their parents, but says they are unlikely to present any public show of unity in the near term. All four were together in December for the christening of Beatrice’s daughter, Athena Mapelli Mozzi. However, when Beatrice and her husband, Edoardo Mapelli Mozzi, stepped out for a celebratory pub visit following the ceremony alongside guests, her parents did not join them.

Suggestions that the princesses have cut off contact entirely are pushed back on by those close to the family. In late January, Beatrice and her daughter Sienna were seen joining Andrew for a horseback ride around the grounds of Windsor Castle, shortly before he vacated Royal Lodge on Feb.

Ferguson is also believed to have moved out of Royal Lodge and is expected to spend time abroad before deciding where to live next. She is said to be considering options in the Windsor area, where she has close friends.

“They are older and mothers themselves now and will be thinking this is appalling,” Jobson says of Beatrice and Eugenie. “They have always been very defensive of their mother. ​There’s a sense that they would put an arm around their mom, but dad will be left to get on with it.”

“I have sympathy for them,” Jobson adds. “It’s not their fault who their parents were associated with.​”