Prince Harry is opening up about the reality that he may never be able to return to his homeland with his wife, Meghan Markle, and children, Prince Archie and Princess Lilibet, in the wake of his latest legal setback over security arrangements in the U.K.

After losing his legal appeal against a government decision that stripped him of automatic police protection when visiting Britain, the Duke of Sussex, 40, told the BBC on May 2: “I can’t see a world in which I would be bringing my wife and children back to the U.K. at this point.”

It’s a striking admission from a prince who once called the United Kingdom home and served in the British military for a decade.

“The things that they’re going to miss is, well, everything,” Harry continued. “I love my country. I always have done. Despite what some people in that country have done.”

In his latest remarks, Harry highlighted the emotional impact this has on his children, Archie, 5, and Lilibet, 3.

“I miss the U.K., I miss parts of the U.K., of course I do,” he said. “I think that it’s really quite sad that I won’t be able to show my children my homeland.”

In his official statement following the judges’ decision, Harry said, in part: “The U.K. is my birthplace and will always be part of who I am. It is a place I love, and the country where my son was born. I’ve only ever wanted to continue my charitable work in supporting the causes and people that mean so much to me, and for my children to know the beauty of my homeland. I remain committed to a life of public service.”

The Duke and Duchess of Sussex relocated to California in 2020 after stepping back from their roles as senior working royals. The couple has since built a new life in Montecito, California, with their children. But Harry has continued to fight a legal battle over the loss of publicly funded police protection when he visits the U.K. — a decision made by the government’s Royal and VIP Executive Committee (RAVEC) following his royal exit.

Though he has returned to Britain on a number of occasions, his visits have been brief, heavily guarded and without Meghan or their children. He has argued that without secure protection, it is too risky to bring his young family with him.

Those words strike a poignant chord for a man whose identity has been so deeply tied to Britain and the monarchy. Prince Harry was once a central figure in the royal family — and someone who often connected with the public through his military service and charitable work.

But the years since his royal departure have been marked by deepening family tensions, public scrutiny and an ongoing fight over security — something Harry has said is non-negotiable for the safety of his wife and children.

The court ruling delivered on May 2 upheld the government’s decision, with the judge noting that Harry would be assessed on a case-by-case basis for protection, rather than automatically granted it. The verdict effectively shuts the door on one of the final hopes Harry had for more stable and secure visits home.

Despite the outcome, Harry remains resolute in his commitment to his family’s safety — even if it means sacrificing a closer connection with his homeland and his family, particularly his father, King Charles, with whom he remains estranged amid his cancer diagnosis.

“I would love reconciliation with my family,” he told the BBC. “There’s no point in continuing to fight anymore.

“Life is precious. I don’t know how much longer my father has. He won’t speak to me because of this security stuff. But it would be nice to reconcile.”

As Harry’s legal battle reaches a pivotal moment, it has only highlighted that his rift with his family is far from healed.

“I don’t think there is any rapprochement,” a source close to the palace previously told PEOPLE. “Nothing has changed.”

Added another source about the father and son: “They are distant.”