Princess Diana “hated” spending Christmas at Sandringham with the royal family, according to her biographer Andrew Morton.

Morton, who wrote the bestseller Diana: Her True Story, said her disdain for the traditional royal Christmas celebrations began as early as her first Sandringham Christmas in 1981, five months after she married the future King Charles on July 29, 1981. By Christmas of that year, the Princess of Wales was already pregnant with their first child, Prince William, who would be born six months later on June 21, 1982. Despite suffering from morning sickness, Princess Diana still took the time “to buy her new family members thoughtful and expensive gifts,” Vanity Fair reported.

She was “mortified” on Christmas, though, when she discovered that members of the royal family only gave one another gag gifts. (After all, what can one really get the Queen?) Case in point — she gave her sister-in-law Princess Anne a cashmere sweater and got a toilet paper holder in return.

“It was highly fraught,” Princess Diana told Morton. “I know I gave, but I can’t remember being a receiver. Isn’t that awful? I do all the presents, and Charles signs the cards. [It was] terrifying and so disappointing. No boisterous behavior, lots of tension, silly behavior, silly jokes that outsiders would find odd, but insiders understood.”

“I sure was [an outsider],” Diana added.

The Princess of Wales’ hairdresser Richard Dalton backed up the sentiment that Diana hated Christmas at Sandringham, telling Kitty Kelley for her book The Royals, “The princess just hated going to Sandringham for Christmas. She told me it was freezing cold and dinner had to be over by 3 o’clock. ‘It’s 3 and time to watch me on TV,’ she’d say, imitating you-know-who. The royal family had to watch the Queen’s Christmas message on television. Diana said it was a command performance.”

As the Prince and Princess of Wales’ marriage became more and more fraught with tension, “I used to get calls from her on Christmas Eve and she was alone,” a friend of Diana’s told Tina Brown for her book The Diana Chronicles. “Whenever we talked it was all about tactics. What to do next.”

Diana’s “highly fraught” Sandringham Christmas experiences were dramatized in the 2021 film Spencer, where Kristen Stewart plays the royal over the course of three days at Sandringham in 1991, “when an anguished Diana resolved to end her broken marriage,” PEOPLE previously reported.

Can’t get enough of PEOPLE’s Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

A dramatization though it is, it’s worth noting that 1991 was the last Christmas Charles and Diana spent at Sandringham before separating. On Dec. 9, 1992, the Prince and Princess of Wales formally separated, and Diana spent her first Christmas away from sons Prince William and Prince Harry just weeks later.

For more on Princess Diana’s holidays with the royal family, pick up PEOPLE’s Special Edition: Christmas at the Palace: A Royal Family Album.