Maggie Smith may not appear in Downton Abbey: The Grand Finale, but her presence is felt throughout the film.

In a notable moment, Hugh Bonneville, who portrays Robert Crawley, Lord Grantham, references one of Smith’s most iconic lines as Violet, the Dowager Countess. “The weekend — I’m glad Mama isn’t alive to hear you say that word,” he quips when speaking of plans with Lady Mary (Michelle Dockery).

It’s a funny and heartwarming wink to a piece of dialogue that helped turn Downton Abbey into a phenomenon. But Bonneville actually hated the line.

“It’s my least favorite line in the movie, and I was hoping it was gonna be cut,” he tells Entertainment Weekly.

It’s not because he doesn’t think it’s funny, but rather because of the daunting task of trying to live up to Smith’s memory. “How can you quote Maggie?” he continues. “She did it so iconically.”

Bonneville wasn’t surprised at the dialogue’s inclusion, however. “I could sense it coming,” he says with a laugh. “And it was down to me to reference it.”

Mostly, Bonneville’s reticence came from his great reverence for the late Smith and her legacy as an actress. “We were very lucky to work with one of the greatest actresses of her generation,” he says. “For me to have been able to call her ‘mummy’ on and off for 15 years was a great privilege. She was one of a kind, and I feel very blessed to have been in her orbit for a while.”

Downton Abbey: The Grand Finale, which is now in theaters, is the conclusion of the Downton saga, at least for this generation of characters. As such, it’s bursting with nods to some of the most memorable parts of the story over its 15-year history.

“The movie is full of those little back references,” Bonneville notes. “There’s a reference to Mr. Pamuk from season 1, which is quite delicious, and various others along the way. It was inevitable that lines like that would pop up.”

Though the cast has been saying goodbye since the TV series ended in 2015, they all felt a more permanent sense of an ending while making this film. “It’s a fond love letter to all the audiences that have followed us for so long,” Bonneville reflects. “The compassion with which Julian Fellowes writes is second to none. He’s created this fictional world in this uniquely British setting in which all the accidents of human nature play out, but at its heart, there’s a generosity of spirit. I’m proud to be part of that story.”