Lord Carnarvon has spoken about some of the stars of Downton Abbey(Image: Getty) The owners of Highclere Castle, where Downtown Abbey was filmed, have fired a subtle dig at…