William the Conqueror gave the manor of Toddington to the de Tracy family in return for services that they had rendered to him, and it remained in that family for hundreds of years. William de Tracy of Toddington was one of the four knights who murdered Thomas à Becket in 1170. It was de Tracy who struck the first blow and wounded the Archbishop with his sword. The original manor house was replaced by the present building in 1830, and it was used as a school for many years. It is now owned by Damien Hirst, the world renowned artist, and is being renovated under a huge wrapping of scaffolding and polythene.