The castle was built by the powerful Bishop Roger de Caen in the early 12th century, but it was seized by the crown at about the time of King Henry I's death in 1135 and Stephen's troubled accession to the throne. The castle was given to Sir Walter Raleigh by Queen Elizabeth I. The gallant knight built Sherborne Lodge in the deer park close by (now privately owned). The buildings were mostly destroyed in the civil war, but you can still see a gatehouse, some graceful arcades, and decorative windows.