Haydn worked in this château built on the site of a medieval castle and owned by the Esterházy princes. The Esterházy clan was a great Hungarian family with vast estates that ruled over Eisenstadt and its surrounding area. They claimed descent from Attila the Hun. The Esterházys helped the Hapsburgs gain control in Hungary; so great was their loyalty to Austria, in fact, that when Napoleon offered the crown of Hungary to Nic Esterházy in 1809, he refused it.

The castle, built around an inner courtyard, was designed by the Italian architect Carlo Antonio Carlone, who began work on it in 1663. Subsequently, many other architects remodeled it, resulting in sweeping alterations to its appearance. In the late 17th and early 18th centuries, it was given a baroque pastel facade. On the first floor, the great baronial hall was made into the Haydnsaal, where the composer conducted the orchestra Prince Esterházy had provided for him. The walls and ceilings of this concert hall are elaborately decorated, but the floor is of bare wood, which, it is claimed, is the reason for the room's acoustic perfection.