When Jan Václav Novohradsky of Kolowraty bought the castle in 1675, he moved the residence of the noblemen of Kolowraty to Velké Dvorce, where he built a large Baroque castle. The castle was first reconstructed by Count Leopold Kolowrat-Krakowsky, after an extensive fire in 1907.
The reconstruction took place from 1907 to 1910 giving the castle its current Baroque form. In 1910, Leopold’s son, Count Alexandr Josef Kolowrat-Krakowsky, founded his film studios here. In 1912, after another fire at the castle, Count Alexandr moved to Vienna. Although the castle was reconstructed again, Count Jindřich Josef Kolowrat-Krakowsky, brother of Alexandr, moved the residence to the nearby hunting lodge Diana.
During World War I. the castle served as a military hospital, and during World War II. it was converted into a Hitlerjugend dormitory. After the war, the castle was supposed to serve as a retirement home, but the reconstruction was not completed. After 1989, the castle was given back to Count Jindřich Kolowrat-Krakowsky, who sold it in 1992.