The Commandant's House was built in 1806 as the quarters for the commodore of the Brooklyn Navy Yard. Rumor has it that Charles Bulfinch, the Federal-era starchitect responsible for the iconic dome of the U.S. Capitol, was behind the design. There is scant evidence to prove Bulfinch's involvement, save for the dubious claim that the proportions of the dining room are exactly the same as those of the Oval Office.
In 1841, Captain Matthew C. Perry, commander of all U.S. naval vessels in New York Harbor, moved into what was then called Quarters A of the New York Navy Yard. Perry would go on to convince Japan to open their borders to trade with the U.S., a feat he accomplished by sailing several large naval vessels into Tokyo Bay.
The house, which was once painted a "creamy yellow," was home to successive Navy Yard commandants for nearly two centuries.
In 1974, eight years after the Navy Yard was decommissioned, the building was designated as a National Historic Landmark. The property's transformation from military quarters to private residence began in 1996, when Charles Gilbert and Jennifer Jones became its first civilian owners.
It cannot be cheap to maintain a huge old house that, due to its landmark status, cannot be changed. That is probably why the home is frequently used for TV and movie shoots, most famously as the home for Nucky Thompson in Boardwalk Empire.