In 1763 Henry Culyer, scion of a wealthy Dutch family involved in the sugar trade, built a massive six-story, brick sugar house at the corner of what was Prince Street (later Rose and William Street) and Duane Street. The building was impressive for colonial New York, standing six-stories tall. It was used for both refining sugar and the storage of sugar and molasses.

The Culyer family remained Loyalists during the Revolution and as a result, lost their property (including this sugar house) when the Act of Forfeiture was passed. Sold at auction, the building was purchased by William Rhinelander, another son of an old New York family made rich by sugar.

Improbably surviving the Great Fire of 1845 and the general destruction of much of the city's colonial architecture, the sugar house was still standing in the second half of the 19th century. Once the tallest building in the area, by the 1890s, when it caught the attention of photographer Robert Brackow, it was dwarfed by taller structures and was obviously suffering from neglect. The building was finally razed in 1892 and replaced by a new Rhinelander Building.

Fun fact! Not all of the 1763 building was lost! Urban legend/local lore in the 19th century suggested that this sugar house was used as a prison during the Revolutionary War. It was not, but the Livingston Sugar House on Liberty Street was used as a prison. Regardless of the provable facts, stories persisted. When the building was demolished two windows were saved. One was donated to the National Society of Colonial Dames in the State of New York and placed near the Van Cortlandt mansion in the Bronx. The other window was installed at the new Rhinelander Building and when that building was razed in 1968 and the street demapped for the Civic Center redeveopment, it was placed within the Brutalist structure of One Police Plaza (where it can still be seen today.)