Sugar House Prison Window, just east of the Van Cortlandt House, undated postcard.

In 18th century New York City a number of sugar houses—constructed for the storage of imported sugar and molasses— were commandeered during the Revolutionary War. The five-story brick sugar house at Rose (now William) Street and Duane Street was constructed in 1763. According to local lore, this sugar house was used by the British army as a prison during the war. Recent scholarship, thought, suggests that perhaps this specific sugar house was not, in fact, used as a prison.

Regardless of this evidence, for more than two centuries the building was considered part of Revolutionary War history. The Duane Street building fell into serious disrepair in the late 19th century and was demolished in 1892. Given the history of the building, two of its "prison" wall sections with windows were saved. One was installed in the Rhinelander Building, and later One Police Plaza when it replaced the Rhinelander Building in 1973. A second wall section with single window was moved to Van Cortlandt Park, where it remains next to the Van Cortlandt House.