Penitentiary and City Hospital, on the south end of Welfare Island; as seen from the Queensboro Bridge, March 5, 1933.

In 1832, on what was then called Blackwell's Island, a large penitentiary was constructed. The long imposing arcaded structure looked a bit like a castle with crenelations at the roofline and turrets. This complex included areas for both male and female prisoners. In the 1830s hundreds of prisoners were head there and spent their days in labor, often breaking rocks. Boss Tweed was imprisoned here in 1874, though he had a private unlocked room and a secretary.

Though the island prison likely had a long history of poor conditions and corruption, it was not until the early 20th century that the conditions were widely made public by Katharine Davis, the female Commissioner of Correction in 1914. Despite the accusations of overcrowding, drug-dealing, and favoritism the prison survived until reforms by Mayor La Guardia in the mid 1930s finally prompted the closing of the facility and the removal of prisoners to the nearly complete complex on Riker's Island.