Once one of the largest office buildings in the city, the Whitehall Building was constructed in two phases. Planning for the larger second structure began immediately after the original 20-story building opened to great success in 1904. The 31-story addition quintupled the building's size. Cass Gilbert's newly completed U.S. Custom House had generated a new demand for office space in the district to the east of Broadway which had formerly been dominated by the port and its warehouses and small-scale industrial facilities.

In the original building, the frontage on Battery Park guaranteed offices light and expansive views. Henry J. Hardenbergh (also the architect of the Dakota Apartment Building and the Plaza Hotel) created a boldly scaled design equal to its prominent riverfront site. The larger addition, by Clinton and Russell, has stylistic affinities with Hardenbergh's earlier work. The building has recently been converted into a mixed-use commercial and residential complex.