The address 50 Broadway is famous in skyscraper history as the site of the Tower Building, the first building in New York, and arguably in the world, to be constructed with iron framing. The skeleton frame was able to fully support the extraordinarily narrow building's walls and floors through the use of wind bracing diagonal beams. Unlike in Chicago, where engineers and builders embraced metal skeleton construction, in New York, the new technology was met with conservative reaction. Some protested that the tower would be unsafe, even after the Department of Buildings had granted its permit for construction.

At one point during the building's construction, after the walls had been raised to the height of 158 feet, but before the structure had been roofed in, a great windstorm struck. A crowd was said to have gathered, awaiting the structure's imminent collapse. Gilbert, who had climbed the scaffolding to the building's 11th floor to measure vibrations with a plumb line at the height of the storm, reported to the New York Times in a heroic, if perhaps self-serving narrative printed the following day, that the building "stood as steady as a rock in the sea." The Tower Building's façade was in a heavy, Romanesque style with an exaggerated, arched entranceway. It was demolished in 1914. A 37 story office building replaced it in 1927. In order to attract brokerage and investment firms, a special entrance was built at 41 New Street so that brokers and clearinghouse runners could "make quick contacts in emergencies."