The New York Free Circulating Library opened in 1878 with a collection of over 6,000 books donated by George W. Vanderbilt. Upon its opening, it was one of the first free public resources of its kind in New York. Vanderbilt, who funded much of the project, oversaw the construction of the building which would serve as an ornament to the neighborhood. The red-brick structure emulated the style of the Flemish Renaissance and was adorned with wrought-iron numerals of the year the building was constructed. In 1901, it was incorporated into the New York Public Library System.

At the height of the library's popularity, it attracted over a hundred visitors every day and circulated about 126,000 volumes annually. The library was closed for a time in 1908 as it worked to disinfect books that could transfer communicable diseases. It reopened and would eventually host community events such as a "story-hour" for children and an exhibition of drawings of pre-war Korea by artist Sam-Kih Min. The library permanently closed in the 1960s when services were transferred to nearby Jefferson Market Courthouse.

In 1967, artist and pioneer of the neo-Dada movement Robert Delford Brown purchased the library for $125,000 as a home and headquarters of the First National Church of the Exquisite Panic, Inc. (FUNKUP for short). FUNKUP was a place of collaborative projects and what Brown referred to as "Happenings," or event-as-art pieces. As one of his most dramatic pieces, Brown staged "the Great Building Crack-Up" in Hunt's former library.

The "Crack-Up" was a "collision between a 19th-century building and a 20th-century building," as described by a plaque placed on the outside of the Church in 1995. The meeting was a juxtaposition of the modern and the classical. Here, the work of Richard Morris Hunt would collide with modernist Paul Rudolph's interior. The building was mostly gutted in order to accommodate design changes characteristic of Rudolph's style: flowing spaces through multiple levels, with a free-plan layout and plenty of glass bricks. After the fact, Brown noted that his interactions with Rudolph were "full of conflict," though he selected a designer with such a dogma to "see the sparks fly."

In the mid-1990s, Brown moved out of the Church and the building was sold to television writer and producer Tom Fontana. Fontana saw little value in Brown's renovation. He said of it that it was "like living in an Escher painting." Fontana spent two years reversing Rudolph's intervention, though Hunt's original drawings were unobtainable. The building is still Fontana's residence on its top two floors, which maintain Rudolph's open plan. The ground floor and basement are his writing staff's offices and editing suite. Mahogany ceilings and bookshelves, copied from the only untouched room in the building, envelop much of the interior. The modern residence Rudolph designed appears to have gone back in time.