Few traveling to Governors Island can fail to miss Castle Williams, the striking fort resting on the Island's coast.