The vacant 21-story building downtown has a history that includes the discovery of asbestos in the building and multiple hurdles with building inspections.