Home‎ > ‎News‎ > ‎

Blackout - Postmortem Report and Next Steps

posted Jul 26, 2019, 2:34 PM by Marc Donner

West Side Blackout

In our previous update, dated July 18, we reported on several system failures experienced by 205 West End Avenue in consequence of the July 13 blackout.

In this report we will summarize what we have learned in investigating these failures and what plans management has agreed with the board to address the failures.

The Cogen System

While we have been testing the cogen system quarterly, it failed on July 13th.

The root of the problem was a failed cooling pump and a failed battery.  Because of the blackout the network connection used by the maintenance company was down, making it impossible for them to diagnose the problems.  And the blackout conditions made travel time for their technician much longer than anticipated, so he arrived at 205 after power was restored by the utility.

We will be taking two primary actions to remedy this weakness:

  1. We will test the blackout capability of the cogen system monthly instead of quarterly until such time as we are confident that the system is as reliable as we want it to be.  In fact, we conducted a test, which was successful, on July 19th.
  2. We will train most or all of the staff in how to transition the cogen system to blackout emergency generator status.  Five members of our staff had been trained.

The Emergency Lighting System

We learned several important things:

      The emergency lighting system is only rated to provide 90 minutes of lighting.

      Many of our lights failed in significantly less time than the rating.

      Most other buildings affected by the blackout experienced similar failures of these emergency lighting systems.

The board and management have agreed to pursue the following plan of action:

      We are evaluating passive luminescent tape that we can apply in the stairways.  These absorb ambient light and reemit it for about 90 minutes.

      We will test samples of our backup lighting systems to see how reliable their batteries are.

      We will explore the marketplace in cooperation with AKAM to see if there are more reliable products available to replace our current emergency lighting system.

The Elevators

The elevator system is designed to lower the car to the nearest floor and open the doors.

The elevator controller opens the doors of occupied cars once they are safely at a floor.  The controller relies on weight sensors built into the cars to tell which cars are occupied.

The motors that move the elevators, along with their power supplies and their computerized controllers are all located in a structure on the roof of the building called a head house.  Because the system produces a lot of heat, each head house is equipped with an air conditioner to keep the equipment at an appropriate operating temperature.

One of the A/C units had failed, so the temperature in that head house was too high, resulting in the emergency control system failing.  As it happened, the one elevator that was occupied at the time of the blackout happened to be under that head house.  The elevator did lower itself to within a few inches of the floor below, but failed to open its door.

Building staff have taken these actions in response:

  1. Replaced the failed A/C unit in the head house.
  2. Added the two head houses to the regular inspection schedule.
  3. And we will develop and implement a test for safe recovery behavior and add it to our monthly test program.

Front Desk Telephones

The power outage rendered our FIOS installation inoperable.  The telephone system complexity is too great for us to be able to adopt a simple solution, so we will work to enhance the reliability of our front desk phones.

We intend to work with our telephone service provider to establish the ability for our main number (1-212-496-5218) to ring through to several instruments at the desk, including one or more cell phones, regardless of the availability of power at the front desk.

Trust But Verify

Our system is well designed and should be capable of weathering a blackout like the one we experienced on July 13th with far fewer failures.

Obviously, good design alone is not enough.  While we had implemented a testing regime that checked what we thought were the critical systems on a quarterly basis, this schedule was clearly inadequate.

In addition, it is clear that our testing regime must include more components and more potential failure modes than we had previously though necessary.  Lesson learned.

We will be implementing a monthly blackout test schedule going forward.  The board will review these monthly results with management at each board meeting.

As time passes we will develop a way to report on our test results to the community.