Following on from Simon Lockhart’s reminder of his ticket from September 11th 2001, this is the one that I was involved with:
Ticket Number: 20010912-2 Ticket Status: UPDATE Ticket Type : Unscheduled Ticket Source: TEN-US NOC Ticket Scope : Site Site/Line : New York Ticket Owner : TEN-US NOC Problem Fixer: Telehouse NY Ticket Opened: 20010912 12:08 UTC Problem Start: 20010911 20:35 UTC Ticket Update: 20010918 05:13 UTC Ticket Closed: Problem Ends : Ticket Summary: Status of 25 Broadway Problem Description: This ticket is being issued to track the status of the infrastructure at the DANTE World Services PoP in New York. Individual circuits will be dealt with separately. Updates will be in the "actions" section below. Affected: DANTE World Service Actions: cziarhe 20010918 05:13 UTC Load was switched from the Con Ed generator to the critical generator as planned. Unfortunately, after about 45 minutes the generator again began to overheat and the load was switched back. This was achieved without interruption to service. It is now believed this could be due to limescale build-up in the radiator. This is to be rectified by using an acid wash through the cooling system and again running the generator under load at 22:30 UTC tonight (Tuesday). A plan is also being developed to bring in a new generator should the situation with the critical generator persist. Time to Fix: (Hours:Mins) Fix: History: cziarhe 20010917 21:53 UTC Telehouse NY believe the problems with their "critical" generator have been caused by faulty thermostats. These have now been replaced and the generator has been spun up for a basic test. The next step is to try a load test, which will begin at 22:30 UTC. If this is successful, then load from the "essential" generator, which supplies the air conditioning will be transferred to the Con Ed generator for maintainence to take place on this generator, which has also been running overtemperature. Assuming all this works successfully, Telehouse plan to run the Con Ed generator in conjunction with either the essential (Air Con) or critical (equipment) generator to make the best use of available fuel. Further fuel supply problems are not thought to be likely as there are 17 tankers in the city, and a further 30 generators are available. The IP NOC will monitor the initial switch to the critical generator to ensure equipment remains in operation, and reinstall the workaround with KPN should it be needed. cziarhe 20010916 14:59 UTC The latest update from Telehouse reports that an extra 2000 gallon fuel tank is expected to be connected to the Con Ed generator soon to allow for longer refuelling intervals. in addition, Con Ed's refuelling process to all buildings in the area has now settled down, so it should be as reliable as possible in the circumstances. An engineer is currently onsite investigating the problems with Telehouse's own generator. cziarhe 20010916 12:05 UTC Regular fuel deliveries appear to have been secured for the Con Ed generator. There were deliveries at 02:00 UTC and 11:30 UTC, and another is scheduled for 21:30 UTC. The status of the Telehouse generator is currently unknown. cziajom 20010915 20:16 UTC Contacted Telehouse NY. Informed that Con Ed generator is now up, with enough fuel until Tuesday (18 Sep). Engineers are still working on the Telehouse generator. NY Routers are starting to come up. cziarhe 20010915 18:25 UTC Despite what we were informed of earlier, the Con Ed generator ran out of fuel at 16:45 UTC. Telehouse's generator then started up, but overheated again. There is a fuel truck just outside the cordoned area, but it is not being allowed through, and unfortunately we have no ETA for the fuel, or time-to-repair for the Telehouse generator. cziarhe 20010915 18:15 UTC Power was lost again at 17:45 UTC, we are contacting Telehouse for information. cziarhe 20010915 15:52 UTC The current status relayed to us by Telehouse is as follows: The Con Ed generator is monitored around the clock by Con Ed personnel, who are responsible for refuelling it. As of 15:00 UTC there was approximately 15 hours worth of fuel remaining. The water pump on Telehouse's own generator has been replaced and the unit has been run for about 30 minutes to test. Telehouse are taking this opportunity to perform some routine maintenance (changing oil and filters, pressure washing the radiator) to ensure it is ready should it be needed for another extended run. Fuel currently available onsite should last until Thursday at current consumption rates. Another delivery is being planned for Monday. Engineer access is possible, but the procedure is slightly different due to movement restrictions still in effect. cziarhe 20010915 08:55 UTC Power was returned via the Con Edison generator at approximately 02:45 UTC this morning (Saturday). All DANTE World Service equipment is functioning without any failed power supplies. Telehouse also expect their own generator to be fixed soon, so there will be two sources of power again, as the Con Edison generator will remain until grid power is restored. cziarhe 20010914 18:06 UTC Con Edison have delivered the transformer and connected it to be building's power supply, they expect to have the generator itself online by approximately 21:00 UTC this evening (Friday). Unfortunately the water pump for Telehouse's own generator has been difficult to source and is currently being shipped from Pennsylvania with an expected arrival time at 25 Broadway of midnight tonight UTC. Installation is planned for tomorrow. Hopefully the Con Edison generator should allow the DANTE World Service to resume at some point this evening. cziarhe 20010914 08:11 UTC To clarify the previous entry, the generator from Con Edison was expected onsite yesterday evening US time for connection this morning their time, i.e. early to mid-afternoon European time. cziaseo 20010914 06:16 UTC The generator failed because of the water pump. Estimated repair time is late tomorrow (Friday, 9/13). In the meantime, Con Ed has committed to provide generator from the street and we expect delivery tonight. Estimated connection is sometime tomorrow morning. cziajom 20010914 04:57 UTC An engineer is on-site to try and repair generator. However, still not able to give any indication of a resolution time. Further information will be given, as it becomes known to us. cziarhe 20010914 01:50 UTC Power failed again at 01:40 UTC. Apparently there is a generator technician onsite, and Telehouse are talking to Con Edison about borrowing a generator in the event the current one cannot be fixed. cziarhe 20010914 01:39 UTC Power has returned at 01:30 UTC. More information will be supplied as we have it. cziarhe 20010914 00:20 UTC Have spoken to Telehouse Ops, they are waiting for an engineer for the diesel generator and will let us know via email as soon as they have more information. This ticket will be updated when the information reaches us. cziajom 20010913 23:11 UTC Received msg from M+ (23:06 BST) to contact WorldCom. Contacted WorldCom. Engineer stated that generator had to be powered off due to overheating .... and that if generator cannot be fixed, then they will have to await the arrival of a new generator. Therefore not possible to give any restoration time for power. Ticket # 447111 issued by WC. Asked WorldCom to contact us, as they get further information. cziarhe 20010913 21:45 UTC As of 21:40 UTC, Telehouse have been required to shut down the generator due to a rapid increase in temperature, this has meant a loss of power to all DANTE World Service equipment. We currently have no time for restoration of power. cziarhe 20010913 16:24 UTC Fuel has arrived on-site and is currently being transferred into Telehouse's storage tanks. Once this is complete there will be sufficient diesel for another four days of generator power. cziarhe 20010913 14:29 UTC The fuel delivery truck is currently preparing to leave Staten Island for 25 Broadway, however, progress is expected to be slow. Should this fail, Con Edison are prepared to deliver fuel to 25 Broadway. They are still unable to give an estimate as to when utility power will be restored. cziarhe 20010912 21:35 UTC Commercial power to 25 Broadway was interrupted when debris from WTC 7 (The Salomon building) fell on the substation. It is not expected to be restored before the weekend, so delivery of the fuel mentioned in a previous update is being investigated. Telehouse believe they will be able to allow personnel onto the site at some point tomorrow (Thursday). cziarhe 20010912 18:23 UTC Telehouse have secured an additional 4500 gallons of diesel fuel and are attempting to get permission from the Port Authority of New York to transport it to 25 Broadway. cziarhe 20010912 13:31 UTC Telehouse have informed us that if commercial power is not restored, it is unlikely that they will be able to replenish the fuel supply when it becomes depleted. In addition, the area in which 25 Broadway is located has been evacuated. Essential staff are remaining on-site, but should they be forced to vacate the building, Telehouse say the ability to continue to provide emergency power would be impacted. cziarhe 20010912 12:17 UTC Sorry, "18:35 UTC" in the previous entry should read "18:35 UTC on Friday September 14th". cziarhe 20010912 12:13 UTC As of 20:35 UTC on September 11th, 25 Broadway is running on the emergency diesel generator. Initial information was that they had 24 hours worth of fuel, but drain is currently much lower and we have been informed that they could last up to 70 hours, which takes service up to 18:35 UTC, although this is a maximum. It is not clear whether refuelling will currently be possible, or when commercial grid power will be restored.