Seventeen years ago

Following on from Simon Lockhart’s reminder of his ticket from September 11th 2001, this is the one that I was involved with:

Ticket Number: 20010912-2                  Ticket Status: UPDATE
Ticket Type  : Unscheduled                 Ticket Source: TEN-US NOC
Ticket Scope : Site                        Site/Line    : New York
Ticket Owner : TEN-US NOC                  Problem Fixer: Telehouse NY
Ticket Opened: 20010912 12:08 UTC          Problem Start: 20010911 20:35 UTC
Ticket Update: 20010918 05:13 UTC
Ticket Closed:                             Problem Ends :

Ticket Summary: Status of 25 Broadway

Problem Description:

This ticket is being issued to track the status of the infrastructure
at the DANTE World Services PoP in New York.  Individual circuits
will be dealt with separately.  Updates will be in the "actions" section


DANTE World Service


cziarhe	20010918 05:13 UTC
Load was switched from the Con Ed generator to the critical generator
as planned.  Unfortunately, after about 45 minutes the generator
again began to overheat and the load was switched back.  This was
achieved without interruption to service.  It is now believed this
could be due to limescale build-up in the radiator.

This is to be rectified by using an acid wash through the cooling
system and again running the generator under load at 22:30 UTC
tonight (Tuesday).

A plan is also being developed to bring in a new generator should
the situation with the critical generator persist.

Time to Fix: (Hours:Mins)



cziarhe	20010917 21:53 UTC
Telehouse NY believe the problems with their "critical" generator
have been caused by faulty thermostats.  These have now been replaced
and the generator has been spun up for a basic test.  The next
step is to try a load test, which will begin at 22:30 UTC.  If
this is successful, then load from the "essential" generator, which
supplies the air conditioning will be transferred to the Con Ed
generator for maintainence to take place on this generator, which
has also been running overtemperature.

Assuming all this works successfully, Telehouse plan to run the
Con Ed generator in conjunction with either the essential (Air
Con) or critical (equipment) generator to make the best use of
available fuel.  Further fuel supply problems are not thought to
be likely as there are 17 tankers in the city, and a further 30
generators are available.

The IP NOC will monitor the initial switch to the critical generator
to ensure equipment remains in operation, and reinstall the workaround
with KPN should it be needed.

cziarhe	20010916 14:59 UTC
The latest update from Telehouse reports that an extra 2000 gallon
fuel tank is expected to be connected to the Con Ed generator soon
to allow for longer refuelling intervals.  in addition, Con Ed's
refuelling process to all buildings in the area has now settled
down, so it should be as reliable as possible in the circumstances.

An engineer is currently onsite investigating the problems with
Telehouse's own generator.

cziarhe	20010916 12:05 UTC
Regular fuel deliveries appear to have been secured for the Con
Ed generator.  There were deliveries at 02:00 UTC and 11:30 UTC,
and another is scheduled for 21:30 UTC.

The status of the Telehouse generator is currently unknown.

cziajom	20010915 20:16 UTC
Contacted Telehouse NY. Informed that Con Ed generator is now up,
with enough fuel until Tuesday (18 Sep). Engineers are still working
on the Telehouse generator. NY Routers are starting to come up.

cziarhe	20010915 18:25 UTC
Despite what we were informed of earlier, the Con Ed generator ran
out of fuel at 16:45 UTC.  Telehouse's generator then started up,
but overheated again.  There is a fuel truck just outside the cordoned
area, but it is not being allowed through, and unfortunately we
have no ETA for the fuel, or time-to-repair for the Telehouse generator.

cziarhe	20010915 18:15 UTC
Power was lost again at 17:45 UTC, we are contacting Telehouse for

cziarhe	20010915 15:52 UTC
The current status relayed to us by Telehouse is as follows:

The Con Ed generator is monitored around the clock by Con Ed personnel,
who are responsible for refuelling it.  As of 15:00 UTC there was
approximately 15 hours worth of fuel remaining.

The water pump on Telehouse's own generator has been replaced and
the unit has been run for about 30 minutes to test.  Telehouse
are taking this opportunity to perform some routine maintenance
(changing oil and filters, pressure washing the radiator) to ensure
it is ready should it be needed for another extended run.

Fuel currently available onsite should last until Thursday at current
consumption rates.  Another delivery is being planned for Monday.

Engineer access is possible, but the procedure is slightly different
due to movement restrictions still in effect.

cziarhe	20010915 08:55 UTC
Power was returned via the Con Edison generator at approximately
02:45 UTC this morning (Saturday).  All DANTE World Service equipment
is functioning without any failed power supplies.  Telehouse also
expect their own generator to be fixed soon, so there will be two
sources of power again, as the Con Edison generator will remain
until grid power is restored.

cziarhe	20010914 18:06 UTC
Con Edison have delivered the transformer and connected it to be
building's power supply, they expect to have the generator itself
online by approximately 21:00 UTC this evening (Friday).

Unfortunately the water pump for Telehouse's own generator has
been difficult to source and is currently being shipped from Pennsylvania
with an expected arrival time at 25 Broadway of midnight tonight
UTC.  Installation is planned for tomorrow.

Hopefully the Con Edison generator should allow the DANTE World
Service to resume at some point this evening.

cziarhe	20010914 08:11 UTC
To clarify the previous entry, the generator from Con Edison was
expected onsite yesterday evening US time for connection this morning
their time, i.e. early to mid-afternoon European time.

cziaseo	20010914 06:16 UTC
The  generator failed because of the water pump. Estimated repair
is late tomorrow (Friday, 9/13). In the meantime, Con Ed has committed

to provide generator from the street and we expect delivery tonight.

Estimated connection is sometime tomorrow morning.

cziajom	20010914 04:57 UTC
An engineer is on-site to try and repair generator. However, still
not able to give any indication of a resolution time. Further information
will be given, as it becomes known to us.

cziarhe	20010914 01:50 UTC
Power failed again at 01:40 UTC.  Apparently there is a generator
technician onsite, and Telehouse are talking to Con Edison about
borrowing a generator in the event the current one cannot be fixed.

cziarhe	20010914 01:39 UTC
Power has returned at 01:30 UTC.  More information will be supplied
as we have it.

cziarhe	20010914 00:20 UTC
Have spoken to Telehouse Ops, they are waiting for an engineer for
the diesel generator and will let us know via email as soon as
they have more information.  This ticket will be updated when the
information reaches us.

cziajom	20010913 23:11 UTC
Received msg from M+ (23:06 BST) to contact WorldCom. Contacted
WorldCom. Engineer stated that generator had to be powered off
due to overheating .... and that if generator cannot be fixed,
then they will have to await the arrival of a new generator. Therefore
not possible to give any restoration time for power.

Ticket # 447111 issued by WC.

Asked WorldCom to contact us, as they get further information.

cziarhe	20010913 21:45 UTC
As of 21:40 UTC, Telehouse have been required to shut down the generator
due to a rapid increase in temperature, this has meant a loss of
power to all DANTE World Service equipment.  We currently have
no time for restoration of power.

cziarhe	20010913 16:24 UTC
Fuel has arrived on-site and is currently being transferred into
Telehouse's storage tanks.  Once this is complete there will be
sufficient diesel for another four days of generator power.

cziarhe	20010913 14:29 UTC
The fuel delivery truck is currently preparing to leave Staten Island
for 25 Broadway, however, progress is expected to be slow.

Should this fail, Con Edison are prepared to deliver fuel to 25
Broadway.  They are still unable to give an estimate as to when
utility power will be restored.

cziarhe	20010912 21:35 UTC
Commercial power to 25 Broadway was interrupted when debris from
WTC 7 (The Salomon building) fell on the substation.  It is not
expected to be restored before the weekend, so delivery of the
fuel mentioned in a previous update is being investigated.

Telehouse believe they will be able to allow personnel onto the
site at some point tomorrow (Thursday).

cziarhe	20010912 18:23 UTC
Telehouse have secured an additional 4500 gallons of diesel fuel
and are attempting to get permission from the Port Authority of
New York to transport it to 25 Broadway.

cziarhe	20010912 13:31 UTC
Telehouse have informed us that if commercial power is not restored,
it is unlikely that they will be able to replenish the fuel supply
when it becomes depleted.

In addition, the area in which 25 Broadway is located has been
evacuated.  Essential staff are remaining on-site, but should they
be forced to vacate the building, Telehouse say the ability to
continue to provide emergency power would be impacted.

cziarhe	20010912 12:17 UTC
Sorry, "18:35 UTC" in the previous entry should read "18:35 UTC
on Friday September 14th".

cziarhe	20010912 12:13 UTC
As of 20:35 UTC on September 11th, 25 Broadway is running on the
emergency diesel generator.  Initial information was that they
had 24 hours worth of fuel, but drain is currently much lower and
we have been informed that they could last up to 70 hours, which
takes service up to 18:35 UTC, although this is a maximum.

It is not clear whether refuelling will currently be possible,
or when commercial grid power will be restored.

By Rob

Just another network engineer that enjoys motorcycles and travelling.