Month: October 2016

BES Operation Horror Story 4 – Lack of Training, Situational Awareness and Protection Misoperation.

The following one-line is relevant for this story:

SLLA ==== CNTC —- PALM —- FATC

After an all-weekend line reconductoring outage on one of the two parallel 115 KV lines between the SLLA and CNTC substations everything seemed to be smooth sailing that afternoon. For the duration of the outage, all the load in what constitutes a single 115 KV loop around the region as shown on the one-line above, is being fed radially from the SLLA substation through the remaining parallel line by opening the breaker of the CNTC-PALM 115 KV line at CNTC and dividing the load just as simulated in the  load flow runs. The 38 KV lines between the same substations have N.O. ACB’s in them to avoid parallel flows over their ratings.

The time comes to energize the line, the breaker at SLLA was closed and the line was energized, the open breaker at the other end at CNTC failed to operate close through SCADA multiple times. The breaker at SLLA is opened and the breaker at CNTC closes on by SCADA. When the breaker at SLLA was commanded to close, it fails to close but luckily a substation maintenance crew wrapping up their work right at the substation was called to verify the “inoperative” GCB, they found nothing wrong with it other than the closing coil isn’t getting energized whenever the breaker control handle in the control house is rotated to the close position.

Easy-peazy, the substation maintenance supervisor resolves to energize the GCB’s closing coil directly, the operator agrees on this and the breaker at SLLA did close, coincidentally the breaker on the parallel line to SLLA at CNTC trips.
The CNTC load was left radially fed by the freshly reconductored SLLA – CNTC line while the parallel line was also radially energized with no load from SLLA.
The operator still unaware of the overall picture issues a close command on the open CNTC breaker of the CNTC -SLLA line, it closes! The GCB of the reconductored parallel line at SLLA trips too!

The substation crew notices a led flag on one of the line protection relays at SLLA. After some measurements, the relay techs determine the breaker isn’t closing because the line is not synchronizing to the system.

The operators got lucky because when their phases got swapped – by pure chance – the rotation remained the same as the original system’s.

Other utilities have not been so lucky and they’ve left a couple of towns and industries with their phase rotation in the opposite direction…for days!

BES Operation Horror Story 3 – Physical Security Fail

A transmission operator worked in the Energy Control Center for, at least, five years before relocating to the continental US eventually getting certified as a Security Coordinator for one of the five ERO regions.
One day while on vacation he finds a quick break to visit his buddies at his old place of work.
Gets there, door is propped open, walks into the deserted ECC and calls for someone; no answer. Tired of waiting sits down in one of the two unlocked EMS consoles. The phone rings and again no one answers so he decides to pick up; it’s a supervisor closing a work order in a transmission line. Closes the work order step in the line clearance document.

A line voltage hi-limit exceeded alarm appears along its audio indication, after a couple of minutes he acknowledges the alarm.

The transmission operator finally emerges…cooking back in there in the kitchen.

BES Operation Horror Story 2 – International Embarrassment!

An executive visit to a South-American country is in schedule, one of the three executives backs off so I was volunteered to go. Our mission is to reach an understanding, to start studying the economics of laying an underwater transmission cable interconnecting both systems.
Everything is going well, presentation after presentation of markets, prices, fuels, etc. On the second day, what I considered the highlight of the trip (for me anyway), a visit to their energy control center, up in the image above.
A mezzanine over the control center gives you a vantage point, an eagle-eye view of everything going on. After their Operations chief recites through what seems the standard spiel, he’s about to end the presentation then I start with specific system operation questions; generation, loads, EMS, frequency bias, ACE, contingencies, SPS’s, UFLS, etc. All my questions are answered very professionally and then, with an very seriously concerned face, “our system is very stable akin to an infinite bus, unlike your island in the Caribbean”.
Before I can recover from what he just said and try to politely answer or not his statement, behind him I happen to notice the ACE going to around +100 MW and the frequency up to about 60.18 Hz. Aha! How often the ACE goes +/- 100? I casually ask and he answers “once a month, maybe”. “Oh, just like right now” I say. That was a too-easy cheap backhand shot I had to take.
As he turns toward the glass pane, the mapboard now has 2 breakers flashing green at one substation and down at the bottom another 2 green flashers…then they all turn solid green so I ask about what just happened.
Takes a look and tells me that two transmission ties with a neighboring country to the south tripped and I know he keeps talking but my attention is already locked in the mapboard and the situation below.
One of the bottom breakers flashes red then goes dark, the same with the other sometime later. (Aha! Lines’ energized)
The big screen to the right of the mapboard has the transmission alarms summary page up, one of the open breakers turns red then green and I see the fail to operate alarm popping up in the display, this happens twice. Then the exact thing happens with the other breaker…My questions keep coming as if nothing happened:
So, are there any other ties to this country? No
Not even at any other voltage? No
“Well the breakers in your country are not closing because both areas are out of sync” I, matter-of-fact say.
The operations chief’s face turns cherry red, grabs a flip-phone out of his pocket and angrily barks a couple of instructions to the operators below…while trying not to raise his voice.
The submarine cable project never happened, the economics did not support the investment.

BES Operation Horror Story – From 60 to 0 Hz in 30 minutes

From 60 to 0 Hz in 30 minutes

I was in my office in the operations department training three new recruits to become transmission system operators, one of their tasks involved data input in another boring MS Access form. It’s about 8:15 in the morning and the lights blink when instinctively my heart skips a beat but the lights remained on so I continued about the training. Another coworker walks in and says they had a 230 KV line trip and reclose; a once every other month event – nothing to worry about too much.
Work was getting a little bit routine in the transmission clearance office but to keep things interesting I had an analog frequency meter over my monitor just in case and earlier that morning I was showing the trainees how to change the alarm settings on the meter by adjusting two movable indicators on the dial, crude but did the job. I remember those alarms got to be distracting at first so I set them up at 57.5 Hz(the lowest step of UFLS where it would never go!) and 60.2 Hz. About an hour later I jump from my chair as the alarm goes off and a quick gander to the now left leaning needle 57 Hz!
Nah, probably a malfunction but I walk as fast as I could to the Energy Control Center and the card reader doesn’t work, through the glass wall and doors the operators have a deer-in-the-headlights look on their faces and after what seemed an eternity the door finally opens

54.2 Hz

the frequency meter read. The operations division head tells me in a broken voice to open the substation 1346 breaker (the local 115/13 KV substation that also serves the ECC, about 30 MW’s of load- we still had a wired control of it from the mapboard). As I walk towards the mapboard and right before I select the breaker to open I look back again at the frequency meter:

40.2 Hz

The breaker opens and the faint drone of the emergency generator running is the only thing heard amidst an eerie silence as I glance back

0.0 Hz

Someone received a call from the alternate control center in the south of the island, they were still on! “go to the helipad they’re going to pick up and fly you to Ponce” with a broken-trembling voice the division head said, it was the first time I had to fly in a helicopter with no doors. In no time I was operating a system controlling the frequency by instructing hydro generators on one phone while giving instructions to larger thermal units on the other.

From 3,300 to 800 MW. What happened?

When the 230 KV line was first tripped by a tree in the right of way, the automatic reclosing of it was blocked by the operator under instructions to avoid “the switching disturbance” in an election year. The line tripped 20 minutes later and it remained out. Meanwhile a high percentage of the system generation remained tied to the system through a single 230 KV line, this line went immediately to 98% of its rated capacity. The operator redispatched the generation to minimize the possible effect of the contingency but this line tripped about 25 minutes later (it was later determined the line sagged to the point of breaking clearance against a distribution line).
When the second contingency happened, the system split into two separate systems, the generation-poor side to the northeast blacked out but the generation-rich one to the southwest remained online and stable.
One of the operators in training never came back.