After the daily meeting, and a trip with Lou to grab stakes, ran back up to rsw08 to reset its power. Found that all INPUT power in the fuse box was 0. Was mystified, then got back to the ops center to find that the entire network was dead except for 3 towers (rne06, tnw09?, and tnw12, the latter 2 presumably paired with rne06). Suspected a widespread power failure, so went to upper Orange Grove to look at breaker panel servicing tse13. Found Lou with power and nothing obviously tripped, thus a journey up the mountain was needed....
Drove the path: rsw01/02/03/tse13/12/11/10/09. Found one blown DC supply (rsw03), but didn't have a spare, so had to return later on. The differential protection had tripped at all of the tall masts. Resettting appeared to bring these back to life.
Next drove up to tnw11 and walked to rne07. Found another blown DC supply, and still didn't have a spare so had to drive back to ops and up again. That brought rne07 up. Also had to hit the differential protection at tnw10.
Finally, drove back up to rsw (since I had seen none of the rsw masts come online). Nothing had tripped on the main breaker panel (near rsw06) and all the fuses checked okay. Nevertheless, I cycled all of the breakers and differential protection switches. After that, checked rsw06. The differential protection had tripped (for the tower, but the DLR lidar was okay), yet input power was still 0 on all phases. Same with rsw05. Conclude that service by the electrical contractor is needed.
Thoughts:
- This network is unservicable. Who would put a major network access node in a location where you have to hike to?
- The power system is unservicable. At every one of the towers, you have to carry an extension ladder to access the fuse box which needs a tumbler-type key. (Including rne07 that you have to hike up to.)
- Our DC supplies all should have differential protection circuits. Fortunately, we have a lot of these supplies – at least the 12V versions.
- There needs to be a better system to distribute information in a timely fashion. I resorted to creating a fake daily briefing message, because I knew it would be sent to all participants. The "boss phone" is fine for receiving updates, but not the other direction.