Site visit May 4 (network outage, TRHs, tower)

Karl Schwenz, Chris Golubieski, Gordon Maclean

Objectives:

• Diagnose and fix reason that system has been offline for last few days (no data in Boulder after April 27).
• Replace data system box in order to take the old box back to Boulder for a tuneup.
• Replace top TRHs which have not been working well. Add new radiation shields.
• Replace barometer with external unit.
• Make anti-climb door easier to latch.

Arrived at site, May 4 10:30.

Chris climbed tower and replaced the TRH transducer/PIC units, and replaced the radiation shields with picnic plates .

Replaced the barometer. New unit is B1 with chassis outside of the DSM. Old was B9, inside the DSM.

Karl fixed the anti-climb door. Works like a breeze now. If door is ever difficult to latch give a hand knock on the bottom right corner to shift it to the left. Hinge bolts should be replaced with correct size on a future visit.

System was up and taking data. (Reporting about 1000 spurious interrupts per second, then later 500 after Chris replaced some TRHs). The pocketec disk has data files for April 28 to the present, so no loss of data. Could not ping manitou server, 192.168.100.1, at other end of fiber link. Power cycling the fiber/copper media converter at the tower did not help. In retrospect I should have also cycled power on the 5 port switch. Cycling power on the media converter in the seatainer also did not bring it back. Only the power LED was lit on the converter at the tower, not SDF (signal detect fiber), SDC (signal detect copper), or RXC/F (receive copper/fiber). I am not sure whether the SDC LED was on in the seatainer.

Site note: There is a strong smell of what I believe is butanol in the seatainer where the turbulence tower fibers are terminated. This is the first seatainer as one drives into the site. There are several butanol bottles in that seatainer.

The manual for the fiber/copper network converters (Transition Networks, model E-100BTX-FX-05) describes the dip switch options. I've cut and pasted some of the manual to the end of this entry. It seems that we should disable the link pass-through and far-end fault options on both converters. Link pass-through and far-end fault options can cause a copper interface to be disabled if a fault is detected on the remote copper interface or on the fiber. These options appear to be for a building network with good network monitoring equipment. Perhaps a power outage, cable disconnect, or powering down a network switch could cause the converters to think there is a fault in the copper ethernet or the fiber, and then the copper interfaces would be disabled. The manual says nothing about whether the interfaces are automatically brought up if the fault disappears.

Dip switch 1 is auto-negotiation of speed and duplex and that was left enabled=UP. Set switches 2, 3 and 4 DOWN on both converters, to disable pause control, link pass-through and far-end fault.

Pause control should be enabled if ALL devices attached to the media converters have it. I didn't know if the ethernet switches have it, so I disabled it (dip switch 2 DOWN). I now think that was a mistake and that all modern ethernet switches have pause control. So I suggest that we set switch 2 UP on both units, and power cycle them on the next visit.

With 1=UP, and 2,3,4=DOWN and power cycling both units, the link came back and we could ping the seatainer. Swapped data boxes and then couldn't ping. Eventually power cycled the 5 port switch at the tower and things worked again.

New data box is swapped in and all sensors are reporting. Taking the original pocketec back to Boulder so the system at the tower now has a different pocketec unit.

From the manual for the converters:

Pause Control
The Pause feature can improve network performance by allowing one end of
the link to signal the other to discontinue frame transmission for a set period
of time to relieve buffer congestion.
NOTE: If the Pause feature is present on ALL network devices attached to the
media converter(s), enable the Pause feature on the media converter(s).
Otherwise, disable the Pause feature

Link Pass-Through
The Link Pass-Through feature allows the media converter to monitor both the
fiber and copper RX (receive) ports for loss of signal. In the event of a loss of
an RX signal (1), the media converter will automatically disable the TX
(transmit) signal (2), thus, “passing through” the link loss (3). The far-end
device is automatically notified of the link loss (4), which prevents the loss of
valuable data unknowingly transmitted over an invalid link.


near-end <-1-> local media -2->  remote media <-3-> far end
device         converter   <-4-  converter          device

original fault   local media cvtr       remote disables
on copper link   sends loss signal      ethernet device
(1)              over fiber (2)         (3)

Far-End Fault
When a fault occurs on an incoming fiber link (1), the media converter
transmits a Far-End Fault signal on the outgoing fiber link (2). In addition the
Far-End Fault signal also activates the Link Pass-Through, which, in turn,
disables the link on the copper portion of the network (3) and (4).

                      original
                      fault on
                      fiber (1)
near-end <-4-> media     -1->  media converter <-3-> far end
device         converter <-2-  converter             device
               A               B

media converter                               media converter B
A disables copper                             detects fault on (1),
link (4)                                      disables the copper (3)
                                              sends far-end fault
                                              signal to A over fiber (2)