Jakob had pointed out that most of the network was down.  (I've been on vacation and hadn't been monitoring.)  Apparently stations went down in the last lightning storm (again!), either Fri or Sat.  I just:

• found a dead power supply in rne01 and replaced it with the one from rne02.  Of course, rne02 is a <repeater> so much of the network is still down.
• reset differential protection on tse13
• reset differential protection on tse11
• reset differential protection on tse09

Of course, this is now <tear down>, so we shouldn't be working to <maintain> the network.  Nevertheless, several groups are still running, using this period for post-cals, etc.  Eventually, DTU will rework the network for operation after we are gone.

While touring my family yesterday, stopped by tse04 and noticed that TRH.2m fan wasn't working.  Cycling power (unplug/plug at DSM) brought it back.  This done about 1230 (17 Jun).  Not that we need the data now, but hopefully this will be a check on whether Ifan monitoring was working.

No changes in instruments, everything seems to have been working as usual for the last few days.

I had to cycle the power on rsw02 twice in the last 24 hours, after it got hung up and stopped responding to pings, so we lost a few hours of data for P.2m.rsw02.

The substitute Ubiquiti radio at tnw05 eventually stopped working for some reason.  When I visited the site on Sunday to fix it, it came back after cycling the power.  Then it went out again by the next day.  So I visited the site again on Monday and swapped the original radio back into operation.  Since then that radio has not had any problems.  Go figure.

I leave today, Kurt and Dan are on their way here with another truck.

Tcase.in from the KZRAD has been missing since 2017-06-02.  It started showing problems on 2017-05-26.  So this afternoon I finally installed the last spare NR01 as a backup, from about 13:45 to 14:45 WEST on 2017-06-10.

It is only level by eye, and there was some disturbance to the ground to wash the radiometers. (I only washed the NR01, I don't know why I didn't think to wash the KZ also...)

I assumed it was better to mount it further from the dark horse beam, but now I see that the legs are probably in view of the NR01 downlooking radiometers.  Let me know if I need to change it.

Sonics and gas analyzers

I think all the sonics are working.

v01 10m IRGA

The v01 10m irgadiag has regular stretches of non-zero values, maybe 40% of samples are good.  I don't know if that's worth trying to fix, given the chance a new head could make it worse.  Maybe it's just a spider web, so if we get a chance to climb we should take a look.

v07 20m LiCOR

The LiCOR at v07 20m (port 3 on v07t) was not reporting, fixed by cycling power.

Other issues

tnw05u

I replaced the Ubiquiti radio at tnw05.  It was able to connect to its AP without climbing the tower to mount it.  Since the original had already been upgraded to firmware 8.1.4, the replacement was also upgraded to 8.1.4, and then I just saved the config from the original and restored it to the replacement.  Traffic to and from tnw05b and tnw05t has been normal, including rsyncs, however for some reason now tnw05u does not respond to ping, ssh, or https access on the WLAN interface.

Sonics and gas analyzers

All sonics appear to be working. The new METEKs needed cal files, so the netcdf data prior to the cal file additions are probably missing values. I think we also fixed an inconsistency in the cal file paths, so sites with more than one dsm may not have been applying the most accurate site-specific boom bearings.

TRH sensors

No change.

Other issues

tnw05u

The radio continued to hang up rsync connections today, even after a few configuration changes to match it exactly with a working radio, rsw06. The latest attempt was to upgrade to the latest firmware, 8.1.4, but even that did not work. We may try adding another radio in its place.  This is being tracked in ISFS-152.

Generating stats_5min.xml

Isabel has written a python script to generate the stats xml file from the sensor list, to reduce the chance for human error in generating in manually. She found several missing TRH sensors and one sonic at the wrong height. Once we've compared the netcdf output using the generated xml file, the manual file will be replaced.

Sonics and gas analyzers

No known problems at this time. Unlike the RMYoungs they replaced, the METEKs at tnw07b 4m and v07 8m do not report ldiag, so those are all missing in the QC tables, but I don't know at the moment how to fix that.

TRH Sensors

The only sensor known to be down at this point is the 60m on tse11.

Other issues

v05

We visited v05 to investigate why it was offline, even though the Ubiquiti was still reachable. Eventually we rebooted the Ubiquiti and the connection came back. The DSM had been up the whole time so no data were lost.

tnw05 rsync

The tnw05u radio continues to be a problem, hanging up rsync connections. No explanation or fix yet.

v04

Tcase.in is still down and nothing has been done about it. We still need to work on mounting a NR01 on the dark horse with the KZ.

Sonics and gas analyzers

v07 8m

Replaced RMYoung with a METEK, using the same port since all are filled. The arrow is pointing towards the tower in direction of boom. We discovered that the fuse to the serial interface boards would blow eventually once the METEK was running, so we replaced the 1A fuse with a 3A. The configurations have been updated and all appears to be working now. Port 1 is jumpered for RS485.

tnw07b 4m

Replaced RMYoung with a METEK, using port 5, now jumpered for RS485.

TRH sensors

No change.

Other issues

rsync

We have added rsync monitoring to nagios, and discovered that a few sites are days behind on rsync.  It turned out the problem was the Ubiquiti not allowing certain traffic, same symptoms as documented in ISFS-152. Two of the four systems were tnw05t and tnw05b, the same ones as reported in the original issue.  The problem keeps happening within several hours after rebooting the Ubiquiti, so there must be something particularly wrong with the radio at that site.

Sonics and gas analyzers

tnw09 10m

Replaced CSAT3A sonic head at 10m on tnw09. At first it reported all bad samples. So we plugged in the original again, thinking maybe some spider webbing had been interfering, but it still reported bad samples also. So we left the replacement installed, and lo and behold, a few hours later the diagnostic bits turned to zero and remained zero the next day. Go figure.

v07 8m

Had a METEK tested and ready to go, did not have time to install it.

tnw07b 4m

Had a METEK tested and ready to go, did not have time to install it.

Other issues

ARTSE

I have one full EC150 tested and set aside.

Sonics and gas analyzers

v07 8m

RMYoung on port 1 is still bad, but now we know that it is actually at 8m.

tnw09 10m

CSAT3 IRGA winds on port 2 still about 50% flagged.

tnw07b 4m

RMYoung on port 1 keeps going in and out. It looks like the bad samples peak in the afternoons, similar to v07 8m.

tse01 10m

No sign of problems since going out for several hours on 2017-06-02.

TRH

tse11 60m

No change.

Other issues

There seems to be a problem with the winds in the high-rate dataset not being oriented correctly to geographic coordinates, so I need to investigate that.

tse05 finally stayed up overnight.

v04 Tcase.in is still out. I will figure out how to mount our last NR01 on the dark horse and just record it as an additional radiation sensor.

Around 2017-06-03,14:00 UTC, Isabel and I visited v07 to attempt to replace the flaky RMYoung in port 1 with a less flaky RMYoung. We were able to use a ladder to reach the 4m boom and swap in the new sonic, but that did not improve anything on port 1. We discovered a fuse was not quite plugged in the whole way on port 3, so plugged it in. And at some point we "lost" all the serial ports. Probing the bulgin pins only showed 5V on some pins and no pins with 12V. So just to be sure, we powered down the DSM and replaced all the fuses leading up to port 1, including the 7.5A and 3A fuses on the power panel. After that all the ports resumed working, but I don't know if that means we really did have a partially failing fuse somewhere (seems doubtful), or something else got into a funky state.

We still saw almost all flagged samples from the RMYoung on port 1. So then we discovered that the configuration was incorrect. Port 1 is at 8m and not 4m, so we had replaced the wrong sonic. We unclipped all the cable loops hung on the tower and mapped them to their sensors, here's what we found:

So ports 1 and 3 were swapped, and I've fixed the configuration to match.

This means the sonic that really has been failing during the day is at 8m, so we cannot replace it without climbing. During our visit, I thought we determined eventually that the replacement did not work any better, but now I'm not so sure that we were looking at the right sonic, so maybe we still have one RMYoung with 2% flagged samples which could replace the 8m. Otherwise we have a METEK we could install, whenever we're able to climb.

Note that we disturbed the flow for the 2m CSAT sonic during our visit because we were working right next to it.

Sonics and gas analyzers

v07 4m

The plan is to replace it with the last RMYoung. 2% bad samples all the time would be better than 2% good samples during the day.

tse01 10m

CSAT3A on port 2 is reporting all bad samples as of 2017-06-02,08 UTC. So we only have to replace whichever of head or box is bad, or both, but we have spares. Cycling power did not help.

tnw09 10m

Lots of flagged samples still. This is a CSAT3 IRGA. Up to %30 of the winds are flagged in qctables, need to check on the gas diagnostic.

tnw07b 4m

Diagnostic still mostly bad.

TRH Sensors

tse11 60m

Still needs to be replaced.

Other issues

tse05 power

Isabel and I installed a second solar panel at tse05 around 16z. The panel is more horizontal and is aimed more west, since the first panel is aimed more east. It Looks like voltage jumped up a tenth, but maybe there will not be enough charging left today to get it through tonight. Maybe tomorrow night.

Sonics and gas analyzers

rsw02 20m

Replaced around 16z with help from Orson (ND).

v07 4m

Since this is a RMYoung, the problem could be related to voltage, but we have no more 14V adapters. The easiest replacements are the last working RMYoung, which reports about 2% bad samples in the ops center, and a Metek, since the Meteks mount onto the Samortechnica booms.

We visited the site around 17z, and measured 12.45V at the port 2 bulgin pin, and 12.85V at the power panel input pin. So I'd guess the RMYoung is getting enough volts since it is only 4m up. Vdsm is always more than 12V, even though it is measured at a soil mote 5m from the DSM.

I forgot to bring a ladder to look at the adapter in the junction box, so at the moment I'm assuming it's a 14V and it's performing well enough that the voltage should be high enough for the RMYoung.

tnw09 10m

Diagnostic flipping back and forth.

tnw07b 4m

Diagnostic mostly bad.

tse10 10m

Some spiking.

TRH Sensors

tse04 60m

INEGI replaced it, working now.

tse11 60m

Still needs to be replaced.