S3 was still down this morning, so it was a good excuse to take the new boat out for a spin. ifdown eth1/ifup eth1 got the network back up and all was well. As usual, data were saved locally, so no data were lost.
Despite the fun, this is getting old, so I've now installed (updated and put into crontab) the net_check script on every system in the marsh. Previously, it had only been running on S8. As another test, I did this all on eddy which is now running VPN (thanks, Gordon).
9/26/16
Summary: Light winds and clear skies again today. Station 3 was not reporting this morning. A short trip to the field has it back online.
Actions past 24 hours:
- Yesterday (9/25) Quiet day in the command post.
To dos:
- Reshoot boom angles (just because...)
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: offline
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
I've been away this afternoon, and the netcdf files were down, so I don't know how long this connection has been down. Hopefully, it will come up later. If not, it will be a perfect use for my new boat!
Netcdf files hadn't been updated at EOL since this morning.
data_stats sock:barolo showed no data.
nidas_udp_relay was running on eol-rt-data, though it had been restarted this morning at 09:50 MDT. It wasn't due to a reboot, it's been up for 50 days.
On eol-rt-data, "data_stats sock::30010" showed data coming in. Or you can do, from any system at EOL:
data_stats sock:eol-rt-data.fl-ext.ucar.edu:30010
These errors started showing up in /var/log/isfs/isfs.log, every 10 seconds:
Sep 25 09:41:10 barolo dsm_server[44405]: ERROR|SocketConnectionThread: IOException: inet:128.117.188.122:30010: connect: Connection refused
Eventually the socket open succeeded, but then this error:
Sep 25 09:50:11 barolo dsm_server[44405]: WARNING|SampleInputStream: inet:128.117.188.122:30010: raw sample not of type char(0): #bad=1,filepos=0,id=(609,25461),type=28,len=779247971
As with reading disk data, the reader skips forward one byte and looks for a good sample.
Not sure why the corrupt data, and why it didn't recover. Would be good to look at the logs on eol-rt-data.
Did a kill -TERM of dsm_server on barolo, and ran check_vertex_procs.sh by hand, rather than waiting for crontab.
Updated crontab to check the procs every 15 minutes, rather than 30.
Date Time | Position | Tare (g) | Wet (g) | Dry (g) | rho (g/cm^3) | Moist (%) | Moist (%) | Comments |
|---|---|---|---|---|---|---|---|---|
9/24 09:38 | S9 3-6cm | 8.2+73.3 | 159.2 | 99.2 | 0.27 | 90.3 | 67.4 | Had to undercut sample to break it out of root mat |
| 9/27 15:50 | S15 3-6cm | 8.2+73.3 | 162.9 | 103.9 | 0.34 | 88.8 | 64.7 | Undercut; tenacious roots; rusted corer; core slid w.r.t. rings |
| 10/11 14:30 | S1 3-6cm | 8.2+73.3 | 165.1 | 99.7 | 0.27 | 98.4 | 62.4 | Tenacious roots |
tare = c(8.2+73.3,8.2+73.3,8.2+73.3)
wet = c(159.2,162.9,165.1)-tare
dry = c(99.2,103.9,99.7)-tare
vol = c(3)*pi*(5.31/2)^2
moist = 100*(wet-dry)/vol
rho = dry/vol
m = moist
grav.moist = m
grav.comp = m
ec5 = c(67.4)
matplot(grav.comp,ec5,xlim=c(0,60),ylim=c(0,60)); abline(0,1,lty=2); abline(-8,1,col=3,lty=2)
9/25/16
Summary: Calm and clear skies today. All stations working normally this morning.
Actions past 24 hours:
- Yesterday (9/24) Reset tower 12 cell modem, took a soil sample and retrieved the S1 QSoil sensor which isn't reading.
To dos:
- Reshoot boom angles (just because...)
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: offline
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
Tonight, noticed long periods of wind outages at S8. The rserial output show that these period have data values of 999.99 with an error code=04 (the manual says that the value of this code provides "no useful information to the user"!). However, I also see reasonable data interspersed with the bad periods. I suspect a bird is roosting in the array (despite Dan's heroic effort at making bird spikes) and that this problem will go away. Just to do something, I ddn/dup and eio 6 0/1, to no effect.
We've noticed that tower's battery capacity is marginal to bridge the cloudy days that we've had recently. Thus, we decided simply to add a battery to tower. This was just done a few minutes ago. We managed to keep the first battery connected through this change, so the station and power mote configuration stayed up.
9/24/16
Summary: Light winds and cloudy skies today. Station 12 was not reporting in this morning. A field trip found the cell modem was not working. Reset the tower and left when it all appeared to be working. Recovered the QSoil sensor from Station 1 and took a soil sample from Station 9 while getting some boom angles from 1, 9, 10, and 12 to round out this trip.
Actions past 24 hours:
- Yesterday (9/23) investigated TRH and decided the fan current sensor board was faulty. Found corrosion on several pins in connectors and one wire had corroded and broken off under the heat shrink covering a splice. Repaired the wire, replaced the board, tested and found the unit runs slowly but shows 329mA of current which is not right either so it will have more time spent on it soon.
To dos:
- Reshoot boom angles (just because...)
- troubleshoot mote from S9
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: offline
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
S1: We knew that Qsoil here has been mostly down (a few good samples have come through). No obvious issues with cables/connectors/etc., but still bad Qsoil values (nan). Pulled entire probe out of the ground and brought back to trailer. (In the process, disturbed Gsoil, which did change briefly by as much as 0.5 W/m2, but seems to have settled back in.) In trailer, find that the Qsoil PIC continuously resets when a probe is connected, and never is able to give it the full 3.3V excitation pulse. Either the EC-5 is dead, or something is wrong with the PIC front-end to the probe. (The rest of the PIC runs fine.) Further debugging needed, but in the meantime, no Qsoil at this site. (Since we don't have a spare EC-5, the only other option is to take the Qsoil PIC to another site (9 or 15) and swap it in as a test. I'm sorry I didn't think of this when we were at S9 this morning...)
Thought about taking core, but rather useless if no Qsoil!
Shot boom angles (though forgot to bring monopod, so a bit shakey).
S9: Grabbed a "soil" core from here at 0938. Had to undercut the core in order to remove it. (The entire corer pulled out, leaving the plug behind the first time.) Mostly a peat-type mesh of roots. Core now being processed in lab.
Shot boom angles.
S10: Shot boom angles.
S12: We knew that this had been off the net since last night. The cell modem had a yellow status light. Unplugging and replugging the modem caused its LEDs eventually to go to green, but had to reboot to get the system to connect. After the reboot, all was fine. "lsu" showed that local storage had been working fine, so no data lost.
shot boom angles.
Attempted to walk to S13 to shoot angles, but couldn't cross the ditch.
S8 data were offline from about 07:45-08:03 this morning. Given that the crontab net-check script is scheduled to run at :37 after every hour, this is not what brought it back. Thus, this was just a typical Verizon hiccup, not what we dealt with 2 days ago. Local data storage was unaffected.
9/23/16
Summary: Issues with S5 TRH fan again. Low winds with mostly cloudy skies for calm conditions. Made a morning trip to Station 5 to investigate the TRH ultimately replacing it with the unit that had been brought back previously but hadn't shown any issues.
Actions past 24 hours:
- Yesterday (9/22) since last report Station 8 has operated normally.
To dos:
- Reshoot boom angles (just because...)
- troubleshoot mote from S9
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: offline
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
Died about 11pm last night. I just restarted it with control-R, and it seems happy again.
We have no idea why this is happening – always at this site, three different TRHs tried. I guess we'll throw together a DSM script to send control-R to it periodically.
9/22/16
Summary: All stations are operating normally this morning with sunny skies and light winds.
Actions past 24 hours:
- TRH from S5 has been running on the bench since 14:00 9/21 without issue. It will stay running on the bench today. Some corrosion is showing on the sensor back plate so it's likely light corrosion on a contact caused the previous problem.
- Two trips to scenic S8 to investigate why the cellular modem was not working has resulted in suggestions to use an external antenna and some discussion on why it has suddenly had issues.
To dos:
- Reshoot boom angles (just because...)
- troubleshoot mote from S9
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
We just did a quick trip to the marsh (just about an hour base-to-base) to revive S8's networking. I got it running with: su; ifdown eth1; ifup eth1. I tried switching the USB positions of the stick and the modem, but they don't fit the other way.
After talking with Gordon, I'll try to implement his old script to check networking and do ifdown/up to attempt to revive. 2 trips to S8 in a day is fun, but I don't want to make a habit of it.