Blog from May, 2015

Rudy just pointed out that at 18:00 today, data from everything on the BAO tower stopped.  I also note that I can't log in to flux.  Thus, it is either a total power outage at the tower, or something got messed up with eol-rt-data.

Rudy noticed that these sensors were down.  After resetting power on both:

200m: TRH30 9.49 65.31 8 0 1238 129 26\r\n

(fan has low current)

300m: TRH16 -40.14 0.00 9 0 0 0 30\r\n

(sensor is essentially dead)

Both sensors are going through frequent reset cycles as well.  I don't know if we even have replacement sensors ready to go.  (Or whether we should bother with fixing this with one week left in the project, officially.)

(It probably is obvious, but there have been a lot of thunderstorms today that may have helped these sensors die.  Both sonics at each level are still chugging away, though they all are reporting a 0x0f in the status, which may indicate wet transducers?)

Was at the bao site from about 1315-1415 today to:

1. Fix soil mote 1, that was in a reboot cycle thought to be due to a shorted Tsoil probe.  Forgetting that there were 2! Tsoil probes at this site, I just unplugged the first one I saw.  It appears that I randomly removed the correct one (the one that has had a bad reading for a while), however it still took several reboots and resets of various things (including the DSM) to get everthing up.
2. Take soil core samples as a wet extreme.  The soil was quite sticky, though 20cm was perhaps a tiny bit drier.  I'll update the gravimetric entry with the new weights.
3. Give a tour of the site to Jielun and 2 visitors.
   
   

Sensor status: Everything is wet...

T: ok

RH: ok

Ifan: ok

spd: ok

P: ok

co2/h2o: wet

csat u,v,w: lots of missing data

csat ldiag: lots of flags

soils: Tsoil.x.4.4cm.bao high

Wetness: ok

Rsw/Rlw/Rpile: ok

Voltages: ok

sstat outputs: ok, ok

Sensor status: Soil Vmote went down

T: ok

RH: ok

Ifan: ok

spd: ok

P: ok

co2/h2o: wet

csat u,v,w: some missing data

csat ldiag: lots of flags

soils: NA

Wetness: ok

Rsw/Rlw/Rpile: ok

Voltages: Vmote.soil.bao is down

sstat outputs: ok, ok

Rudy just noted that T.300m was down.  It was much more than that.  The pocketec was offline and data acquisition wasn't working.  mount and dup got it going again, but there appear to be no data (TRH or sonic) during the last 27 hours (since 5:30PM Friday).

P.S. rsync appears to have run automagically to bring the 15May files from the pocketec.

two power outages today.  150m now doesn't come up – no ping or ssh.  Bruce is on his way up the tower to see what is going on (hopefully an issue with his power cube providing DC power to the DSM).  200m came up, but has time starting at Jan 1 2000.  I've just rebooted 200m to try to fix this.

P.S. Rebooting still has time in 2000.  This DSM doesn't have a GPS and it was timeserving from 150m.  Gordon is fixing this.

The TRH at 300m went nuts between 17:45 and 18:00 MDT today, May 13. The check_trh.sh script on the DSM repeatedly power cycled it because Ifan was low:

May 13 17:45:46 300m root: Ifan is 9 . Power cycling port 5
May 13 17:46:29 300m root: Ifan is 9 . Power cycling port 5
May 13 17:46:45 300m root: Ifan is 8 . Power cycling port 5
May 13 17:47:01 300m root: Ifan is 2 . Power cycling port 5
May 13 17:47:18 300m root: Ifan is 1 . Power cycling port 5
May 13 17:54:12 300m root: Ifan is 7 . Power cycling port 5
May 13 17:54:29 300m root: Ifan is 0 . Power cycling port 5
May 13 17:54:50 300m root: Ifan is 6 . Power cycling port 5
May 13 17:55:06 300m root: Ifan is 0 . Power cycling port 5
May 13 17:55:22 300m root: Ifan is 0 . Power cycling port 5
May 13 17:55:39 300m root: Ifan is 0 . Power cycling port 5
May 13 17:55:55 300m root: Ifan is 0 . Power cycling port 5
May 13 17:56:11 300m root: Ifan is 0 . Power cycling port 5
May 13 17:56:27 300m root: Ifan is 0 . Power cycling port 5
May 13 17:56:44 300m root: Ifan is 0 . Power cycling port 5
May 13 17:57:00 300m root: Ifan is 0 . Power cycling port 5
May 13 17:57:16 300m root: Ifan is 0 . Power cycling port 5
May 13 17:57:33 300m root: Ifan is 2 . Power cycling port 5
May 13 17:57:49 300m root: Ifan is 0 . Power cycling port 5
May 13 17:58:05 300m root: Ifan is 0 . Power cycling port 5
May 13 17:58:21 300m root: Ifan is 0 . Power cycling port 5
May 13 17:58:38 300m root: Ifan is 0 . Power cycling port 5
May 13 17:58:54 300m root: Ifan is 0 . Power cycling port 5
May 13 17:59:10 300m root: Ifan is 0 . Power cycling port 5
May 13 17:59:26 300m root: Ifan is 0 . Power cycling port 5
May 13 17:59:43 300m root: Ifan is 0 . Power cycling port 5
May 13 17:59:59 300m root: Ifan is 0 . Power cycling port 5
May 13 18:00:15 300m root: Ifan is 0 . Power cycling port 5
May 13 18:00:32 300m root: Ifan is 0 . Power cycling port 5
May 13 18:00:48 300m root: Ifan is 0 . Power cycling port 5
May 13 18:01:04 300m root: Ifan is 0 . Power cycling port 5
May 13 18:01:20 300m root: Ifan is 0 . Power cycling port 5
May 13 18:01:37 300m root: Ifan is 0 . Power cycling port 5
May 13 18:01:53 300m root: Ifan is 0 . Power cycling port 5
May 13 18:02:09 300m root: Ifan is 0 . Power cycling port 5
May 13 18:02:25 300m root: Ifan is 0 . Power cycling port 5
May 13 18:02:42 300m root: Ifan is 0 . Power cycling port 5
May 13 18:02:58 300m root: Ifan is 0 . Power cycling port 5
May 13 18:03:14 300m root: Ifan is 0 . Power cycling port 5

After all that it seemed to recover. at 6:30 pm the data looked like so, with a reported Ifan of 36 mA:

TRH16 9.33 80.63 36 0 1226 161 114\r\n
TRH16 9.33 80.63 37 0 1226 161 116\r\n
TRH16 9.29 81.09 37 0 1225 162 115\r\n
TRH16 9.29 81.09 35 0 1225 162 111\r\n
TRH16 9.25 81.08 35 0 1224 162 111\r\n
TRH16 9.25 81.08 37 0 1224 162 116\r\n

• rad.bao was down for most of Fri & Sat – don't know why – Xbee reset should have been running
• tsoil.bao was spotty on Fri – don't know why
• Sat most sonics (including 2D at 10m.bao) were pretty bad in rain/snow
• need to add p.5m.bao and tc.5m.bao to WWW plots.

Just a reminder that the EC150 bandwidth parameter in the project config needs to be set to match the EC150 settings, since it is used to determine the sample delay time.  Often this won't be a problem, however we are now correlating the nanobarometer and EC150 winds for CABL and it turned out that we had set bandwidth to 25, rather than 5Hz.  This has now be set to the correct value, causing w'p' cospectra no longer to show a phase lag.