Completed to-do list for today
Site 1
Capped off a connector at the mote with tape to protect it.
Site 8
Measured azimuth angles for 3m and 10m 3D sonics. They measured 222 degs and 308 deg, respectfully.
Site 11
We found the data had kept recording at this site, although it wasn't be seen over the network. The modem would not connect, and the net_check.sh script (runs at 40 minutes after the hour) would not bring the connection up. Unplugging and re-plugging the modem did the trick. The CPU at this site is a viper.
On the status web page for the modem, the signal strength was -102 db (showing between 2 and 3 bars) with SNR of 0 for the existing 6" antenna. We connected a larger 12" whip antenna, and held it up by hand. There was no increase in gain so we left it with the 6" antenna. Perhaps hand holding the antenna was not a good test (a worse ground plane than the installed antenna?).
The system battery voltage measured was approx 13 volts (the power monitor on S11 is out).
Site 15
Measured azimuth angels for 3m and 10m sonics, they were 225 deg and 309 deg respectfully.
Cleaned the radiometers at 4:46 pm EDT.
Wet over night, 105% RH. Partly cloudy today and cool, light breeze.
To dos:
- Reshoot boom angles at S15 and others sites that previously have only had one
- Second soil core at S15? (Steve do you still want this? Gordon and I were only aware of one at S1 which was done today)
- Check out failure of S11. It's not reporting, but we don't know why.
Station Status: S11 is down.
T/RH: 42m TRH came back after being power cycled. Others okay (note that S1,2,9,10 are wired not to report Ifan).
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
3D RMY: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: Ready to manually read sensor
Rainr: ok.
Vbatt: Vbatt.tower at 12.6V on 10/9
Clear skies, light wind. Beautiful Day
To dos:
- Reshoot boom angles at S15 and others sites that previously jave only had one
- Second soil core at S15? (Steve do you still want this? Gordon and I were only aware of one at S1 which was done today)
Station Status: All up.
Main Tower
Gordon just got the TRH sensor at 42 m on the main tower up and running by restarting it.
Station 1
Soil core at S1 completed. Baking now.
Successfully installed TP01 and capped off end and taped it up to keep it out of water and mud during high tide.
Station 13
Cleaned the EC150 3D sonic cable end with contact cleaner and cleaned the cable receptacle. It restarted with success and is collecting data.
Reshot boom angles of sonics at 10 m and 3 m. The 2D sonic at 10 m had an azimuth of 319 deg. The 3D sonic azimuth was 227 deg.
Sensor Status:
T/RH: 42m TRH came back after being power cycled. Others okay (note that S1,2,9,10 are wired not to report Ifan).
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
3D RMY: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: Ready to manually read sensor
Rainr: ok.
Vbatt: Vbatt.tower at 12.6V on 10/9
Kate and Gordon
Removed CSAT SN 1117 on s13. Replaced it with EC150, sonic head 1311, gas head 1277. This was in order to get an additional water vapor measurement.
Updated the XML, installed the new isfs-field-vertex package on s13. Moved the interface panel jumper from RS232 to RS422 for port 6. Set emode 6 422
in pre_dsm.sh for s13.
Unfortunately this unit does not seem to be working. Sonic and gas head LEDs in the EC100 stayed red. It worked in the trailer...
If I removed the gas head connector in the EC100 electronics box, then powered it up, the sonic LED would go green.
At first we heard the gas head making a bit of a rough noise, like worn bearings, which we also heard then testing it in the trailer.
Then after removing/reconnecting the cable at the EC100 we did not hear the gas head, and the gas LED was red.
Unloaded voltage at the connector was 13.6 V. Don't know the voltage under load, since I could not reach the screw terminals on the connector
with the DVM probes.
Could not display the data on over a serial connection to the DSM console, since the DSM is strapped too close to
the tower. Since time was short, we left it in that state, sonic LED green, gas LED red.
On arrival back at the trailer, see that the sonic data is coming at 10 Hz, but is all hex FFFF.
This sensor is configured as ID 13,55, with the duplicateIDOK=true option in the XML, so that two
sonics are defined for this site, with the same u,v,w,tc,diag,spd,dir variable names, but from different sensors. In this
way we don't need a second config XML.
Clear skies, but windy, cool.
Kate arrived. SteveO leaving today.
To dos:
- Reshoot boom angles at 60m and CSAT S8 and S15 We've set up the binos on a tripod.
- Soil core at S1, a second at S15
- Install and "manually" read another TP01. Steve is taking two TP01 wisard boards back to Boulder for rehab.
- Install EC150 at S13. We've checked that it is ready to go.
Station Status: All up.
Sensor Status:
T/RH: 42m TRH now not reporting. Power cycle does not bring it back. Others okay (note that S1,2,9,10 are wired not to report Ifan).
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
3D RMY: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: Ready to manually read sensor
Rainr: ok, reported about 1/4" today.
Vbatt: Vbatt.tower at 12.6V on 10/9
Summary: Rain ended at 1330, after dumping 3.6" during the last 24hrs. Strong, gusty winds – 15m/s at 3m; 25m/s at 49m.
Actions past 24 hours:
- Kate now in Delaware
- barolo stayed down after systems work yesterday until 1340 -- plots/tables now being regenerated.
To dos:
- All actions waiting for wind to die down to less extreme levels
- Reshoot boom angles at 60m and CSAT S8 and S15 We've set up the binos on a tripod.
- Soil core at S1, a second at S15
- Install and "manually" read another TP01. Could be left running at a station hard-wired to a serial port (just made cable for this task).
- Install EC150 at S13. We've checked that it is ready to go.
Station Status: All up.
Sensor Status:
T/RH: 42m Ifan has come back, but we don't completely trust that fan is working again (after being dead for many days). Others okay (note that S1,2,9,10 are wired not to report Ifan).
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: went wonky during rain today, as expected
3D RMY: Heavily spiking during rain, came back when rain stopped
EC150: usual problems during rain
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: Ready to manually read sensor
Rainr: ok, reported about 1/4" today.
Vbatt: ok, manually read Vbatt.tower in afternoon at 12.6V, which is fine (despite the rain/clouds).
Summary: Light rain most of the day. All stations working as expected this morning.
Actions past 24 hours:
- Yesterday (10/7) Reinstalled TRHs at S4&S7.
- Second trip to marsh failed to read S15 TP01. Not only does the probe seem not to be good, it appears to have blown out the wisard board TP10.
- Andy went bye-bye
- Boulder computers went down for systems work (hence couldn't work on this status until the evening). barolo is still down, so no plots/tables.
To dos:
- Reshoot boom angles at 60m and CSAT S8 and S15 We've set up the binos on a tripod.
- Soil core at S1, a second at S15
- Install and "manually" read another TP01. Could be left running at a station if we could hard-wire it in to a serial port.
- Install EC150 at S13. We've checked that it is ready to go.
Station Status: All up.
Sensor Status:
T/RH: 42m Ifan has come back, but we don't completely trust that fan is working again (after being dead for many days). Others okay (note that S1,2,9,10 are wired not to report Ifan).
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: went wonky during rain today, as expected
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: Ready to manually read sensor
Rainr: ok, reported about 1/4" today.
Vbatt: ok, manually read Vbatt.tower in afternoon at 12.6V, which is fine (despite the rain/clouds).
Andy gave a tour of the site to new recruit (Gordon)
Add TRH to S4.
Replaced TRH on S7 since it failed for several hours last night.
Summary: Partly cloudy and light winds. All stations working as expected this morning.
Actions past 24 hours:
- Yesterday (10/6) Raised DSM and battery boxes at 14 stations. All have now been raised.
To dos:
- Install TRH at S4. Replace TRH at S7.
- Reshoot all boom angles (just because...Have done most sites so far)
- Soil core samples (took some from S15, could redo S9, S1 waiting for a new Qsoil)
- Do spot sampling with TP01
Station Status:
S3: Pi DSM is working.
S15: Also working.
Sensor Status:
T/RH: 42m Ifan OK. Data from TRH at S7 dropped out from 18:47 to 23:09 EDT last night. Came back on its own.
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
Summary: Sunny and light winds. All stations working as expected this morning.
Actions past 24 hours:
- Yesterday (10/5) Boat trip to raise battery boxes and dsm's to improve water level clearance. Added a new soil probe at S1 and removed the TRH from S4 to fix the fan pcb which showed heavy corrosion.
To dos:
- Reshoot all boom angles (just because...Have done most sites so far)
- Soil core samples (took some from S15, could redo S9, S1 waiting for a new Qsoil)
- Install TP01 when fixed
- Fix WWW plots that die too frequently
Station Status:
S3: Pi DSM is working.
S15: Also working.
Sensor Status:
T/RH: Lots of issues: 42m now Ifan=0, don't know if this is real;
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: S3 looked bad 2 days ago when initially restarted through Pi. Now not sure.
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: S15 was up at least a bit yesterday, but not sure when we left.
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: S9 and S15 ok; S1 is dead (a new EC-5 sensor may be coming from Boulder)
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
This has to be a record: 14 stations in 5 hours (~0900-1420). Purpose was to raise all DSMs and battery boxes, in anticipation of Hurricane Matthew's visit this weekend. Also inspected every TRH sensor to see if proactive action was called for...
S15: (DSM already raised; Battery box on wood platform Andy made). TRH ok (fan is on).
S8: (DSM and cooler raised). TRH ok (fan is on)
S6: same
S1: (DSM and cooler raised). TRH fan board was uncoated, so we anticipate problems later. Replaced with spare we brought. Installed Qsoil that arrived yesterday (YAY!) Since I didn't want to re-disturb the hole with the Gsoil probe, created a new slot to place the EC-5. There was at least 2cm of standing water, so the placement at 5cm depth was a total guess. I <think> I got the prongs in horizontally.
S4: (DSM and cooler raised). TRH fan board also uncoated, removed to fix in base.
S5: (DSM and cooler raised). TRH ok (fan is on)
S2: (DSM already raised; cooler raised). No action on TRH since replaced yesterday (fan is on)
S3: (DSM already raised; cooler raised). TRH ok (fan is on)
S13: (DSM and cooler raised). TRH ok (fan is on)
S12: same
S11: same
S10: same
S9: same
S7: same
(We raised DSM and cooler and replaced TRH at S14 yesterday, so no need to revisit.)
Summary: Moderate winds and partly cloudy. Hasn't rained yet which probably means it's about to. Set a course record for sites visited in a single trip by going to all but 14 to raise DSM and battery boxes a few inches for better water clearance. Placed new soil probe at S1. Took the TRH from S1 to replace the fan pcb. S10's TRH isn't reading fan current but is still working.
Actions past 24 hours:
- Yesterday (10/3) A boat trip saw the DSM placed at S3, the TRH and crossarm reinstalled at S2, TRH replacements done at S9 and S14, and S15 getting some work to get it reporting again.
To dos:
- Reshoot all boom angles (just because...Have done most sites so far)
- Soil core samples (took some from S15, could redo S9, S1 waiting for a new Qsoil)
- Install TP01 when fixed
- Fix WWW plots that die too frequently
Station Status:
S3: Pi DSM is working.
S15: Also working.
Sensor Status:
T/RH: Lots of issues: 42m now Ifan=0, don't know if this is real;
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: S3 looked bad 2 days ago when initially restarted through Pi. Now not sure.
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: S15 was up at least a bit yesterday, but not sure when we left.
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: S9 and S15 ok; S1 is dead (a new EC-5 sensor may be coming from Boulder)
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
Summary: Typical day with intermittent rain and light winds and partly cloudy. A boat trip saw the DSM placed at S3, the TRH and crossarm reinstalled at S2, TRH replacements done at S9 and S14, and S15 getting some work to get it reporting again.
Actions past 24 hours:
- Yesterday (10/3) Worked on DSM for S3 and modified TRH's for swaps at S9 and S14. The modified TRH's bypass the Ifan pcb so they won't read any current but they work otherwise.
To dos:
- Reshoot all boom angles (just because...Have done most sites so far)
- Soil core samples (took some from S15, could redo S9, S1 waiting for a new Qsoil)
- Install TP01 when fixed
- Fix WWW plots that die too frequently
Station Status:
S3: Pi DSM is working.
S15: Also working.
Sensor Status:
T/RH: Lots of issues: 42m now Ifan=0, don't know if this is real;
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: S3 looked bad 2 days ago when initially restarted through Pi. Now not sure.
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: S15 was up at least a bit yesterday, but not sure when we left.
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: S9 and S15 ok; S1 is dead (a new EC-5 sensor may be coming from Boulder)
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
This visit was to take care of numerous instrument and infrastructure issues:
S3: Installed Pi DSM with old Cradlepoint and old Verizon modem (after lots of testing/configuration yesterday and this morning). Station is now completely up. This is a bit surprising, since we had some CSAT3 rebooting errors a few days ago, but it looks okay now (knock on wood). I think we forgot to dress the cell modem external antenna cable. Also finished installation of raised DSM with a bottom support rod.
S2: Reinstalled TRH crossarm/cable/sensor that I had removed Sunday to get the TRH off. (I ended up drilling out the setscrew, but didn't damage the helicoil much.) Note that fan is wired always on and reports Ifan=0. Raised the DSM.
S14: Swapped out TRH. Found that the fuse was blown and replaced it stealing from the GPS port. Raised the DSM. Also raised the battery box.
S9: Swapped out TRH. This fan also always on with Ifan=0.
S15: Powered off, then on with only cell modem connected to USB. It worked. Plugged in Bluetooth to mote. It worked. Found that data had successfully been recorded on the on-board SD card. Powered off, moved SD card to Acer, copied raw_data files to Acer, reinstalled SD card on Titan, powered up again (with both modem and Bluetooth), and everything came up again.
So now, the only sensor issues are Qsoil.s1 (which should arrive in the mail today) and the TP01s.
We'll go back to Home Depot for more support rods and battery box shims.
At the moment, this is up. With hints from syslog during a crash, and then Googling, I've done:
/boot/cmdline.txt – added "dwc_otg.fiq_fix_enable=0", though this may not have been needed (update: yes, it is...)
/etc/default/avahi-daemon – set detect parameter to 0 to turn off avahi .local checking (couldn't find where unicast .local is being defined, so just turned the check off)