During the battery swap at tower this morning, power apparently was cycled to the power mote. This seems to have caused it to reverted to Wisard output, but the xml assumes it is serial. I first tried mp=2, which I believe is what we use for the sensor motes, but this apparently outputs uncalibrated data. After flailing for a while, I finally came up with mp=1, which is back to calibrated serial data. Finally, apparently, eeupdate doesn't work for power motes, so I expect it to lose this setting again if power is cycled.
To pick up the change of swapping motes at S9, I restarted dsm_server on isfs@barolo. However, I failed to notice that statsproc stopped and didn't auto restart. (I <thought> process_restarter or some-such was supposed to ensure that these processes stayed up?) Anyway, I just guessed at the statsproc command (statsproc -z sock:localhost), since there is not a statsproc start script that I could find, and files are now being updated again.
(Of course, I also can't use 'git' here to push changes made on eddy(?), so I had to manually made the same change back in Boulder to pull/push/pull...)
9/21/16
Summary: Issues with tower battery power and station 8 links made for a busy start to the day. Low winds with continued heavy cloud cover but no significant rainfall last night. Remnants of TS Julia are staying southeast but still effecting this area.
Actions past 24 hours:
- Yesterday (9/20) since last report some minor display software changes to Cockpit have been made.
- Station 8 required a reboot to restore modem functions.
- Tower battery controller decided 12.2 was too low and turned off tower sensors last night. A fresh battery has it running again, possibly several days of overcast skies and rain have reduced the solar panel's effectiveness.
To dos:
- Reshoot boom angles (just because...)
- troubleshoot TRH from S5
- troubleshoot mote from S9
Sensor Status:
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
This was the TRH that we just swapped in yesterday to avoid fan issues in the previous sensor! Fan was turned off by the PIC, presumably because Ifan exceeded the 80mA threshold. Control-R brought it back. Now reading ~28mA, which is a bit low, if anything. Strange... What could it be about this station that it kills TRHs??
Quick run to the swamp (since able to go under the bridge) to S8. Station was up and had collected data with no problems, just couldn't ping. Couldn't read the modem LEDs due to position with respect to USB stick. (We should swap these positions sometime.) Tried unplugging and replugging in modem with no change. Ended up rebooting – data gap from 0934-0936. Now all is well.
Apparently, the battery had drained due to the low sun. It was running when I left at 8pm, but stopped shortly thereafter. This morning, the "load disconnect" LED was red on the sunsaver and the battery measured 12.2V. Disconnecting and reconnecting things didn't help. Swapped batteries with one in the base measuring 12.7V and the station came back up. Took advantage of this opportunity to dump about 2cm of water from the bottom of the battery box.
S8 dropped its network connection while we were in the area. Cause unknown. We didn't visit the site, but apparently should have... We're hoping that it will fix itself.
Didn't keep close track of time – was at S15 first at 15:27, ended back from the trip about 18:00...
Out via Red Mill Creek, with the tiki duckblind shortcut.
S15: console was dead, power cycled DSM, everything came up except Mote, power cycled mote, then all sensors up. However, station wasn't on network. Decided to leave and hope it came up by itself later. Noted that the wetness sensor was covered about halfway up the handle.
S9: mote was always in cycling mode, rebooting after setting A/D gains. Pulled all sensors off (found a bit of moisture in the Gsoil connector), still rebooting. Pulled entire ribbon connector off and mote, still rebooting. Decided this mote needs some base love.
Swapped in spare mote we had brought (ID=9). Saw rare messages with rserial. Thought it might be in bluetooth console mode (stupid, because the Bluetooth module LED was off) and tried changing using 2 presses of the white switch. Eventually, got Bluetooth console enabled, paired with phone (key=1234), and set to 232 console mode (pp=0; eeupdate), still not much on rserial. Eventually got it through my thick skull that the mote was in ASCII mode and rserial wasn't seeing the correct, Wisard, terminators. Did ddn, minicom to port, set Wisard mode (mp=0; eeupdate), and all was fine. Brought old mote (ID=1) back to base. (Back at the base, changed isfs_env.sh to include 1&9 in the definition of MOTE1 and restarted dsm_server.)
Found out from Gary that S15 still hadn't come up on the network, so....
S15: Found that the cell modem had two green LEDS: data solid on, status on, then blinking. Pulled modem off USB and reconnected, LEDs went through cycle, ending in state that we had found it (2 greens), still no ping google. rebooted DSM using reboot command, modem went through yet another LED cycle, google immediately came up.
S5: Swapped TRH with spare we had brought (ID=49) even though the old sensor seemed to be working. Seems okay, so no harm done. Will evaluate sensor we brought back in the base to determine the source of its odd fan behavior during the rain.
Back via under-the-bridge.
9/19/16
Summary: Rain from remnants of TS Julia is still intermittent and expected the rest of today.
Motes at 15 and 9 are not reporting, efforts are being made to fix from the command trailer instead of making another saturating boat ride.
Shift change: Steve S. leaves today
Actions past 24 hours:
- Yesterday (9/19) boat tour to clear NR01 lenses
To dos:
- Reshoot boom angles (just because...)
Sensor Status:
- Motes from 9 and 15 lost data starting last night. 9 fixed itself this morning. Suspect a software issue with the bluetooth module in 15 (Gary is working on this), though it is odd that its onset is coincident with rain.
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: ok
Vbatt: ok
Someone came up with the random idea that we could test the new laser scanner just before tear-down by having it scan the entire array. It probably would be best to take multiple views, but it appears that the clump of trees just across the channel NE of S6 could have stable ground and a view of everything related to the project (all 15 stations, the wind turbine, and even most of the tower). This would be a semi-circle scan at reasonably low* elevation angle, out to about 1km (except the tower, which is 1.13km). (*60m at 1.13km is an elevation angle of only 3 degrees.) When we next go out in good visibility, I'll check it out.
About 1630, s5 TRH fan was being turned off. We upped the fan threshold current level from 80 to 150ma and now the fan is going (reading ~110ma). T and RH have been reading fine all the time, though who knows how good the signal is with no aspiration. We'll have to visit this site, with the idea of replacing this fan (and will have to reset the current threshold back at that time).
P.S. It died soon after we did this, getting above 150 as well. So, I've just reset to 80mA and we have it on the list to be replaced.
P.P.S. Note that with low overcast skies, the radiation error will be small anyway, so these data still can be used.
P.P.P.S. Now that it has been several hours since the rain ended, I tried a reset (^R) again. This time, the fan is back at about 38mA and stayed up. Also, I didn't see much of a change (if any) in temperature after it restarted. So...we might say that it was just the rain and now works, but I'm still suspicious.
As of about 1500 today, the EC150 h2o and co2 channels have stopped reporting. No doubt they are offscale with lots of water on the lenses. Hopefully, both will come back when the rain stops.
P.S. Now that the rain has stopped for a while, both signals are back, though both higher than they should be – likely still with water on them. We'll look again in the morning.
We used radiometer cleaning as our excuse to go boating this morning and thus visited s15, s9, and s1 (in that order). However, nearly continuous rain during these visits meant that a) the up-looking surfaces were wet immediately after being wiped b) totally overcast sky won't be good for measuring any effect of cleaning c) rain was already doing a pretty good job of cleaning without human intervention. I did not need to use any water from the squeeze bottle.
As part of this process, Steve S has been releveling the sensor in "roll". At s9 and s1 I also adjusted the legs to attempt a leveling in "pitch".
s15: ended 10:04 (local)
s9: ended 10:33 (local)
s1: didn't record – approximately 11:00
P.S. From the data, call the times:
s15: 0959-1003
s9: 1028-1033
s1: 1047-1053
P.S. The wetness sensor at s15 was completely out of the water (though the tide was coming up), though it had a few weeds adjacent to it and clearly had some mud splatters on it. I pulled most of the weeds, but left the mud (arguing that it would just get muddy again soon)
9/19/16
Summary: Rain started as soon as we got to the trailer 8:30
Off and on all day; nevertheless did a boat tour to 15, 9, 1
All stations look good this morning, until the rain started
Actions past 24 hours:
- Yesterday (9/18) was a "dead day" (day off)
- Shift change: Andy arrived Thurs, Gary left Sat, today is Steve O's first day, Steve S. leaves tomorrow
To dos:
- Reshoot boom angles (just because...)
Sensor Status:
- Lost everything from s9 for 10min overnight in the UDP data; I'll check if the raw_data are okay
- CHECKED: rsync data are fine.
T/RH: ok
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)
Rainr: isn't being covared, doesn't show tips in cockpit, even though it is raining. Hmmm.
CHECKED: archive shows tips from sensor (1.67" so far!); not being covared yet (but will).
cockpit issue was just that RAINR_RANGE wasn't defined and defaulted to +/-10, but one tip = 183mm/hr. Changed.
Vbatt: ok
9/18/16
Summary: Sunny, light winds out of the SSW
All stations look good this morning.
Actions past 24 hours:
- Gave Terry a tour of the sites
To dos:
Sensor Status:
T/RH: ok
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: not responding
Vbatt: ok