Blog

S1: We knew that Qsoil here has been mostly down (a few good samples have come through).  No obvious issues with cables/connectors/etc., but still bad Qsoil values (nan).  Pulled entire probe out of the ground and brought back to trailer.  (In the process, disturbed Gsoil, which did change briefly by as much as 0.5 W/m2, but seems to have settled back in.)  In trailer, find that the Qsoil PIC continuously resets when a probe is connected, and never is able to give it the full 3.3V excitation pulse.  Either the EC-5 is dead, or something is wrong with the PIC front-end to the probe.  (The rest of the PIC runs fine.)  Further debugging needed, but in the meantime, no Qsoil at this site.  (Since we don't have a spare EC-5, the only other option is to take the Qsoil PIC to another site (9 or 15) and swap it in as a test.  I'm sorry I didn't think of this when we were at S9 this morning...)

Thought about taking core, but rather useless if no Qsoil!

Shot boom angles (though forgot to bring monopod, so a bit shakey).

S9: Grabbed a "soil" core from here at 0938.  Had to undercut the core in order to remove it.  (The entire corer pulled out, leaving the plug behind the first time.)  Mostly a peat-type mesh of roots.   Core now being processed in lab.

Shot boom angles.

S10: Shot boom angles.

S12: We knew that this had been off the net since last night.  The cell modem had a yellow status light.  Unplugging and replugging the modem caused its LEDs eventually to go to green, but had to reboot to get the system to connect.  After the reboot, all was fine.  "lsu" showed that local storage had been working fine, so no data lost.

shot boom angles.

Attempted to walk to S13 to shoot angles, but couldn't cross the ditch.

S8 data were offline from about 07:45-08:03 this morning.  Given that the crontab net-check script is scheduled to run at :37 after every hour, this is not what brought it back.  Thus, this was just a typical Verizon hiccup, not what we dealt with 2 days ago.  Local data storage was unaffected.

Died about 11pm last night.  I just restarted it with control-R, and it seems happy again.

We have no idea why this is happening – always at this site, three different TRHs tried.  I guess we'll throw together a DSM script to send control-R to it periodically.

We just did a quick trip to the marsh (just about an hour base-to-base) to revive S8's networking.  I got it running with: su; ifdown eth1; ifup eth1.  I tried switching the USB positions of the stick and the modem, but they don't fit the other way.

After talking with Gordon, I'll try to implement his old script to check networking and do ifdown/up to attempt to revive.  2 trips to S8 in a day is fun, but I don't want to make a habit of it. 

During the battery swap at tower this morning, power apparently was cycled to the power mote.  This seems to have caused it to reverted to Wisard output, but the xml assumes it is serial.  I first tried mp=2, which I believe is what we use for the sensor motes, but this apparently outputs uncalibrated data.  After flailing for a while, I finally came up with mp=1, which is back to calibrated serial data.  Finally, apparently, eeupdate doesn't work for power motes, so I expect it to lose this setting again if power is cycled.

To pick up the change of swapping motes at S9, I restarted dsm_server on isfs@barolo.  However, I failed to notice that statsproc stopped and didn't auto restart.  (I <thought> process_restarter or some-such was supposed to ensure that these processes stayed up?)  Anyway, I just guessed at the statsproc command (statsproc -z sock:localhost), since there is not a statsproc start script that I could find, and files are now being updated again.

(Of course, I also can't use 'git' here to push changes made on eddy(?), so I had to manually made the same change back in Boulder to pull/push/pull...)

This was the TRH that we just swapped in yesterday to avoid fan issues in the previous sensor!  Fan was turned off by the PIC, presumably because Ifan exceeded the 80mA threshold.  Control-R brought it back.  Now reading ~28mA, which is a bit low, if anything.  Strange...  What could it be about this station that it kills TRHs??

Quick run to the swamp (since able to go under the bridge) to S8.  Station was up and had collected data with no problems, just couldn't ping.  Couldn't read the modem LEDs due to position with respect to USB stick.  (We should swap these positions sometime.)  Tried unplugging and replugging in modem with no change.  Ended up rebooting – data gap from 0934-0936.  Now all is well.

Apparently, the battery had drained due to the low sun.  It was running when I left at 8pm, but stopped shortly thereafter.  This morning, the "load disconnect" LED was red on the sunsaver and the battery measured 12.2V.  Disconnecting and reconnecting things didn't help.  Swapped batteries with one in the base measuring 12.7V and the station came back up.  Took advantage of this opportunity to dump about 2cm of water from the bottom of the battery box.

S8 dropped its network connection while we were in the area.  Cause unknown.  We didn't visit the site, but apparently should have...  We're hoping that it will fix itself.

Didn't keep close track of time – was at S15 first at 15:27, ended back from the trip about 18:00...

Out via Red Mill Creek, with the tiki duckblind shortcut.

S15: console was dead, power cycled DSM, everything came up except Mote, power cycled mote, then all sensors up.  However, station wasn't on network.  Decided to leave and hope it came up by itself later.  Noted that the wetness sensor was covered about halfway up the handle.

S9: mote was always in cycling mode, rebooting after setting A/D gains.  Pulled all sensors off (found a bit of moisture in the Gsoil connector), still rebooting.  Pulled entire ribbon connector off and mote, still rebooting.  Decided this mote needs some base love.

Swapped in spare mote we had brought (ID=9).  Saw rare messages with rserial.  Thought it might be in bluetooth console mode (stupid, because the Bluetooth module LED was off) and tried changing using 2 presses of the white switch.  Eventually, got Bluetooth console enabled, paired with phone (key=1234), and set to 232 console mode (pp=0; eeupdate), still not much on rserial.  Eventually got it through my thick skull that the mote was in ASCII mode and rserial wasn't seeing the correct, Wisard, terminators.  Did ddn, minicom to port, set Wisard mode (mp=0; eeupdate), and all was fine.  Brought old mote (ID=1) back to base. (Back at the base, changed isfs_env.sh to include 1&9 in the definition of  MOTE1 and restarted dsm_server.)

Found out from Gary that S15 still hadn't come up on the network, so....

S15: Found that the cell modem had two green LEDS: data solid on, status on, then blinking.  Pulled modem off USB and reconnected, LEDs went through cycle, ending in state that we had found it (2 greens), still no ping google.  rebooted DSM using reboot command, modem went through yet another LED cycle, google immediately came up.

S5: Swapped TRH with spare we had brought (ID=49) even though the old sensor seemed to be working.  Seems okay, so no harm done.  Will evaluate sensor we brought back in the base to determine the source of its odd fan behavior during the rain.

Back via under-the-bridge.