Blog from October, 2013

10/13/13
Summary:
Second IOP on the night of October 11-12

10/11: 
Replaced floor kh2o
Nothing obviously wrong with P.nne.flr, but Sebastian swapped with P.ssw1.flr
Replaced battery at P.sw.flr
Checked battery voltages at all pressure motes on floor.
Finished replacing dessicant in all radiometers

10/12
Measured guy wire tensions.
Recycling power fixed TRH.20m.rim and Rpile.out.far
Gordon diagnosed and fixed problem with power on the serial ports for the
power-monitering motes
ETC.

10/13
FLR kh2o still dropping out.  Is the serializer having problems?

T/RH: T.20m.rim often a little high, but also sometimes in line with others.  When does this occur?
          RH.40m.near and RH.35m.rim perhaps a bit high.
P: ok
csat u,v: ok
csat ldiag: some ldiag > 0 morning of Oct 12
csat w, tc: ok
kh2o: FLR still flaky; FLR fluxes noisy but believable
motes: FAR ok for past 3 days
           NEAR ok for past 3 days
           FLR all off and on; rad currently missing
Wetness: dew or frost night of October 11-12 (IOP)
radiation: ok
Tsoil: Tsoil.4.4cm.near has offset since 10/12 10:50
         Tsoil.2.5cm.flr & far (linear avg) does not fit profile
Gsoil: ok
Qsoil: ok
Cvsoil: ok
2D sonic: Dir.10m.flr at times appears to be off by 180 deg.

10/12/13

14:10 Visited NEAR to measure guy wire tensions and remove power jumpers on serial ports for power motes.

16:50 Tested FAR pyrgeometer by plugging into radiation mote.  Worked okay.

10/12/13 14:38-14:45

Added a third stake to the soil mote 'mast'.

Removed serial FET plus power jumpers on serial ports for power monitor motes.

Removed down-facing pyrgeometer which was flat-lined for Rpile.

Opened the pyrgeometer in the lab and checked the thermopile voltage and continuity of the connectors.  All looked okay.

17:18 After testing pyrgeometer at NEAR, reinstalled at FAR.  Chances are we could have 'fixed' the pyrgeometer simply with by switching the power off and on.

10/12, twh

Gordon and I visited the RIM station to measure guy wire tensions, add strap to hold the door closed to the upper mote, and to replace the 20m TRH.  

The TRH was fixed by removing jumpers unnecessarily providing power to the power monitors.  See related blog post.

10/12/13 Gordon and Tom measured guy wire tensions.

RIM about 13:00, very low wind speeds

NEAR about 14:10, still low wind speeds

Measured from low to high

10/21/13 Steve S&O; NEAR about 12:00, RIM about 12:30

Marked in orange are the only 2 that changed significantly.

From Sebastian:

IOP 2, 11 October 2013
16:10 MST removed sensor from ssw1
16:25 MST connected ssw1 sensor to nne
16:43 MST connected nne sensor to ssw1

Went to rim-tower to replace the TRH at 20m since it has been reading way off scale, and we have not been able to reset it with ctrl-R or "eio 15 0/1". See https://wiki.ucar.edu/x/6RtaE and https://wiki.ucar.edu/x/FRdaE

It is connected to port 15. The power monitors for rim and rimup are in ports 14 and 16. I wondered if somehow the power monitors are "back" powering the interface board through their serial cables. To test this, I removed the jumpers labeled FET and +V for ports 14 and 16. The power motes kept coming in, so, as expected, they are getting their power from the battery. After removing those jumpers, I can now power cycle the TRH with eio, and reset it with ctrl-R. After that it looks like the TRH is reporting good data, so didn't replace it.

Prior to removing the jumpers, at some point the rim dsm crashed as I've seen once or twice before on this project, reporting "too much work for irq 4", which is the irq of the second serial board. I believe this happens after an "eio 15 0", which might exacerbate the "back" power problem. See https://wiki.ucar.edu/x/jwtaE

Since this seemed to help rim, I then removed the power monitor jumpers at near, far and base. All those motes continued to report.

Post-Project Update, JohnM. 15Nov13.

I cannot reproduce the reported 'back-trh-power' problem coming from the power monitor motes.   I did a test using the rim dsm, a trh plugged into ttyS15 and the same 2 power monitor motes connected into ttyS14/16 and with all the jumpers re-installed, although there were no other sensors installed and the exact same cables cannot be determined.   Nevertheless this is the same sub-configuration that Gordon had with 2 separate battery boxes for pwrmote ID8 and ID9 and one running the dsm.  I tried eio 15 0/1 with eio 14/16 0/1, etc. and everything worked as advertized.    I really didn't think the problem reported could have been due to the power monitor motes because as he mentioned, they're powered independently by the batteries themselves and the interface cable has only gnd,tx,rx on it: i.e. no power.    I suppose there is a possibility that there was a bad cable but i'm doubtful of that too because it is hard to see how either a bad cable could have been working, which it was, or how it could have back-fed power to port 15 forcing it's way through the power control fets specific to each port.   What exactly happened is a mystery.   Could there have been an issue in the Diamond board that caused either both, or the wrong digital i/o line to be pulled?   Was the dsm restarted or 'tickled' or was the power shutdown while the jumpers were changed and the trh worked on?   Could one of the fet jumpers for the trh have been loose or intermittent in cold/warm and possibly contact re-established?    Etc.,etc.   I don't know but feel it's safe to say the power monitor motes cannot back-feed power through their serial port to a dsm.   And if in doubt they can also be setup for wireless (bluetooth, xbee or wifi) data delivery.

10/10/13 at 16:50.  Reseated the up-looking pyranometer shield, which was not completely seated.

10/11/13, twh:

Arrived 4:39 pm.  Cleaned kh2o and reseated all connectors (initially looked okay).

4:49 - 4:53 pm. Cleaned radiometers, wetness sensor.

4:58 - 5:08 pm. Replaced dessicant in pyrgeomters.  The cylinders were screwed in so tight that I wonder whether they had ever been serviced.

10/11/13, twh: 4:14 - 4:20 pm

Cleaned kh2o, radiometers, wetness sensor.

Soil mote stand blown over about 45 degrees to the north.  This stretched the soil sensor wires.  Did it disturb the sensors?.   Mote stand needs a third stake on the south.

We are not sampling the battery voltages at the outlier barometer and sonic stations, and so they have to be measured by hand. We can keep a record here.

NNE
Oct 11: healthy power level: 14.4 V at radio power connector. Data from this barometer is suspect however.

SW
Oct 11: Dead, 4.0 V. Replaced battery. Checked connections. New battery: 16.3 V

SSW1
Oct 11: not great: 11.2 V. Checked connections in battery box. Voltage increased to 11.6

SSW2
Oct 11: healthy: solar 19V, load 14V, battery 14V

SSW3
Oct 11: healthy: 14.7 V at radio power connector

SSW4
Oct 11: healthy: solar 17.7, load 14.2, battery 14.2

Sebastian measured the voltages of the rest of the pressure motes:

WSW at 9:00 am MST 10/12: 13.86 V

W around 9:15: 10.71 V; 11.25 V after fiddling with the connectors to and from charging chip

S at 9:35: 14.18 V

Tom, Gordon arrived at tower in crater floor, 10:30am. Calm, very nice day.

Cleaned radiometers and wetness at 10:40
Replaced dessicant on pyg and pyrs
Level of uplooking KZ pygeometer is good. Pyranometer doesn't have a level.

Checked mounting of zigbee radios in all motes - looked good. All GPS jumpers are removed.

Replaced 1101 krypton with 1394. Cleaned 1394 before mounting. Voltage level 1.1V

At 12:33 checked level on SPN1, which was good.

Went to NNE to check barometer, which has shown some bad data. An easy way to find NNE is to follow the wavy line of hobos in that direction. A hobo is right next to NNE.

Voltages at NNE look good. 14.4 V where the power cable plugs in to the PC board. 3.3V at the other test point. Can't see anything obviously wrong with this barometer. Left NNE at 11:54.

Went to SW-p, which has been out fpr a day or so. Voltage at board power connector was 4.04 V. One connector inside the battery box appeared to be loose. After fixing that, the voltage increased to 7.6 V. So it appears that the is was just a loose connector issue. Replaced the battery. Voltage at radio board was 16.3V.

SSW2-p/c This site has a regular battery and solar panels. The large solar panel had blown over against the other so that it was facing down. Moved it a bit more out of the wind. At the controller: solar 19V, load 14V, battery 14V.

SSW3-p 12:58. 14.67V at the radio power connector.

SSW4-p/c 1:05. At charge controller: solar 17.7, Load 14.2, battery 14.2

SSW1-p 1:20 11.2 V at the radio power connector. Opened up battery box. Seemed the the voltage at the points labeled PV was 0.4V. Fiddled with connectors. Just before closing the lid, saw 11.9 at the battery, 12.3 at the "PV" test points and 11.6 at the radio. Did not replace the battery since it seems that it is now being charged.

Before leaving, checked the motes. SPN1 was not coming in. Tom said the LEDs look OK. Power cycled that mote.

Also noticed that we had failed to reconnect the sensor cables and had left the soil mote turned off when checking whether the radio was seated on the board.

2:00 left floor.

10/10/13

Summary:
TRH at 20m.rim is reporting garbage
P.(nne,sw).flr reporting garbage
kryptons questionable at FLR and NEAR (per SPO)
Motes at FAR and NEAR doing well, FLR rad and SPN1 currently ratty
Tsoil.3.1cm.near has recovered
Rpile.out.far flat-lined

T/RH: 20, 35, 40m T.rim too high after (02:05, 03:45, 02:05), 10/10
                Gordon fixed 40m with software reset at 12:30.
                Tom likewise fixed 35m at 15:25.
                40m RH at NEAR perhaps a bit high.
P: nne.flr went walkabout around 09:00 10/8
     sw.flr went walkabout 05:00 10/10
csat u,v: ok
csat ldiag: some ldiag > 0 during high wind yesterday and this morning with precip
csat w, tc: ok
kh2o: FLR flaky; FLR fluxes noisy but believable
motes: FAR ok 
           NEAR ok
           FLR rad  & SPN1 off and on
Wetness: ok
radiation: Rpile.out.far flat-lined since 10/7 09:00
Tsoil: Tsoil.3.1cm.near has offset since 10/7 12:15
                Spontaneously recovered 10/8 15:25
Gsoil: ok
Qsoil: FLR, FAR were < 0; now > 0 after rain
Cvsoil: ok
2D sonic: ok, FLR wind direction has been corrected

10Oct13, ~16z

Continuing rad/soil mote outages appear related in at least some cases to the 3hourly xbee resets automatically done by the motes.   Because the gps's and the local storage options are now turned off, having radio outages is problematic.   In an attempt to improve the situation the timing was changed:

Flr: id1 & id2 & id17: xbee reset = hourly (xr=3600), gps=off (gr=0)

Near: id3=xbee reset hourly,   id8 & id10=xbee reset off; gps=off(gr=0)

Far: id4 & id22: xbee reset = 2hours (xr=7200),  gps=off (gr=0)

With this mix, hopefully the outages will be less and we can see if the reset timing is still the critical factor instead of hardware issues.

We may want to consider having the dsm's send 'hb' heart-beat commands to the motes and re-enable xbee reset values (which only reset if there is no heart-beat received).

14Oct13, ~1Z GMT

Noticing that Tom reported in the mote status that flr was having additional problems I logged in and rserial'd to the xbee's.

ID1 and 2 were reporting but not 17 as noted.   I tried a 'hb<cr>' command and then a simple <cr> and 17 began reporting.

Further commands issued and normal responses were received.

Changed xbee reset timing: was xr=3600 (1hour), to xr=43200 (12hours).   Maybe they should be turned off, however, what i notice is that things work much better after commands are received from the base radio, so we should consider the cron-'hb' i believe.

14Oct13, ~19Z

flr:  The flr motes had outages last night and at least 2 rebooted.   The setting above hadn't gotten 'eeupdate'd so i reset the xr=43200 and stored the setting.   While interacting with them, id2, which had been 'down', again began sending data.

near:  Daily status reported outage of the spn.   All 3 were reporting when i logged in.  I changed id3 to disable the xbee reset, xr=0, to match the other 2 and not be what appears to be problematic hourly resets.   Gordon's cron 'hb' heartbeat is running so we might try setting xr=12hrs, but not yet.

I refueled the ISFS generator at BASE today at 12:30pm local on thursday the 10th.

I checked fuel levels and found the smaller upper tank empty, and the lower at 1/2, and refueled both, so it should be good for another three days, expect to refuel again on SUNDAY the 13th.

The rest of the site looks like it weathered the storm passage ok, a little evidence of moisture and the expected winds, but no snow down at this altitude.

We (Damien from Leosphere and myself) are here utilizing bench space to work on the Lidar computer replacement.

Let me know(email or cell) if there is anything else I can check on for you guys while I am here this afternoon.

Thanks, Tim