...just a few hours ago at 7pm, after running solid since I worked on it on Monday (5 days ago). The other 3 (with the software upgrade) were working fine. As usual, a pio brought it back, so all 4 are working again. Unlike usual but as expected, the networking now didn't go down, since we rewired the router.
- Our prompting and parsing upgrades to the snow pillows are now complete. ncharts now shows Loads and SWE. I'll update qctables soon to add SWE (DONE). Pillow #3 still is running the old v3 code, though I have instructions from Matt to try to connect to it differently when we next return.
- After a week of erratic readings from two of the Apogees, all are now operating properly (without us doing anything!). After consulting with Apogee, our best guess is that water got into the cables. Apogee is sending us 4 replacements with their "newer" style (though the current ones also should have been the new style).
- One of Ethan's lidars on the uw tower still is not connecting. Ethan plans to give us a new SD card (that probably will rename this to pi7) to try to install on our next visit. We also will try to move the placement of his Pis in the DSM box to avoid antenna blockage.
With this handful of tasks, we are trying to decide if it is worth making another trip out to Crested Butte this month (as we had scheduled) or wait for the Dec. visit. Clearly, the snowpack is developing, which would be nice to document with the above user sensors. None of these tasks require climbing above 3m and could be done using the ladder (though this is awkward).
Starting with thoughts as Isabel and I drove back from the late Oct visit:
- One climbing harness in the barrel has leg clips, the other has closed leg loops. It would be nice for both to have the clips, i.e. replace one with another harness. (Will actually prefers the closed loops, hence why this harness is there.)
- It would be nice to have an Allen key set on a tool tether.
- It also would be nice to have a canvas "chalk" bag that doesn't have the messy fleece liner of the only bag currently in the barrel.
- We need to generate a walking path map (now in the site portion of this wiki) and place a copy in the JHA binder
- Ethan will give us a new SD card for the one of his Pis that isn't connecting. He also suggested shifting the position of the Pis inside the DSM box to get better antenna placement.
- Make a mote console cable for Jessica and students to use with the spare set of Tsnows – maybe give to Ethan first?
Cold (14C) when we arrived 8:40 ish. Above freezing and sloppy as we left about 13:40. Lots of blue sky. Making snow by the hotel.
Isabel and Steve
Tasks:
- Seal & Peel all sonics, in a semi-random order (bottom to top). Initially, stuff sollidified in the caulk tube, so we cut off the spouts. Hopefully, this will help.
- Cleaned 9m radiometers from 0957-1005 (times from plot of rawdata, now entered into cal_files). Used isopropyl alcohol since it was cold. Domes looked clean already, with lots of fan ventilation, though the air didn't feel warm to me.
- d DCDC changed to 20W to power Xstream's Ubiquiti
- Tried to upload v4 software to snow pillows, but got a "Failed to read the controller ID" error on the last pillow (#3). Unable to fix, so left it running v3 code.
- Gary uploaded change to NIDAS to pre-pend a message type prefix, that should allow us to parse the snow pillow data, that seems to work.
- Also overnight, added a cal file for the CSAT3BH, so winds should be correct (though it hasn't been added to statsproc yet).
- Installed the wifi repeater, connected to cm. Had to go through the set-up menu again (Gary had done in the lab) to get it to connect to the Cradlepoint, but now up.
- Nevertheless, one of Ethan's Pis still won't connect. We cycled power on both Pis on uw by unplugging and plugging in their USB-C power cables. (Twice on 4, reseated SD card, LEDs on ethernet same on 3 and 4, so we think it is still getting data from the lidar.)
- Discovered that uw DSM wasn't getting GPS messages (rs G had no response). Power cycle brought everything back. systemctl restart gpsd didn't work.
- Also realized that the uw Apogee data have been quite erratic for the last week. Reseated its screw connector and both Binder connectors on the popsicle stick (also pio cycled power somewhere in there), but still see bad values. P.S. Downwind's apogee also is odd now.
- Soil core taken.
- Back at RMBL, removed 8x CSAT3A cassettes for the shop to work on. Stored CSAT3BH box in white quonset tent.
Left Boulder around 10, got to the site by 3:30 with a stop at RMBL to grab the ladder and try to get into the lab building (no luck). The road to RMBL is mostly clear, the road to kettle ponds is mostly still covered in snow. There's at least a chance that the current snowpack at the site will be there all winter, so we're using the official paths to access towers and guy points while minimizing tracks inside the triangle of d, ue, uw. Currently the site is mostly snow-covered, some parts significantly deeper than others:
Took guy wire tensions on all towers (added to tower page – they appear to be holding well) and climbed uw to add the heated sonic at 3m. I'll let Steve fill in the other stuff he took care of while I was making a hash of cable management for the CSAT3BH.
Other tasks:
- Flipped 1m.uw sonic head about 16:40. Followed Jacquie's suggestion to power down during the change, so it would be apparent in the data. Will update the cal file after finishing this blog.
- Moved the network switch power on cb and cl to the DCDC output, to avoid inadvertent pio power cycles (especially with the snow pillows on cb).
Lost sunlight around 4:30 and was getting too cold to work by 5:30. Back out tomorrow to do the rest of the site visit tasks.
Just changed the data rate for the snow pillows to 0.002s, rather than 0.02s. I only did this on cb's config – haven't pushed this to git. (I'm hoping to sample this snow event without further data gaps in the pillow measurements.)
Sometime, we'll want to add prompting for the other pillow outputs as well (e.g. SWE).
We haven't yet gotten into a mode of regular daily status messages, so just some general notes here:
- I've been looking at QC tables several times a day and have noticed no problems, with the exception of the snow pillows
- Snow pillows (no particular pattern of which pillow) have stopped reporting several times a day. They always come up with a power cycle (pio, using a type-ahead to avoid permanently losing the network connection). We are hoping that a software upgrade will fix this, and the cable and software may be in the mail to us now.
- The CSAT3BH (minus one cable connection) is on the NCAR loading dock as I write this. We hope to configure it in the coming days and take a quick trip back to Crested Butte to install it late next week (along with the software upgrade).
- The nanobarometers that are sealed off have large pressure variations, due to temperature changes. This is confusing on both ncharts and QCtables, but is normal.
Found that at least one Tsnow is sending corrupt data messages, presumably the same issue we saw in the previous 2 projects on the soil wisard sensors. This is happening even though both the Tsnows and the soils are now running versions of the PIC code with double buffering (v2.38 and v1.08, respectively). For the moment, I've set limits on the Tsnow data in sensor_catalog to remove wildly incorrect values, though I guess if the really bad corrupt messages are happening, that more subtle corruption also can occur.
After this change, I restarted both statsprocs...
Pillows 2&4 stopped last night, brought back with power cycle. Other sensors still running.
Hoping that this visit can occur while the road is still open...
Just to put this in one place: (number of minutes)
- DONE Flip uw.1m sonic (the head to the cassette) (5)
- 3/4 DONE Upgrade software on snow pillows (20)
- DONE Install CSAT3BH (15)
- DONE Seal all CSAT cassette mounts (120)
- DONE Clean radiometers (2m NR01 and 9m K&Zs) (10)
- DONE Another soil sample (10)
- DONE Install wifi extender, attached to cm dcdc power (and swap in 20W dcdc converter) (15)
- DONE In cb and cl, move the ethernet switch power from bank1 to the dcdc terminal on the power panel (bring extra dcdc converters?) (5)
- DONE Check new pillow prompting code that Gary may install (unknown)
- DONE Read guy tensions (20)
- Remove cassettes from spare CSAT3As, if time allows (15)
Total: (235 min = 4h, but could do some tasks in parallel)
What to bring:
- individual allen wrench set with 5/32" (on shelf)
- snow pillow cable and windows laptop (on shelf)
- CSAT3BH and mount and mounting bolts (on shelf)
- Seal and Peel and caulk gun (grab another gun from home)
- Kim wipes and alcohol (on shelf)
- soil kit (on shelf)
- wifi extender and cables and dcdc converter (from Gary)
- more dcdc converters (have 1 extra 20W for d to power Xtream's Ubiquiti and 2 extra 10W just in case cb and cl don't have them)
- my tool bag (in car)
- tire chains (in truck)
From the wind direction data, I was pointed to a problem with this sonic, and photos show that it is upside down. This time, I think the "cassette" was incorrectly positioned – probably a year ago, but we hadn't noticed it. It would be a 5-min fix to remount, especially since it is low on the tower, but Chris and I decided to come home, rather than add over an hour of driving with a loaded pickup over the rutted road. We'll need to address this in our next site visit.
This also brings up the question of whether we should have sealed the cassette mounts as Campbell had advised last spring. We'll need to consider this in the coming weeks.
In the meantime, I changed the cal_file to "flipped", so the data now appear to be correct.
...but a "pio BANK1 0; sleep 5; pio BANK1 1" revived them.
It is curious that data still appear as 0 in qctables, even though rs shows no data coming in. Maybe just the timing of me looking at it...
Weather: even warmer, same breezes came up during the day
People: everyone left the site by 4. We're done with set-up!
Tasks:
- Helped Ethan get his Pis connected and powered
- Swapped the ec150.3m.c since it was reading high
- Documented tensions (ue had loosened some, but all still okay – see table elsewhere in this blog)
- Took soil core at 1130 (see comment below)
- Capped off unused pressure ports
- Packed up and removed the on-site staging pile
- We're hoping for the best!