Blog from November, 2022

As Matt pointed out, the TRH.12m.c fan died yesterday.  To avoid further possible damage to the fan, I've logged in and set the fan duty cycle to 0, which hopefully will effectively turn the fan off.  If we get a warm day, I'll try to turn it back to 40% to see if the fan restart.

BTW, the process to change the duty cycle is:

ssh cu

rs 4

control-U (enter TRH settings menu)

DUT 40

EXT (exit TRH settings menu)

control-c (exit rserial)

exit (exit ssh session)

Some odd artifacts in Rsw in the transition between when direct sun is blocked by the mountains, but the sun hasn't set. 

First, from several days ago, I see the amount of incoming (diffuse) radiation higher at, say 15:40 than it was at 15:00.  Perhaps the amount of cirrus cloud coverage changed during this period?

Second, today I see Rsw.out.9m higher than the incomings at 15:00.  Usually, I'd blame this on snow covering the incoming sensors, but Rsw.in.9m is within a CVF4 housing that <should> be keeping it clean.  Also, Rsw.in from the NR01 tracks Rsw.in.9m quite well, even though it has no housing and a different construction.  Thus, I'm inclined to think that this is a real reflection somehow.

P.S. Note the large change in albedo between these two days, due to the recent snow.

P.P.S. The huge dip in Rsw.in.uw at 09:30 is due to the earlier-mentioned shading by the uw tower.

It is obvious from Rsw.in.uw that the NR01, which is mounted on a boom pointing to the WNW of the tower, is getting shaded by the tower and/or stuff on the tower, in the mid morning.  I could use the Leica scans to generate the boom angle and then the dat.azel() to generate the sun angle ephemeris to create a mask of times to remove these data.  Alternately, we could just put in the project report not to use these data (which are a low priority anyway, having not been requested by the PI).  I was never concerned that the NR01 be a perfect measurement, since its purposed was just to ingest data from the Wetness sensor.

As of 10:30 today, the deepest Tsoil at 31.9cm is now below zero (for the first time this project).  This should keep the tower anchors pretty stable, but no soil cores now until spring.

During the last 4 days especially, we have had large diurnal swings in SWE.  I can understand snow melting off in the morning, even now melting completely to get to a SWE of 0cm, but I can't understand a build up at night again to previous values, especially since the last 3 days have been completely clear days (and presumably nights) with no snow.  This appears to me to be a temperature artifact – it is getting pretty cold (almost down to -20 C) at night, though the snow pillows are supposed to apply a temperature correction, I thought.

Got the finest dusting of snow overnight, partly cloudy today.

After packing up and cleaning the cabin we paid a last visit to the site. Data check this morning showed no pwrmon data from cl starting yesterday around noon, so we went to the spares room again and grabbed a spare victron and victron cable. Then out to site around 9. Started by swapping out the body of Ethan's lidar pi 4 with the body of pi 8, which seems to have a better wifi signal inside the same box. Pi 8 doesn't have a cable to connect the GPS to, but Ethan says that's not a problem. Then looked at victrons. We suspected that the victron usb cable in cl may just have been loose or unplugged, as we had been climbing all over the lower part of C yesterday and could easily have nudged it. Will reseated both ends of both cables, and then (once I remembered I had to stop and start nidas to get usb devices to show up) data was flowing again. So, as far as we know, everything is working at site as of 10:30 today. Also while at site we added a bunch of zip ties to the barrel, and took some site pics.

Out to Mt. Crested Butte by 11:15, home to snowy Boulder by 4:15. Thanks to Steve and Gary for their remote help the past couple days!

At the site between around 8:45 and 3:00 today. Sunny, calm in the morning and increasingly windy as the day went on.

Started by looking at the 5m csat/ec150 on CM, which was completely unresponsive. Will climbed up to look in the box, which showed red light for gas but green for power and winds. He tried reseating the sonic, ec150, and bulgin cables and power cycling, which didn't bring it back. We decided the next step would be to replace the ec100 box, thinking that if we weren't getting messages from it at all it would likely be a box problem. New box that we swapped in had the same problem–no data at all, lights showing red for gas but green for power and sonic. Tried replacing the bulgin cable too, no change. Tried changing the config to swap the trh on port 1 with the csat/ec150 on port 0, did successfully get trh data through port 0, did not get csat data through port 1. Tried plugging the 5m csat into port 4 on CL (which is set up for 3m csat, which was working), didn't give data. Tried the 3m sonic in port 0 on CM, working fine, so we know that dsm port can ingest RS422. Running out of ideas, so we tried powering off the ec100, unplugging the gas head, and powering on again. Works! Nans for gas, of course, but we're getting data. Since it turns out the problem was in the ec150, we swapped back to the original ec100 box and bulgin cable. Will pivoted the boom up and replaced the ec150 head, now everything is working. We are taking the bad ec150 head back to Boulder so we can figure out why it's killing ec100 boxes, leaving the spare ec100 box (which we tried but ended up not using) in the spares room at RMBL.

Will also looked at the 15m ec150 since we had been concerned about it giving negative values for h2o. He didn't see any residue on the glass, and currently there's no cleaning supplies at the site or at RMBL, so we didn't try cleaning it. Later heard from Steve that it's a bias and didn't need replacing.

Replaced apogee cables with the newer versions at C, UW, D. Swapped the original popsicle stick back in at UE, after we finally got it detached from the old cable last night.

Got Ethan's lidar pis on the network. At UE all I had to do was power cycle pi 1 by unplugging and replugging the cable. At UW I put the SD card from pi 8 that Ethan had given me into the box for pi 4. That didn't seem to do anything. Later tried pulling the pi out of the dsm box and letting it hang below, to see if that would allow it to connect to wifi. That seemed to work, potentially because now the pi was oriented horizontally rather than vertically. Had to close the dsm box before we left, so tried putting the pi in more horizontally (not sure how successful this was, since in the cold the ethernet and power cables to the pi didn't bend well). However it ended up seems to work, and it's still connected now.

Attempted to do a firmware upgrade on snow pillow #3 by plugging directly into the electronics board in the snow pillow box, bypassing the external SDI connector. Opened up the box (voiding the warranty) only to discover that the connector on the dongle they gave us doesn't match the connector on the board inside the box:

So no luck there. Tried once more the original way, but didn't work (same error as Steve got last time). Incidentally, I was completely blocking the apogee on C as I was trying to get into the snow pillow #3 box, between like 1 and 1:30pm.

Left at 3 as we had lost the sun and it was getting cold fast. Stopped in at the salamander room at RMBL and left the spare csat we had taken just in case, as well as the good spare ec100 whose ec150 we used. Left the extra apogee popsicle stick in the user spares box.

Left Boulder 8:30ish, to Gothic around 2. Road to Gothic is in great condition, some compacted snow but easy driving. Helps that it's in the shade by 1:30 or so now. Stopped by the Barclay building as our reservation emails instructed, but nobody there. Eventually ran into Ben Schmatz who told us we don't need to check in and we don't need a parking permit. Stopped by the Maroon cabin to drop off some stuff, then out to Kettle Ponds. Road is only patchily snow-covered, with 2 or 3 inches of snow max. With the cold weather (19 f) it's frozen, not wet. Better driving conditions than last time either Will or I were here.

Got to the site at 3, right as we were losing sunlight behind Gothic Peak. We checked guy wire tensions (will add to page) and then added the new cable to the apogee at UE, so at least we could tell Chris if the cables were working. Had a very hard time detaching the bender cable end from the popsicle stick since the heat shrink on the popsicle stick board covered the connection, so eventually gave up for now and swapped in the spare popsicle stick Chris sent with us to go with the new cable, just so we could see if things are working. Looks like the apogee is reporting similar values as before the cable swap, so I'm assuming the new cables are good. We took the old cable and original popsicle stick back to the cabin, where we finally managed to cut the heat shrink back far enough to detach them. We'll put the original popsicle stick back in at UE tomorrow. Left the site by around 4, since in the shade and breeze it was getting pretty cold.

Site looking generally good, less snow covered than two weeks ago. Some of the snow pillows are partially bare. Looks like there are some fairly recent human footprints, mostly over by the power panel, as well as some animal pawprints through the triangle between towers. Don't they know to look at the walking path map??

Trying to get a current picture of what needs to be done. Edit as necessary

• Since we will be climbing all towers, check guy tensions on all towers
  • Tensiometer and climbing equipment is all in the barrel. Make sure the barrel is accessible (I don't think this will be a problem this visit)
  • Keep to paths to minimize impact on snow inside triangle. Make sure we bring map + leave in barrel.
• Replace apogee cables on all towers.
  • Use rs during switch to make sure it's reporting same values before and after cable swap.
  • Try to make sure popsicle sticks/cable connections aren't horizontal, so they can shed water better.
  • Tape over both ends of popsicle sticks, esp on UW
  • Spare IR popsicle stick to user spares at RMBL
  • If we need zip ties, grab from RMBL storage (don't think there are many/any at site)
  • can check if there's currently water in bender connectors when replacing cables
• Snow pillow #3 software update: are we still trying to do this, now that we know it doesn't completely solve the pillow dying issue? Yes
  
  • instructions in documentation
  • Dongle cable: plug power in to pi - connector on serial front panel, or in bank1
  • On pillow box, open four screws on box lid (breaking warranty), plug dongle right into board rather than through sd1 connection (unplug bulkhead inside box?)
  • Can also do visual inspection of connectors inside the box
• Troubleshoot Ethan's lidar pis
  • pi 4, which has never connected: try replacing sd card with the sd card from pi 8. If that doesn't work, replace the whole thing with pi 8
  • pi 1, which was connected until recently: try power cycling via unplugging/replugging the power cable. If that doesn't bring it back, try replacing sd card with the one from pi 7. If that doesn't work, replace the whole thing with pi 7.
  • Should we plan to log in to the cradlepoint on site to see if pis have connected, or can this be done remotely? If we're going to log in onsite, does it make sense to bring wstar out? Would be annoying to try logging in to the cradlepoint interface from a phone (if it's even possible..?)
  • pi 4 is on UW, which tower is pi 1 on? ue
• 5m.c sonic troubleshooting
  • try replacing the EC100 box first, then if still bad try replacing the csat/ec150 heads. Before taking box off check lights in it/check power connection
  • were we going to try adding the updated cassette mount???

To bring from Boulder:

• copy of site paths walking map
• apogee cabling
• spare popsicle stick for apogees (to RMBL)
• callab windows laptop, dongle to go between laptop and snow pillow, usb stick with firmware upgrade, documentation. need extension cord for power
• two spare pis in enclosures from Ethan
• polk to haul ec150/csats in cases
• truck: make sure we have chains, bring a full sized shovel just in case

For the first 2 gravimetric samples, the results are:

I'm not quite sure why the densities are so much different, but the soil does have a lot of variability.  This would show a bias of about 8%, which is annoying, but not that large in our experience.

I'm intending to make this a standard daily plot, but just to try, here is a comparison between the average of the 4x vertical velocities at 10m (red) with the vertical velocity computed from the divergence of the velocities measured along the sides of the triangular prism defined by the d, uw, and ue towers.  I've weighted the 1, 3, and 10m measurements by 2m, 3.5m, and 4.5m, respectively, in the vertical, averaged the horizontal velocity components from the two towers that define each side, computed the projection of the wind perpendicular to the orientation vector between the towers (as measured by the laser theodolite), and divided the computed sum of the flow through these 3 sides by the horizontal area of the triangle (green).  Given the errors inherent in this calculation, I'm amazed by how well these measurements compare!

In the past 24 hrs, I've cycled power on the pillows twice, once to restore pillow 4 (that died 9 Nov afternoon) and once to restore pillow 3 (that died early this AM).  Note that pilllow 4 has the new software, so that wasn't a complete fix, unfortunately.

Also last night, the 5m.c sonic died completely.  Previously, it was giving data at 0.5 Hz and only from the sonic, despite me checking that the EC100 was set to Unprompted Data Rate of 20 samp/s.  The webcam image doesn't show anything obviously wrong to me.  This morning, I've tried pio 0 0/1 and ddn/dup, to no avail.

...just a few hours ago at 7pm, after running solid since I worked on it on Monday (5 days ago).  The other 3 (with the software upgrade) were working fine.  As usual, a pio brought it back, so all 4 are working again.  Unlike usual but as expected, the networking now didn't go down, since we rewired the router.

• Our prompting and parsing upgrades to the snow pillows are now complete.  ncharts now shows Loads and SWE.  I'll update qctables soon to add SWE (DONE).  Pillow #3 still is running the old v3 code, though I have instructions from Matt to try to connect to it differently when we next return.
• After a week of erratic readings from two of the Apogees, all are now operating properly (without us doing anything!).  After consulting with Apogee, our best guess is that water got into the cables.  Apogee is sending us 4 replacements with their "newer" style (though the current ones also should have been the new style).
• One of Ethan's lidars on the uw tower still is not connecting.  Ethan plans to give us a new SD card (that probably will rename this to pi7) to try to install on our next visit.  We also will try to move the placement of his Pis in the DSM box to avoid antenna blockage.

With this handful of tasks, we are trying to decide if it is worth making another trip out to Crested Butte this month (as we had scheduled) or wait for the Dec. visit.  Clearly, the snowpack is developing, which would be nice to document with the above user sensors.  None of these tasks require climbing above 3m and could be done using the ladder (though this is awkward).

Starting with thoughts as Isabel and I drove back from the late Oct visit:

• One climbing harness in the barrel has leg clips, the other has closed leg loops.  It would be nice for both to have the clips, i.e. replace one with another harness.  (Will actually prefers the closed loops, hence why this harness is there.)
• It would be nice to have an Allen key set on a tool tether.
• It also would be nice to have a canvas "chalk" bag that doesn't have the messy fleece liner of the only bag currently in the barrel.
• We need to generate a walking path map (now in the site portion of this wiki) and place a copy in the JHA binder
• Ethan will give us a new SD card for the one of his Pis that isn't connecting.  He also suggested shifting the position of the Pis inside the DSM box to get better antenna placement.
• Make a mote console cable for Jessica and students to use with the spare set of Tsnows – maybe give to Ethan first?