Blog from December, 2021

qctables has shown that only the dc CS125 is reporting.  This is because the message format was never changed for the sensors that UU provided.  Have just changed all formats to "11", except for dcs that is currently unreachable.

For the record, the format was set to "5".  Also, the procedure was:

rserial /dev/ttyDSM0 (or whatever)

open 0 (enter user menus)

1 (select message output)

1 (select message format)

11 (set format to 11)

0 (up one menu)

9 (save and exit)

While in this menu, recorded the serial numbers for most sensors.

Just to start a list of what to do when/if(!) we get back next week – a fairly long list for 3 days!:

Site issues:

• dcs is totally down.  It appears to have lost power on 21 Dec soon after being powered up and subsequently died on 26 Dec.  Could the Victron have V/I limit settings set incorrectly?  Update: Sebastian says that his power (connecting to our AC outlet) is fine.  This further suggests that this is a Victron issue, rather than AC power.  DONE, Victron was bad. Replaced.
• pc is totally down.  Similarly, it lost power on 15 Dec soon after being powered up and died 23 Dec.  AC not getting from power panel to job box – cable issue?  DONE, extension codes became unplugged under the snow.
• lc power cord was cut by plow.  Might stay up until we get there and replace the cord. 
• up may or may not have a power drop, so might need to create another solar farm. (Do we have cables?)

Existing sensor issues:

• Most sonics are reporting NA, though they did work initially.  Assuming this is ice related (try more heating?), but might check with Campbell to see if a configuration change would help.  See table below
• Most EC150s also show missing data.  However, in at least one case, I've found the EC150 data to be okay – just missing because statsproc processes it in a group with the sonic data that ARE bad.  Hopefully, fixing the sonics will bring back the EC150 values.
• A thermocouple appears to have been plugged in at sh on 19 Dec, but seems to have died on 21 Dec – just 2 days later.  While it was plugged in, the data look good, though not exactly like tc (maybe a good thing).  I do see noise above 3Hz during the day and 8Hz at night.
• Rfan.3m.prs is bad (T/RH ok).  Housing should be replaced?  DONE, old housing had no fan!
• Rlw.in.2m.prs not reporting (in the CVF4 housing) – another Binder issue?  DONE, mote wasn't sampling, then Rlw.in wanted to be opened up all the way and reassembled.
• Tsoil.mw is all -273.  Hopefully, just a loose Binder connector. DONE, was loose Binder.
• P.2m.prs was missing horizonal pipe mount. FIXED
• HRXL.cc stopped reporting 24 Dec.

Sensor changes:

• reconfigure radiometers into operational mode at dcs and prs.  Have new mounting for 0.5m radiometer "paddles".
  
  • move CVF4 Rlws to darkhorse
  • move other paddles to 0.5, 7, and 32m (booms for 7 and 32 are on site at each site) – lots of cables to redo.  Sebastian will help.
• attach thermocouples and loggers to towers at dcs and prs.  Eric said he'd do most of this work.
• raise both TTs

Standard operations:

• finish recording serial numbers (a table has been started in this wiki)
• take site photos (maybe Dan has already done?)
• shoot boom angles (compass, then Leica)
• take the first set of gravimetric measurements (though freezing soil probably will have moisture that the EC-5 doesn't see).

Software:

• test heating scripts
• it appears that I'm close to parsing the RAD_LOGGER message correctly, but we should check with Sebastian.  I suspect that my "Rpile" variable from him really is an uncalibrated pile voltage.
• why are there still "Early sample...prior to sorting window" errors using prep?  I thought our new timing fixed all this!
• why are there frequent "EMERGENCY|SampleOutput: inet:128.117.43.122:30010: IOException: inet:128.117.43.122:30010: send: Message too long, disconnecting" error messages on the DSM consoles?  Could these be from the OTT, that reports something like a 32k byte sample?
• create irga_co2/h2o.dat QC files by height for supersites
• update all csat.dat QC files once boom angles are known.

Sonic status:

Weather:

Rain throughout the day, followed by wintery mix in the evening

Tasking:

• Radiometers at DCS and PRS are plumb as of 1:30pm and 3pm MST today, respectively
• Punchlist items at both supersites
• Connected Lake Creek Power! 
• Added two solar panels to Upper Provo and swapped batteries, unfortunately they will not see sun for the next week 
• Base trailer deep clean

I will be leaving tomorrow, signing off for this trip. Great job everyone, see you in the new year!

I've just made a flurry of config changes on the dsms to accommodate how things are in the field:

• Finally got parsing of Sebastian's rad_logger message working – still not sure about the variables, but it is a start.  Changes made on dcst's sensor_catalog.xml
• Changed all v2.7 motes at PRS to use id numbers.  Set id=90 for the 2 Rsw's, id=91 for the 0.5m Rlw's, id=92 for 2.0m Rlw's, id=93 for 7m Rlw's and id=94 for the 32m Rlw's.  Appropriate changes made to cfact.xml on prsg and prst.  I think this is the way we will want to operate throughout this project.

It would be great if someone would push/pull these changes back to github. 

Lest I forget over the holidays!

• Green and White trucks!
• At least one tub of ISS stuff (to be left in Steve's office Dec 29 by Lou and Bill
• Two socket wrenches that Steve stole from the base toolchest
• The 2nd soil kit?  (It has the portable scale)
• Two base-plates with poles for the 0.5m radiometers at each supersite

Weather:

Clear throughout the morning, followed by scattered mixed precip in the afternoon.

Tasking:

• Loaded a GSA truck with unused cable, which Tony drove back to Boulder and left in the cage at Foothills
• Chris and myself went to the Center Creek satellite site and checked off a variety of punchlist items
• Chris departed for CO around noon
• Continued work at PRS, all ventilators powered and other items completed

The radiometers at both supersites are still a bit out of plumb, that will be the number one priority for tomorrow. I will make a blog post noting the time that each site's radiometers are plumb. Huge thanks to Chris and Tony for all of their hard work!

Weather:

Clear, calm, cold. Briefly got above freezing midday.

Tasking:

Last full day of this set up, great job everyone!

• Finished DCS, including power. Extended tower to tension guy wires, then retracted but left tilted up for the holidays. Everything was reporting(when we left)!
• Almost finished PRS. Retracted tower that we left extended for the holidays, also left tilted up for the holidays. Will visit tomorrow morning to finish some punchlist items

All DSMs are on the net except for UP and LC, which are still waiting on utility power.

Upon arrival this morning, 17m EC100 status lights were blinking and output was nonexistent. Once box warmed up(with sensors disconnected), the error resolved itself.

Output has resumed but H2O still seems abnormally high.

Before the holidays, just taking a peak at the available data on NCharts

I think a test that the upper radiometers are obscured is that Rsw.out > Rsw.in.  At low sun angles (sunrise, sunset – not a Fiddler on the Roof song), this could be real, but I'm pretty sure I've seen it associated with snow on the shields.

Weather: 

Clear, cold, and calm. Apparent inversion with some smog throughout the day.

Tasking:

• PRS for the day. Tower is up and extended for the night! Most wiring is completed, save for the final termination of the ventilator power
• DCS power drop is now live

Tomorrow is our last full day, we will be tilting and extending DCS along with a variety of other tasks

Weather: 

Foggy morning, clear and cold all day.

Tasking:

• Completed punchlist items at DCS site
• Instrumented and fenced PRS tower

Tomorrow will be final dressing and tower erection at the supersites.

Weather: 

Foggy morning followed by a clear, calm, and cold day.

Tasking:

• Continued straightening up the base trailer and transferring items to storage
• Prepped booms for install at PRS
• Installed CS125 at MW, completed punchlist items. 
• Replaced both TRHs at Pine Canyon
• Replaced 0.5m TRH at Upper Provo

See Sites, Stations List for updated punchlist by site.

Dan, Tony, Chris took the day off.  Steve and Gary visited DCS for a couple of hours in the morning.  Then we were in meetings...

DCS:

• Attached rad logger cable, but didn't work.  2 days earlier, I saw -5V on the logger's pin2 of the D9, but I didn't see this at the DSM end on the Bulgin (on any pin).  I <could> see -5V on pin3 of the D9 from the DSM.  Has to be something simple.  I also am not sure of the baud rate or the data rate (1/min?), but using minicom I didn't even see garbage after 2min.
• Installed batteries and power board in the job box.
• Powered up all of the DSMs (though not together due to power cabling).  The only issue was the fan not working on TRH.1m.
• Gary reconfigured the modems and we set "rfkill unblock wifi" on each DSM, so both cell and wifi connections should work.
• Found that the V2.7 mote connected to the Rsw's had its power switch off, so turned it on.  Also saw that the data output was in "ascii-ish" with mp=1.  Changed to mp=0 (the standard Wisard format)

With most of the satellite sites reporting, started looking at a few variables:

• T has been below freezing most of the time, with a low at CC of -22C earlier in the week
• RH has been near 100% a lot recently with snow.  Note several sites reporting RH a bit over 100%, which is not unusual for us.  I haven't worked out if this would be due to RH with respect to ice...
• Maximum speeds were 10m/s last week, though often below 1 m/s.
• Pressures track each other and are in the correct order with station height.  The topography in this valley is somewhat deceptive:  From high elevation to low, the order is LC, CC, DC, PC, MH, UP, PRS, MW, SH, DCS.
• Rsw.out is at least 50% of Rsw.in this week, and sometimes much higher.  A good indication of snow on the ground!
• ldiag often is close to 1 for some sensors.  This would indicate that heating/Teflon tape isn't completely effective.  We'll have to check/work on this.  CSAT.mw is always 1 (not working).