Blog

Once again things mostly working in the morning. LC had been off the net overnight but came back up on its own this morning. Snow sensor at DCS and CC were not reporting, tried rebooting both dsms, which looks like it did nothing at DCS and only brought the snow sensor back briefly at CC. Annoying, but I haven't figured out anything else to try about these.

Spent the morning on ISS, then visited some satellite sites in the afternoon:

• PC: took a soil sample.
• MW: site photos, sonic height, ec100 serial numbers, soil sample. Took about 4 tries to get the soil corer deep enough in the frozen ground to get a sample, the horses were very entertained. Forgot a compass to get the boom angle.
• SH: site photos, sonic height, ec100 serial numbers. Took a look in the pumphouse again, looks like they've replaced the leaky hose.

IOP starting tomorrow at noon. Sebastian has requested that we replace the broken 7m thermocouple at DCS if we have time, which as of now I think we will, so aiming to do that in the morning.

Isabel had noticed that qctables hadn't updated today.  (Eventually) realized that the netcdf files hadn't been updated since 21:52 last night and that "dsp" wasn't working on barolo.  killed and restarted dsm_server, which seems to have gotten things unstuck.  We'll see if qctables looks okay in a few minutes.

Some clouds and haze, warmest day since I've been here.

Data check this morning (once we got data going again) didn't show any major issues. SP had been offline, but came back up when the modem script triggered a reboot. After reboot, though, it wasn't collecting data until I manually started nidas, more details on jira: https://jira.ucar.edu/browse/ISFS-533

Other than that not much to report, I spent most of the day getting caught up on ISS work.

Clear and cold in the morning, warmer and cloudier in the afternoon.

Did some blog catch-up of site pics, serial numbers, etc this morning. Data check showed HRXL at cc and dcs were down, but rebooting the dsms brought both of them back up. No other new sensor problems.

Did more troubleshooting at PRS with help onsite from Bill (thanks Bill!) and help over the phone from Chris (thanks Chris!). Turns out the problem on prsr with getting good data immediately after boot but then getting nothing is that at boot the port is configured correctly, but after a minute or so it gets configured for RS232 and therefore you don't get any more sonic data. This is due to interference with the HRXL usb-serial, which comes up as ttyUSB0, which apparently conflicts with ttyDSM0. I added a udev rule for the usb-serial at this site so it comes up as ttyHRXL, which seems to have solved the problem. I don't have anything plugged into port 0 right now, but checking with dsm_port_config shows that it remains set for rs485. Didn't do any looking at port 7, which was not reporting ott data the other day.

Also at PRS tried out the stepladder we found in the closet in the base trailer, but it's not tall enough to reach the 3m thermocouples. Got serial numbers from the 2m ec100 but couldn't reach the box for the 3m one.

Gary submitted a Jira issue that this sensor was reporting every 10s, rather than 60.  I've just changed it by sending it the command "CS/I/60".

Preparing:

• inventory: job boxes (3 in storage), manfrottos, otts (5 in lidar trailer, 1 in base), shts (only 13 in base), ???
• TT license plates (now in base) (dcs now done)
• OTTs to base from lidar trailer
• break cables out of ice (3 spots for DCS)
• pack up battery charger
• pack up soil oven

Timing:

• Steve runs to Logan to pick up repaired sonics?. The repairs have been delayed.
• All drives to/from site (especially TTs) should occur in the morning, when the ground is harder.  Perhaps this means caching stuff on site and removing the caches in the mornings?
• Steve already there; Clayton arrives 2/23 (last day of ops)
  
  • tear down entire site into pickup and Penske; transfer pickup stuff into base
    
    • start with satellites, since Sebastian won't be available Thurs AM 
    • remove heating stuff from CSATs
    • add washers to CSATs?
    • good luck removing soils!
    • figure out how to pack NR01s with ventillation systems
  • provide some assistance to remove PI sensors
    • sort out NCAR vs. UU radiometers/ventillators (relatively easy: CVF3s to UU, CVF4s to us; Bindered sensors to us, rest to UU)
    • return HRXLs to UU – just cut cables?
    • return CS125s (DC is ISS's – E3412)
  • packing:
    • Rohn towers/bases/guys to 16' Penske
    • solar/job box to 26' Penske
    • manfrotto/square booms to 26'
    • other booms (including all TT) to 16'
    • cables sorted to base
    • DSMs to 16'
• Dexter available 3/1ish; Tony, Chris arrive 3/2ish
  • move solar panels, job boxes, manfrottos, etc from 16' to 26' Penske 
  • coil up supersite power cables, pack in 16' Penske
  • help remove cc solar panels, if not already done
  • remove fencing to 16'
  • pull TTs out to BW road, if not already done
• someone pulls Ubiquiti from school roof
• need to pull out everything from DCS by 1 Mar

Need to arrange:

• Trailer Transit for base to SWEX (Dan did, pickup on 4 Mar afternoon)
  • loaded with
    • all sensors
    • a few CFACT DSMs as spares
    • cell modems pulled from CFACT DSMs
• Trailer Transit for TTs back to Boulder (Dan did, 26', pickup on 4 Mar from BestWestern)
• Penske from Boulder to bring SWEX infrastructure (Dan did, pickup 2/28 from Loveland)
  • PAM tripods; booms; clamps
  • more job boxes/batteries?
  • SWEX DSMs
• Penske in Heber to return to Boulder (Dan did, 16', pickup 2/23 from Strawberry).  Load infrastructure directly into this.
  • power cables/transformers
  • Rohn towers
  • thermocouple motes
  • anything dead (solar panels with bad connectors, etc.) "bad stuff"
  • v2.7 stuff, Kipp&Zonens, CVF4s and cables
  • bulk of CFACT DSMs
  • round booms

Issues:

• Leave ventillation on NR01s, plugs into DSM AUX for power
• Bill wants his CS125 to go to SWEX

Organized by location:

• 16' Penske
  • Power cables (all), probably need to be removed from field via pickup
  • transformers
  • TT generators
  • Rohn towers (all), base plates, guys/straps
  • most CFACT DSMs
  • fencing stuff
  • thermocouple interfaces
  • sonic heater stuff
  • CVF4s
  • Kipp&Zonens
  • used SHT85s
  • v2.7 motes
  • round TRH booms?
  • TT brackets
  • bad stuff
    • some TRHs
    • a DSM
    • a mote
    • the LC sonic?

• 26' Penske
  • solar panels
  • job boxes
  • manfrottos
  • sonic booms
  • NR01 booms
  • create site boxes?
• base
  • sonics/EC150s (EC100s with EC150)
  • TRH cases
  • QDPs (top shelf)
  • Otts and stands
  • Soils (1 tub)
  • DC power and data cables (sorted)
  • other sensor cables
  • cell modems
  • spare CFACT DSMs
  • NR01s (with ventillation)
  • v2.8 motes
  • cassette brackets
• Lidar trailer
  • ISS CS125

Site-by-site:

Anything muddy, plan to drive in/out in morning.  Rent a UHV to shuttle in and out of MH and CC?

For a satellite site carry in:

• ladder
• crowbar (optional)
• toolbag
• boxes
  • CSAT case
  • EC150 case
  • site box
  • 3xcardboard boxes

Carry out:

• job box
• 2 batteries + charger
• rohn section and base plate
• manfrotto
• ott stand
• 7–9 stakes, 1 auger?, ground rod
• 4 booms
• CSAT
• EC150+EC100+T beehive
• 2xTRH (cardboard boxes)
• NR01 (another cardboard box?}
• site box with
  • QDP+fittings
  • soils+mote
  • DSM
  • spline ball
• ladder
• tooibag

Supersite trips:

1. remove sensors
2. remove trailer
3. remove transformers and generator
4. remove power cable
5. remove fencing

Sunny and cold in the morning, sunny and warmer in the afternoon. Not much fog for the IOP last night. Sounds like we may not have another IOP for the next several days as conditions don't look favorable.

Visited CC this morning to see about the unresponsive HRXL. I had thought the problem might have been that the usb-serial connector was still plugged in to the serial card USB ports, but it was already plugged directly into the pi. Tried ddn, unplug, replug, dup but that didn't bring it back, so eventually I rebooted the dsm, which did bring it back. Annoying, but at least that's something that can be done remotely. Jira link for more info: https://jira.ucar.edu/browse/ISFS-527

While I was at the site I took a soil sample and took some site pictures (that someday I will hopefully have time to upload!).

Just after the IOP ended at noon I went to PRS to check out the unresponsive ott. I found that, once again, it's a port problem on the prsr dsm. More details in Jira here: https://jira.ucar.edu/browse/ISFS-529

For now, I left the ott plugged in to port 6 of prsg so we can collect data while we figure this dsm problem out. We're now out of spare ports on prsg so everything else better keep working...

IOP starting today at noon. Clear and cold in the morning, getting warmer through the day.

Steve and I paid several visits to PRS today to investigate thermocouple nans at 3m and 7m and an unresponsive EC100 at 2m:

• added thermocouple probe at 3m (previously did not have one)
• found 7m thermocouple cable not attached in data logger, attached it, but found that the probe was broken so we had to lower the tower to replace the probe. Tower lowered between approx 9:30 and 10:30 local. Cable management is very easy at PRS now that we have dressed the cables:
• 
• unresponsive EC100 turned out to be a prsr DSM port issue. EC100 was powered w/ green status lights but did not give any data on port 0. Good 3m EC100 didn't give data on port 0, 2m EC100 did give data on port 3. Tried replacing the dsm but replacement dsm had the same issue, so ended up moving the 2m EC100 to port 0 on prsg. More details on jira: https://jira.ucar.edu/browse/ISFS-529

Steve paid a visit to Soldier Hollow and replaced some screws in the mount for the old-style sonic and added a new TP01 probe (left existing one in place but unplugged) while I was debugging the prsr dsm.

Dropped Steve off at the airport shuttle this afternoon. Thanks for everything!

Misc stuff today

• cleaned up base a bit more
• Give Isabel a tour of satellite sites:
  • DC: took soil core; took photos; shot boom angle
  • LC: took soil core; took photos; shot boom angle
  • UP: took soil core; took photos; shot boom angle
  • PC: took photos; shot boom angle; reseated Tsoil sensor side Binder connector (preventative from a few days ago)
  • MH: took photos; shot boom angles
• Lowered DCS TT to replace 7m thermocouple, which turned out to be okay (just poor connection in logger box, down from about 1440-1650.
• Isabel rebooted cc, which brought the HRXL back to life.

Mostly sunny, temps teens to 40s(?).

A short visit each to dcs and prs to mount and connect the new HRXL sensors.  However, only dcs worked (briefly) then died on a ddn/dup cycle (see below).  Both had tested fine in the base with my modified wiring (since these came with potted cables attached).

Sebastian reprogrammed the TC loggers, so a lot of playing with new baud rates to get them to work.  (Initially, he set them back to 9600, which wasn't fast enough for 20Hz data.  Eventually, he went back to 115200.)

A LOT of config/R changes to keep up with (finally!) variable definitions for TC and RAD loggers, v2.7 radiometers found not to be deployed as per their I2C addresses (which are coded to in and out), etc.  Note that qctables now compares (derived) Rlw values, rather than Rpile and Tcase, since the RAD logger does not output Rpile.

Our first attendance of the daily project zoom meeting organized by the PIs.

An "ephemeral" IOP is called for Tues noon, so we'll need to replace broken TCs at 7m at both supersites tomorrow.

A nice, sunny day.

PRS:

• Reconnected prst to the network.  I think what did it was that I unplugged it to test the modem in my laptop and then plugged it back in, so I suspect waiting for the modem reboot script to time out would also have worked. At some point ran into issues with hostapd but upon further investigation I think those are just because the hostapd config file needs to be updated.  Also at some point did a "ddn" without a "dup".  Did the "dup" later in the afternoon.
• Attached TC logger to prsr port 5 (not port 6 as in dcsr, since it wasn't available).  Changed config on git and pulled to the dsm.
• Took soil sample.

SH:

• Installed the CSAT3A (old style) sent by Chris yesterday.  (Thanks, Chris!). Modified the prototype "gold" cassette to accommodate the different head style with some cool machining and yet another trip to Ace Hardware, but still needed to shim with washers.  This was the last ISFS sensor to be installed!  Sebastian brought another sonic of the same type (thanks, Sebastian!), that we'll hold on to as a spare until Campbell repairs ours.
• NEED TO REPLACE CSAT boom mounting screw with one a bit shorter (0.5" 1/4-20).  Actually, the long bolt as well (2.5" 3/8).
• Took soil sample.

Other:

• Took delivery of 2 more HRXLs that UU just purchased.  These have integral, potted, cables attached.  We have one prepared set of cabling.  Figured out how to connect things, so will install this at DCS tomorrow.
• Sometime midday the 5G modem in the window of the base trailer got too hot in the sun and shut off, so temporarily moved it down to the bench to be out of the sun, but that means the internet is worse. Could think about a better solution to this.

Started chilly, then nice, then windy and cold, then a nice evening.

Dan has started PTO – Thanks for everything!!

PRS:

• lowered TT
• reseated connector to Rlw.xx.7m and got it going
• dressed cables up TT and closed off spiral cable guides
• raised TT (finally!), up about 10AM
• removed Sebastian's darkhorse

CC:

• unplugged and plugged in USB serial dongle, then restarted NIDAS.  Got HRXL going
• did a crude compass boom angle shoot of 237deg magnetic (248 true) looking into the boom.
• However, something that we did stopped cc from being on the net.  Probably have to reboot it.  Might wait until we go out to install a thermocouple.

DCS:

• replaced TRH housings at 1m, 3m (they didn't have fans!). Now both are working
• reconfigured mote IDs on Rsw.xx.2m and now see data
• lowered TT
• replaced TRH housing at 17m that was working (so data would have been fine, just had been reading too high Rfan).  Replacement is normal.
• installed thermocouples at all 3 levels
• closed off spiral cable guides
• raised TT
• connected TC_LOGGER to dcsr, port 6 and changed config (but didn't get data until I also changed the baudrate to 115200 – different than the RAD_LOGGER!)
• installed thermocouple connectors at 1, 2, 3m
• installed an actual thermocouple at 3m
• discovered that AC power was off (when Utah started plugging lots of things in).  It had been down since about noon 2 days ago.  Turned out to be a loose plug at the power panel.  Have tried to connect, but don't get a solid "click" when trying to turn.  We'll have to keep an eye on this.

SH:

• asked Chris to FedEx an early CSAT3A to us to replace the one that has been removed.  It will need a special mount.

UP:

• this snow depth sensor appears to have fixed itself, with no action by us.

TODOs:

• replace a sonic at SH (tomorrow AM)
• get cc back on the net
• connect TC logger at prs
• sprinkle thermocouples on satellite sites

A bit of fog in the morning, along with some drizzle that cleared later in the day.  I saw a rainbow.

The push today was to get the supersites finished, since today was supposed to be the first day of ops.

DCS:

• wired fan power for NR01 (had been missed before)
• secured thermocouple connectors
• (radiometer were cleaned by Sebastian yesterday)
• raised tower (but without thermocouples themselves)

PRS:

• leveled 0.5m radiometers
• installed the rest of the 2m radiometers and dressed cables
• installed 7m and 32m radiometers
• Sebastian cleaned all radiometers
• Sebastian created a platform for his lidar
• replaced prototype cassette mount with final design on one sonic, to better fit the thermocouple mount
• found that the 7m sonic had been mounted upside down on its cassette, and fixed it
• added telescoping tower thermocouple cabling (and thermocouples themselves), but logger is still not hooked up
• started to raise tower, but found that the cables tangled.  Pulled the tower back down and reworked some things, but this took enough time that we were not able to fully erect it.  (We didn't want to raise it when it was too dark to see what was happening).  It is about 15m high.

In the meantime, Dan corrected a power issue for Gannet's group and drove to Logan to return bad sonics to Campbell.

So...another productive day and pretty much the end of set-up, but didn't actually become operational.

Also, a server issue in Boulder is preventing us from getting a complete picture of instrument outages, of which I'm sure there are still several.

cold rain/mist all day, that got a lot of stuff (including the field crew!) soaked.

Dan did a bunch of things (he can edit):

• further work to diagnose faulty sonics
• rented a skidsteer and dug out the road to the DCS gate
• parked the rental trailer at the sounding site for ISS
• Removed faulty sonic from sh
• Installed snow depth at mh

In the meantime, Isabel and I had a supersite day:

• PRS
  
  • set the jumper on port 2 to bypass the part of the serial port board that was broken. 17m ec100 now online without having to replace the dsm.
  • stripped everything from Sebastian's darkhorse and rerouted cables
  • Installed 0.5m Rlw pair.  This required moving the ventillator wires from being bundled with 7m and 32m for the intercomparison, to the 2m Rsw and Rlw ventillation system.
  • Installed uplooking 2m Rlw.in, but removed 2m Rsw.out, since the mounting scheme for Rsw.out.2m had to change
  • That was all we could do, since we didn't have the 7m and 32m booms, or plates to finish the Rxx.out.2m mounting (we later recovered plates from dcs, that weren't used)
• DCS
  • swapped in the spare configured modem (M73), since the old one wasn't connecting and both data and status lights were blinking alternating red and green. Took restarting NIDAS before the Pi recognized it.  dcst was on the net after ddn/dup when I checked at the site, but  now seems to be down again.
  • exchanged the 35m serial cable to rad_logger with a 5m one, since now much closer
  • reinstalled 0.5m Rlw pair to allow more cable length
  • dressed cables to darkhorse.  In the process, found that the NR01 fan hasn't been connected.  At this point, it would be easiest to connect it to the aux power on dcsg
  • lowered and tilted down the tower, to allow thermocouple cable installation.
  • ran cables to 7m, 17m, and 32m, but didn't have the screws to mount the connectors to the sonics.
  • tilted up the tower, but didn't raise.

Still needed:

• PRS
  • 1/4-20 bolts and plates (now have) to implement Dan's big plate mounting of Rxx.out.2m, then dress darkhorse cables
  • rad booms (need to retrieve from storage).  I think I have bolts to mount paddles to these
  • split link for 0.5m guy (have turnbuckle)
  • all thermocouple stuff
  • may want an extra 5m bulgin in case we run out of slack for the 32m radiometer
  • raise tower
  • may need amp power plug for NR01 fan
• DCS
  • thermocouple screws
  • thermocouple logger and cabling for 1,2,3m (UU will do), along with all thermocouples
  • amp power plug for NR01 fan
  • raise tower

Found that the real-time data stream hasn't been alive for the last 24 hours.  No raw_data/isfs_ data files since 2022 01 05 04 00 and data_stats failed.  This also caused webplots to fail.  dsm_server was running, so check_cfact_procs didn't do anything.  I just killed dsm_server and reran check_cfact_procs, which seems to have brought everything back to life.

As user isfs on barolo:

pgrep dsm_server (returns the process id number of the dsm_server process)

kill -9 <pid>

check_cfact_procs