We had always wanted the EC150 to be operating in automatic heater mode. According to our SE wiki entry, the heater volts should be set to -2. One (7m.dcs?) that I checked several days ago seemed to be frosting up and had the setting at -1, which I don't know the meaning of.
Just to be sure, I've just reset all 3 levels of both TTs to be -2 via:
rs x
<cr><cr><cr>
P<cr>
2718<cr>
S<cr>
M -2 <cr>
Y<cr>
I used a really wide screen and looked for ASCII messages at the end of the lines to confirm my entries, but basically you are typing in the blind (and have to do this quickly before the EC150 times out) while binary messages are streaming by.
If we see frosting-up behavior on other EC150s, we can repeat this on the other sensor positions.
Sunny and clear in the morning, but getting cloudier as the day went on. Cool and windy by the late afternoon.
IOP starting at noon today (though it was not officially decided on until after noon today).
Data check looked good this morning, without even any broken thermocouples. At some point today I rebooted dcsr and cc to bring back HRXLs. pc was off the net for a couple hours this afternoon but seemed to come back fine.
Got a ubiquiti link working between the elementary school and the base trailer today. Internet is now much faster, if you use ISS-GUEST rather than the 5G modem's wifi network.
Not much to report, I mostly took the day off!
Data looked OK this morning except that we had the same problem with netcdfs not being generated on barolo. Restarting dsm_server and statsproc once again fixed that issue. PC and SP both went off the net during the day but came back after the 4hr timeout in the dsm_watch_network script, so it must have been a modem problem. SP once again didn't have nidas running after it rebooted, see jira: https://jira.ucar.edu/browse/ISFS-533
Rebooted cc and dcsr to bring back HRXLs this evening. Now all HRXLs are reporting but we'll see how long that lasts.
Potential of an IOP tomorrow but conditions are uncertain so they are delaying making the call until tomorrow. Not ideal for us since it's not a lot of notice, but in this case all thermocouples are still working and there are no other sensor issues, so we don't have much to do to prepare.
Some high clouds and sunny.
Morning data check showed no data from HRXLs at cc, sp, dcs. Rebooted the dsms which brought the HRXLs back to working at all 3 sites (for now). Also this morning updated the dsm_watch_network script to reboot the dsm after 4 hours off the net instead of 12, to make it easier to keep an eye on data.
This morning Liz and I went to PRS so I could test out my fix for the port problems on prsr (and to pay a visit to the cows). With the udev rule set so the HRXL comes up as ttyHRXL port 0 is back to functional, so plugged the 2m ec100 back in to prsr. Tried the ott in port 7 on prsr again, but still not getting data from it through that port, even with dsm_port_config showing that the port is set correctly. So ott is still connected to prsg, port 6.
This afternoon visited SP and took site photos, boom angle, sonic height, serial numbers, and a soil sample. Pretty hard to get soil cores from the frozen ground but Liz got it done with the sledge!
No IOP tomorrow but there's talk on slack of a possible IOP on Wednesday.
Mostly clear with some haze, colder morning than the last few. IOP starting at noon today looking at non-fog cases.
Liz arrived yesterday and joined us this morning.
This morning Liz and I visited DCS to replace the broken thermocouple at 7m before the IOP started. We took advantage of the cold morning to drive most of the way to the site, so we could refill the gas in the generator. Tower lowering and raising went smoothly and everything is reporting after replacing the broken thermocouple. Tower was down between ~10am and 11:45 local. It looks like the missing data problem from the 7m EC150 resolved itself while we were lowering the tower and was likely just frost (sorry Steve). Got some more ec100 serial numbers while the tower was down. Everything was good when we left the site but dcst has not been on the net since 10:15 this morning (while we were on site–but I can't think of what we did that would have an effect on the modem). Assuming it's a modem problem and will come back up later tonight, since I was logged in and checking data after it went off the net.
Memorial hill was off the net this morning when I checked data, but came back on its own before I had time to visit it.
Once again things mostly working in the morning. LC had been off the net overnight but came back up on its own this morning. Snow sensor at DCS and CC were not reporting, tried rebooting both dsms, which looks like it did nothing at DCS and only brought the snow sensor back briefly at CC. Annoying, but I haven't figured out anything else to try about these.
Spent the morning on ISS, then visited some satellite sites in the afternoon:
- PC: took a soil sample.
- MW: site photos, sonic height, ec100 serial numbers, soil sample. Took about 4 tries to get the soil corer deep enough in the frozen ground to get a sample, the horses were very entertained. Forgot a compass to get the boom angle.
- SH: site photos, sonic height, ec100 serial numbers. Took a look in the pumphouse again, looks like they've replaced the leaky hose.
IOP starting tomorrow at noon. Sebastian has requested that we replace the broken 7m thermocouple at DCS if we have time, which as of now I think we will, so aiming to do that in the morning.
Isabel had noticed that qctables hadn't updated today. (Eventually) realized that the netcdf files hadn't been updated since 21:52 last night and that "dsp" wasn't working on barolo. killed and restarted dsm_server, which seems to have gotten things unstuck. We'll see if qctables looks okay in a few minutes.
Some clouds and haze, warmest day since I've been here.
Data check this morning (once we got data going again) didn't show any major issues. SP had been offline, but came back up when the modem script triggered a reboot. After reboot, though, it wasn't collecting data until I manually started nidas, more details on jira: https://jira.ucar.edu/browse/ISFS-533
Other than that not much to report, I spent most of the day getting caught up on ISS work.
Clear and cold in the morning, warmer and cloudier in the afternoon.
Did some blog catch-up of site pics, serial numbers, etc this morning. Data check showed HRXL at cc and dcs were down, but rebooting the dsms brought both of them back up. No other new sensor problems.
Did more troubleshooting at PRS with help onsite from Bill (thanks Bill!) and help over the phone from Chris (thanks Chris!). Turns out the problem on prsr with getting good data immediately after boot but then getting nothing is that at boot the port is configured correctly, but after a minute or so it gets configured for RS232 and therefore you don't get any more sonic data. This is due to interference with the HRXL usb-serial, which comes up as ttyUSB0, which apparently conflicts with ttyDSM0. I added a udev rule for the usb-serial at this site so it comes up as ttyHRXL, which seems to have solved the problem. I don't have anything plugged into port 0 right now, but checking with dsm_port_config shows that it remains set for rs485. Didn't do any looking at port 7, which was not reporting ott data the other day.
Also at PRS tried out the stepladder we found in the closet in the base trailer, but it's not tall enough to reach the 3m thermocouples. Got serial numbers from the 2m ec100 but couldn't reach the box for the 3m one.
Gary submitted a Jira issue that this sensor was reporting every 10s, rather than 60. I've just changed it by sending it the command "CS/I/60".
Preparing:
- inventory: job boxes (3 in storage), manfrottos, otts (5 in lidar trailer, 1 in base), shts (only 13 in base), ???
- TT license plates (now in base) (dcs now done)
- OTTs to base from lidar trailer
- break cables out of ice (3 spots for DCS)
- pack up battery charger
- pack up soil oven
Timing:
Steve runs to Logan to pick up repaired sonics?. The repairs have been delayed.- All drives to/from site (especially TTs) should occur in the morning, when the ground is harder. Perhaps this means caching stuff on site and removing the caches in the mornings?
- Steve already there; Clayton arrives 2/23 (last day of ops)
- tear down entire site into pickup and Penske; transfer pickup stuff into base
- start with satellites, since Sebastian won't be available Thurs AM
- remove heating stuff from CSATs
- add washers to CSATs?
- good luck removing soils!
- figure out how to pack NR01s with ventillation systems
- provide some assistance to remove PI sensors
- sort out NCAR vs. UU radiometers/ventillators (relatively easy: CVF3s to UU, CVF4s to us; Bindered sensors to us, rest to UU)
- return HRXLs to UU – just cut cables?
- return CS125s (DC is ISS's – E3412)
- packing:
- Rohn towers/bases/guys to 16' Penske
- solar/job box to 26' Penske
- manfrotto/square booms to 26'
- other booms (including all TT) to 16'
- cables sorted to base
- DSMs to 16'
- tear down entire site into pickup and Penske; transfer pickup stuff into base
- Dexter available 3/1ish; Tony, Chris arrive 3/2ish
- move solar panels, job boxes, manfrottos, etc from 16' to 26' Penske
- coil up supersite power cables, pack in 16' Penske
- help remove cc solar panels, if not already done
- remove fencing to 16'
- pull TTs out to BW road, if not already done
- someone pulls Ubiquiti from school roof
- need to pull out everything from DCS by 1 Mar
Need to arrange:
- Trailer Transit for base to SWEX (Dan did, pickup on 4 Mar afternoon)
- loaded with
- all sensors
- a few CFACT DSMs as spares
- cell modems pulled from CFACT DSMs
- loaded with
- Trailer Transit for TTs back to Boulder (Dan did, 26', pickup on 4 Mar from BestWestern)
- Penske from Boulder to bring SWEX infrastructure (Dan did, pickup 2/28 from Loveland)
- PAM tripods; booms; clamps
- more job boxes/batteries?
- SWEX DSMs
- Penske in Heber to return to Boulder (Dan did, 16', pickup 2/23 from Strawberry). Load infrastructure directly into this.
- power cables/transformers
- Rohn towers
- thermocouple motes
- anything dead (solar panels with bad connectors, etc.) "bad stuff"
- v2.7 stuff, Kipp&Zonens, CVF4s and cables
- bulk of CFACT DSMs
- round booms
Issues:
- Leave ventillation on NR01s, plugs into DSM AUX for power
- Bill wants his CS125 to go to SWEX
Organized by location:
- 16' Penske
- Power cables (all), probably need to be removed from field via pickup
- transformers
- TT generators
- Rohn towers (all), base plates, guys/straps
- most CFACT DSMs
- fencing stuff
- thermocouple interfaces
- sonic heater stuff
- CVF4s
- Kipp&Zonens
- used SHT85s
- v2.7 motes
- round TRH booms?
- TT brackets
- bad stuff
- some TRHs
- a DSM
- a mote
- the LC sonic?
- 26' Penske
- solar panels
- job boxes
- manfrottos
- sonic booms
- NR01 booms
- create site boxes?
- base
- sonics/EC150s (EC100s with EC150)
- TRH cases
- QDPs (top shelf)
- Otts and stands
- Soils (1 tub)
- DC power and data cables (sorted)
- other sensor cables
- cell modems
- spare CFACT DSMs
- NR01s (with ventillation)
- v2.8 motes
- cassette brackets
- Lidar trailer
- ISS CS125
Site-by-site:
| road | carry | power | job boxes | ladder | comments | order | |
|---|---|---|---|---|---|---|---|
| lc | muddy | medium | light AC, some digging and lots of ice/snow | 1 | 6' | straightforward, better in AM extra clamp on mast | |
| cc | snow crust (use snowmobile!) | drive up? | solar (10 racks), break ice/snow | 3 | no | ||
| dc | ok | drive up | light AC, ice/snow | 1 | no | straightforward | |
| sp | driveable | drive up | light AC, some snow/ice | 1 | no | straightforward, better in AM | 2 |
| sh | paved | drive up | light AC, snow/ice | 1 | no | straightforward | 1 |
| mh | icy patches (use chains), open bike gate | drive up | light AC, snow/ice | 1 | 3' | ||
| pc | paved | shortish | light AC, some digging, no snow | 1 | no | straightforward | 4 |
| up | driveable | drive up | solar (2 racks), no snow | 1 | no | access when frozen | |
| mw | ok | short | light AC, no snow | 1 | 6' | straightforward | 3 |
| prs | muddy | drive up | heavy AC, transformers, no snow | 2 | 8' | a lot of work, plus fence | 5 |
| dcs | muddy | drive up | heavy AC, transformers, break ice 3 spots | 2 | 8' | a lot of work, plus 3xfences | 6 |
Anything muddy, plan to drive in/out in morning. Rent a UHV to shuttle in and out of MH and CC?
For a satellite site carry in:
- ladder
- crowbar (optional)
- toolbag
- boxes
- CSAT case
- EC150 case
- site box
- 3xcardboard boxes
Carry out:
- job box
- 2 batteries + charger
- rohn section and base plate
- manfrotto
- ott stand
- 7–9 stakes, 1 auger?, ground rod
- 4 booms
- CSAT
- EC150+EC100+T beehive
- 2xTRH (cardboard boxes)
- NR01 (another cardboard box?}
- site box with
- QDP+fittings
- soils+mote
- DSM
- spline ball
- ladder
- tooibag
Supersite trips:
- remove sensors
- remove trailer
- remove transformers and generator
- remove power cable
- remove fencing
Sunny and cold in the morning, sunny and warmer in the afternoon. Not much fog for the IOP last night. Sounds like we may not have another IOP for the next several days as conditions don't look favorable.
Visited CC this morning to see about the unresponsive HRXL. I had thought the problem might have been that the usb-serial connector was still plugged in to the serial card USB ports, but it was already plugged directly into the pi. Tried ddn, unplug, replug, dup but that didn't bring it back, so eventually I rebooted the dsm, which did bring it back. Annoying, but at least that's something that can be done remotely. Jira link for more info: https://jira.ucar.edu/browse/ISFS-527
While I was at the site I took a soil sample and took some site pictures (that someday I will hopefully have time to upload!).
Just after the IOP ended at noon I went to PRS to check out the unresponsive ott. I found that, once again, it's a port problem on the prsr dsm. More details in Jira here: https://jira.ucar.edu/browse/ISFS-529
For now, I left the ott plugged in to port 6 of prsg so we can collect data while we figure this dsm problem out. We're now out of spare ports on prsg so everything else better keep working...
IOP starting today at noon. Clear and cold in the morning, getting warmer through the day.
Steve and I paid several visits to PRS today to investigate thermocouple nans at 3m and 7m and an unresponsive EC100 at 2m:
- added thermocouple probe at 3m (previously did not have one)
- found 7m thermocouple cable not attached in data logger, attached it, but found that the probe was broken so we had to lower the tower to replace the probe. Tower lowered between approx 9:30 and 10:30 local. Cable management is very easy at PRS now that we have dressed the cables:
- unresponsive EC100 turned out to be a prsr DSM port issue. EC100 was powered w/ green status lights but did not give any data on port 0. Good 3m EC100 didn't give data on port 0, 2m EC100 did give data on port 3. Tried replacing the dsm but replacement dsm had the same issue, so ended up moving the 2m EC100 to port 0 on prsg. More details on jira: https://jira.ucar.edu/browse/ISFS-529
Steve paid a visit to Soldier Hollow and replaced some screws in the mount for the old-style sonic and added a new TP01 probe (left existing one in place but unplugged) while I was debugging the prsr dsm.
Dropped Steve off at the airport shuttle this afternoon. Thanks for everything!
Misc stuff today
- cleaned up base a bit more
- Give Isabel a tour of satellite sites:
- DC: took soil core; took photos; shot boom angle
- LC: took soil core; took photos; shot boom angle
- UP: took soil core; took photos; shot boom angle
- PC: took photos; shot boom angle; reseated Tsoil sensor side Binder connector (preventative from a few days ago)
- MH: took photos; shot boom angles
- Lowered DCS TT to replace 7m thermocouple, which turned out to be okay (just poor connection in logger box, down from about 1440-1650.
- Isabel rebooted cc, which brought the HRXL back to life.
Mostly sunny, temps teens to 40s(?).
A short visit each to dcs and prs to mount and connect the new HRXL sensors. However, only dcs worked (briefly) then died on a ddn/dup cycle (see below). Both had tested fine in the base with my modified wiring (since these came with potted cables attached).
Sebastian reprogrammed the TC loggers, so a lot of playing with new baud rates to get them to work. (Initially, he set them back to 9600, which wasn't fast enough for 20Hz data. Eventually, he went back to 115200.)
A LOT of config/R changes to keep up with (finally!) variable definitions for TC and RAD loggers, v2.7 radiometers found not to be deployed as per their I2C addresses (which are coded to in and out), etc. Note that qctables now compares (derived) Rlw values, rather than Rpile and Tcase, since the RAD logger does not output Rpile.
Our first attendance of the daily project zoom meeting organized by the PIs.
An "ephemeral" IOP is called for Tues noon, so we'll need to replace broken TCs at 7m at both supersites tomorrow.