Today was another clear, calm, and mild day, with some high clouds moving in in the late afternoon and a bit more haze than yesterday. The daytime high ended up being in the low 40’s - the start of a forecast warming trend over the next week that should see high’s near 50. The snow continues to melt and some fields are nearly clear of snow.
We began the day at the ISS2 sounding trailer troubleshooting a power failure - the GFI tripped on the outlet. Afterwards we visited LC and CC to collect soil samples and measure the boom angles with the GPS. While at LC we visually inspected the sonic because of the intermittency of the wind measurements. I expected to see frost, as there was considerable frost at the ISS2 site, but there was none at LC. From what we could see, there were no obvious obstructions on the transducers, but we did not have a ladder. The sonic status LED inside the EC100 box was solid red, which is not surprising since the ldiag flag has been at 1 during the periods of missing data such as when we were at the site this morning. It’s pretty consistent at this point that ldaig=1 from shortly after sunset to shortly after sunrise. Perhaps a change of the sonic in in order?
A number of sites dropped off the network over the course of the day, but according to Nagios they have all recovered. It also appears that the 17m thermocouple at PRS broke this morning, so we will have to replace it before the next IOP. Otherwise everything is running smoothly. We still need to collect a 2nd round of soil samples at SP and PRS, which I will probably wait until Jacquie arrives to do, and I need GPS boom angles from MH and Leica measurements from PRS, both of which I may do tomorrow before Jacquie arrives.
Today was clear and warmer, with a daytime high just above freezing and a bit more haze than the past few days. Winds were very calm all day and based on the afternoon sounding were calm into the free atmosphere above the BL.
John and I spent the first part of the day doing some outreach with a local high school science class, discussing the CFACT project and giving a tour of the Upper Provo site. John potentially set up another outreach opportunity for next Friday where he would give some middle school students a tour of the Deer Creek Super Site and ISS trailers.
Throughout the day there continued to be no issues with ISFS sensors, so I decided to take full Leica measurements of the DCS towers. I took 4 measurements per sonic (7, 17, 32, 1, 2, 3m) and 1 measurement per TRH (7, 17, 32, 1, 2, 3m), in addition to 4 GPS positions and one trailer point for reference. The entire process of 35 measurements and setup took a little over 2 hours.
Jacquie pointed out in a blog post that the data from the sonic at LC is rather sporadic, a trend that I’ve been monitoring myself. It seems to be happening overnight, so I suspect it’s an icing/moisture issue - ldiag = 1 during most (all?) of these periods. We will likely pay a visit to the site to collect a soil sample and measure the boom angle by GPS in the next couple of days, so we will assess the sonic at that time as well.
Today the PI’s announced there will be no IOP’s for the next 48 hours and no science meeting tomorrow, making it a hard down day.
Not much change in the weather today. Partly cloudy in the morning that became mostly clear the rest of the day with light northerly winds. The daytime high stayed below freezing again. The next few days look to stay cold, but slowly begin a warming trend into next week.
The morning data check showed no issues. An ephemeral fog IOP began at noon and by then all ISFS instruments seemed to be functioning properly. Sebastian plugged in his thermocouples at PRS and DCS in the late afternoon and they seem to all be functional as well as of this writing. The only real news to report from today is that the power returned to the MW site, as evidenced by the increase in Vbatt and Icharge at the site around 8:40 am. With this development all sites are back on AC or solar power and we will not need to swap batteries. We will continue to keep an eye on the Vbatt for all sites in case any issues crop back up. We also picked up the repaired soil corer T-bar from the welding shop, so we can return to collecting soil samples.
Due to the IOP we launched soundings at 2:15 and 4:15 pm and continued to check on the status of the ISFS instrumentation throughout the day.
Today we awoke to a dusting of snow that fell sometime after midnight last night. This morning was colder than the past few days with partly cloudy skies that slowly cleared throughout the day. The winds at the surface were from the north and quite cold, while the high temperature stayed below freezing.
The morning data check showed no issues, except that the battery at DC was getting low. We swapped out batteries at the site before noon, plugged the site into the AC power that had finally been restored, and took an additional Leica measurement. We then dropped off the T-bar to the soil corer at a local welding shop to get it fixed. It should be done by the end of the week or sooner.
John and I spent a bit of time working on the lidar, which has a moisture problem and software issue keeping it offline. We used the soil oven to bake another pack of desiccant and added it to the lidar, which will hopefully help with the moisture problem.
There is an IOP scheduled for tomorrow beginning at noon. As of this evening there are no issues with the ISFS.
All of the CFACT modems need to be switched to a different Verizon data plan to avoid excessive charges, and that is going to require changing their static IP addresses. I don't know exactly when this will happen, but the change was approved this morning. At worst, all the modems stop connecting until the DSM script restarts them, either in 15 minutes or with a reboot at 4 hours. More likely, the impact will be minimal. The modems should reconnect on their own, get assigned a new IP address, and the dynamic DNS update script will update the dyndns.org address for each DSM automatically. If you happen to have hardcoded the IP address somewhere, such as in a ssh config or browser bookmark, then it will need to be replaced with the DNS name, like cc-isfs.dyndns.org.
Nagios will tell us if any DSMs do not come back as expected.
This is a good time to remind everyone who connects to the DSMs to be careful about authentication. If ssh reports that a host key has changed, then it's entirely possible it's because the previous IP address for a DSM has been assigned to someone else's host, and it's not a good idea to start sending credentials to an unknown host. 
If you're ever not sure, one way to confirm a DSM's address is to first log in through the ssh tunnel, since only our DSMs are able to authenticate with eol-rt-data to create the tunnel port.
There was finally a slight change in the weather today to colder and partly cloudy with breezy winds. The clouds mostly stayed over the mountains to the west.
The morning data check showed that MW was off the network. At first I assumed it was the typical modem issue, but Nagios showed that it had been offline for 9 hours (since around midnight). A visit to the site revealed that the power had shut off and the batteries had drained over the last few days. We think it’s likely that the breaker has been tripped, however we could not find a tripped circuit in the exterior breaker box meaning that the breaker we need to access is probably in the garage. Liz coordinated with Dan to call a contact for the site, who revealed that the owner is aware of the issue and is in the process of getting it fixed. In the meantime we swapped batteries and will monitor their charge going forward.
In troubleshooting the MW site, I also noticed that NCharts was not showing any data after Jan 31. I contacted Gary and he was able to fix the problem quickly, which was an error in the NCharts configuration for the project end date.
In the afternoon I dropped Liz off at the airport, returning in time to help John with the afternoon sounding. Many thanks to Liz for all her help with ISFS, especially helping me get back up to speed with supporting an ISFS project. Thanks Liz!
The PI’s were discussing having an IOP tomorrow (Wednesday), but have now decided to consider an IOP starting on Thursday at noon.