4 good soundings today at 00, 06, 12, 18 UTC. Still intermittently having issue with sondes not being able to get a GPS lock. At the 06 sounding we had no luck with the first sonde and launched the second one, then it looks like at the 12 sounding Kate successfully launched the one we had been having problems with in the morning and then launched another sonde at the same time. Had no problems with the 00 and 18 soundings.
Inventory at sounding site: 32 balloons, 36 sondes, plenty of parachutes. No more info stickers from the Portuguese, will ask around at ops center tomorrow.
Inventory in container near ops center: 50 balloons, 40 sondes.
Weather hot and mostly clear, wind from the W/NW.
4 good soundings today at 00, 06, 12, and 18 UTC. I had an issue with the 06 sounding that the GPS data from the sonde didn't show up, but when I selected bad sonde and then started again in Gauss with the same sonde it worked fine. The same thing happened to Kate at the 12 UTC sounding, so she used a second sonde. Her first (unused) sonde worked for me so I used it for the 18 UTC sounding. I couldn't recondition the T sensor since the battery was already attached but everything else was fine.
Tomorrow Kate is going to inventory the rest of the sondes in the ISS container.
Weather today and tomorrow is warming up with increasing winds from W/NW.
4 good soundings today at 00, 06, 12, and 18 UTC. Ascent rates are still a little low with 43-43.5 ft^3 of helium.
Weather was cooler and somewhat cloudy, calm.
John left for Lisbon after the noon sounding. (And I forgot to do the daily update until now, oops.)
Sounding site - we3 have settled on 43 cu. ft. of He in the balloons for proper ascent rates. OU conducted super sounding from 1500 UTC on the 27th to 0300 UTC on the 28th, reporting no surface data available at the 2100 UTC sounding. Issue fixed by Gary by next sounding at 00 UTC.
Profiler site - Temperature data collected from the Hobo logger in the profiler shed shows max temperature of 85 F over the last two days. Shed checked today, air conditioner running, comfortable temperature in the shed.
Sodar site - Temperature set point on IceCube air conditioner unit increased to try and get more load sharing with the other AC unit. No issues with the electronics in the box.
John leaving for Lisbon tomorrow after 1200 UTC sounding.
The surface data at iss2 sounding site stopped around 16z. The culprit turns out to have been a bad system time on the DSM, causing samples to get dropped as too different from absolute time on the data manager. I don't know what caused the DSM system time to jump so far off, but I've adjusted the chrony configuration to correct even large offsets. Surface data have been restored, and if this happens again, one thing to try is to restart chrony on the dsm. See ISS-562 for details.
SODAR site went offline around 3z. The 20170526 SODAR data was not transferred by then, so the SODAR data look very far behind. I visited the site and cycled power on the TP-Link, that restored the Internet connection, and then I needed to update the allowed hosts on eol-rt-data to allow the tunnel to connect. Connection restored at 2017-05-27,14:36.
All ISS sites operational. The new He adapter was received and the regulators switched out. Balloon filling now takes less than 3 min. vs. 15-16 min. with other regulator, much to the relief of sounding operators. Started with 30 cu. ft. and increasing until desired ascent rates reached. Looks like value will be between 40-45 cu. ft. Moved the temperature logger from the sounding trailer to the wind profiler shed to monitor/log temperatures there. Gary has some concern about the high values seen in the recorded CPU temperatures and would like to correlate them with temperatures inside the shed. Weather has cooled some so not expecting any thermal issues in the next few days. Mentioned at daily meeting that the 4 lidars at the upper orange site are all on the same circuit breaker and to consider moving loads to multiple breakers when convenient. No input received.
Isabel and Kate up to speed on soundings with Gary learning.
For some reason, the iss2 DSM lost the ability to keep it's IP address between boots. I discovered many days ago a situation where the DSM came up using DHCP instead of 192.168.0.100, so I ran set_ip again to set the static address. The interface was configured correctly and the DSM was running fine. When the power was cycled on the DSM a few days ago in an attempt to fix missing surface data, it came back up with DHCP again and got address 192.168.0.132. The realtime network stream to the data manager still works, but the DSM looks down because it is not at the expected address and rsync does not work. In the end, I had to fix the DSM by reverting to an older interface configuration, but now it boots with the right address. (The details will be in JIRA, since the underlying problem is not fixed.)
I also ran apt-get update and apt upgrade on the DSM, in case that would fix the problem.
While investigating that problem, I discovered an unidentified host at 192.168.0.120. It turns out the GAUS PC IPMI IPV4 interface was enabled in firmware (ctl-E during bios boot), and it was set to 192.168.0.120, causing it to respond to pings even when off. So I've disabled that, and hopefully it was enabled for a reason. I'm a little concerned as to how it got enabled in the first place.
I fixed the expected DSM sample ids on iss2, so nagios checks are all green now and should stay green. Surface ingest and GAUS broadcast messages are working again.
I also noticed the 20170526_06z launch was mislabelled as 20170527_06z, so that will have to be sorted out later.
Day 24 of the Perdigao IOP. Today was another hot day, with mostly clear skies and calm to light winds. A few thin, high clouds were visible through much of the day, with isolated towering Cu forming after 1500. Thunderstorms formed late in the day well to the S of Perdigao, and slowly moved north towards the area. A weakly organized MCS finally reached the area around 0000, with frequent lightning. The storm persisted in the area until at least 0100. Due to the lightning, the 00 UTC 25 May 2017 sonde was scrubbed.
As a result, only three sondes were launched today, at 06 UTC, 12 UTC & 18 UTC, with no issues.
Day 23 of the Perdigao IOP. Today was hot with clear skies and light winds. The 18 UTC sonde measured 45 C in the trailer prior to launch.
Four regular sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no issues.
John reset the WV DIAL and realigned the Ubiquiti antenna, which resolved the issue with the instrument not reporting since the power loss the night before.
Day 22 of the Perdigao IOP. Today was very warm with few Cu during the afternoon and mostly clear skies in the morning and evening. Surface winds were calm to very weak.
Four regular sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no major issues. The 12 UTC sonde had trouble acquiring GPS and also lost winds between 790-640 mb. Three additional radiosondes were launched at 15, 21, and 03 UTC as part of the OU supersonde schedule. There were no issues with these either.
Notre Dame re-installed their scanning lidar in the afternoon. At some point in the evening the breaker tripped in the power box at the road and all of the lidars lost power. This potentially happened around 20 UTC. No effect was noticed in the ISS trailer. I reset the breaker in the morning of 23 May and the lidars powered back up.
Day 21 of the Perdigao IOP. Today was warm with mostly clear skies through mid-afternoon, then mostly clear skies with haze (perhaps dust from N Africa, as was forecast). Surface winds were mostly weak and southerly.
Four sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no issues. The potential supersonde schedule was scrubbed for today, but is being discussed for tomorrow.
Day 20 of the Perdigao IOP. Today was another pleasant day, with mostly clear skies and warm temperatures. Surface winds were weak, while winds aloft were veering, from the NNW to SSW.
Four sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no issues. Supersonde schedule is being discussed for Sunday or Monday, with decision to be made after weather briefing on Sunday.
Day 19 of the Perdigao IOP. Today was another pleasant day, with clear skies and fairly cool temperatures. Winds were weaker today than yesterday. There was a small wildfire N of sounding site this morning, which created a smoke plume through the valley during the 12 UTC launch. The fire was extinguished by mid afternoon.
Four sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no issues.
Day 18 of the Perdigao IOP. The weather was extremely pleasant today, with clear skies and cool temperatures. Winds were a bit gusty out of the north through the afternoon, but calmed by late evening.
Four sondes were launched today, at 00 UTC, 06 UTC, 12 UTC & 18 UTC, with no issues.