...and then there were two.
Justin and I got out to the site early to prep the base trailer and clean up. Disconnected power cable from base trailer and stowed securely on reel. Locked down all panniers. Removed and disassembled patio/stairs. Closed shutters. Disassembled and stowed back stairs.
Sent off tower trailer at noon as the driver was in town already. Scheduled base trailer pick up for tomorrow morning, as driver says they are arriving early as well. Containers and ISS trailer pickup is still on for Monday morning. Driver numbers for reference:
ISFS Tower Trailer: Rick Tanner Truck# 3594 at 909- 519-0768
ISFS Base Trailer: Mike Damschroder Truck # 3846, his cell is 419 -202-3472
ISS Trailer: Colt Perry Truck# 3800, his cell is 912 -293-0242
ISFS & ISS Containers: Nick, his cell is 208-206-1819
Packin' it up...
Huge day. ISS helped us pickup all the solar panels with their empty Uhaul. David, Bill, Justin and myself disassembled the TT and towed it to a safe pickup location near the airport FBO. Mark Peterson approved of the temp location for the TT. Clayton helped us with running equipment back to base and packing the container. Sam disconnected power to the base trailer and I recorded the kWhr usage. Network equipment is now removed from trailer. All of our equipment is now removed from the field. Spoke with trailer transit and container transit and we are on schedule for monday pickups. Justin and I plan to do final wrap up tomorrow.
All rohn towers and TT down. All sensors and tower infrastructure moved to base. Working hard to get the big stuff done before storms and plummeting temps Saturday.
Another perfect day in the desert. All towers in the main line down. T0 and TT are planned for tomorrow. Also hours and hours of carting equipment back to base. Halle crushed it on her first field day. We are all in general agreement of her awesomeness.
The sun comes up...the towers go down.
More organizing, always the organizing. David O. helped us take down 9 towers from the line in the morning. We lunched. Clayton, Justin and I took down 9 more in the afternoon. 17 left in the line. Tomorrow we get Halle.
I see that my first post was entered as a "page". Should have been a blog post. Here's what I wrote:
9/24
Day one. The organizing.
Gave JHA and overview of teardown project to Terry, Clayton, and Justin. Removed everything from the "sea-tainer". Staged all pelican and CSAT3 cases in the field. At first glance, we have more CSAT3 cases that CSAT3s on towers(?). I feel the answer will reveal itself in time. Took down three towers (started at opposite end). With four people there are no issues with a controlled walk down.
Shane mentioned today's conditions were perfect for data collection. We will continue to organize tomorrow and start bringing down more of the array.
9/25
Day two. "Commence the lowerings."
Gave JHA presentation to Allison, Bill, Josh, Antonio and David. Bill gave Allison a thrilling tour of the base and ISS setup. Then we set out to the array for teardown ops already in progress by Justin and Clayton. Gave Allison some background of my experience thus far as a EOL tech. Also, of our general operations as the surface team (as Tony understands it...differing to Bill for science explanations). We have both radiometer stations, Chenning's line, the chem shelter, and 15 towers down. Looking good.
Staff: Antonio and Rick. Chenning and Bruce present most of the day. Josh arrived in Las Vegas 3 hours ahead of his luggage, and got out to Tonopah late afternoon.
Very quiet day on the ISFS front. Plots coming in, QCharts look good. Nagios complaining about disk space on t0t as usual - one more day to hang on. Batteries at tt look good after yesterday's swap-out and repositioning of the solar panel.
A few glitches on the ISF side: the sounding system module (SPS?) decided to have a temper-tantrum and refused to initialize a sonde. After a stint in time-out (power cycle), it behaved properly. Likewise the laptop running the remote desktops into the lidars decided to log itself out, for mysterious reasons, so we restarted those processes.
Antonio and Rick on the morning and afternoon soundings.
Antonio and Rick go out to the towers and apply peppermint oil to the battery coolers in an attempt to convince the unwanted guests out there that they really are unwanted. We'll see if they get the message.
Rick's last day on site: he reflects misty-eyed about the experience and almost wishes he could be around for tear-down. That passes in about 785 milliseconds.
We noticed that the QC charts were publishing a lot of NAN's on the trailer tower last night. When we got in this morning however, we saw that plenty of data was coming in through the DSMs. It looks like they are operating like normal now. So we are thinking that maybe the tower batteries are on the fritz and the tower ost power overnight in the absence of solar power. Rick and I will ride out there to check after the morning sounding.
We replaced the batteries at the trailer tower with charged batteries. The trailer tower DSMs were reporting no NANs after the sun came up. Chenning also showed us that the Victrons report a history of the power supplied by the solar panels. we found that the panels at the trailer tower were supplying a relatively low amount of power the past few days. Today, the panels have supplied more power than in previous days. We wonder if there was a malfunction in the Victron that was solved when we power cycled it this morning? We don't know. We'll keep an eye on it. Everything seems in order for now.
Staff: Antonio and Rick. Chenning and Jielun in the morning. Jielun departs mid day.
Unbelievably quiet day today. No Nagios alerts (Gary had taken care of the disk space issues on t0t and ustar), plots updating according to plan, equipment performing as intended.
Sunny, few clouds, temps in the 70's, calm in the morning, sporadic gusts in the afternoon.
Antonio and Rick on the morning and afternoon soundings.
Estimated population density per square mile around the site: 6.
Rick holds down the fort in the morning -- Antonio, Chenning, and Jielun make science outreach presentations to three classrooms at the Tonopah middle school.
Rick and Bruce (lastname?) handle the morning sounding:
Nagios still complaining about t0t usbstick and ustar disk space utilization being near capacity. Not eminently critical, but will be in another couple of days. I want to talk with Gary about how to deal with this.
QCcharts look OK. Plots look OK and updating. The W'*W' plots for the past 3-4 days have been showing an anomaly: For tower 33 (+/- 1), the reported values seem to be random in the midnight-dawn hours, but somewhere around dawn, become more coherent with the rest of the array until the next midnight.
Antonio, Chenning, Jielun arrive at the site around noon. By all accounts, their presentations to the students were well received and successful.
Antonio and Rick on the PM sounding. Antonio squashes rumors that circulated at last night's dinner about a shortage of balloons, by locating several boxes of balloons in the container.
Overall, a pretty quiet day in paradise.
The Sabbath has ended.
At around 11am, chenning noticed that the QC plots had not updated since 9:00am. We confirmed that data was still coming into the local drives int he DSMs, and then we confirmed that data was coming into ustar. So we thought that there may be an outage or something at the NCAR servers. We waited a little while and then the QC plots for the past few hours all updated at once. The QC plots are updated properly now.
The disk space on t0t was about 97% filled today. There was around 2 Gb of space left, and the Nagios reported that t0t populated a average of around 5Gb a day (between 4 and 6 depending). Gary said in a note that he sent to Chris, that Rick subsequently forwarded to me that we could delete bulky hotfilm data files of the t0t drive so long as we made sure they were backed up on the ustar drive.
t0t had bulky hotfilm data files from 9/12 through today (9/17). We made sure they were backed up on ustar, then deleted all the hotfilm data form 09/12 and 09/13. This freed about 12 Gb of space on the t0t drive, which we think is plenty to last until Gary's glorious return to our working ranks, and can perform his dark magic rituals to solve the problem, or he may just delete more redundant files.
Came in. Everything worked great. Nothing broke. Went home.
The malfunctions in the ISFS system seem to respect the Sabbath, and so too did our blog posting. Shomer Shabbos, brothers. Peace love and unity 
Onsite today was Rick, Antonio, and Chris.
Weather was mostly sunny in the morning with gradually building clouds; high temp in the mid 70s.
Chris went over all ISFS operations with Antonio and Rick; Rick was planning on taking over the ISFS duties, and Antonio taking care of ISS ops.
The 2m hot firm was acting up; it was reporting ~ 7 volts (instead of being between 1.5 and 2.5 volts). We removed the 2m hot film and it measure ~5 Ohms (the correct value). We checked the resistance at the coax cable leading into the Dantech bridge , and it was reading a short (0 Ohms). We took apart the coupler between the thin and thick coax cables (out on the boom), and then reattached everything. This seemed to fix the problem. We then taped this coupler up (as it had been at the other levels); it probably got some water in the connector from the previous evening's rain. All for hot film are now reporting correctly. I did talk to Steve, and the one hot wire (not hot film) in the base trailer, would be a suitable potential replacement for a hot film if another one dies.
The Purdue students moved their UAV operations from the array to behind the trailer tower, and were flying the UAVs at the three different levels.
Tomorrow (Friday) will be their last day of operations.
Chris leaves tomorrow (Friday) morning.
Chenning and Antonio found that the wifi was very slow this morning. Tested the speed at around 100 Kbps. Called Merci and she mentioned that they tend to see our are affected regularly on Fridays. She said that a technician is looking into things. In the meantime, internet speeds are increasing to enough that we can access the wiki in a reasonable amount of time.
While checking the QC tables, Antonio noticed that the on t24, spd, w, and tc were each reporting 26 NA's int he past 6 hour period. This might suggest that the anemometer on tower 24 is down. Rick and Antonio pointed a browser at t23 and found that CSAT3_IRGA_BIN co2 and h20 were reporting large numbers of nan values. And CSAT3B dir, spd, tc, u, v, w, were all reporting 0 valid and lots of invalid samples. Looking at t26 and t20 showed that CSAT3A_BIN had all data were sampling slow at 20Hz, where 60 Hz is expected. Curious...Rick and Antonio ride to face the problem.
When we got to the tower, we couldn't find the t23 subnet. After some poking around, we found that the DSM lost power. Voltage on the batteries read 11 V. Victron is probably bad. We are not hosting a Black widow in that particular cooler so we will swap out the batteries and victron after lunch.
Took ISFSVIC2 from the charging station. Replaced batteries and victron. DSM came back live. All the sub-systems are sampling data. However, we saw that the gas analyzer (CSAT3_IRGA_BIN) is periodically recording a continuous set of NANs... about 1/4 of the time. Power cycling doesn't resolve, and re-seating the controller cable doesn't resolve. Chris Roden assures us that the gas analyzers do this form time to time, especially when it gets windy. So the problem is resolved. For now.
Chris onsite today with Rick taking the day off.
Weather was mostly sunny in the morning with gradually building clouds. In the afternoon we were mostly cloudy, with rain late in the day.
Stephan and Nathan's students were again flying UAVs on the west end of the array ( same location and stations as yesterday).
Equipment generally looked good. Stephan took some time to look at his lidar, and noticed an issue with the vertical profiles. It looked like horizontal profiles were good, but vertical ones were misbehaving. He was discussing this with Bill and was doing some trouble shooting.
For some reason the sounding computer rebooted this afternoon, and then initially had an issue connecting to the Vaisala sounding equipment. With Josh's help we got it sorted out prior to the 3:00 sounding.
The Purdue students again performed both balloon releases for the soundings.
The below photo credit goes to Stephan De Wekker:
