Chenning and Antonio found that the wifi was very slow this morning. Tested the speed at around 100 Kbps. Called Merci and she mentioned that they tend to see our are affected regularly on Fridays. She said that a technician is looking into things. In the meantime, internet speeds are increasing to enough that we can access the wiki in a reasonable amount of time.
While checking the QC tables, Antonio noticed that the on t24, spd, w, and tc were each reporting 26 NA's int he past 6 hour period. This might suggest that the anemometer on tower 24 is down. Rick and Antonio pointed a browser at t23 and found that CSAT3_IRGA_BIN co2 and h20 were reporting large numbers of nan values. And CSAT3B dir, spd, tc, u, v, w, were all reporting 0 valid and lots of invalid samples. Looking at t26 and t20 showed that CSAT3A_BIN had all data were sampling slow at 20Hz, where 60 Hz is expected. Curious...Rick and Antonio ride to face the problem.
When we got to the tower, we couldn't find the t23 subnet. After some poking around, we found that the DSM lost power. Voltage on the batteries read 11 V. Victron is probably bad. We are not hosting a Black widow in that particular cooler so we will swap out the batteries and victron after lunch.
Took ISFSVIC2 from the charging station. Replaced batteries and victron. DSM came back live. All the sub-systems are sampling data. However, we saw that the gas analyzer (CSAT3_IRGA_BIN) is periodically recording a continuous set of NANs... about 1/4 of the time. Power cycling doesn't resolve, and re-seating the controller cable doesn't resolve. Chris Roden assures us that the gas analyzers do this form time to time, especially when it gets windy. So the problem is resolved. For now.
Chris onsite today with Rick taking the day off.
Weather was mostly sunny in the morning with gradually building clouds. In the afternoon we were mostly cloudy, with rain late in the day.
Stephan and Nathan's students were again flying UAVs on the west end of the array ( same location and stations as yesterday).
Equipment generally looked good. Stephan took some time to look at his lidar, and noticed an issue with the vertical profiles. It looked like horizontal profiles were good, but vertical ones were misbehaving. He was discussing this with Bill and was doing some trouble shooting.
For some reason the sounding computer rebooted this afternoon, and then initially had an issue connecting to the Vaisala sounding equipment. With Josh's help we got it sorted out prior to the 3:00 sounding.
The Purdue students again performed both balloon releases for the soundings.
The below photo credit goes to Stephan De Wekker:
Rick and Chris onsite today.
Weather today was mostly sunny with some low puffy clouds, and high in the mid 70s
Cleaned the Solar panels and NR01s today. Noticed the one of the solar panels ( t20 ) had some weird burn marks in the silicon. Have not noticed this impacting the charging of the batteries, but something to keep and eye on and possibly tests when back in the Boulder.
Based on the pictures from yesterday's blog, Steve and I had a discussion about moving the drone testing to minimize impact caused by changes in the downwind roughness produced by the UAV activities. Chenning and I had a discussion with Stephan and Nathan about the location of the students, vehicles and tents. Today the UAVs were moved between t0 and t8, and the students were seated between the solar panel and the road, and the vehicles and tents were moved to about 150' west of t0.
Drone pictures for reference
small drone:
Large drone:
We got Nathan's students involved with balloon launching today and Stephan gave a short lecture on sondes.
Rick and Chris on-site today. Weather was partly cloudy in the morning, and mostly cloudy in the late afternoon with some showers in the area.
After the balloon launch, Rick and I went to the trailer tower to fix the 28m TRH (Old SSN was 011). We double checked the fuse with was OK. We dropped and lowered the tower. The fan for the 28m TRH was still running, but no data as coming out of the sensor. I showed RIck how the TRH sensors are replaced. The first two new TRHs that we tried were bad (one the sensor had fallen out of the housing), and we finally found a good one (SN 052). We verified the data, and then raised the tower back to its fully extended position.
Stephan De Wekker and Nathan ?? (From Purdue) were here today with a gang of bout 15 undergraduate UAV pilots.
They were flying drones between and about 5 meters behind the towers.
They were flying 8 drones between the following towers:
49 & 48,
48 & 47,
47 & 46,
46 & 45,
45 & 44,
44 & 43,
43 & 42,
42 & 41
They will be flying drones for this entire week (Sept. 11- 15) , and will leave on Friday or Saturday.
Weather in the afternoon:
Only Rick onsite today (thanks for taking over), Chris took the day off.
I did notice Saturday night that the Trailer Tower battery voltage was getting low (12.3). This may just have been due to the clouds days, but I asked Rick to swap the batteries today. We will look over everything on the power system tomorrow (Monday).
Rick noticed that the 27m TRH had failed; he cycled power on the DSM, and tried to check the fuses, but found nothing definitive. We will investigate more tomorrow, and swap out the TRH if needed.
I had a nice day hiking up to the highest point in Nevada.
Any guesses on the elevation of Nevada's highest point??
Its Boundary Peak at 13,140' (or 4005 meters)
Rick and Chris onsite today.
Weather today was cloudy with sprinkles in the morning, changing to partly cloudy in the afternoon.
I consistently monitor the voltages on the QC tables, to make sure that the battery voltages increase as the sun comes up:
Today, T23 battery voltage continued to decrease. I went to the array, and checked the solar panels and solar output - both were good.
On inspecting the Victron, the battery cable came loose from the Victron screw terminal. I reconnected this wire, replaced both batteries with fresh batteries, and restarted the DSM.
Then put the old batteries from T23 (which had discharged to ~12.5 volts) on the charger.
We have had several issues with wires coming loose from the Victron screw terminals - we should think about ways to improve this connection.
Rick and I occasionally still prefer the old school method of releasing balloons:
Site Visitor:
Tomorrow (Sunday Sept 10) - I plan on taking a rest day.
Rick and Chris onsite today.
Weather today was Sunny with the only a few clouds on the horizon. In the morning there were light winds from the south. In the afternoon, winds increased slightly, but not as strong as yesterday.
All looked good on the desert front and equipment is behaving.
We did drive out to the array, and checked the updated ratchet strap on T0 new to make sure there was no settling. The T0 tower was still plumb and level. We went to the trailer tower, and I gave Rick a tutorial on the it. We started the generator to make sure all was OK with it.
We cam back to the site at sunset to take some sunset array pictures.
Then we decided - "When in Tonopah - do as the Tonopahins do", so we went to the Friday night dirt track races. Chenning, Rick and I had a fun filled evening eating dust and covering our ears - it was loud, but very entertaining:
Rick, Josh and Chris onsite today. Josh left us this evening, and I dropped him off at the Tonopah Greyhound bus terminal (Chevron station). The "bus" (large vanish thing) was about 30 mins late arriving, but eventually Josh made his way south to Las Vegas.
Weather today was Sunny with the only a few clouds on the horizon. In the morning there were light winds from the south. In the afternoon, winds increased to about 17 mph; they were forecast to die back down in the evening, so the hot-films were left in place. Gary has been working on an increased sample frequency on those, so we will see how this goes.
The rest of the day was spent with balloon launches, Josh relaying all of his responsibilities to both Rick and myself, a reboot of ISS1 data server, and checking out all of the ISS equipment.
I am testing the hotfilm recording at 4KHz instead of 2, and because the network throughput sometimes is too slow to prevent buffer overflows on the labjack, I've moved the hotfilm process back to t0t. So it is not a problem if hfd on ustar shows no data or if the hotfilm service is not running. Please do not try to start the service on ustar. When the hotfilm process is running on t0t, you can use the hfd alias on t0t to look at the current hotfilm voltages. While I'm testing, the hotfilm process may not always be running, and it could run in alternate locations.
Josh and Chris onsite today. Rick joined us this afternoon, and receive a refresher course in ballooning.
Weather was Sunny with the only a few clouds on the horizon with light winds from the south in the morning, and a bit stronger winds in the afternoon.
- All data from the ISFS towers looked good
- We did have one voltage reading issue on T17 caused by the USB cable disconnecting from the Victon (when we moved it the previous day). We reconnected the USB, and start seeing voltage data.
- Updated the strapping on T0new to lower the ratchet and move some of the bulk further from the tower and sonics.
We first added the new strap, and tensioned it. Then removed the old strap (with the higher ratchet)
Both straps paralleled:
Finished strap:
- Charged all of the Dewalt batteries to get them ready for tear-down (also best for the batteries to leave in a charged state)
- checked both baths for the DTS fiber, and cleaned up some old flags and string
Otherwise not much going on except some balloon releases.
Josh and Chris onsite today. Isabel left us this morning for the drive back to Las Vegas and flight back to Denver, although she was still providing remote tech support.
Sunny with the only clouds on to see on the horizon with light winds from the southeast.
- All data from the ISFS towers looked good
- Josh gave Chris a refresher on launching balloons, since Josh will be leaving Thursday evening.
- We opened all of the coolers and tried to move Vicrons on top of the batteries in the coolers that were not aggressively guarded by Spiders. There were three coolers that we decided the Vicrons looked great right where they were (because the resident Black Widow retreated into the Victron)
- We checked the tension on all of the straps. There were a few that were loose enough (slack and slightly drooping) to require a few additional clicks.
Did NOT make further adjustments to adjust for level or tighten straps - only adjusted about 4 straps that actually had slack / droop. I wanted to avoid / minimize any movement. - One set of straps had been rubbing enough to abrade about 1/4 into each strap. These straps were taped together at the junction with some electrical tape .
- Chenning and I inspected the new T0 and tightened one strap (SW strap) which was drooping. We also checked level on this tower and adjusted to bring it to level.
We will revisit it tomorrow to check if we can improve the strapping on the SW side to move the ratchet part lower to the ground. We plan on doing this be setting up a parallel strap and then removing the original one. - I pulled out all of the crates that had been under the trailer tower and poured out any standing water, and allowed the cases to dry out.
When dry, I pushed these further under the trailer so that run-off from the sides of the trailer would not flow on to the top of the crates.
- Lastly, we were having some internet trouble today, with intermittent slow downs (between 13:00 and 15:00). Nagios was showing connectivity alternately jumping between green, yellow, red and back again. Speed tests where showing between 50Kbps, and over 1Mbps. I called Mimo, but did not receive a definitive answer into the cause or resolution. We will continue to monitor this.
Teardown crew will need to be very careful - wearing gloves, and paying close attention to all boxes and coolers. Over half of the coolers have black widows in them, and I also found a little scorpion friend under one of the boxes:
Josh and I onsite today, joined by Chris who got in mid-afternoon.
Sunny with a few scattered clouds, brilliantly clear after yesterday's rain. Winds very light, much calmer than last couple days.
Staying busy today to earn our holiday pay. After morning data check (see previous post), headed out to the array:
- put hot film probes back at t0, done at approximately 11:15. Probe holders were taped over and seem all dry inside. All four hotfilms now reporting, though the 0.5m one is reporting a little higher voltage than the others (~2.1 vs ~1.8). I don't know if this is a probe problem or if this is the actual conditions, and I don't know if there's anything we can do about it with no spares.
- investigated why t23 is offline. When we opened the cooler to swap out batteries, we found that one of the wires from the victron to the batteries had come completely out of its screw terminal, so presumably the batteries weren't charging, which is why the station was offline. Re-attached the wire, swapped in charged batteries, and t23 is back online.
- finally got around to swapping the EC150 head at 28m on the trailer tower. The EC150 head we swapped in is SN 1385. Reporting fine after the switch, and so far now the 28m co2 looks like it's more similar to the other t0 co2 measurements, but a little early to know for sure. Tower lowering and raising went smoothly, no trouble from the cables. When we arrived the NE guy wire seemed very loose, and the end of the guy wire that was threaded through the turnbuckle had come out. So after we finished raising and re-tightening each guy wire by 10 turns, I tightened that one some more to get it closer to the other tensions. Tower looks plumb as far as we can tell.
- cleaned off all the solar panels, which were pretty dirty after yesterday's rain.
Looking for a spare victron (that we did not end up needing) at base this morning, we found that the spare sensors bin under the base trailer had filled up with water due to a crack in the lid. We put the motes and soil sensors and cables out to dry, and the NR01s in their special cases seem unscathed. The spare victron box next to it has a couple black widows in it. Next time we should probably keep our bins of spares inside the trailer...
Gave Chris the rundown at site this afternoon, and now Josh knows everything about ISFS so he can fill in the rest. I'm headed back to Boulder tomorrow morning.
Looks like it did eventually rain at the site yesterday evening. Checking the data this morning found 2 problems:
- t23 is offline (all of it, ubiquiti too)
- CSAT3A/IRGA data at 7m on tt is not showing up in qctables
Logging in to ttb, no data at all from port 0 (the 7m sonic). Checking with dsm_port_config shows that the port was set for RS232, which it shouldn't be. Doing a ddn/dup sets port 0 back to RS485_FULL, and now we're getting data again.
Looking in files on ttb it looks like the last data from the 7m sonic before I restarted nidas was 9/4 00:44 UTC, or around 5:45 last night. At that time I see a chunk of messages about the power monitor in dsm.log, but nothing about DSM0, so I'm not sure what happened to set the port to the wrong mode.
Sep 4 00:44:42 localhost level,message[1299]: ERROR|ttb:/dev/ttyPWRMONV: Exception: EOFException: /dev/ttyPWRMONV: read: EOF Sep 4 00:44:42 localhost level,message[1299]: NOTICE|closing: /dev/ttyPWRMONV, #timeouts=0 Sep 4 00:44:42 localhost level,message[1299]: NOTICE|Sensor state for ttb:/dev/ttyPWRMONV:DSMSerialSensor is now SENSOR_CLOSED Sep 4 00:44:42 localhost level,message[1299]: INFO|8 active sensors Sep 4 00:44:52 localhost level,message[1299]: NOTICE|opening: /dev/ttyPWRMONV Sep 4 00:44:52 localhost level,message[1299]: INFO|UnixIODevice::open : entry Sep 4 00:44:52 localhost level,message[1299]: INFO|UnixIODevice::open : exit Sep 4 00:44:53 localhost level,message[1299]: NOTICE|Autoconfig is not enabled or is not supported. Sep 4 00:44:53 localhost level,message[1299]: NOTICE|But we must still set the termios, xcvr config, turn it on, etc. Sep 4 00:44:53 localhost level,message[1299]: NOTICE|ttb:/dev/ttyPWRMONV:DSMSerialSensor: Putting sensor into measurement mode Sep 4 00:44:53 localhost level,message[1299]: NOTICE|Sensor state for ttb:/dev/ttyPWRMONV:DSMSerialSensor is now SENSOR_OPEN Sep 4 00:44:53 localhost level,message[1299]: NOTICE|Sensor state for ttb:/dev/ttyPWRMONV:DSMSerialSensor is now SENSOR_ACTIVE Sep 4 00:44:53 localhost level,message[1299]: INFO|9 active sensors
Vbatt data from t23 shows voltage dropping off yesterday afternoon before it went offline, so suspect either dead batteries or maybe water in the cooler interfering with the victron.
Off to the array we go!
Josh and I onsite.
Cool and windy today, mostly cloudy in the morning but clearing up a bit at the site in the afternoon. Thunderstorms and pea sized hail in town this evening, but I don't think the rain made it to the site.
Went out to the array after the morning sounding. We replaced the batteries at tt with fresh ones, since the voltage was getting pretty low after a couple cloudy days. We brought the EC150 head out to replace at 28m, but with wind gusts and storms in the area we put that off for another day. I connected the callab laptop to the CSAT3B at t48 to check on shadow correction settings. Looks like there is an option to enable or disable shadow correction from the instrument, which on this one is currently set to disabled. We swept off all the solar panels (a little dust accumulated, but not too bad), and I took one more (unsuccessful) stab at getting the PurpleAir to work.
Storm building over town, just as we left site this afternoon.
Josh and I onsite.
Cool and mostly cloudy today, still with strong winds from the south. No real rainfall, but you can feel the moisture in the air.
Out to the array midmorning to try plugging the t5 dsm power supply into the serial card rather than the power panel, at Gary's suggestion. Turns out this doesn't work. While out at the array also did some more looking at the PurpleAir sensor, but no luck getting it to connect to t44.
In the trailer, looked at the two csat3s that Josh and Jacquie replaced last week. When I plugged them into the spare dsm, they are both still giving nans for everything except diagbits, which is 31 (and ldiag, which is 1). So I guess letting them dry out a bit wasn't enough to bring them back.
This afternoon we had some guests on site! Last night we had dinner with Wes and Paula (the drone operators from OU) and a bunch of people they knew, who are in the area because they're working as biologists at Nellis Air Force base. Today they came out for a tour of the site and to help with the 3pm balloon launch, and to tell us all about the lizards and small mammals in the area.
