Another hot, clear day in the desert. Josh and I onsite. Chenning and Jielun mostly around the base today, as the drone is not flying.
Data check shows everything looking fine.
Out to the array today between about 1:45 and 2:45. Went to the trailer tower to lower it, but by the time we found where the keys were (operator error) and realized we would have to go get the gas can for the generator, we didn't have time to do that today. We're planning to head out again on Friday now that we know what we're doing and will be prepared for it. While out there, we cleaned the NR01s (t47 one at 2:26pm local, t0 one at 2:48pm local). We also tried swapping the original PurpleAir sensor from the array back in, since at base it was working just fine connected to a spare dsm's wifi, but out on the array it just doesn't work. So, still using the spare we swapped in a couple days ago.
Besides that I spent some time getting SOS data off the Leica for Steve and finishing off the presentation I'm going to give as part of Jielun's M2HATS event in town this evening. We'll see how big of a crowd we get for that...
Happy birthday Josh! Another sunny, cloudless day. Trailer continues to be mouse-free and outhouse continues to be spider-free (though there are plenty of them elsewhere).
All looking good in the data, except that when I checked the hotfilms after the 3pm sounding the 0.5m one (which was having intermittent problems yesterday) was now reading about 0.3.
Out at the array checked the resistance of the 0.5m hotfilm, and got values around 1.1 kΩ (compared to the 1m hotfilm, which was around 7Ω as expected). We tried swapping in the extra bridge to hotfilm mount cable, no change. We measured resistance again at the hotfilm mount, without the cable to the bridge, and got the same value. We removed the probe, didn't see anything obviously wrong with it, tried reseating. While the probe was out I tried to blow on the probe mount in case there was water in it, but didn't notice any. Switching in spare probe SF9 returned resistance values to around 7Ω as expected, and the probe is now measuring about 2 volts, same as the other hotfilms. Once we verified that fixed the problem, we changed the dip switches in the bridge for the new probe at 0.5m. Data after around 5:10pm local should have the correct switch settings.
So I think now we're out of spare hotfilm probes, unless we could make the one that's wire and not film work, but Steve says we don't have the correct bridge settings for that one.
Jielun has also been wondering about whether the guy strap near the 2m hotfilm is the cause of the weird data she's seeing, especially since the ratchet itself is at about 2m. We're talking about redoing the guy wires on that tower entirely, but for now I've tried taping over the ratchet (to smooth its shape) and the extra strap (to prevent it flapping).
Josh and I onsite. Another clear, warm day with light winds. Not enough aerosols for the lidars!
All trucking along well data-wise, but don't worry, Steve is busy thinking of things for us to do. Looked into the low tc at t29 (see blog post) but so far no resolution. Saw Gary's post about the 0.5m hot film, but by the time I was getting ready to go out and switch the probe, it's back to working again. Steve says the short might also be in the cable between bridge and probe (we could try a spare) or in the probe holder (we don't have spares). I've been mostly playing with the PurpleAir we removed from the array yesterday and working on a presentation for the M2HATS open house on Wednesday.
Consistently reading a degree or two lower than other sonics:
I don't see any noticeable differences in u/v/w between it and nearby sonics.
t29 is a CSAT3A + EC150 combo.
Tried power cycling, doesn't seem to make a difference.
Channel 0 (0.5m) started hitting 7 Volts on 2023-08-25 at 9:42 UTC, or 2:42 last Friday morning local time. It was intermittent at first, but as of this morning it is continuous. Presumably a short has developed somewhere, likely in the probe. If there is a spare probe, then I think that's the first thing to try swapping out.
The first sign of a short in channel 0 seems to happen here:
(base) [daq@ustar ~]$ data_dump -i 200,510-513 -x hotfilm.xml /data/isfs/projects/M2HATS/raw_data/hotfilm_20230825_* |& egrep ' 7\.' 2023 08 25 09:42:43.1870 1 200, 510 12 2.1569 2.0216 7.2967 2023 08 25 09:46:02.1870 0 200, 512 12 1.8874 1.8348 7.2177 2023 08 25 10:10:30.1875 1 200, 510 12 2.007 1.8459 7.2689 2023 08 25 10:14:52.1875 0 200, 512 12 1.9778 1.7318 7.2174 2023 08 25 11:03:13.1880 1 200, 510 12 1.962 1.8282 7.2673 2023 08 25 22:21:08.2910 1 200, 510 12 2.3411 0.68967 7.2026 2023 08 25 22:21:31.2910 1 200, 510 12 2.3765 0.73897 7.2588 2023 08 25 22:21:48.2910 1 200, 510 12 2.4567 0.85462 7.1738 2023 08 25 22:21:55.2915 1 200, 510 12 2.3388 0.76678 7.2604 2023 08 25 22:22:04.2915 1 200, 510 12 2.5594 0.80248 7.091 2023 08 25 22:22:05.2915 1 200, 510 12 2.5424 0.7614 7.1185 2023 08 25 22:23:36.2915 1 200, 510 12 2.5616 0.74718 7.0755 2023 08 25 22:23:37.2915 1 200, 510 12 2.6161 0.74118 7.1119 2023 08 25 22:23:39.2915 1 200, 510 12 2.4647 0.78953 7.085 2023 08 25 22:23:44.2915 1 200, 510 12 2.4616 0.80912 7.225 2023 08 25 22:23:55.2915 1 200, 510 12 2.3531 1.0297 7.2455 2023 08 25 22:23:59.2915 1 200, 510 12 2.2909 0.75256 7.2955 2023 08 25 22:24:57.2920 1 200, 510 12 2.3089 0.76836 7.2838 2023 08 25 22:25:09.2920 1 200, 510 12 2.4819 1.54 7.1864
I didn't see any other problems with ch2 except for those two in the dump excerpt above, whereas ch0 keeps showing up intermittently at 7V over the following days. I don't know why the min goes close to zero, as if the bridge is disconnected, but the max is 7, normally indicating a short.
Josh and I onsite, another day of clear skies and winds from the south. The puddles in the road to the site are finally all dried out.
Another quiet day for ISFS, data all looking good. We went out to the array twice this afternoon to check out the PurpleAir sensor, first to swap out its SD card, then again to replace the sensor after we found that it was taking data, but with completely wrong dates. The new PurpleAir on the array is successfully connected to wifi at t44 and sending data to the PurpleAir website, which is good, since this one doesn't have a working SD card slot.
Catching lizards today out at the array:
And some pics from moon night yesterday evening!
Josh and I onsite, clear with a south wind.
Data checks all looked good this morning. We went out to the array to clean the lenses on the Ott (at Bill's suggestion), other than that nothing much for ISFS! No new mouse evidence so we're hoping the situation is resolved.
Reva left this morning, so now just Josh and I from ISF.
Again partly cloudy and not too hot, a little windier today (still out of the south).
Out to the array today between around 11 and 12:
- cleaned off all solar panels, including new t0 and the trailer tower. They weren't terribly dirty, but definitely some dust buildup in the corners.
- wiped the lenses of the EC150s at 1m and 4m on t0, to see if that helps with the low values they've been reporting. 11:30-11:40
- replaced the TRH housing at t47, which was reading Rfan values ~2500. With new housing all looks good. Replaced around 11:45.
- paid another visit to the drone operators on the way back.
Then back to the base trailer to work on the mice situation. We wiped down all the countertops with bleach and vacuumed the trailer, so now at least we'll know if there are new signs of mice. We also tried adding some steel wool to a couple of the places we think they might be getting in (cable ports, conduit leaving the electrical panel). While vacuuming we did discover one male black widow in the base trailer (behind the bin of spare dsms in the SW corner of the trailer), so keep in mind when moving boxes around. We've also seen them in the sounding/ISFS storage container and in the portapotty. Watch out!!
These seen in wind speed for most of the day, but especially the afternoon. They appear to a much lesser extent in temperature and I don't see them in.pressure (oddly). Maybe Bruce noticed something interesting too today?
Onsite today with Josh, and Reva (here just for a quick visit). Jacquie left to head back to Vegas today, thanks Jacquie!
Weather good, partly cloudy. Calm in the morning, moderate south wind by the afternoon.
We went out to the array a couple times. On our first visit we gave Reva the tour, had an initial look at the bad 4m hot film, and brought the callab laptop out to connect to the t1 csat3a with the wrong sampling rate. In the Campbell device config utility, the options for sampling rate were 10Hz, 20Hz, and "motor rate". I tried motor rate, which results in data at 60Hz. The bandwidth filtering parameter on that sonic is currently set to 20Hz, since the only faster option is 25Hz, which Steve says would be weird with 60Hz data. This was at about 2pm.
We got back to the 4m hotfilm after the 3pm sounding. At new t0 we checked the resistance of the 4m probe (16 ohms, while the good probe at 2m was 7 ohms). Since we were measuring upstream of the bridge electronics, presumably this was a problem in the probe. We confirmed by switching the 4m and 2m probes to the opposite bridges, which resulted in good data from the 4m bridge and bad data from the 2m bridge. So I replaced the 4m probe (SF5) with a good spare (SF7), which is now giving good data. The wire on the bad SF5 probe is still intact, and visually it looks ok. I realized after getting back to base that I'd set the switches in the bridge wrong, so went back out to fix it. Data after 5:00 local is with the correct switch settings for SF7. Thanks to Steve for the remote support!
Other than that, we did the 10 and 3 sonde launches, showed Reva around the site, and talked about ideas for a sonde system using hydrogen. We also stopped to check out the drone operations out near the array.
Sorry Jacquie, I haven't taken any pictures yet...
Staff: Jacquie, Josh
Weather: Sunny, light winds from the south, high 70's.
Notes:
- We replaced the bad CSAT's at t1 and t22 with the remaining spares.
- We replaced the TRH casing at t41 (the ladder is now parked there)
- The radiometers have been cleaned.
| t1 | CSAT3A SN 1312 | EC100 SN 1800 | PPE000020722 |
|---|
| t22 | CSAT3 SN 0539 | PPE000016613 |
|---|
t41
- Replaced the TRH casing. The SHT sensor is the original.
The Shelf of Bad Sensors
All the bad sensors have been labeled.
One more thing: we noticed that the glass on the bad t1 CSAT3 was broken.
The 4m hotfilm (ch3) started reporting suspicious values about 6pm local time last night. It was averaging over 2 V, then dropped off to 0.6, and at the moment is 0.2 V. If it were a broken film, then I would expect the voltage to be at 2.474 V or so, and if the bridge were powered off or disconnected I would expect near 0. A short in the film or in the probe would measure around 7 V. So I'm not sure how to interpret the current measurements. Anyone have any ideas? I suppose checking the film integrity is a good start. Otherwise maybe it's worth switching to a different bridge?
Jacquie and Josh just replaced the t1 CSAT3 with a CSAT3A. I made this change in the local config directories on t2 and ustar, but have not checked them in. (All I did was change the sensor type from CSAT3 to CSAT3A_BIN.). I've also restarted dsm_server (not sure if I needed to do this) and statsproc on ustar.
Note, however, that this is a config change breaks the analysis of previous data. (I think we did something similar, changing a sonic type, during burn-in.). I think we'll need to change configs.xml, perhaps only with an environment variable, to implement this change, but we'll also need to break up any statsproc_redo runs. Maybe Gary will think up some better way to deal with this.
P.S. I made the config change (in /net/isf/isfs/projects/M2HATS/ISFS/config), but haven't checked it in since ck_xml fails. Maybe this is okay??
BTW, the new EC100 box is set for 20sps, rather than 60sps that the other EC100s are now set to. I tried to change this remotely through the serial port, but couldn't. Someone should power up ainu and use ECMon to make this change. Maybe this could even be done on the tower using the ladder!
Staff: Jacquie, Josh
Weather: Better than at Boulder.
Notes:
- It was a long day and I couldn't get out to the site until ~3pm.
- We took soil samples at all three sites. The soil was still wet and it was trivial to pull out a core sample.
- We reinstalled the hotfilms. Steve checked the data fidelity and we were able to get all of them working.
Bad CSAT3's at t1 and t22
We unplugged/replugged ports and the cables going into the CSAT3 white boxes and still couldn't establish good measurements. We ended up taking down the CSAT boxes and looking inside the boxes. We couldn't find any corrosion except made some on the external ports. Below is what the inside looks like.
So we plan to try and replace both CSATS with the following spares we found:
CSAT 3 Spare
CSAT 3A Spare
- Steve and Chris please be on mobile standby tomorrow as Josh and I work through replacing the sensors.
- All suggestions, protocols, and tips are welcome.
- I assume the SN needs to be noted?
Staff: Jacquie, Sam
Weather: Wonderful! Cool and cloudy while at the site. Windy. Clouds started clearing late in the afternoon.
Looking back at the array and trailer tower. Taken ~10am
Notes:
- The towers survived hurricane Hilary!
- We visited the site in the AM before Sam departed (safe travels!)
- Josh arrives tonight.
- We addressed a few priority issues:
tt
- Replaced the batteries and power cycled the DSM's. The beacon is now powered and lit.
- However, the GPS in ttb is still not operating. We replugged, let the GPS sit outside the box for a while, but non of those worked.
t23 DSM
- The cooler was filled with a few inches of rain and the victron was waterlogged. We replaced the victron and everything powered-up.
- I check one of the other coolers just to see if it also had water and found a black window spider and it's babies (not doing that again!).
- Just the CSATS were not coming online so we unplugged/replugged the cables on the CSAT ports.
- We also power cycled the soil mote which was also offline.
t1 DSM
- The CSAT3 and CSAT3A was offline so we unplugged/replugged the cables on their ports. Having the print out in the DSM boxes indicating which port the sensors are on was great!