Blog

Josh and I onsite. Weather is cooler, with clouds developing during the day. Visibly hazy, especially when looking down across the valley the site is in, from the highway on the way out. Strong winds (we saw gusts of 18 m/s at 4m on dsm dashboard), blowing dust. Got a few raindrops on the windshield as we were leaving the site this afternoon.

Went out to the array today, mainly to (once again) mess with the PurpleAir sensor. We had been thinking of lowering the trailer tower to replace the 28m EC150, but with today's conditions it didn't seem safe to lower and raise the tower. We'll be back to that once the wind has died down enough, which sounds like it'll be Sunday. We did take a look at the trailer tower to see if it was swaying a lot in the high winds, but it looks like it's doing fine. The guy wire on the downwind side is visibly looser than the others, so it must be tilted a bit, but it doesn't seem like the tower itself is moving. The 7m sonic on its long boom is definitely bouncing around.

Josh onsite, taking care of business while I took a day off. Thanks Josh!!

Data all good in the morning when I checked it.

Josh and Jielun removed the hotfilm probes around 12:30pm local as we started getting heavy winds in the afternoon. They covered the probe holder ends with electrical tape so hopefully they won't end up full of blowing dust.

Another hot, clear day in the desert. Josh and I onsite. Chenning and Jielun mostly around the base today, as the drone is not flying.

Data check shows everything looking fine.

Out to the array today between about 1:45 and 2:45. Went to the trailer tower to lower it, but by the time we found where the keys were (operator error) and realized we would have to go get the gas can for the generator, we didn't have time to do that today. We're planning to head out again on Friday now that we know what we're doing and will be prepared for it. While out there, we cleaned the NR01s (t47 one at 2:26pm local, t0 one at 2:48pm local). We also tried swapping the original PurpleAir sensor from the array back in, since at base it was working just fine connected to a spare dsm's wifi, but out on the array it just doesn't work. So, still using the spare we swapped in a couple days ago.

Besides that I spent some time getting SOS data off the Leica for Steve and finishing off the presentation I'm going to give as part of Jielun's M2HATS event in town this evening. We'll see how big of a crowd we get for that...

Happy birthday Josh! Another sunny, cloudless day. Trailer continues to be mouse-free and outhouse continues to be spider-free (though there are plenty of them elsewhere).

All looking good in the data, except that when I checked the hotfilms after the 3pm sounding the 0.5m one (which was having intermittent problems yesterday) was now reading about 0.3.

Out at the array checked the resistance of the 0.5m hotfilm, and got values around 1.1 kΩ (compared to the 1m hotfilm, which was around 7Ω as expected). We tried swapping in the extra bridge to hotfilm mount cable, no change. We measured resistance again at the hotfilm mount, without the cable to the bridge, and got the same value. We removed the probe, didn't see anything obviously wrong with it, tried reseating. While the probe was out I tried to blow on the probe mount in case there was water in it, but didn't notice any. Switching in spare probe SF9 returned resistance values to around 7Ω as expected, and the probe is now measuring about 2 volts, same as the other hotfilms. Once we verified that fixed the problem, we changed the dip switches in the bridge for the new probe at 0.5m. Data after around 5:10pm local should have the correct switch settings.

So I think now we're out of spare hotfilm probes, unless we could make the one that's wire and not film work, but Steve says we don't have the correct bridge settings for that one.

Jielun has also been wondering about whether the guy strap near the 2m hotfilm is the cause of the weird data she's seeing, especially since the ratchet itself is at about 2m. We're talking about redoing the guy wires on that tower entirely, but for now I've tried taping over the ratchet (to smooth its shape) and the extra strap (to prevent it flapping).

Josh and I onsite. Another clear, warm day with light winds. Not enough aerosols for the lidars!

All trucking along well data-wise, but don't worry, Steve is busy thinking of things for us to do. Looked into the low tc at t29 (see blog post) but so far no resolution. Saw Gary's post about the 0.5m hot film, but by the time I was getting ready to go out and switch the probe, it's back to working again. Steve says the short might also be in the cable between bridge and probe (we could try a spare) or in the probe holder (we don't have spares). I've been mostly playing with the PurpleAir we removed from the array yesterday and working on a presentation for the M2HATS open house on Wednesday.

Consistently reading a degree or two lower than other sonics:

I don't see any noticeable differences in u/v/w between it and nearby sonics.

t29 is a CSAT3A + EC150 combo.

Tried power cycling, doesn't seem to make a difference.

Channel 0 (0.5m) started hitting 7 Volts on 2023-08-25 at 9:42 UTC, or 2:42 last Friday morning local time.  It was intermittent at first, but as of this morning it is continuous.  Presumably a short has developed somewhere, likely in the probe.  If there is a spare probe, then I think that's the first thing to try swapping out.

The first sign of a short in channel 0 seems to happen here:

(base) [daq@ustar ~]$ data_dump -i 200,510-513 -x hotfilm.xml /data/isfs/projects/M2HATS/raw_data/hotfilm_20230825_* |& egrep ' 7\.' 
2023 08 25 09:42:43.1870       1 200, 510      12     2.1569     2.0216     7.2967
2023 08 25 09:46:02.1870       0 200, 512      12     1.8874     1.8348     7.2177
2023 08 25 10:10:30.1875       1 200, 510      12      2.007     1.8459     7.2689
2023 08 25 10:14:52.1875       0 200, 512      12     1.9778     1.7318     7.2174
2023 08 25 11:03:13.1880       1 200, 510      12      1.962     1.8282     7.2673
2023 08 25 22:21:08.2910       1 200, 510      12     2.3411    0.68967     7.2026
2023 08 25 22:21:31.2910       1 200, 510      12     2.3765    0.73897     7.2588
2023 08 25 22:21:48.2910       1 200, 510      12     2.4567    0.85462     7.1738
2023 08 25 22:21:55.2915       1 200, 510      12     2.3388    0.76678     7.2604
2023 08 25 22:22:04.2915       1 200, 510      12     2.5594    0.80248      7.091
2023 08 25 22:22:05.2915       1 200, 510      12     2.5424     0.7614     7.1185
2023 08 25 22:23:36.2915       1 200, 510      12     2.5616    0.74718     7.0755
2023 08 25 22:23:37.2915       1 200, 510      12     2.6161    0.74118     7.1119
2023 08 25 22:23:39.2915       1 200, 510      12     2.4647    0.78953      7.085
2023 08 25 22:23:44.2915       1 200, 510      12     2.4616    0.80912      7.225
2023 08 25 22:23:55.2915       1 200, 510      12     2.3531     1.0297     7.2455
2023 08 25 22:23:59.2915       1 200, 510      12     2.2909    0.75256     7.2955
2023 08 25 22:24:57.2920       1 200, 510      12     2.3089    0.76836     7.2838
2023 08 25 22:25:09.2920       1 200, 510      12     2.4819       1.54     7.1864

I didn't see any other problems with ch2 except for those two in the dump excerpt above, whereas ch0 keeps showing up intermittently at 7V over the following days.  I don't know why the min goes close to zero, as if the bridge is disconnected, but the max is 7, normally indicating a short.

Josh and I onsite, another day of clear skies and winds from the south. The puddles in the road to the site are finally all dried out.

Another quiet day for ISFS, data all looking good. We went out to the array twice this afternoon to check out the PurpleAir sensor, first to swap out its SD card, then again to replace the sensor after we found that it was taking data, but with completely wrong dates. The new PurpleAir on the array is successfully connected to wifi at t44 and sending data to the PurpleAir website, which is good, since this one doesn't have a working SD card slot.

Catching lizards today out at the array:

And some pics from moon night yesterday evening!

Josh and I onsite, clear with a south wind.

Data checks all looked good this morning. We went out to the array to clean the lenses on the Ott (at Bill's suggestion), other than that nothing much for ISFS! No new mouse evidence so we're hoping the situation is resolved.

Reva left this morning, so now just Josh and I from ISF.

Again partly cloudy and not too hot, a little windier today (still out of the south).

Out to the array today between around 11 and 12:

• cleaned off all solar panels, including new t0 and the trailer tower. They weren't terribly dirty, but definitely some dust buildup in the corners.
• wiped the lenses of the EC150s at 1m and 4m on t0, to see if that helps with the low values they've been reporting. 11:30-11:40
• replaced the TRH housing at t47, which was reading Rfan values ~2500. With new housing all looks good. Replaced around 11:45.
• paid another visit to the drone operators on the way back.

Then back to the base trailer to work on the mice situation. We wiped down all the countertops with bleach and vacuumed the trailer, so now at least we'll know if there are new signs of mice. We also tried adding some steel wool to a couple of the places we think they might be getting in (cable ports, conduit leaving the electrical panel). While vacuuming we did discover one male black widow in the base trailer (behind the bin of spare dsms in the SW corner of the trailer), so keep in mind when moving boxes around. We've also seen them in the sounding/ISFS storage container and in the portapotty. Watch out!!

These seen in wind speed for most of the day, but especially the afternoon.  They appear to a much lesser extent in temperature and I don't see them in.pressure (oddly).  Maybe Bruce noticed something interesting too today?

Onsite today with Josh, and Reva (here just for a quick visit). Jacquie left to head back to Vegas today, thanks Jacquie!

Weather good, partly cloudy. Calm in the morning, moderate south wind by the afternoon.

We went out to the array a couple times. On our first visit we gave Reva the tour, had an initial look at the bad 4m hot film, and brought the callab laptop out to connect to the t1 csat3a with the wrong sampling rate. In the Campbell device config utility, the options for sampling rate were 10Hz, 20Hz, and "motor rate". I tried motor rate, which results in data at 60Hz. The bandwidth filtering parameter on that sonic is currently set to 20Hz, since the only faster option is 25Hz, which Steve says would be weird with 60Hz data. This was at about 2pm.

We got back to the 4m hotfilm after the 3pm sounding. At new t0 we checked the resistance of the 4m probe (16 ohms, while the good probe at 2m was 7 ohms). Since we were measuring upstream of the bridge electronics, presumably this was a problem in the probe. We confirmed by switching the 4m and 2m probes to the opposite bridges, which resulted in good data from the 4m bridge and bad data from the 2m bridge. So I replaced the 4m probe (SF5) with a good spare (SF7), which is now giving good data. The wire on the bad SF5 probe is still intact, and visually it looks ok. I realized after getting back to base that I'd set the switches in the bridge wrong, so went back out to fix it. Data after 5:00 local is with the correct switch settings for SF7. Thanks to Steve for the remote support!

Other than that, we did the 10 and 3 sonde launches, showed Reva around the site, and talked about ideas for a sonde system using hydrogen. We also stopped to check out the drone operations out near the array.

Sorry Jacquie, I haven't taken any pictures yet...

Staff: Jacquie, Josh

Weather: Sunny, light winds from the south, high 70's.

Notes: 

• We replaced the bad CSAT's at t1 and t22 with the remaining spares.
• We replaced the TRH casing at t41 (the ladder is now parked there)
• The radiometers have been cleaned. 

t41

• Replaced the TRH casing. The SHT sensor is the original.

The Shelf of Bad Sensors

All the bad sensors have been labeled. 

One more thing: we noticed that the glass on the bad t1 CSAT3 was broken.

The 4m hotfilm (ch3) started reporting suspicious values about 6pm local time last night.  It was averaging over 2 V, then dropped off to 0.6, and at the moment is 0.2 V.  If it were a broken film, then I would expect the voltage to be at 2.474 V or so, and if the bridge were powered off or disconnected I would expect near 0.  A short in the film or in the probe would measure around 7 V.  So I'm not sure how to interpret the current measurements.  Anyone have any ideas?  I suppose checking the film integrity is a good start.  Otherwise maybe it's worth switching to a different bridge?

Jacquie and Josh just replaced the t1 CSAT3 with a CSAT3A.  I made this change in the local config directories on t2 and ustar, but have not checked them in.  (All I did was change the sensor type from CSAT3 to CSAT3A_BIN.).  I've also restarted dsm_server (not sure if I needed to do this) and statsproc on ustar.

Note, however, that this is a config change breaks the analysis of previous data.  (I think we did something similar, changing a sonic type, during burn-in.). I think we'll need to change configs.xml, perhaps only with an environment variable, to implement this change, but we'll also need to break up any statsproc_redo runs.  Maybe Gary will think up some better way to deal with this.

P.S. I made the config change (in /net/isf/isfs/projects/M2HATS/ISFS/config), but haven't checked it in since ck_xml fails.  Maybe this is okay??

BTW, the new EC100 box is set for 20sps, rather than 60sps that the other EC100s are now set to.  I tried to change this remotely through the serial port, but couldn't.  Someone should power up ainu and use ECMon to make this change.  Maybe this could even be done on the tower using the ladder!