We had 3 reasons to get out to some towers: fix 2m TRHs, take soil cores, and check on other coolers (since 2 have been found with water):
tse01: cooler had some water; took core
tnw02: 2m TRH fan wasn't working – replaced
tse02: cooler had lots of water; took core
tse04: 2m TRH fuse was blown – swapped from another port
tse06: 2m TRH swapped, but still not working. Likely a blown fuse, but is at 30m and didn't have climbing gear. Left new TRH in place.
tse07: took core
tnw09: cooler full of water; took core
tnw06: cooler full of water
Yesterday, tnw05 died. Again, found a LOT of water in the battery cooler. Dumped it out, went back to the ops center to get a spare controller, and reinstalled on a second visit. Everything eventually came up, though I rebooted tnw05t again to help get its clock in sync. Also, tnw05t wasn't pinging. Moved its ethernet cable to another ethernet switch port in tnw05b and it came up – perhaps this cable was stressed.
Clearly, we should proactively visit every site with battery boxes to check for water issues.
HOWEVER, in looking at the tower, I note that the NR01 at 20m is directly above the solar panel mounted at about 15m. I think that the orientation of this sensor should be moved 180 degrees, though then it would view even more of the clearing at the base of the tower.
The ARL LiCors have been stuck in customs for weeks. We expect them to finally arrive early next week. Speaking with Chris Hocut today, they have shipped 5, but would like to deploy 3 on their towers. That leaves 2 for us to deploy. Tina K. Chow selected the 2 rsw masts for the locations for these licors. I just hope they arrive before Andy and I leave, so Gary and Kate don't have to deal with them.
The 10m sonic here fixed itself overnight. I'm mystified as to why the CSATs do this – not finding a signal to trigger on for hours at a time, but I'm not complaining much.
The TP01 also died 2 days ago. Turned out to be a cold solder joint on the higher-current resettable fuse we had installed on the motes. Replaced this fuse and now is working.
rne04: This RMY has error flags on almost every sample. In the past, we've found this to be the case if the voltage is below 12V. This tower is the only one with RMYs that isn't AC or Solar powered. (It is powered by a 24VDC supply at a lidar.) To solve the power problem, we tried routing 24VDC from the DCDC converter to the DCDC input on the serial frontpanel, but found that it was pulled down. Last night, I tried routing directly from the 24VDC input and blew fuses. This morning determined that the 24VDC issues must be due to something on the frontpanel, so tried 12VDC again. We brought a spare sensor and cable which worked fine on the ground at 12VDC, so we replaced both on the tower. The new sensor still gave bad data on the tower.
Back in the ops center, realized the VERTEX problem with 15V protection diodes on the front panel was the source of the power problem. Even after removing this (from one port), found that the 20W DCDC converter would pull the 24V power down, but a 10W DCDC converter worked fine. Ran back up the hill once more, switched DSM boxes (keeping the same Pi and hat). Now, the sensor is getting solid 24V. (actually 24.18)VDC. But...the sensor still gives bad data.
This must be a grounding issue. Indeed, nowhere is "-" attached to Earth. Earth on the sensor isn't attached to anything and the sensor ground lug isn't attached. The DSM isn't grounded either. This all sounds bad, but several other towers (e.g. the new solar-powered TNW masts) are wired essentially the same way and running fine. Perhaps it is related to this also being a "staked-in" tower. Andy and I will see if we can come up with a workable strategy to ground this system.
10m CSAT is in constant reset mode since midday yesterday. TP01 completely died at the same time. Just visited and found a liter of water in the battery cooler (can you say "VERTEX"?). Powered station down, dumped water out, cleaned up a bit, and restarted station, but no change in sensor status. Guess we have to go back prepared with spares.
INEGI provided climbers to replace TRHs on the tall masts today. Andy was their ground support/check. We had a bunch, all have been replaced by "bypassed" sensors:
- tnw07.60m
- tnw10.40m
- tse04.40m
- tse04.60m
- tse13.60m
- rsw03.40m
I have put BYPASS sensor types in the .xml, though we may now need make a time-varying .xml (using configs.xml)...
Noticed for the last day that this site has been down. Turns out that the solar panel was wired backwards, so we haven't been charging since it was set up last week – just running down the 2 big batteries. The charge controller hadn't turned on the output by the time I left (a few minutes after fixing), but with good sun this afternoon, it hopefully will turn on before evening (and be able to run all night?).
Andy and I split up. He went with INEGI to replace TRH sensors and got to all the towers on the SW ridge before lunch.
I first visited v05, that wasn't pinging. The DSM was up and logging data and the Ubiquiti could ping from outside. (I didn't remember the IP address to try pinging the station's Ubiquiti from the wired side.) Couldn't get ifup/down eth0 to work. Rebooting the DSM didn't help. Eventually just cycled power on the Ubiquiti (using the power side of the POE cable), and everything came back up.
Reinstalled the Ubiquiti at tse05 (see previous logbook entry). It is now up.
Last, went back to rne04. Andy's +24V wiring was fine, but reading only 20V, thus wondered if the total Ubiquiti + RMY power draw was too much for the 10W DC/DC converter. Since at this site we have +24DC coming in (from a power supply at UND's "Lionsgate" lidar), I redid Andy's wiring to come directly from the station's input 24V power. The Ubiquiti power went back up to 24V. However, the sonic now blows fuses. (I went through two of the 1A fuses.) This isn't normal – all of our other RMY's have run fine through 1A fuses, and the current should even be lower at 24V. (Yes, I checked the specs for RMY 81000 that it should run fine at 24V.) I conclude that either the sensor is bad, or it is miswired at the sensor. (I opened up the Bulgin connector and it seems okay to me.) Climbing will be needed.
Yesterday, spent most of our effort erecting the "ladder" site at tse05 (with a lot of help from our friends). It is highly compromised, but hopefully will still be useful:
- Using the ladder, the sonic and NR01 are at about 1.5m, rather than 20m!!
- The sensors are not level. With my phone, I measure:
- CSAT: pitched up to the SE by 1.8deg, rolled with top to the SW by 0.3deg.
- NR01: pitched down to the NW (same as the CSAT) by 2.4deg, rolled down to the SW by 1.8deg
- (still, only about 2deg off in this terrain is pretty good)
- There are a LOT of rocks. I explored several m^2 before finding a core location. The soil installation is thus not precise (e.g. in depths and soil repair) and not totally representative.
- The NR01 view contains the solar panel, the battery box, trampled brush, and branches above and below from the adjacent tree. I tried to place the solar panel roughly perpendicular to the NR01, and to hide the battery box under a fern, but I'm sure both still can be seen.
- Since the height is so low, I wanted to be away from trees. This site is in a gap between trees on a terraced slope, but the trees probably channel the flow to be directly up or downslope. Also, there is a several m^3 sized rock 5–10m upslope from the sonic.
Data immediately started being collected locally, but had trouble getting on the Ubiquiti network. (Turns out that at least part of the problem was that this adaptor never had a WiFi password set. Another issue may be trees in the field of view.) Returned today to place the adaptor in a different location (strapped to a tree!) and all is well.
<Finally> added a sonic to rne04. To mitigate the concern about the holding power of one stake, I attached a "temp" (rope) guy to a reasonably large tree. The tower felt secure while I was on it.
However, the data are garbage. It acts as though the power is low, although the way it is wired the voltage (which measures 11.8V) should have been pretty close. Nevertheless, Andy went back and wire/jumpered it to 24V. Even with that change, the sonic data are garbage.
Tomorrow, we plan to climb again, check the wiring (possible miswired power ground?), and replace the sonic if necessary. I had hoped for this tower to be done, but we still are <just> at 41 towers.
- tnw02: added last sonic; fixed cables-into-wrong-ports. Now completely operational (#40).
- rsw05: cycled power – hadn't been collecting data since 0145 this morning, even though had power. Seems okay now.
- tnw04: completely instrumented today. Rewired INEGI's solar power system for 12VDC. Now completely operational. (#41)
- all sonics set with "box" facing the tower.
- Top 2 levels (20m, 12m) set with boom to NW (to avoid hitting the solar panel while hauling up the sensor)
We're reasonably proud of ourselves!
20m.rne07 is reporting non-zero diagnostic flags since this afternoon and thus all data are being flagged as bad. pio didn't help.
P.rsw02 is reported as bad, but data look good logging into the dsm. Not sure what configuration isn't working.
Qsoil.tnw09 still negative.
Several TRHs out (I'll update the list later)
Spent the rest of the day at tnw02. Got everything up except the 2m sonic (because we left its boom at the ops center). However, rain (again) drove us out, so I mis-plugged sensors in. Since some involve 422 jumpers, I can't just change the config from here. We'll need another trip (tomorrow).
v07t: powered up each component in sequence. As I was connecting the sensor power, heard a brief squeal, then normal. Everything coming in except Li7500.20m. Suspect that a fuse may have blown up there, but didn't have a spare. Recall that it has been on our TODO list to ground this box 
. Left this sensor down, but everything else should be back.
rsw02: went to figure out why 20m sonic was dead. Stupidly decided to climb and replace the RMY (including the messy rewiring of the cables while up the tower). After doing all that, still no data. Swapped fuses (and power tests okay). Ended up changing from usb2 to usb3 and data are fine. Had to set "5be" for this new sonic. Andy has the new serial number. Of course, the orientation (pitch and roll) may have changed.
tnw02: dropped off a few things – recovered the extra climbing claws from the climbing box we had left there. Back to ops to get a few supplies (and gas) and will go back out now.