Another day of waiting...
Our tensionmeter shipment is due in tomorrow, so still no climbing. We have asked INEGI to arrange for Samortecnica to be ready to work on tower retensioning on Friday. Complicating things are that rain is expected on-and-off starting tomorrow, breaking our run of great (but chilly) weather. Another complication is that we lose 3 of our crew members on Friday. We have had several discussions about staff scheduling, but a lot seems to depend on how Friday goes...
A field crew went out to work on power and made a lot of progress. The electricians also have been continuing to add power drops, so several towers are now up. Unfortunately, the electricians are having a hard time getting a good ground at the tse13 power drop (their last measurement was 200 ohms and they want 20), so they still haven't run the power cable from the new box location to the mast. DTU has prepped this, and at least one other box, so that the tower can come up as soon as power goes to it.
In the ops center, almost all of the towers that are built have been prepped. We also worked on software and scripts for webplots and data transfer.
We spent a bit of time helping DTU to position their lidars.
I'm told that Cornell now have 3 of their 4 lidars powered up, 2 are communicating (over cell phones). They also have installed the seismometers, though there still are no power drops on the sw ridge.
Notes to myself...
- Change the ridge repeater from tse04 to rsw06 (because tse04's construction is delayed): Steve Oncley
- pull power "Y" cable from tse04t box.
- swap tse04t's 20W DC-DC converter with a 10W DC-DC conver
- climb to install "Y" cable and swap in 20W DC-DC converter on rsw06
- add access point Ubiquiti to rsw06
Still waiting for tensiometers to arrive from Boulder. In the meantime...
- Kurt/Lucas/Laura/Bob? ran up and down the mountainside trying to power up stations.
- Dan helped the INEGI crew instrument tse05. The last 60m tower is now up and instrumented. Its solar-power system needs a couple of batteries and then should be working.
- Andy and I prepped 4 more towers.
- Ted and Santiago prepped Ubiquitis, tweaked the network configuration, and arranged for a bittorrent stream of data to Boulder
- Gary doesn't know what he did, but he stayed warm working outside in the parking lot...
Portuguese outfitted 60m tse-6 (tower 22) mast with DTU sonics and NCAR trh sensors. Was unable to power up and test due to lack of batteries for the solar system. Batteries will be provided by Portuguese.
Due to conflict with location of 10m sonic, it was moved to 16m. TRH was moved to 16m to match sonic measurement level.
Currently left to do:
- Install Batteries and wire solar system Daniel Buonome
- Install Barometer inlet at 2m Daniel Buonome
- Verify Operation of tse06 Daniel Buonome
We met with the tower company again today and appear to have a path forward to lift our climbing stoppage. The company will fix the saddles, when obvious problems are noted. They will retension towers with us monitoring with our tensiometers (these should be in FedEx to us now, to arrive Wed.). Clearly this will take some time, but is the solution we wanted.
Today's tasks:
- Prepped tse05 (the last 60m tower) for the INEGI crew to install tomorrow. They installed 2 of the 3 solar panels on this tower today to get ready.
- Connected AC power to rne02 and rne03, which are now reporting (though rne02's 20m RMY is not reporting good data).
- More ops center network work. We did have a brief outage that took a while to recover from since I let a key go up the mountain.
- Prepped the remaining (all?) user sensor booms
DTU and INEGI crews are now at the site, ARL having left over the weekend. Cornell is still here as well.
As documented in Fixing CSAT3s, Steve found a method of restoring data from the CSAT3 sonics. I tried to figure out if we could get rid of the warning message about mismatched signatures, in case that was related to cal data or something else that was still corrupted. That is documented in a JIRA issue: ISFS-140. The end result is that Steve looked at data on a DSM for four CSAT3 sonics, and now we're convinced that they work:
1119, 1120, 1122, 0538
1121 was never connected to a DSM with oversampling enabled, so it should still be fine, and will be tested when staged with a tower.
A light work day, since we are still under a work stoppage for tower climbing. The crew spent about 4 hours attaching cables to the RMY sonics, assembling solar panel frames, assisting Cornell, and organizational tasks. The ops center WiFi access point is now in its final position working well. Gary got the ISS profiler site running again.
Tomorrow is a down day for our crew.
We're now under a "stop work" order for any tower work, due to safety issues noted by our crew. After a discussion with the president of Samortecnica, with Jose Palma translating, it appears that guy wire tensions have been set by "feel", which is not up to US standards, even though design documents do exist that specify the working tension. Also, "saddles" to secure the guy wires have not been installed properly, leading at least one guy wire to become totally loose. This issue was not resolved. The next step will be a further discussion on Monday with a person from Samortecnica who speaks English. Our current thinking is for work on the tall towers to continue to be performed by the INEGI crew. (They plan to work on the last 60m tower on Tues.)
In the meantime, a bit of work was done:
- Installed a Litebeam at the ops center that is working as a primary access point
- All of our CSATs are now back to functioning.
- Soil sensors installed at 2 sites (tnw09 and rse06)
- Prep work: putting all boxes on mounting plates, assembling top guys
With networking going, found that we could connect to tse13 and tse07. tse07 needed a reboot to allow logins, for some reason and now is fully functioning. Found that tse13 sensors weren't reporting, even though everything else was fine. Turns out that there are no SHTs installed in the housings – I could have sworn that we checked that (several times). Installed the 2m, but couldn't do more due to climbing restriction.
Over the weekend, we'll have a light workload doing what we can on the ground.
I stayed in the ops center, so I'm not completely sure of all that happened today....
- TRHs should now be up on tnw07, finishing the installation of this mast begun in Dec. by INEGI with the DTU equipment.
- Finished off ground wiring left over from yesterday at tse07 and tse10.
- Powered up tse13, but still couldn't get its WiFi to pair with the ops center. We may be collecting data from here now.
- Started to install rne07, but tower was determined not to be climbable. José Carlos is being contacted.
- Finished prep of rne06, tnw09 and tnw11, but had some issues:
- Found that the RMY cables were mislabeled – the orange wire (Bulgin in 6) should go to B2 and A1 and A2 should not be connected. Also had to change settings on the ARL RMYs. One was at 19.2kbaud, rather than 38.4K and the usual message format. Wake correction is turned on. There were 3 different levels of "WT" threshold – 0, 1, and 20 cm/s, but this is not used in UVW output mode.
- Found that another 2 of our CSAT3s were reporting 8000 on all channels. After a long phone call with Ed at CSI, determined that nidas initialization of these sensors somehow is putting them into a state in which they can't acquire a signal. Is this perhaps the same issue we had pre-VERTEX where the CSAT3s using version 5.0 can't run at 20/60 Hz anymore? I thought we solved this. In any case, this is not a hardware issue, though it appears that it will take a bit of work to revive the ones we've mis-initialized.
- ARL and Cornell also were busy, and now have their Ubiquitis and cell-modem SIM card, respectively, but I don't know their details.
ddn
minicom usbX
& to stop binary output
T to enter terminal mode
disconnect sonic head
D or & (I'm not sure) to get to "......" mode
x to break out
aq 9 to set a data rate that doesn't use 60Hz (10Hz in this case)
reconnect sonic head
^ to check that field 4–6 are similar
^ to stop this diagnostic mode
D to set data mode
& to start collecting data
Another day, another 2 towers. One crew did tse10 on the east side of the valley, another tse07 on the west side. What is important is that tse07 is our first solar-powered station and, as such, is the first to have continuous power. Thus, TODAY IS OUR FIRST DAY OF OPERATIONS!!! As part of the tse07 installation, we took a soil sample, which also should be the first to be used for Qsoil comparison.
Except for tnw07 (the 60m tower instrumented with DTU sensors), we have now installed everything that we "prepped" in December. Tonight, I started prepping another 4 towers.
In other news:
- Samortecnica gave us an update on their tower progress, including that the last 60m tower is now in place (the last 100m tower still needs to be started). ARL also gave us an update on their towers. The Google-Drive "Tower Status" has been updated. 33 of the project-total 52 towers in the main array are now built (and 18.5 of them are now instrumented).
- Santiago got a wireless router working in the ops center, so we now can connect DSMs directly to the tower network as we stage/prepare them.
- We now have access to some of the power drops and will try to get some of the AC towers going.
- ARL (and Jose Carlos, of course) helped Cornell drop off their equipment (primarily lidars) throughout the array.
Thus, a lot of activity is going on.
Similar to yesterday, all staff (except Santiago) went to the field to become familiar with a different type of tower set-up. Finished tse08. It should even be straightforward to get power here reasonably quickly. Transported all equipment to tse07 for tomorrow's deployment. Back in the ops center, mostly prepped tse10. Also, Santiago gave up on using the Cisco routers (even after a firmware update) and has fallen back to using a spare older-model router we had brought. This router is now being used in the ops trailer for primary connection to the tower network. He and Ted later picked up a second router for use in the ops center staging area.
The primary issue is that, in prepping tse07, we found 3 straight CSAT3s that all report "00". We ended up using one from Petra (OU), that worked just fine. Her shipping box had more foam than the standard CSAT3 case – possibly vibration in shipping broke all of our CSAT3s??? Hard to imagine. When the dust settles, I'll go back to looking at them, which may require shipping back to Utah 
. This experiment wasn't designed to have spare instruments, just redundancy in the measurements that are being made...
Rebecca and Sara are now here, and José Carlos is in the area. We'll all meet tomorrow AM.
First day for the entire crew in the field. Everyone came so that different perspectives could be shared about tower operations. All this was done at rsw04, which is now ready for power. We powered it up with a battery for about 30min to check out everything. (ARL now is using DTU's portable generator.)
Errors found and fixed at rsw04:
- eolupdate had reverted to the wrong serial ports for the sonic and licor
- licor DID changed to reflect height of 10m
- TP01 had been plugged into incorrect mote port (stupid me!)
- 15m cables were fine to get to the rad and soil motes
Back in the ops center:
- Finished prep of our next two towers: tse07 and tse08.
- Found a bad Tsoil probe (reporting all -0.001)
- Found that the PTB110 connector inside the EC100 was loose (just like VERTEX s15!)
- EC100 needed to be changed to binary mode
- Found that the EC100 had to be grounded both to the DSM and to the CSAT3A head, when using a long serial cable
- Changed to longer cables and configuration, since tse07 is now a 30m tower (picklist had shown 20m)
- One CSAT3 (older) seemed bad and another became bad during testing. We're hoping that it will act better deployed on a tower.
- One bad port found on tse08's DSM, so METEK was moved to a different port. I think this is the first bad port we've seen?
- Updated software on tnw07's 2 DSMs
- Reset 16 EC100 boxes to binary mode (needed for about half of these boxes – very strange)
- Prepped Ubiquiti's for tnw09, tnw11, and rne07, which we hope to get to on Wed.
Interacted a bit with both Samortecnica's 3-person crew and ARL's crew (saw Chris just before he left). Samortecnica was working on rsw05, 06, 07 today. ARL has the SW energy balance 10m tower up, though not completely connected; one 30m tower up; at least 3 other 30m towers are erect, but not yet telescoped to full height as their crew is adding instrumentation. So...progress is being made!
Ted and I continued prep/network tasks in the ops center. We also visited tse13 to rewire the power (and hopefully build the first leg of the WiFi network), but couldn't actually get AC power up since the fusebox was locked. Both 24V and 12V power supplies (barely) fit in the lower part of the DTU box along with an outdoor-rated power strip. This will have to be changed once DTU installs their "differential protection" circuit.
rsw04 should be ready to go, though I need to double-check the mechanicals and then pack. The other 4 towers still need software upgrades and systems checks. All sensors are prepped.
Ted was able to get radio pairing to the ops center access point using (multiple) frequencies in the range suggested by IMPA (to avoid possible interference with their radar). Now need to copy this configuration to the rest of the Ubiquiti gear.
The other NCAR staff dropped by to get oriented to the ops center as well.
Kurt, Laura, Andy, Dan, Lucas, Bob, Ted, and I arrived via Lisbon today. Ted and I worked at ops center from about 2 to 5:
- After much gnashing of teeth, Ted found that the cheap ISFS router that we had brought as a backup works fine between our LAN and the Telco's router. The more expensive Cisco routers do not work. Ted will buy another router tomorrow, but in the meantime, we now have a DSM in the ops center able to connect to the network (and execute "eolupdate"). This was done by temporarily running an ethernet cable through the cafe inner door to the ballroom – not a permanent solution.
- I unpacked the December airshipment and rearranged stuff a bit to make some space between INEGI, DTU, ARL, and Cornell stuff.
- I inventoried our prep work: We left tnw07, tse07, tse08, tse10, and rsw04 pretty much ready to go, though I want to "restage" all of them with our checklist. I got most of the way through the checklist for rsw04.
Tomorrow's tasks:
- Finish staging the above 5 towers
- Establish routers in the ops seatainer and in the ballroom.
- Check that all EC100s are set to binary.
- If time permits, update rest of tower nanobeam software (will need some for lidars and profilers as well)