As documented in Fixing CSAT3s, Steve found a method of restoring data from the CSAT3 sonics. I tried to figure out if we could get rid of the warning message about mismatched signatures, in case that was related to cal data or something else that was still corrupted. That is documented in a JIRA issue: ISFS-140. The end result is that Steve looked at data on a DSM for four CSAT3 sonics, and now we're convinced that they work:
1119, 1120, 1122, 0538
1121 was never connected to a DSM with oversampling enabled, so it should still be fine, and will be tested when staged with a tower.
A light work day, since we are still under a work stoppage for tower climbing. The crew spent about 4 hours attaching cables to the RMY sonics, assembling solar panel frames, assisting Cornell, and organizational tasks. The ops center WiFi access point is now in its final position working well. Gary got the ISS profiler site running again.
Tomorrow is a down day for our crew.
We're now under a "stop work" order for any tower work, due to safety issues noted by our crew. After a discussion with the president of Samortecnica, with Jose Palma translating, it appears that guy wire tensions have been set by "feel", which is not up to US standards, even though design documents do exist that specify the working tension. Also, "saddles" to secure the guy wires have not been installed properly, leading at least one guy wire to become totally loose. This issue was not resolved. The next step will be a further discussion on Monday with a person from Samortecnica who speaks English. Our current thinking is for work on the tall towers to continue to be performed by the INEGI crew. (They plan to work on the last 60m tower on Tues.)
In the meantime, a bit of work was done:
- Installed a Litebeam at the ops center that is working as a primary access point
- All of our CSATs are now back to functioning.
- Soil sensors installed at 2 sites (tnw09 and rse06)
- Prep work: putting all boxes on mounting plates, assembling top guys
With networking going, found that we could connect to tse13 and tse07. tse07 needed a reboot to allow logins, for some reason and now is fully functioning. Found that tse13 sensors weren't reporting, even though everything else was fine. Turns out that there are no SHTs installed in the housings – I could have sworn that we checked that (several times). Installed the 2m, but couldn't do more due to climbing restriction.
Over the weekend, we'll have a light workload doing what we can on the ground.
I stayed in the ops center, so I'm not completely sure of all that happened today....
- TRHs should now be up on tnw07, finishing the installation of this mast begun in Dec. by INEGI with the DTU equipment.
- Finished off ground wiring left over from yesterday at tse07 and tse10.
- Powered up tse13, but still couldn't get its WiFi to pair with the ops center. We may be collecting data from here now.
- Started to install rne07, but tower was determined not to be climbable. José Carlos is being contacted.
- Finished prep of rne06, tnw09 and tnw11, but had some issues:
- Found that the RMY cables were mislabeled – the orange wire (Bulgin in 6) should go to B2 and A1 and A2 should not be connected. Also had to change settings on the ARL RMYs. One was at 19.2kbaud, rather than 38.4K and the usual message format. Wake correction is turned on. There were 3 different levels of "WT" threshold – 0, 1, and 20 cm/s, but this is not used in UVW output mode.
- Found that another 2 of our CSAT3s were reporting 8000 on all channels. After a long phone call with Ed at CSI, determined that nidas initialization of these sensors somehow is putting them into a state in which they can't acquire a signal. Is this perhaps the same issue we had pre-VERTEX where the CSAT3s using version 5.0 can't run at 20/60 Hz anymore? I thought we solved this. In any case, this is not a hardware issue, though it appears that it will take a bit of work to revive the ones we've mis-initialized.
- ARL and Cornell also were busy, and now have their Ubiquitis and cell-modem SIM card, respectively, but I don't know their details.
ddn
minicom usbX
& to stop binary output
T to enter terminal mode
disconnect sonic head
D or & (I'm not sure) to get to "......" mode
x to break out
aq 9 to set a data rate that doesn't use 60Hz (10Hz in this case)
reconnect sonic head
^ to check that field 4–6 are similar
^ to stop this diagnostic mode
D to set data mode
& to start collecting data
Another day, another 2 towers. One crew did tse10 on the east side of the valley, another tse07 on the west side. What is important is that tse07 is our first solar-powered station and, as such, is the first to have continuous power. Thus, TODAY IS OUR FIRST DAY OF OPERATIONS!!! As part of the tse07 installation, we took a soil sample, which also should be the first to be used for Qsoil comparison.
Except for tnw07 (the 60m tower instrumented with DTU sensors), we have now installed everything that we "prepped" in December. Tonight, I started prepping another 4 towers.
In other news:
- Samortecnica gave us an update on their tower progress, including that the last 60m tower is now in place (the last 100m tower still needs to be started). ARL also gave us an update on their towers. The Google-Drive "Tower Status" has been updated. 33 of the project-total 52 towers in the main array are now built (and 18.5 of them are now instrumented).
- Santiago got a wireless router working in the ops center, so we now can connect DSMs directly to the tower network as we stage/prepare them.
- We now have access to some of the power drops and will try to get some of the AC towers going.
- ARL (and Jose Carlos, of course) helped Cornell drop off their equipment (primarily lidars) throughout the array.
Thus, a lot of activity is going on.
Similar to yesterday, all staff (except Santiago) went to the field to become familiar with a different type of tower set-up. Finished tse08. It should even be straightforward to get power here reasonably quickly. Transported all equipment to tse07 for tomorrow's deployment. Back in the ops center, mostly prepped tse10. Also, Santiago gave up on using the Cisco routers (even after a firmware update) and has fallen back to using a spare older-model router we had brought. This router is now being used in the ops trailer for primary connection to the tower network. He and Ted later picked up a second router for use in the ops center staging area.
The primary issue is that, in prepping tse07, we found 3 straight CSAT3s that all report "00". We ended up using one from Petra (OU), that worked just fine. Her shipping box had more foam than the standard CSAT3 case – possibly vibration in shipping broke all of our CSAT3s??? Hard to imagine. When the dust settles, I'll go back to looking at them, which may require shipping back to Utah 
. This experiment wasn't designed to have spare instruments, just redundancy in the measurements that are being made...
Rebecca and Sara are now here, and José Carlos is in the area. We'll all meet tomorrow AM.
First day for the entire crew in the field. Everyone came so that different perspectives could be shared about tower operations. All this was done at rsw04, which is now ready for power. We powered it up with a battery for about 30min to check out everything. (ARL now is using DTU's portable generator.)
Errors found and fixed at rsw04:
- eolupdate had reverted to the wrong serial ports for the sonic and licor
- licor DID changed to reflect height of 10m
- TP01 had been plugged into incorrect mote port (stupid me!)
- 15m cables were fine to get to the rad and soil motes
Back in the ops center:
- Finished prep of our next two towers: tse07 and tse08.
- Found a bad Tsoil probe (reporting all -0.001)
- Found that the PTB110 connector inside the EC100 was loose (just like VERTEX s15!)
- EC100 needed to be changed to binary mode
- Found that the EC100 had to be grounded both to the DSM and to the CSAT3A head, when using a long serial cable
- Changed to longer cables and configuration, since tse07 is now a 30m tower (picklist had shown 20m)
- One CSAT3 (older) seemed bad and another became bad during testing. We're hoping that it will act better deployed on a tower.
- One bad port found on tse08's DSM, so METEK was moved to a different port. I think this is the first bad port we've seen?
- Updated software on tnw07's 2 DSMs
- Reset 16 EC100 boxes to binary mode (needed for about half of these boxes – very strange)
- Prepped Ubiquiti's for tnw09, tnw11, and rne07, which we hope to get to on Wed.
Interacted a bit with both Samortecnica's 3-person crew and ARL's crew (saw Chris just before he left). Samortecnica was working on rsw05, 06, 07 today. ARL has the SW energy balance 10m tower up, though not completely connected; one 30m tower up; at least 3 other 30m towers are erect, but not yet telescoped to full height as their crew is adding instrumentation. So...progress is being made!
Ted and I continued prep/network tasks in the ops center. We also visited tse13 to rewire the power (and hopefully build the first leg of the WiFi network), but couldn't actually get AC power up since the fusebox was locked. Both 24V and 12V power supplies (barely) fit in the lower part of the DTU box along with an outdoor-rated power strip. This will have to be changed once DTU installs their "differential protection" circuit.
rsw04 should be ready to go, though I need to double-check the mechanicals and then pack. The other 4 towers still need software upgrades and systems checks. All sensors are prepped.
Ted was able to get radio pairing to the ops center access point using (multiple) frequencies in the range suggested by IMPA (to avoid possible interference with their radar). Now need to copy this configuration to the rest of the Ubiquiti gear.
The other NCAR staff dropped by to get oriented to the ops center as well.
Kurt, Laura, Andy, Dan, Lucas, Bob, Ted, and I arrived via Lisbon today. Ted and I worked at ops center from about 2 to 5:
- After much gnashing of teeth, Ted found that the cheap ISFS router that we had brought as a backup works fine between our LAN and the Telco's router. The more expensive Cisco routers do not work. Ted will buy another router tomorrow, but in the meantime, we now have a DSM in the ops center able to connect to the network (and execute "eolupdate"). This was done by temporarily running an ethernet cable through the cafe inner door to the ballroom – not a permanent solution.
- I unpacked the December airshipment and rearranged stuff a bit to make some space between INEGI, DTU, ARL, and Cornell stuff.
- I inventoried our prep work: We left tnw07, tse07, tse08, tse10, and rsw04 pretty much ready to go, though I want to "restage" all of them with our checklist. I got most of the way through the checklist for rsw04.
Tomorrow's tasks:
- Finish staging the above 5 towers
- Establish routers in the ops seatainer and in the ballroom.
- Check that all EC100s are set to binary.
- If time permits, update rest of tower nanobeam software (will need some for lidars and profilers as well)
I heard via Bill that Jose Carlos wanted me to meet someone at the 60m valley tower to test something at noon. When I finally found the tower (after getting some directions from Jose over the phone), I discovered it was the INEGI tower crew setting up tnw07. They had brought the two tnw07 DSMs and and 5 TRHs and wanted to install them on the tower along with the DTU box and Gill sonics. Even though I had not heard this would be happening, they have installed our equipment before, and it looked like they had found what they needed from the ops center and knew what they were doing. After some time watching them prep, they told me we did not have the right cable lengths and some other problems, so they were not going to install our equipment. So they asked me to take it all back.
I left the tub with the DSMs, booms, and TRH rack are all together under one of the DSM setup tables. I don't know if this site was ever staged by us in the ops center.
Mote 12 was the one that stopped working when initially installed at rsw02, but it got installed on the radiation stand at rsw04 to cap the soil sensor cables. I installed mote 8 in its place to bring it back to EOL for testing.
No progress on the rsw04 tower. The tower sections are still laying on the ground.
Mostly clean-up today, then Dan and I left to go home. Gary is still on site, nominally assisting ISS. We heard yesterday that our air shipment has been received in Porto. INEGI will deliver the two boxes to the ops center by the time we return in Jan.
rsw01: added Bulgin caps, but realized we needed Binder caps as well. Stole from motes we intended to use elsewhere.
rsw02: fired up and had 3 issues. Can't see GPS (on an Titan DSM – shouldn't "rs G" still work?). Seems to be plugged in with daughter board to front panel properly. RMY at 20m is spiking – about 1 in 10 samples is bad in the brisk winds we had at the time. They are detected (diagnostic flag is "A", not "0"), so can be removed, but still not what we want. This sensor had spikes when tested in the ops center as well, though they had gone away when I took it out of the shipping box. Likely this is a weak transducer. We could replace it, but eventually won't have any spares. Finally, mote#12 would start up, then die (no blinking LEDs). Replacing with a different mote worked fine. We think that this is the second mote this has happened to.
rsw03: brief stop to read Setra serial number (and get berries from the tree)
rsw04: installed 4-component radiometers and soil sensors at darkhorse, though tower is not yet up. This will require a 15m cable to the soil mote.
tse12: back to finish up from last night. Installed soil sensors. Raised the Ubiquiti to have better line of sight to rne02. Found that EC100 again was in ASCII mode, so had to drop the box to reprogram through the USB port. In general, we need to be more rigorous in our prep. (I think this will be my job in Jan.)
I've created a table of the sites in the EOL Perdigao web page, with some comments.
We left the ops center with 4? towers prepped, primarily the tnw07 60m tower for when INEGI returns.
A 2.8-tower day (all 20m), with us not quite finishing the third tower due to darkness. Completed rsw01 and rsw02 in the morning/early afternoon, then back to the ops center to reload and work on tse12. There are no more short towers built that we are aware of. We will return to rsw01 and rsw02 tomorrow to add some missing caps and install the radiometers and soil sensors at rsw04. (The darkhorse is already placed.) We also will return to tse12 to install its soil sensors and a few other odds-and-ends. Thus, there are now 6 tall towers and 6 short towers fully installed.
We'll also do some clean-up in the seatainer and ops center before Dan and I return to Lisbon for our flights home on Sunday. Gary will stay a bit longer to help out ISS.
Two more towers up today. Would have done 3, but forgot to completely prep rne02, which took some time (including breaking down two pallets of stuff in the ARL seatainer). Staged booms and sonics to rne01 and booms to rne02. Have now deployed the CSAT3A with and without an EC150 and RMY with the Samortecnica booms, and figured out systems for both.
We're finding different versions of RMY from the ARL shipment – some need "5BE" and others "178" to configure the output. There are 2 or 3 versions of interface circuit board, though all seem to need to be wired the same. Have found one RMY whose received (DSM to sensor) signal seems flakey, though it is working now.