Blog

We finally have our first short tower instrumented!  Samortecnica "topped off" several towers on the ne ridge, so we started on them.  They made a lot of progress today, so we now have from 4–9 short towers to work on during the next 3 days, and hopefully more will be built while we're working.  In the meantime, we've been preparing data systems and sensors, so we have most of these ready to go.  Now, we just have to get busy throwing them onto towers...

A few random details:

• Found that one of the old DSMs that had been prepared from old pieces had the ribbon cable going to Emerald ports 1-4 flipped.  Now DSM ports 5–8 (Emerald ports 1-4) work.
• This same DSM (rsw04) has an old Emerald board that needed to have jumpers changed to configure it to RS422 for the EC100 box.
• Found that we need an extra short 1/4-20 screw on the CSAT3A booms to connect the ground straps (that we finally found in a tub today).
• We're finding lots of EC100 boxes that are not set to binary mode.  This can be done easily by using minicom to connect to the EC100 and typing "+", "Y", "X", "Y".  We found one that we thought we configured, but had reverted.  Perhaps we forgot the "X" step.
• I finally started cooking the first soil sample that I took 2 days ago.
• Since we don't yet have AC power at the towers, we are using DTU's portable generator to power up the towers to check that things are installed correctly.
• We're still waiting for the ops center router to be reconfigured to allow us to attach our network.  (Both the ops center and the towers use the 192.168.1.X subnet.  In hindsight, it would have been better to change our subnet to something more unique.  It would be a lot of work at this point for us to change every DSM and Ubiquiti device.
• We will change the configuration .xml to default to simple cal_files so that we'll be able to look at data on the DSMs without having to create every cal_file now.
• Four people from IMPA (the Portuguese meteorological service) visited the site and ops center today.
• Found that we can buy fruit from the food truck that comes to the Alvaiade square periodically!

A group from IMPA visited today and asked whether our WiFi network would interfere with the two national weather radars.  It is true that their frequencies of 5.63 and 5.64 GHz are within the "DFS" hopping table of the Ubiquitis.  However, Bill Brown judges that with the directed beams of the Ubiquitis, the lower power levels, the distance to the nearest radar being 120km, the scanning of the radar, and the frequency hopping of the Ubiquitis (that will hop off of the radar frequency if it detects the interference), the chance of interference is quite low.  

If necessary, we could deselect the radar frequencies from our hopping table, but this would be a lot of work, probably requiring an in-person visit to each site.

Thus, we are going to proceed by leaving the Ubiquitis in "auto" frequency selection mode for now.

This taken during the initial installation hole at rne01 on Monday about 4pm.  (However, we aren't taking data at this site yet, so it doesn't really matter when it was.)

I didn't get around to weighing until just now (2 days later) so the Qsoil should be lower than the real value.  However, again, we don't have measurements to compare to so this doesn't really matter.  Nevertheless, it is useful to process this sample to get a bulk density measurement...

Tray (tare): 8.2g

Wet+ring+tray: 172.2g

Great weather, and did get out a little, but still limited by the availability of completed short towers...

• Samortecnica did drop off about 10 booms to allow us to work.  They didn't have backplates for the clamps, but INEGI thought we could borrow from the DTU stockpile for the moment...
• The municipality truck came and was perfect for moving the darkhorses.  Nevertheless, their time was limited, so we just had them drop them along the valley road.  Gary and I transported them to their proper locations (rne01, v04, and rsw04) using the pickup for the up-the-hill run and set them up.  
  • At v04, we couldn't locate the tower, so this darkhorse will have to be moved.
  • At rne02, the darkhorse is south of the tower at the maximum cable distance.  It is above low shrubbery, but there are higher pine trees all around.  I <think> the direct sun will clear many of these trees, but there undoubtedly will be times that the radiometers are shaded.  This is typical of the local environment.  I judged that the immediate plot under the darkhorse was similar to the one I selected for the soil sensors that are about 10m away.
  • At rsw04, I chose a location NORTH of the tower, which of course puts the tower shadow across the sensors for some part of the day.  The alternatives were deploying down the slope, or adjacent to a power pole, which I thought would have an even larger effect on the radiation.  Close to the tower, the land cover is greatly disturbed by tower construction operations.  Although we raised it to the maximum possible height, this darkhorse is only just above the tops of shrubs and has a view of large rocks on the surface.  I actually thought that a feature of this location was that rocks blocked the radiative footprint of the nearby road.  All of this seems to me typical of the ridgetop environment.
• In the meantime, Dan worked with the INEGI crew to instrument rsw03 (60m), which is now done.
• After lunch, we participated in the monthly telecon, then had a good discussion with João from Porto about data organization and access.  He now understands our data products and how these will flow from the sensors.
• We learned that the rsw short towers will soon be ready at their final heights (12 and 21m), so we began prepping them.  Among other things, the Li7500 3A fuse problem bit us again, so the prep is still ongoing.  At this point, instrumentation and data systems for rne01, rne02, rne03, tse10, and tse12 are ready to go, and we expect rsw01 and rsw02 to be ready shortly.
• With a bit of spare time, we again picked up the pyrgeometer problem.  Steve Semmer pointed us to a few tests that eventually resulted in him sending us new code to load into the pyrgeometer wisard board.  This worked for the one sensor we tried, so Steve will prepare new code for the remaining 5 sensors.

Lots of activity today, in no particular order:

• Telecom guy came and (after more than an hour) got our bandwidth to 48 Mbps up; 48 Mbps down.  This is close enough to the 50/50 we paid for.  Yea!
• We requested from the telecom guy a set of port forwards, but this has to be done back at the office.  A workorder has been submitted, but for now we can't get our router on the network.
• We tried a bunch to pair tse13 to ops with no success.  Dan climbed and reoriented 2 of the antennae.  Later testing in the ops center showed that we need to set the station adaptors to SA PTMP, rather than SA PTP.  All of the SAs programmed so far will have to be changed, but this isn't a big deal.  (P.S.: By the end of the day, had reprogrammed all except the uncommitted tseXX and rswXX (and still have some vXX and tnwXX to program completely))
• The Metek SODAR was delivered to the ops center today.  (This should be in an ISS logbook entry!)
• 10 Gill sonics purchased by U. Porto were delivered today.  I presume that these will be used on the 60m towers connected to the DTU data system.
• We prepped for most of the towers expected to be ready tomorrow.
• We installed a set of soil sensors at rne01.  The ground was easy to work with, going only to 5cm.
• And last, but not least, we now have access to the circuit breaker panel in the ops center where we can turn on the ballroom lights – that makes a HUGE difference in nighttime lighting!  (It is interesting that cold is much easier to bear if it isn't dark and cold.) 

For tomorrow, we are expecting:

• Samortecnica will deliver our booms – needed to add the DTU and ARL sonics to the short masts.
• Samortecnica will add sections to the ridge towers to get them to 12 and 21m.  Then we can start instrumenting them!
• Samortecnica will pour concrete for the valley short towers.  The idea is to have some of the solar-powered towers up before we go.
• Samortecnica will also work on the remaining 60m towers.
• The municipality will provide a longer-bed truck that ISS will use to move profiler stuff to the profiler site in the morning.  We will use it to transport one darkhorse to each ridge and valley.
• The INEGI tower crew will return to instrument at least one more 60m tower.

A somewhat frustrating day.  We started with the plan to instrument the short towers that have been erected.  (Today, we confirmed that tse12 is built and tse10 has two (unguyed) sections in the air.)   Already, though, two of those towers need booms from Samortecnica that we do not yet have, so we wouldn't have been able to complete this job.  Nevertheless, I gathered what we thought we needed from the seatainer (requiring two pickup trips) and started kitting these 5 towers.  Then, we ran into problems:

• The EC100 boxes (for both the CSAT3As and EC150s), were not set to binary mode.  It took quite a while to figure out that this can be done by connecting the USB port and, via minicom, setting "+", then "X" (with some yesses to confirm) to establish binary mode.
• Even then, the CSAT3As show errors that require the grounding solution we had found in FLAB.  Unfortunately, we can't seem to locate the straps that Rick made.  We have some braid to make up more, but it would be nice to locate the prebuilt ones!
• There were lots of nan's in the data.  These mostly are due to missing cal_files, so we had to make about 8 so far.
• 4-component radiometer values were coming in on the wrong address.  Changed sensor_catalog to set the correct range of I2C addresses.
• Even so, we never got Rlw data – there are no Wisard messages getting through the motes from the K&Z pyrgeometers.  This result was consistent through a change of pygs and motes.  Semmer suggested looking at the Binder wiring, but it seemed to be the same as for the psp's.  We can try to build a test cable, based on the Pi hat USB console module since there apparently are TTL level signals out of the radiometers.
• We noticed that we forgot to bring single-channel wisard boards for the wetness sensors.  We only need 3, for the 3 darkhorses.
• We also will have to figure out how to transport darkhorses with a short-bed pickup!

In the middle of all this diagnosis, moderate rain started up with low cloud covering the ridges, so it didn't seem to be a nice day for driving on the ridge or climbing anyway.  Rain is forecast to continue tomorrow.

We also slipped in a slight reorientation of the base WiFi access point.  I've calculated that it needs to point up by 230m across 2800m, or 4.7 degrees, and the mount only allows pitching up by 3 degrees.  However, by swapping the top and bottom brackets, we can tilt up up to 10 degrees.  This angle is now set to ~4.5 degrees.

We've decided to take our first (and probably only) day off tomorrow.  We assume that Monday will be busy preparing rsw02 for INEGI to instrument, while starting with rne01+ ourselves.  We also need to revisit tse13 to replace the bad POE cable (I verified that it had a short) that Per removed.

• 3x single-channel boards for 4-component wetness
• Do we have short binder-bulgin connectors for NR01s?  (We have 4-pin cables for motes.)
• Binder sensor test cable(s)

Ops->Foz via Sarnadinha: 10min

Ops->Foz via Vale do Cobrao: 7min

DTU team left the site last night.  INEGI is home for the weekend.  Gary arrived this afternoon, so we now have our 3-person NCAR team.  Dan is moving to Castelo Branco to have more exciting night life (than hanging around with me...)

Work today:

• Finished mote fuse replacement
• Finished configuration of all WiFi routers and configuration of WiFi station adaptors for tneXX.  Also assembled the 60-degree antennae with their AirPrisms.  Now just plowing through the rest of the station adaptors.  Have updated Ted's networking page with color codes to indicate the status of each adaptor
• Residual clean up/organization of the ballroom
• Visited tne09 and Dan climbed to turn on tne09t's power switch.  Everything immediately came up, including the Ubiquiti.  Increased Bluetooth console power and can now pair from the ground (though you have to stand about 20m away from the base of the tower).  Dan got a great photo of Vale do Cobrão.  Measured the black rope as 200'.
• At José Carlos' request, drove to ne ridge and confirmed that rne01, rne02, rne03 are all in place, though they are 9, 18, and 9m high, respectively.  We really would have liked the full 10, 20, 10m heights...

P.S. Back at the hotel, prepped all rneXX and rswXX adaptors as well.  Now just have vXX and tnwXX to do.

It was a national holiday today, so INEGI was back in Portugal and won't work tomorrow or the weekend as well.  We assume that Samortecnica also was not working today, but don't know their Friday/weekend plans.

Dan helped DTU instrument rnw06 (60m).  Off-and-on rain all day made this an annoying task for them, but they were motivated by wanting to go home tomorrow.  (They said that, living in Denmark, they are accustomed to working in rain.)

I started the day with a drive out to the east energy balance/profiler candidate site.  Access at the site is actually easier than I thought.  However, with trees around (estimated at ca. 11m high), it would be better to have a taller flux tower.  (We realize that ARL just packed 10m towers for these sites.)

I next got a lesson in using the MultiScanner, that we tested at tse09 (the valley 100m tower).  They had a simple method for georeferencing the MS, but used GPS and remote-connect equipment that we don't have.  (Hmmm)  Afterwards, DTU decided that the most time efficient procedure is to scan just a few points on each boom, rather than autoscanning either the entire tower, or even just a measurement level.

While at tse09, we powered it up with the generator.  Per enabled the UDP ports and the DSM saw all of these data.  Unfortunately, it does appear that the top DSM is switched off, since I couldn't see the blue light on the power switch when the power was supposed to be on.  This is on our task list.

Back at ops, (mostly Dan) and I continued to replace fuses – getting close.  I got a lesson on configuring Ubiquitis.  I tested the METEK connection, using the cabling that DTU made.  Found that our DSMs aren't preconfigured for RS422 on these ports, but this is easy to do here as we prep these boxes.  We still don't have the Samortecnica booms for the METEKS or RMYoungs, that I'll have to ask José Carlos about.

The ops center trailer is mostly set up.  It has power (of sorts), and cabled connections to both the ops center ethernet connection (via fiber) and the WiFi access point looking towards the ridge.  There is a desk&chair, Ted's router, a 220V power strip, DTU's data logging computer, and DTU's 13Tb backup disk (unconnected) in the trailer.  As mentioned yesterday, we are awaiting a network technician to arrive Monday (I think I said Fri previously) to diagnose low bandwidth on the fiber.  Also, Per only saw one LED (out of 4) on the Ubiquiti station adaptor when tse13 was powered up, and suspects that the access point antenna will need to be moved.  Finally, Per noted that port 1 on Ted's router appeared to be dead, so he plugged the Ubiquiti WiFi antenna into the port 2.  Strange.

Per has configured all of the access point Ubiquitis and upgraded their firmware to 7.2.4 (since some of the devices were old and he wanted everything to run the same version.  He left most of the 16db station adaptors for us to configure.

rsw03 is mostly prepped, for when INEGI staff return on Monday.

Two more towers (both 60m) instrumented today, again Dan with the INEGI crew and me with the DTU crew.  DLR and Enercon folks also were on site.  Started a bit late since I had a bad 24V power supply that failed only with load – rather annoying – that took a while to diagnose.  Another issue was that all my boom clamps were too loose by about 1.5mm.  We ended up stealing DTU's duct tape that was wrapped around the tower uprights as filler.  Far from an elegant solution, and used about 1/2 roll of duct tape just for one tower – we'll need to get more.  The smaller uprights on the 60m towers (rather than the 100's) allowed the Ubiquiti hose clamps to fit, so we could use the uprights with no modification.  Climbers were on the tower from ~10AM - 1:30PM to do everything.  (Yesterday, it was more like 9AM-4PM for the 100m.)  

Two more notes about  tse11: Samortecnica appears to have installed it straight, but not level.  My cell phone measured 2–3 degrees off (top displaced 2–3 m at the top of a 60m tower).  Not very good.  Also, an 11m high tree adjacent to the tower forced the 10m level to be moved up to about 12m. 

Climbers were back down for lunch, which included cake to celebrate Per's birthday!

Had enough time later in the day for us and DTU to prep riSW06.  That should go fast tomorrow, though there is extra DTU stuff.  INEGI is forced to take tomorrow off since Conception Day is a national holiday.

So the status is:

• tse09 (100m in the valley): Ready, but DSMs probably switched off
• tse11 (60m on slope): Ready
• tse13 (100m on the ridge): Ready, but only one WiFi access point (to WSW) connected.  Other needs POE cable.
• tnw10 (60m on slope): Ready
• rsw06 (60m north of wind turbine): Prepped for instrumentation tomorrow.
• rsw03 (60m south of turbine): Partly prepped.  INEGI should instrument Friday.
• rne01, rne02, rne03, tse10, tse12: Expected to be built by Friday; we'll instrument over the weekend
• rsw01, rsw02, rsw04, rsw06, rsw07, rsw08: Expected to be built early next week; we'll instrument when available

None of these towers yet have power, though many lines and poles are installed, so we do not expect to leave a functioning network.  The network in the ops center/trailer is now up, though connection speeds are very slow.  A MEO technician is scheduled to visit tomorrow to work on it.

Little tasks:

• Buy the stuff on our needs list.
• Visit east energy balance/NCAS site
• Diagnose bad POE cable
• Finish stuffing mote fuses

(to be updated)

Stuff that we should have, that will require a run into Castelo Branco:

• plumber's putty
• quick-disconnect connectors
• larger hose clamps
  
  

Other things to remember:

• leave DSM power turned on (easy to forget on tall towers)
• test DSMs with WiFi adaptors (to ensure we have cables and can power everything)
• set binary mode on EC100 boxes (done using direct USB cable and minicom)
  
  
  

As planned, crews split to instrument the two 100m towers.  By the end of a long day (working until dark), both towers were complete with both DTU and NCAR instrumentation.  Kudos to the INEGI and DTU tower crews!!!  

At the ridge, we powered up the tower using a portable generator.  Unfortunately, the top DSM didn't power up.  Per Hansen zoomed up the tower and found a bad connection on the input 24V line (we need to remove these screw terminals from the links that have to be made on the tower!).  This got the DSM and hardwire network to come up, but the Ubiquitis were all dead.  Per was able to debug the box to find either a bad POE injector cable or DC splitter (Y) cable that loaded down the power.  He removed the antenna pointing NNW and got the other two Ubiquitis to come up.  After this, I was able to Bluetooth to the bottom DSM (tse13b), ssh over the copper to tse13t, and ping to the ops center antenna (192.168.1.2).  Kudos to Per for debugging a system he's never seen before (and that <we> don't have much experience with either).

I wasn't able to make a Bluetooth connection to the top DSM (though Per's body may have blocked it).  Also I got "destination unreachable" when trying to ping the ops center directly from tse13b. Hmmm. 

Note that there should actually be some valid data on the USB stick for about an hour when the generator had things going from this tower!  Perhaps even from the DTU system as well, assuming that we were receiving its UDP data.

Previously, back at the ops center, the ops trailer was powered up and networking connected to it.  Also, Dan replaced about half of the mote fuses.

As expected, things were busy today with 7 DTU and 5 INEGI staff arriving, though still had time to a) pick up Dan (then put him to work!), b) have a nice lunch, and c) have a nice dinner...

Since one of you has asked, weather has been beautiful (as predicted).  Fog this morning, but burned off quickly.  High temps were about 16C and partly cloudy high clouds made for great views, including seeing a bit of snow on the highest peak in Portugal.  Still about 14C in the ops center, so somewhat chilly.

DSM configuration

Started by showing our set-up for tse13 to DTU and INEGI.  INEGI is concerned about humidity getting through the Gordon hole clamp and condensing and will supply some desiccant packs (though they of course will saturate eventually).  We also discussed using more plumbers putty to seal around the clamp, but have yet to get some.

I am most concerned about connecting to the screw terminal blocks on the 24->12 DC-DC converter, which has been a bit problematic in the ops center and would be even harder at 70m.  INEGI is less concerned, but I'm still considering other options.

Later in the day, packed up tse13 (the ridge 100m tower) stuff and built up tse09 (valley 100m).  In prep for this, Dan and I unloaded all DSMs and the second 1/4 of TRHs from the seatainer.

Network

Per took over the job of configuring the Ubiquiti network modules and I supervised mounting of the WiFi antenna at the ops center (that INEGI staff ended up doing).  It looks good, though still isn't powered up since we don't have power yet to the ops trailer.  Per also dug up a +12V power cube with a European AC plug, so we'll be able to bring up Ted's router as soon as power is available.  José Carlos will contact the network utility to see if bandwidth can be improved (to the 50 Mbps specification they paid for).  José also will arrange for power to the ops trailer soon.

Mechanics

The truck with DTU equipment arrived this morning and was downloaded with a local forklift.  I helped drive two of their boxes to the valley 100m tower, while DTU transported two more to the NE ridge 100m tower.  DTU went back to this ridge tower later in the day to explain set-up of their sensors to INEGI.  The plan for tomorrow is for DTU climbers to install everything (including our stuff) on the ridge tower while INEGI installs the DTU and our equipment on the valley tower.   Dan will advise on our stuff in the valley while I will advise on the ridge.

We note that no towers yet have power, but DTU has a generator we can use to bring things up to check that cabling is correct.  (Though it isn't clear when we would move it between the ridge and valley sites.)

Also, there still are no short towers erected, much less power, so Dan and I have no incentive to split off and do other things.

Other towers

Samortecnica appears to have finished at least one more 60m tower today.  If so, that leaves two more 60m and one more 100m to go.

We heard today that one "valued" member of ARL's team suffered a serious medical issue that will delay them by at least 5 weeks.  They now plan to set up their equipment on 9 Jan. (along with our primary set up).

Other prep

I slipped in reprogramming of all TP01s this morning, as well as replacing fuses on another few motes (much easier with Per's <perfect> tweezers).

Got a late start due to my hotel hosts wanting to show me around (the supermarket, the Vila Velha castle, griffins flying overhead, etc.).  It was great, but I didn't get started working until afternoon (again!).

Started by changing out mote fuses.  Only did 5 of the boxes, since using the soldering iron here without a visor in poor light was difficult.  I'm going to try tonight in my hotel room where lighting is much better.  A visor is in the air shipment that should arrive later this week.

Went to the seatainer, with the thought of pulling tse13's sensors and DSMs.  This turned into an hour-long project to remove most of the upper deck and unpalletize the DSMs.  I think there is pretty good access to almost everything in the seatainer at this point.

Next, assembled everything to go on tse13: 2 DSMs, 7 TRHs with booms, clamps, cables, 2 motes, power supplies & connections (including the weird 24V through 500' of cable and, of course, the rubbermaid jugs), network cables, etc..  Got a scare when the TRHs didn't come up, then realized that the sensors were packed away elsewhere.  Found them and all read about 14C – no wonder I've been chilly working in here!  By now everything is up except the DSM Bluetooth console.  (Using the USB test console module worked fine.)  Have noticed that one of the motes has a yellow (labeled orange) LED (near the middle of the board by RP11) that is solid on when first powered up, but goes away after a power cycle (including pio 7 0/1).  Seems to work when it is on anyway.  Also, the 24->12 DC-DC on the top DSM shut itself off the first time, but now has stayed on for more than an hour.  Scary.  One cute thing is that the GPS's get good reception in the ballroom!  

The DSM assembly took a surprising amount of time, though some of that is the usual finding things in boxes and remembering how everything goes.  I could imagine it going twice as fast in the future (and of course this is just about the hardest tower).  Also, having another pair of hands would have sped things up .

I am bothered by the Bluetooth.  My Mac can connect, though neither the tablet nor my phone could.  Probably a pairing issue, but I had hoped we had figured that out before.

Now leaving the ops center for the day.  Both the INEGI and DTU crowd begin work tomorrow (along with Dan), so things will get busy!

P.S. Light and heat definitely are better in my room, but 91 steps are a lot to do carrying a tub full of motes.  DTU will lend me a tweezers (better than my Swiss Army Knife's) which also should make the mote job easier.