Blog

Ops->Foz via Sarnadinha: 10min

Ops->Foz via Vale do Cobrao: 7min

DTU team left the site last night.  INEGI is home for the weekend.  Gary arrived this afternoon, so we now have our 3-person NCAR team.  Dan is moving to Castelo Branco to have more exciting night life (than hanging around with me...)

Work today:

• Finished mote fuse replacement
• Finished configuration of all WiFi routers and configuration of WiFi station adaptors for tneXX.  Also assembled the 60-degree antennae with their AirPrisms.  Now just plowing through the rest of the station adaptors.  Have updated Ted's networking page with color codes to indicate the status of each adaptor
• Residual clean up/organization of the ballroom
• Visited tne09 and Dan climbed to turn on tne09t's power switch.  Everything immediately came up, including the Ubiquiti.  Increased Bluetooth console power and can now pair from the ground (though you have to stand about 20m away from the base of the tower).  Dan got a great photo of Vale do Cobrão.  Measured the black rope as 200'.
• At José Carlos' request, drove to ne ridge and confirmed that rne01, rne02, rne03 are all in place, though they are 9, 18, and 9m high, respectively.  We really would have liked the full 10, 20, 10m heights...

P.S. Back at the hotel, prepped all rneXX and rswXX adaptors as well.  Now just have vXX and tnwXX to do.

It was a national holiday today, so INEGI was back in Portugal and won't work tomorrow or the weekend as well.  We assume that Samortecnica also was not working today, but don't know their Friday/weekend plans.

Dan helped DTU instrument rnw06 (60m).  Off-and-on rain all day made this an annoying task for them, but they were motivated by wanting to go home tomorrow.  (They said that, living in Denmark, they are accustomed to working in rain.)

I started the day with a drive out to the east energy balance/profiler candidate site.  Access at the site is actually easier than I thought.  However, with trees around (estimated at ca. 11m high), it would be better to have a taller flux tower.  (We realize that ARL just packed 10m towers for these sites.)

I next got a lesson in using the MultiScanner, that we tested at tse09 (the valley 100m tower).  They had a simple method for georeferencing the MS, but used GPS and remote-connect equipment that we don't have.  (Hmmm)  Afterwards, DTU decided that the most time efficient procedure is to scan just a few points on each boom, rather than autoscanning either the entire tower, or even just a measurement level.

While at tse09, we powered it up with the generator.  Per enabled the UDP ports and the DSM saw all of these data.  Unfortunately, it does appear that the top DSM is switched off, since I couldn't see the blue light on the power switch when the power was supposed to be on.  This is on our task list.

Back at ops, (mostly Dan) and I continued to replace fuses – getting close.  I got a lesson on configuring Ubiquitis.  I tested the METEK connection, using the cabling that DTU made.  Found that our DSMs aren't preconfigured for RS422 on these ports, but this is easy to do here as we prep these boxes.  We still don't have the Samortecnica booms for the METEKS or RMYoungs, that I'll have to ask José Carlos about.

The ops center trailer is mostly set up.  It has power (of sorts), and cabled connections to both the ops center ethernet connection (via fiber) and the WiFi access point looking towards the ridge.  There is a desk&chair, Ted's router, a 220V power strip, DTU's data logging computer, and DTU's 13Tb backup disk (unconnected) in the trailer.  As mentioned yesterday, we are awaiting a network technician to arrive Monday (I think I said Fri previously) to diagnose low bandwidth on the fiber.  Also, Per only saw one LED (out of 4) on the Ubiquiti station adaptor when tse13 was powered up, and suspects that the access point antenna will need to be moved.  Finally, Per noted that port 1 on Ted's router appeared to be dead, so he plugged the Ubiquiti WiFi antenna into the port 2.  Strange.

Per has configured all of the access point Ubiquitis and upgraded their firmware to 7.2.4 (since some of the devices were old and he wanted everything to run the same version.  He left most of the 16db station adaptors for us to configure.

rsw03 is mostly prepped, for when INEGI staff return on Monday.

Two more towers (both 60m) instrumented today, again Dan with the INEGI crew and me with the DTU crew.  DLR and Enercon folks also were on site.  Started a bit late since I had a bad 24V power supply that failed only with load – rather annoying – that took a while to diagnose.  Another issue was that all my boom clamps were too loose by about 1.5mm.  We ended up stealing DTU's duct tape that was wrapped around the tower uprights as filler.  Far from an elegant solution, and used about 1/2 roll of duct tape just for one tower – we'll need to get more.  The smaller uprights on the 60m towers (rather than the 100's) allowed the Ubiquiti hose clamps to fit, so we could use the uprights with no modification.  Climbers were on the tower from ~10AM - 1:30PM to do everything.  (Yesterday, it was more like 9AM-4PM for the 100m.)  

Two more notes about  tse11: Samortecnica appears to have installed it straight, but not level.  My cell phone measured 2–3 degrees off (top displaced 2–3 m at the top of a 60m tower).  Not very good.  Also, an 11m high tree adjacent to the tower forced the 10m level to be moved up to about 12m. 

Climbers were back down for lunch, which included cake to celebrate Per's birthday!

Had enough time later in the day for us and DTU to prep riSW06.  That should go fast tomorrow, though there is extra DTU stuff.  INEGI is forced to take tomorrow off since Conception Day is a national holiday.

So the status is:

• tse09 (100m in the valley): Ready, but DSMs probably switched off
• tse11 (60m on slope): Ready
• tse13 (100m on the ridge): Ready, but only one WiFi access point (to WSW) connected.  Other needs POE cable.
• tnw10 (60m on slope): Ready
• rsw06 (60m north of wind turbine): Prepped for instrumentation tomorrow.
• rsw03 (60m south of turbine): Partly prepped.  INEGI should instrument Friday.
• rne01, rne02, rne03, tse10, tse12: Expected to be built by Friday; we'll instrument over the weekend
• rsw01, rsw02, rsw04, rsw06, rsw07, rsw08: Expected to be built early next week; we'll instrument when available

None of these towers yet have power, though many lines and poles are installed, so we do not expect to leave a functioning network.  The network in the ops center/trailer is now up, though connection speeds are very slow.  A MEO technician is scheduled to visit tomorrow to work on it.

Little tasks:

• Buy the stuff on our needs list.
• Visit east energy balance/NCAS site
• Diagnose bad POE cable
• Finish stuffing mote fuses

(to be updated)

Stuff that we should have, that will require a run into Castelo Branco:

• plumber's putty
• quick-disconnect connectors
• larger hose clamps
  
  

Other things to remember:

• leave DSM power turned on (easy to forget on tall towers)
• test DSMs with WiFi adaptors (to ensure we have cables and can power everything)
• set binary mode on EC100 boxes (done using direct USB cable and minicom)
  
  
  

As planned, crews split to instrument the two 100m towers.  By the end of a long day (working until dark), both towers were complete with both DTU and NCAR instrumentation.  Kudos to the INEGI and DTU tower crews!!!  

At the ridge, we powered up the tower using a portable generator.  Unfortunately, the top DSM didn't power up.  Per Hansen zoomed up the tower and found a bad connection on the input 24V line (we need to remove these screw terminals from the links that have to be made on the tower!).  This got the DSM and hardwire network to come up, but the Ubiquitis were all dead.  Per was able to debug the box to find either a bad POE injector cable or DC splitter (Y) cable that loaded down the power.  He removed the antenna pointing NNW and got the other two Ubiquitis to come up.  After this, I was able to Bluetooth to the bottom DSM (tse13b), ssh over the copper to tse13t, and ping to the ops center antenna (192.168.1.2).  Kudos to Per for debugging a system he's never seen before (and that <we> don't have much experience with either).

I wasn't able to make a Bluetooth connection to the top DSM (though Per's body may have blocked it).  Also I got "destination unreachable" when trying to ping the ops center directly from tse13b. Hmmm. 

Note that there should actually be some valid data on the USB stick for about an hour when the generator had things going from this tower!  Perhaps even from the DTU system as well, assuming that we were receiving its UDP data.

Previously, back at the ops center, the ops trailer was powered up and networking connected to it.  Also, Dan replaced about half of the mote fuses.

As expected, things were busy today with 7 DTU and 5 INEGI staff arriving, though still had time to a) pick up Dan (then put him to work!), b) have a nice lunch, and c) have a nice dinner...

Since one of you has asked, weather has been beautiful (as predicted).  Fog this morning, but burned off quickly.  High temps were about 16C and partly cloudy high clouds made for great views, including seeing a bit of snow on the highest peak in Portugal.  Still about 14C in the ops center, so somewhat chilly.

DSM configuration

Started by showing our set-up for tse13 to DTU and INEGI.  INEGI is concerned about humidity getting through the Gordon hole clamp and condensing and will supply some desiccant packs (though they of course will saturate eventually).  We also discussed using more plumbers putty to seal around the clamp, but have yet to get some.

I am most concerned about connecting to the screw terminal blocks on the 24->12 DC-DC converter, which has been a bit problematic in the ops center and would be even harder at 70m.  INEGI is less concerned, but I'm still considering other options.

Later in the day, packed up tse13 (the ridge 100m tower) stuff and built up tse09 (valley 100m).  In prep for this, Dan and I unloaded all DSMs and the second 1/4 of TRHs from the seatainer.

Network

Per took over the job of configuring the Ubiquiti network modules and I supervised mounting of the WiFi antenna at the ops center (that INEGI staff ended up doing).  It looks good, though still isn't powered up since we don't have power yet to the ops trailer.  Per also dug up a +12V power cube with a European AC plug, so we'll be able to bring up Ted's router as soon as power is available.  José Carlos will contact the network utility to see if bandwidth can be improved (to the 50 Mbps specification they paid for).  José also will arrange for power to the ops trailer soon.

Mechanics

The truck with DTU equipment arrived this morning and was downloaded with a local forklift.  I helped drive two of their boxes to the valley 100m tower, while DTU transported two more to the NE ridge 100m tower.  DTU went back to this ridge tower later in the day to explain set-up of their sensors to INEGI.  The plan for tomorrow is for DTU climbers to install everything (including our stuff) on the ridge tower while INEGI installs the DTU and our equipment on the valley tower.   Dan will advise on our stuff in the valley while I will advise on the ridge.

We note that no towers yet have power, but DTU has a generator we can use to bring things up to check that cabling is correct.  (Though it isn't clear when we would move it between the ridge and valley sites.)

Also, there still are no short towers erected, much less power, so Dan and I have no incentive to split off and do other things.

Other towers

Samortecnica appears to have finished at least one more 60m tower today.  If so, that leaves two more 60m and one more 100m to go.

We heard today that one "valued" member of ARL's team suffered a serious medical issue that will delay them by at least 5 weeks.  They now plan to set up their equipment on 9 Jan. (along with our primary set up).

Other prep

I slipped in reprogramming of all TP01s this morning, as well as replacing fuses on another few motes (much easier with Per's <perfect> tweezers).

Got a late start due to my hotel hosts wanting to show me around (the supermarket, the Vila Velha castle, griffins flying overhead, etc.).  It was great, but I didn't get started working until afternoon (again!).

Started by changing out mote fuses.  Only did 5 of the boxes, since using the soldering iron here without a visor in poor light was difficult.  I'm going to try tonight in my hotel room where lighting is much better.  A visor is in the air shipment that should arrive later this week.

Went to the seatainer, with the thought of pulling tse13's sensors and DSMs.  This turned into an hour-long project to remove most of the upper deck and unpalletize the DSMs.  I think there is pretty good access to almost everything in the seatainer at this point.

Next, assembled everything to go on tse13: 2 DSMs, 7 TRHs with booms, clamps, cables, 2 motes, power supplies & connections (including the weird 24V through 500' of cable and, of course, the rubbermaid jugs), network cables, etc..  Got a scare when the TRHs didn't come up, then realized that the sensors were packed away elsewhere.  Found them and all read about 14C – no wonder I've been chilly working in here!  By now everything is up except the DSM Bluetooth console.  (Using the USB test console module worked fine.)  Have noticed that one of the motes has a yellow (labeled orange) LED (near the middle of the board by RP11) that is solid on when first powered up, but goes away after a power cycle (including pio 7 0/1).  Seems to work when it is on anyway.  Also, the 24->12 DC-DC on the top DSM shut itself off the first time, but now has stayed on for more than an hour.  Scary.  One cute thing is that the GPS's get good reception in the ballroom!  

The DSM assembly took a surprising amount of time, though some of that is the usual finding things in boxes and remembering how everything goes.  I could imagine it going twice as fast in the future (and of course this is just about the hardest tower).  Also, having another pair of hands would have sped things up .

I am bothered by the Bluetooth.  My Mac can connect, though neither the tablet nor my phone could.  Probably a pairing issue, but I had hoped we had figured that out before.

Now leaving the ops center for the day.  Both the INEGI and DTU crowd begin work tomorrow (along with Dan), so things will get busy!

P.S. Light and heat definitely are better in my room, but 91 steps are a lot to do carrying a tub full of motes.  DTU will lend me a tweezers (better than my Swiss Army Knife's) which also should make the mote job easier.

Having worked in the ops center this afternoon, realized that we need:

• trash can
• welcome mat (for dirty shoes)
• work lights
• European power strips (at least 2), and possibly a cube tap

Obviously, these can all be obtained locally.

Also: marker board

First, hardwired Mac to router.  Immediately came up and assigned 192.168.1.33 and set up routing.

• whatsmyip.com reports: 62.28.221.142, with ISP Meo
• speed test reported: 10ms ping; 14.5Mbps download; 5.2Mbps upload

Second, connected via WiFi, selecting "CSCR Alvaiade".  Again, immediately came up.

• whatsmyip.com still reported 62.28.221.142.
• speed test reported: 14ms ping; 13.8Mbps download; 7.9Mbps upload (strange that it is higher than when direct-connected!)

So...easy to connect, but pretty low bandwidth.  Since WiFi is reasonably peppy, we could use this to connect the trailer to the ops center connection, but then we'd need a WiFi station adaptor and still have to run a power cable to the trailer.

Couldn't try Ted's router since it came with a 110V plug. I have one adaptor, but there weren't any plugs available by the router in the cafe anyway.  Will wait to try more until we have the network installed in the trailer (which still needs power and thus will wait until tomorrow).

Travel:

• Arrived from a nice flight via Newark, landing at 0810.  Waited for luggage.
• A nice guy at Hertz called for a driver to pick me up at the airport location and take me to the Hertz corporate location, where I rented a half-bed pickup (white!)
• Drove to Brico Depot for a few items (my GPS wouldn't navigate me, but there were billboards at every step of the way).
• Drove to site, arrived at seatainer at 12:20.  Jose Carlos and Jose Palma arrived about 13:20.

Seatainer:

• Everything looked perfect.  Some of the tub walls had collapsed a bit, so some ratchet straps weren't tight, but not a big deal.  4 pickup loads later, everything was in the ops center.  Just removed tubs and a few other boxes.  Everything else left in seatainer for now.

Ops Center:

• Got (one) key for the ops center ballroom, now on my keyring.
• We have use of the entire ballroom.  Interior doors are unlocked to give us access to everyplace except the cafe.  (Except that they have left the cafe interior door also unlocked to give me access to the networking connection tomorrow.)  Only wall sconce lights turn on from inside ballroom.  The overhead fluorescents turn on from the bar hallway.
• A beat-up seatainer is in the far corner of the parking lot as our "permanent" ops center.  As of now, we don't have a key (Jose Carlos will pick one up shortly) and there is no power, communications cables, or furniture.  Jose Carlos can get desks and chairs from a local school and see below about cables.
• Got agreement in principle to mount the WiFi access point on the upper roof.  I have a plan which involves drilling unobtrusive holes in the ops center concrete wall.  Jose Carlos prefers running its ethernet cable to the ops seatainer along the wall (involving drilling more holes) to keep it off the ground.

Communications:

• Connection in ops center cafe has WiFi access point connected, that connects to the world.  Currently, this WiFi connection is unsecured and apparently is being enjoyed by locals.
• Did not have time to try a wired connection – on tomorrow's list.
• Have identified a metal plate used to mount the cafe's vent fan as the easiest place to cut a hole to feed a power and an internet cable to the ops seatainer.  
• Several cable routing options are available to get these cables to the seatainer.

Towers:

• 100m: tse09 and tse13 are up (I visited both).
• 60m: rsw03, rsw07, and tsw10 are up (I saw all 3).  I also saw the tower sections in place for tse11, ready to go vertical.
• 10/20/30m: saw concrete in place for tsw10, rne01, and rne02.  Jose Carlos assures me that there are more.
• Although power lines have been installed, no towers had actual power drops from these lines.
• Jose listed the towers needing solar power.  There are 16.  He plans for INEGI to supply up to 9 solar panel systems, in addition to the 9 or 10 we've prepared.  I note that we will have to make 4-pin AMP connectors for his systems to connect them to our DSMs.  The list is: tse01, 02, 05, 06, 07, 14, 15; tnw02, 04, 05, 06, 08, 09; rne06; v01, 02, 03.

Tomorrow plan:

• Fix mote fuses
• Reprogram TP01 wisard boards
• Test wired networking addressing and throughput, including adding our own router
• Assemble and test everything needed for tse13 (100m tower used as first WiFi repeater).

Andy and occasionally I have spent a lot of time over the last week tracking down error flags that show up on the CSAT3A data, as seen by the "sonic" LED staying red.  Not all the results are consistent, but they generally seem to point to noise originating from the switching AC to DC power supplies that we are using to power the DSMs.  I would expect this noise to be at a similar frequency as the sonic acoustic signal (~100kHz).  

The noise appears to be mitigated by grounding (to earth) both the CSAT3A head and the EC100 box.  Hanging the EC100 on the new mounting plates on a Rohn tower was adequate.  However, the Perdigão towers will be painted and paint is even on the ends where the tower sections connect.  Thus, I don't think we can expect to be able to get a good earth ground by attaching to the tower (even if we scraped away to bare metal).  The only mitigation strategy that I can think of would be either to run our own ground cable down the tower or to attach to the guy wires.  (It doesn't seem to me to be a good idea to ground to a lightning suppression cable, even if it exists.)  I don't think we can prepare to do this, other than to pack a fair amount of braid.  (I have 2 small spools now packed.)

I took a snippet of data from both the Setra and Nanobarometers today in the lab, sitting on the bench.  

I see white noise dominating the spectra down to about 0.1 Hz on the Setra.  I estimate the noise level at 0.02 mb RMS.  I don't see a spec for noise in the datasheet, but the "resolution" is listed as 0.01 mb – not a lot different.   (I don't know how resolution is defined for an analog sensor!)  Of course, it is possible that our A/D and/or cabling is injecting noise, but it should be better than this.

I saw no evidence of white noise on the Nano, with a general monotonic 5/3 decrease.  There was a hint of a step down in frequency response by about a factor of 2 (in power) at about 2Hz.

From both of these tests, I would say that these barometers are working properly.

So...2 boxes (rsw04 & 05) have Titan power panels, Titan front panels, and Viper CPUs.  With testing, I found that the DC-DC control was not being driven by any of vio 0–7 (though vio 3 controlled the Aux power).  With the DC-DC control not driven (and thus at 0V), the DC-DC converter wasn't enabled to provide +24V to the Ubiquiti.  Thus, I came up with a hack of using wire-wrap wire to insert +5V to the "Titan J3" connection on the power front panel.  This successfully turns on the DC-DC converter.  However, since J3 is now not connected, we've lost vio control of Aux.  Thus, I also changed the Aux FET jumpers to leave Aux always on at +12V.  (This probably wasn't necessary, since the Aux FET control has a pull-up and was on anyway, but it seemed the proper thing to do.)

I haven't mapped out the other vio ports to determine which/how ttySx are controlled by vio, but it seems to differ from what vio reports.

Even though rsw02 has an actual Titan CPU, I did the same wiring on that box as well.  Perhaps vio would have worked...

rsw03 is completely different – Viper power panel, Viper front panel, and maybe a Viper.  However, this tower will have a DTU Moxa system, so it needs to have an ethernet switch.  Andy and I placed a switch inside it, which is a tight fit, but it probably still should be secured since it will be placed 30m high.  Unfortunately, this is also the box that Gordon had identified (and I confirmed) to have intermittent data when using the front-panel RJ45 jack.  Thus, the only way to connect to the Viper ethernet would appear to be using one of the precarious adaptors that slip on to the Viper board itself.  (I did find that the outside Bulgin connector worked, but then we would have to route the ethernet signal back into the box to get to the switch!)  This entire front panel has obvious corrosion on the connectors, so an issue with the RJ45 jack perhaps is not surprising.

Ran every head through EC100 number 1817 using ECMon.  Found the following:

I have a call in to Campbell.  I'm hoping that going back a rev (from 100.07) on EC100 firmware will fix 1001–1010.  Heads 2050/2051 may have to be sent back .