Blog

went to RSW 03 to replace TRH's that were reading bad  at 40 m (missing) and 10 m 

• found that ports were not working (2 and 6) so moved to 8 and 10 (2 -8, 6 -10) called Gary to change the configs 
• turns out that both TRH's were pulling to much current and blowing the fuses - will check out the sensors tomorrow to determine the problem.
• visitied RSW 05 for a problematic Metek sonic - also suspect a bad DSM because a replacement sonic and cable showed the same problem. will investigate the Metek parameters tomorrow as concerned with Viper DSM.
• Added TRH's at TNW03 at 10m (port 3) and 2 meters (port 1)
• noticed looking across the ridge that RNE 04 tower is now up. Will attempt to instrument that tower as time allows. Priority is to make sure existing network is operational.
• searched for other towers being erected, but found no signs of activity at TrNW04 or TrSE 15 and 16. Not getting any updates from Inegi about tower progress, or them installing batteries at 2 sites still waiting for power (TrNW01,  or Va03)
• TrSe08 now has power according to the tower update spread sheet but is not operational - will investigate tomorrow? 
• assisted water vapor dial with manpower unloading equipment.

instrumented TrNW06 today. last of the standing towers that I know of.

My apologies for redundant entries - I could not see my earlier posts for some reason, and wasn't sure if they had actually been posted.

Samortechnica was in the area today, working on ??? no idea where.

Enigi will be here tomorrow to install batteries and solar systems hopefully at TNw01, Va03, and TrSE 5?.

we will start replacing bad TRh's at multiple sites, and wearing down the "to do" list

already having people asking to borrow the truck and a driver to assist in various user deployments,

will help when I can, but our work must come first, and not sure how people will take that. Is NCAR running this circus or are we just another participant?

set up most of TNW 5,

(Upper part was done on the 13th)

CSAT3A 2047 and EC100 1798 at 10m 

CSAT3A 2049 EC100 1696, EC150 1384 at  20 m

NR01-T2 s/n 2540 20 m

power mote #37

leaf wetness LW 013

soil moe # 7

Tsoil #09

Echo ERC5 001

TP01 200669

Flux Plate HO 23009

12m RMY 81000 ND/ARL # 650

8m RMY ND/ARL #1377

top DSM #253

mote # 37

6m RMY ND/ARL 1360

4m RMY ND/ARL 498

2m RMY ND/ARL 495

all sonics 12 meters and lower were pointed to NW because of trees (had to trim many branches just to mount the brackets)

RSW 2 visit - 10 m sonic ok, 20 m sonic not good, will replace later

RSW 3 40 m TRH fallty, will replace later.

RSW 4 leaf wetness sensor - added pic processor 

20 m licor RTS/DTS? no jumpers in Viper box to address.

RSW.in - RSW.out - config change? not sure how to accommodate.

RSW 5 added POE cable (ops center made for Ubiquity Ant, now have telecoms but realized that was not needed because

sonic port and Ubiquity ports were switched. 

sentra pwr was not plugged in.

now all is O.K.

not sure what happened to my blog note  last 3 days, I noticed they were missing... 

anyway friday 4/14 V04 hooked up pwr and soil mote

S/n for RNE 03 : DSM # 240

Pwr mote # 33

Csat3a P1005

EC100 #1807

TSe 13 replaced 2m TRH - 40 and 100 M trh's still bad - no safety equipment to go that high with us.

Grounded RNE 07, TNW 11, RNE 6,TNW 10, TNW 9, RNE 1, RNE 2, RNE 3. TSE 11 DSM's

Replaced RSW 3 20 m TRH and recorded S/N's 

finished TNW 07 today

added 4 levels of RMY sonics to tower to finish the array.

also visited TNW 05 to configure the Ubiquity and upper DSM with proper ISP address to put it on the network.

Cleared the sea container of surface group items before the horde arrives tomorrow.

Only one more tower to outfit (TNW 06) until Samortechnica builds more! (we were told today that they will have 6 -7 more towers by the end of the week....)

purchased more ground rods and grounding wire to accommodate.

we set up TNW05 today (and yesterday)

all sonics below 20 m are pointing to the NW because of trees. Had to remove many limbs from a tree that was leaning into the tower just to be able to hang the booms.

lower DSM has wrong IP address, need to bring a laptop up to the site to remedy.

visited many sites yesterday to work on "to do" list, see Gary's posts from earlier today.

The latest cockpit has been installed on eddy and is ready for testing on the large display.  At the moment, we do not know the location of the stand or the long 12' HDMI cable, nor do we know where it should end up.  Once those are figured out (or a new cable purchased), we should be able to get it working with eddy same as at VERTEX.  eddy has a wire ethernet connection to the ASUS wireless bridge, so it should be possible to move eddy anywhere to be near the location of the display.  I think it would also work to set the eddy wireless interface to the same static IP as the wired interface, so long as it is on the tower network (SSID perdigao and not CSCR).

Details on the VERTEX installation, such as desktop configuration for the second display, are in ISFS-113.

We visited rsw05 yesterday to get the DSM on the network. This is a viper v1. I logged in on the console, verified still could not ping radio or anything else.

The config says the METEK is in port 5, but actually found it in port 1, even though port 5 still had the cap with the leash. The mote was correct in port 2. USB at 54%.

vio reports ports are off, even though they clearly are on...?

    daq@rsw05:/media/usbdisk/projects/Perdigao/raw_data$ vio 5
    0
    daq@rsw05:/media/usbdisk/projects/Perdigao/raw_data$ vio 2
    0

After Dan climbed the tower to replace the bulgin ethernet cable with a straight ethernet cable, and came down dressing the cables, we finally realized that the ethernet and metek cables had been swapped from the beginning. That makes a lot of sense, since it explains why there was no signal at all on ethernet rather than just bad signal, and why Dan could hear the Metek at the beginning, very loud, but couldn't after I disconnected what I assumed was the ethernet bulgin.

The Setra pressure sensor was not plugged into power, now the a2d pressure measurements look much better.

We plugged the METEK into port 5 and verified that the messages looked good. The Viper-radio link now works over the new ethernet cable, but probably it would have worked over the bulgin ethernet also. Later, from ustar, I ran update, rsync, tinyproxy, and clean-usb playbooks successfully on rsw05. rsw05 has been up since 2017-04-12, probably the last power outage.

Today, there is no METEK data. Looking back for data on port 5 at rsw05, this is the first data file where messages appear and look good, at least at first:

    [daq@ustar raw_data]$ data_stats rsw05_20170415_150215.dat
    2017-04-16,11:47:44|NOTICE|parsing: /home/daq/isfs/projects/Perdigao/ISFS/config/perdigao.xml
    Exception: EOFException: rsw05_20170415_150215.dat: open: EOF
    sensor                              dsm sampid    nsamps |------- start -------|  |------ end -----|    rate minMaxDT(sec) minMaxLen
    rsw05:/dev/gps_pty0                  36     10      6986 2017 04 15 15:02:15.645  04 15 15:59:59.645    2.02  0.135  1.148   51   73
    rsw05:/var/log/chrony/tracking.log   36     15       237 2017 04 15 15:02:26.327  04 15 15:59:52.910    0.07  0.000 16.199  100  100
    rsw05:/dev/ttyS5                     36    100    174737 2017 04 15 15:02:15.771  04 15 15:55:16.049   54.94 -0.005  0.359    2   89
    rsw05:/dev/dmmat_a2d0                36    208     69288 2017 04 15 15:02:15.597  04 15 15:59:59.999   20.00  0.048  0.052    4    4
    rsw05:/dev/ttyS2                     36  32768       693 2017 04 15 15:02:20.421  04 15 15:59:55.402    0.20  0.195  5.695   11   52

But a few minutes later in this file problems appear:

    [daq@ustar raw_data]$ data_dump -i 36,100 rsw05_20170415_150215.dat
    |--- date time --------|  deltaT       len data...
    2017 04 15 15:02:15.7711       0      44 M:x =   124 y =   519 z =   -38 t =  2218\r\n
    2017 04 15 15:02:15.9941   0.223      44 M:x =   125 y =   523 z =    -4 t =  2257\r\n
    2017 04 15 15:02:16.0165  0.0224      44 M:x =    87 y =   514 z =    -2 t =  2248\r\n
    2017 04 15 15:02:16.0389  0.0224      44 M:x =    68 y =   535 z =   -11 t =  2236\r\n
    2017 04 15 15:02:16.0613  0.0224      44 M:x =   122 y =   528 z =    -8 t =  2203\r\n
    2017 04 15 15:02:16.0837  0.0224      44 M:x =   158 y =   520 z =    -7 t =  2191\r\n
    2017 04 15 15:02:16.1061  0.0224      44 M:x =   112 y =   511 z =    -3 t =  2195\r\n
    ...
    2017 04 15 15:05:47.0883 0.05001      44 M:x =   170 y =   141 z =   -46 t =  2119\r\n
    2017 04 15 15:05:47.1386 0.05031      44 D:x = 32767 y =-32768 z = 32767 t =     0\r\n
    2017 04 15 15:05:47.1611 0.02245      18 E:quality < 10%\r\n
    2017 04 15 15:05:47.1698 0.008694      20 E:data lost 40403\r\n
    2017 04 15 15:05:47.1883 0.01849      44 M:x =   167 y =   150 z =   -43 t =  2114\r\n
    2017 04 15 15:05:47.2384 0.05013      44 M:x =   176 y =   154 z =   -50 t =  2121\r\n
    2017 04 15 15:05:47.2884 0.04997      44 M:x =   170 y =   154 z =   -51 t =  2126\r\n
    2017 04 15 15:05:47.3450 0.05668      44 M:x =   171 y =   149 z =   -48 t =  2116\r\n
    ...

Then finally it goes berzerk:

    2017 04 15 15:06:26.2324    0.05      44 M:x =    55 y =   175 z =    15 t =  2126\r\n
    2017 04 15 15:06:26.2823 0.04994      44 M:x =    60 y =   177 z =    14 t =  2129\r\n
    2017 04 15 15:06:26.3391 0.05681      44 M:x =    64 y =   181 z =    16 t =  2132\r\n
    2017 04 15 15:06:26.3547 0.01561      89 C:\xbf\xa5\xbf\xb7\xbf\x9d\x95\x95\xbfK\xbf\xa5\xbf\xbf\xbf\xbf\xbf\x95\xbf\x17\xbf\xc5\xbf\xbf\x9b\x9d\xd9\x9f\xe5k~\xc0~\xee\xe8+\xfb\xfb\xfb\xfb\xfb~\xfb~\xa5\xbf\xbb\xbf\xbf\xbf\x9f\xbfK\xbf\xa5\xbf\xbf\xbf\xbf\xbf\x9f\xbfW\xbf\xa5\xbf\xbf\xbf\xbf\xbf\xdb\xe5\xebu\xcb]\x95\xbd\xb7-\x97M\xbf\xa7\r\n
    2017 04 15 15:06:26.4005 0.04579      19 E:unknown symbol\r\n
    2017 04 15 15:06:26.4099 0.009346      44 D:x =    67 y =   182 z =    19 t =  2133\r\n
    2017 04 15 15:06:26.4323 0.02237      89 C:\xbf\xa5\xbf\xbf\xbf\x95\xaf\x9d\xbf\x0b\xbf\xa5\xbf\xbf\xbf\xbf\x9d\x91\xbf\x17\xbf\xa5\xbf\xbf\x9b\x9d\xd9\xdb\xe5\xeby\x8b~ 155 Z"-     5 t!\x1d !\xb210MJ\xa2~~~($"0 Z"-     0 T"-    \r\n
    2017 04 15 15:06:26.4786 0.04634      19 E:unknown symbol\r\n
    2017 04 15 15:06:26.4880 0.009367      44 M:x =    68 y =   183 z =    19 t =  2126\r\n

Occasionally there are still good messages. It stops entirely at 15:55:16. Given the theory that the Metek was plugged into the 24V ethernet port for a few weeks, maybe it burned up? I have no other explanation for this behavior.

I suppose we could try a different port.

Added a second DSM to this site to sample all but the analog Setra sensor.  Fixed noisy messages from 10m RMYoung by moving it to a different port.  RMYoung at 20m is reporting non-zero diagnostic flag, and nothing we did fixed it.  The voltage at the DSM mote is 12.7V, but maybe there's too much of a drop up the tower, or maybe the RMYoung is beyond it's useful life.

• Replace the RMYoung at 20m.
• Rearrange the DSMs so that both have a clear view of the sky.  Putting rsw02x directly above rsw02 seems to have prevented GPS lock in rsw02.  Oops.

The JIRA issue on the crashing TItan problem has been updated: ISFS-149.

System updates to ustar and eddy:

• dnf update
• timedatectl set-timezone UTC
• set desktop on eddy to KDE (aka Plasma)
• reboot

LICOR looks to be working fine, no DTR/RTS jumpers to be seen. (This is a PC104 DSM, so maybe the todo task confused it with a Pi?)

Added a pic board for the leaf wetness sensor, cycled power on the mote, then verified messages are coming in with ID 0x8239.

Mote ID2 is radiation, ID1 is soils.

We were not sure what to do about swapped IDs for rsw in and out, since we assume there's no cable swapping to be done, and we probably don't want to reconfigure the individual sensors. Should we just override the NIDAS sensor XML at this site to match up the right IDs?

Kurt and Dan connected v04 to power, I visited later to get it on the net and check the data.

Ubiquiti

• Turned off frequency list.
• Fixed AP MAC, replaced current setting with tnw10 WSW: 44:D9:E7:C2:D5:A8.
• Changed name to Vall_04.

Eventually it linked successfully, signal at -75 dbm, and the dsm could ping the Internet.

DSM

• KZRAD in port 6 looks like a mote binary message.
• 12.8V on one mote (probably soil?), 13.3V on the other
• I see mote IDs in the 0x54 to 0x6b range, so I think KZRAD is working.
• 2m sonic on port 1 looks good.
• 10m winds on port 5 look good (repo config still had port 2).
• USB archive at 5% and growing.

At first there was nothing for 10m LICOR on port 3. Port 3 has 3A fuse and the fuse is good. Port 3 is jumpered for 232. When I checked the fuse that cycled the power on the LICOR, and that seems to have fixed it...

Cannot update eol packages, apparently because somehow some of the directories in the EOL debian ftp archive have become inaccessible. So until I can fix that, I updated the DSM manually by installing xinetd and check-mk, and downloaded and installed the latest package files for isfs-field-erpdigao, eol-daq, and isfs-daq.

The repo config has been fixed to match what was running on the DSM.

At the moment, none of our cal_files have Vazimuth entries, so everything is in instrument coordinates.  This isn't useful for our users.  We are expecting "exact" angles to be measured using the laser multistations during operations, but until then probably need to do something else.  One complication is that we are using Gills, METEKS, RMYs, and CSATs.  I assume the Gills were all set to have north=north.  We set the METEKS and RMYs to have north=along boom away from the tower, though some may have been opposite(!).  The CSATs are all along the boom, pointing away from the towers.  So, we also need to know the tower orientation.  The booms are nominally parallel to the ridge, pointing in the SE direction.  Call this direction 142degrees.

In one case, I oriented 10m.tse12 antiparallel to the other sonics (boom pointing in the NW direction) to avoid a local tree.

Thus:

Gill: (For Gill R3s, V_az is the N arrow direction + 240 degrees.)

   vaz = 0 + 240 = 240

CSAT: (For ATI and Campbell CSAT3 sonics, V_az is the direction relative to true north, straight into the array from the un-obstructed direction, minus 90 degrees.)

CSAT.10m.tse12: vaz = 142 - 90 = 52

RMY: (from me: +v is "from north", oriented perpendicular out from the connection box.  Assuming we mounted the box away from the tower, this is the boom direction away from the tower:)

  vaz = 142 (it seems that 142+180 works better – perhaps we put the connection box towards the tower?)

METEK: (from me: manual seems to show +v is perpendicular to N arrow – essentially East)

  vaz = 142 + 90 = 232 (sign convention appears to be left-handed.  we'll change this in NIDAS.  Also, this makes vaz = 232-180 = 52)

All this now implemented in cal_files and config.

got tnw05 partially setup today (need N.D. and O.U. sonics to flesh out the station).

Access is particularly difficult, but do-able.

Sea containers still have not arrived but Jose Carlos says they will be here this evening.

Not sure which container holds the sensors we need, or where in the container they reside, but we will attempt to retrieve them tomorrow. This site and tnw06 are the only towers erected left to instrument, then it is all maintenance till Samortechnica gets more in the air. Jose tells me that they plan on a big push next week. Time will tell.

I plan on at least two days per tower for these since they have a lot of levels, and solar panels.

tnw05 also has a lot of trees, one right next to the tower that is about 10 meters high, so I plan to place all sonics 12 meters and lower 60 degs to the west, (upper levels still are aimed SE), unless you want me to cut the tree down.