Blog from February, 2015

Not much activity since snow has stopped tower and station work.

flux: Okay

tower: se sonics, TRHs, and DSMs fine.  50m nw sonic dead (need to check connector and possibly replace), 100m nw sonic is good, other nw sonics not mounted yet.

bao: Okay except EC150 which sends intermittently bad data.  Suspect that the terminal block screws inside EC100 box need to be tightened.  Other possibilities are the bulgin cable or the EC100 box itself, but the box was tested in the lab.

ehs: Have verbal permission for a site, but may need written permission.  Waiting for this decision (and snow) before deploying.  Kurt spent today preparing (which involves a few physical and cable changes to make a "cleaner" installation).

www: Created dat.Theta and dat.P on EOL servers.  List of www plots to generate is on Rudy's office marker board.  Julie is desperate for wind direction plots!

The data looks pretty bad.  One would expect bad winds and co2, h2o because of the snow, but it seems to be effecting the data format too.

The UART for that serial port /dev/ttyS4, is reporting data errors. Here's some cktty's done a few second apart. You can see the "fe" (framing errors) count is growing:

cktty 4
4: uart:XR16850 mmio:0x10800000 irq:123 tx:690 rx:56562450 fe:4336624 RTS|CTS|DTR
root@bao config# cktty 4
4: uart:XR16850 mmio:0x10800000 irq:123 tx:690 rx:56564384 fe:4337087 RTS|CTS|DTR
root@bao config# cktty 4
4: uart:XR16850 mmio:0x10800000 irq:123 tx:690 rx:56565358 fe:4337256 RTS|CTS|DTR

To me this looks like a "water in the connector"  problem.

data_dump shows the bad data.  The records should all be 60 bytes, but this dump shows record length of 56, 57, 55 and 53 bytes. 

The EC150  on the bao flux station is configured for binary mode.  This is a new option provided by Campbell, and the first time we have used it (to reduce the data sent over the cellular connection).  

data_dump -i 7,30 -H

2015 02 26 00:59:38.8464 0.1007 56 de 82 67 ff b3 f9 bf f3 d8 12 3e 00 21 f6 03 06 04 0c f8 fc f8 74 c5 ce 14 c3 40 fc 86 80 00 f9 fb 2b b3 9a fc fa 3e 37 4f e1 c3 3c 19 18 80 e0 47 e3 30 60 6f ff 55 aa
2015 02 26 00:59:38.9457 0.0993 57 9f 4e cc 45 da 01 c0 de 1c a8 bf 00 61 03 c0 80 00 80 80 ec f0 74 c5 cc df 34 3f 08 86 00 00 ff bf a5 f6 a7 42 95 f3 fc 3c f1 69 f8 3c 90 17 80 e0 9f 84 03 00 df f8 55 aa
2015 02 26 00:59:39.0456 0.09993 55 e3 9b 28 3e 19 bf 00 c0 45 7f 44 02 22 d5 bf 00 00 00 80 c0 84 7a 35 7a 9a 3e 1f 0c 86 00 00 ff 2b b3 9a fc 12 b8 3c 37 ab f8 3c 19 16 80 e0 44 36 08 78 ff fb 55 aa
2015 02 26 00:59:39.1458 0.1002 53 f1 fc 61 7a 0b db bf 70 09 00 01 4a 05 03 80 00 80 80 48 8f 54 5b 24 c3 40 fc 86 80 00 ff 2b b3 9a bc f4 dc 3c bf 95 f7 3c a6 17 80 e0 90 84 30 60 7e f4 55 aa

This data looks  basically non-recoverable to me.

I haven't had a chance to go through all the sonics, but we (Dan and I) did notice that the 50m sonic in port 2 is going through a cycle of 8000... then 3 secs of stuff... then 8000.  This usually is the sign of a dead transducer/array, though I suppose it could be just a cable issue.  I tried "tio 2 0/1" which looked like it did cycle power, but didn't help (which is what I expected).

Today we:

• mounted the EC150, 
• reworked the guying and winch, 
• added cable ties
• wired up the second battery
• shot boom angles with handheld compass.  2D N is to the SW at an angle of 130deg; 3D into the boom is 132/135 degrees.  (These were with no declination set.  With declination at the BAO of about 11 degrees E, these values should be 141 for the 2D and 143 for the 3D.)
• verified that Qsoil #16 (at 20cm) is I2C address 29. (by unplugging cable 16 and verifying that 29 went to nan).
  
  

Bruce at the BAO realized that flux must have a dhcp server running, because when Dan Wolfe connected his laptop this morning it got a 192.168.0.176 address instead of 10.173.  It does not look to me like the dhcp server is needed on this project, since the DSM addresses are set in /etc/hosts, so I disabled it:

[root@flux dhcp]# systemctl stop dhcpd
[root@flux dhcp]# systemctl disable dhcpd
Removed symlink /etc/systemd/system/multi-user.target.wants/dhcpd.service.

Bruce called earlier about an interference between "flux" and the NOAA network, so he had unplugged flux from the network.  I asked Gary to work with him and flux is now back up.

However, I noticed that "ds" was only finding "300m" and not "bao".  Eventually, I did "dsdn"/"dsup" which appears to have brought the bao data stream back into flux.  Along the way, also found that bao was losing data since the Pocketec didn't come up from the last power outage, but was able to get it back with "tusb".  Thus, we probably lost data from bao for some time this morning (though ncharts showed data there – perhaps servers at NCAR grabbed the raw_data stream?).

flux (server laptop):

• Set up and working, but is interfering with NOAA network and has just been unplugged.  Gary is working with Bruce to fix

Tower:

• 300m DSM is on the network; other DSMs installed but not yet connected to BAO network
• 9 (of 12) sonics mounted (and collecting data locally on DSMs?).  We'll bring out the remaining 3 (that just got back from Campbell) today
• 6 TRHs should be mounted and also collecting data (locally)
  
  

bao:

• All instruments up, except EC150 (and EC100).
• Some mechanical and power issues (power and thus station dying at night).  Kurt and I will visit today to fix.
• Pocketec didn't come up after last power cycle (and doesn't respond using "tusb 1 0/1").  We'll need to cycle power manually...

ehs:

• Julie and I or Kurt will meet with the building superintendent tomorrow afternoon to get site permission.  We'll plant soil sensors and possibly the tripod base at that time, if possible.
  
  

When we get running, we will need to:

• Clean radiometers (every 2 weeks?)
• Clean EC150 (every 2 weeks?)
• Take 2 more gravimetric samples (one in middle, one at tear-down)
  
  

About 0630Z today, bao died due to low power.  Apparently, only one battery is connected and it wasn't enough to keep the load going with snow covering the solar panels.  With some sun today, the station revived at about 1600Z.

soils:

• secure cables with cable ties
• set solar panels to 50 deg
• unplug/plug 10cm Qsoil to verify position

rad:

• level schwartz (darkhorse is level)
• secure cables with ties (now just wrapped)

tower:

• secure modem/bluetooth boxes
• secure cables with ties
• rework winching system; possibly other tower adjustments (including guy wires)
• add EC150 head
• add EC100 box and cables
• shoot boom angles on both sonics (shoot the N-S axis, denoted by the (nearly invisible) red dot and silver/blue line on the label below the north arm).
  
  

power:

• died, so clean snow off panels
• replace battery with fresh?
• connect second battery
• test load/charge with new power meter?

On 19-20 Feb, did "field check" calibrations on our 2 EC150s and the loaner IRGAson from Campbell.  (Our third EC150 is at Campbell for repair.)  For all 3, I performed the following steps:

1. Started by cleaning all lenses with isopropyl alcohol.
2. Supplied Nitrogen to the sample cell at 0.3lpm (what Andy's small cylinder produces), let settle, then clicked on "zero" using ECmon.
3. Supplied CO2 mixture to the sample cell, setting the flow rate also to 0.3lpm.  The bottle Andy gave me was labeled 396.  Read the value after settling, then clicked "span".  Checked that value was now 396.
4. Supplied H2O mixture to the sample cell from the humidity generator with a setting of 70%RH.  I didn't control the flow rate, which was larger than 0.5lpm.  (Also set the temperature set-point to 25C, but this appears not to work.)  Read both T and RH values after settling, and entered these values into a WWW-based dewpoint calculator.  Entered the resulting dewpoint into ECmon and clicked on "span".  Checked that value was now correct.

I also looked at values at 35% RH, just for sanity checking, but wasn't consistent about doing this before or after the dewpoint span.

For future reference, I'll note that the EC150 calibration cell takes 1/4" swage, the nitrogen regulator just has tygon tubing, the CO2 regulator has 1/8" swage, and the humidity generator has large hose barbs.  I made a hose to 1/4" swage fitting for the N2 and largish tubing from the humidity generator and used a 1/8-to-1/4 fitting for the CO2.  The flow gauge used 1/4" swage.  During the humidity calibration, I connected the return flow just by slipping the two tubing pieces together; during the other calibrations it was open.

Notes:

• The IRGAson behaved great.
• 1277 drifted more than usual during the calibration.  I went through the process anyway.  We should replace the scrubber chemicals.
• Before doing the calibration on 1276, I showed Kurt the scrubber chemicals.  This was a mistake, since it exposed the sensor to outside air.  Starting the calibration, there was a 45+ppm offset in CO2 using Nitrogen.  Replaced the scrubber chemicals in both bottles, reinstalled, and waited overnight (the manual says to wait 24 hours, but I only had 15 hours).  Did the "real" calibration the next morning and saw much lower offsets and perhaps less drift as well.
  
  

H2O spans seem a bit large.  I'll send this to CSI for comment.

Gravimetric processing from samples taken using corer.

Note: Dry taken 23 Feb after air-drying in the lab for 3 days, then baking in the toaster oven for an hour.  (The oven got rid of about another 1.5g in each sample.)

The results show all EC5s about 2% low (which could be due to coupling to the soil, just after installation). 

Note2: 9 Mar sample processed as 2 aliquots (since scale range was limited).  The sum of both aliquots is shown in the above table.

Note3: 20 Apr sample also processed as 2 aliquots.  Both are shown above.

Note4: Tried to get a better depth match with 20 Apr samples, moving the 1cm ring to the top.

Note5: 20 Apr EC5 readings taken using minicom – see teardown comment.

Note6: assume that 9 Mar had a math error: 142.333, rather than 152.333???

tare = c(5.58, 2.15, 2.50, 16.477, 8.237+8.260, 8.232+8.214, 8.265+2.139, 2.126+2.379, 2.111+2.094, 2.079+2.107)

wet = c(100.74, 99.59, 89.51, 142.333, 81.960+60.946, 71.380+74.505, 96.082+29.133, 56.184+61.353, 70.604+60.122, 60.887+70.702)-tare

dry = c(83.54, 87.36, 78.51, 127.741, 70.854+52.124, 62.335+65.162, 84.192+25.476, 46.364+50.476, 58.938+49.878, 50.456+58.281)-tare

vol = 3*pi*(5.31/2)^2

rho = dry/vol

moist = (100*(wet-dry)/vol) # density numerically equal to volume mixing ratio since rho_water = 1.

ec5 = c(23.98, 16.38, 13.17, 28.4, 29.47, 28.20, 26.00, 30.1, 32.0, 29.4)

ch = as.character(c(1,2,3,1,1,2,3,1,2,3))

col = c(1,1,1,2,2,2,2,1,1,1)

par(las=1,tck=0.02,xaxs="i",yaxs="i")

plot(moist,ec5,xlim=c(0,40),ylim=c(0,40),col=col,pch=ch); abline(0,1,lty=2)

bias = c(1.5,1.5,4.0,0.5,0.5,-0.5,-2.5,1.5,1.5,4.0)

plot(moist,ec5+bias,xlim=c(0,40),ylim=c(0,40),col=col,pch=ch,xlab="Gravimetric (%Vol)",ylab="EC5, bias adjusted (%Vol)"); abline(0,1,lty=2)

legend(2,38,c("bao.20cm","bao.10cm","bao.5cm","ehs.20cm","ehs.10cm","ehs.5cm"),col=c(1,1,1,2,2,2),pch=c("3","2","1","3","2","1"))

title("CABL Gravimetric Sampling")

As of 5/26/15, I would set biases (add to ec5 to make correct) to be:

bao.5cm: 1.5%

bao.10cm: 1.5%

bao.20cm: 4%

ehs.5cm: 0.5%

ehs.10cm: -0.5%

ehs.20cm: -2.5%

5/29/15: 

I've also tried to remove the diurnal cycle, assuming that it is a temperature affect (since vegetation shouldn't have been very active, especially at the beginning of this experiment).  I use the correction:

Q_correct(z) = Q_meas(z) - xm*Tsoil(z),

where:

xm = c(0.11,0.18,0.17,0.12,0.20,0.18) for bao 5,10,20 and ehs 5,10,20 cm, respectively.

With this temperature correction, the bias correction changes a bit:

Q_correct(z) = Q_meas(z) + xb - xm*Tsoil(z)

xb = c(2.3,2.7,5.0,0.5,-0.5,-2.5) (same order as xm, above)

After these corrections, I get the attached plot: Qgrav.pdf, where the left panels are from bao, the right panels from ehs, the top panels are just a bias correction and the bottom panels are with the combined correction.  Also on these plots are the gravimetric samples as circles.  For ehs, Qsoil was read manually (independent of the data system) at the time of the second soil cores, and these values are plotted as "+" (though I don't have the corresponding Tsoil to apply the combined correction).

As planned, I installed one of the Tsoil pickleforks upside down (and the other normal).  The normal one had the thin, brown coating, the upside-down one was the thicker white epoxy.  I didn't record which was which channel.  However, looking at the data, it appears that the ".x" one is normal and the other is upside down.  I'll modify the config to reflect this arrangement.

Also, looking at the data, my guess is that the 10 and 20 cm Tsoil sensors are swapped.  The cables were labeled that 10cm was SN17 and 20cm was SN20.   I'll make this change as well.

What I don't know about is Qsoil,  10cm SN was 11, 20cm SN was 16.  The current profile makes it appear that the current ID assignments are correct in the data, but I'll have to check this.

Some more notes about this installation:

• dug a small trench to 10cm, took core sample (to be used as an offset check, though the EC-5 won't have settled well).  Depth of this sample approximately 10-13cm.  Core taken 11:12.
• continued digging to 20cm, took another core sample.  Depth of this sample approximately 20-23cm.  Core taken 11:29.
• cleaned one side wall in narrow end of trench and installed T/Q probes at 20cm and 10cm.  Note that we only had a big screwdriver, so the pilot hole for the single Ts were larger than needed.  Thus, these may not be well coupled to the soil (nevertheless, the data look okay).  Used a butter knife for a pilot hole for the 20cm EC5.  The 10cm EC5 didn't need one.
• took a third core sample at a nearby plot.  Depth of this sample approximately 2-5cm.  Core taken 11:57.  Installed EC5 and HFT at about 5cm in this hole.
• removed a "brownie" in this same plot.  Used a butter knife to cut a slot and installed the TP01.  Replaced the brownie.
• cut a nearby slot for a picklefork.  Installed this one in a normal orientation.
• cut a second slot for the second picklefork, installed upside-down (eyeballed to the correct depth).  Note some cracking in this part of the soil.

All this done between (approximately) 10:30-13:00 yesterday (20 Feb).  (Yes, it took a while to work down to 20cm.)

Brought soil cores back to the lab and weighed (wet) about 4–5 hours after the sample was taken (a bit long, so some drying might have occurred).  Couldn't find the oven, so just drying open in the lab.  Note that there were some water drops on the lid of (I think) the 20cm sample.  Thus, it definitely will be drier.  (I realized that I could have measured this by 1. weighing a piece of paper towel 2. wiping off the moisture with the towel 3. weighing the "wet" towel, but I didn't do this.  Just for fun, I'll try to simulate this when I'm next in the lab.)  I'll start a separate entry on processing of these cores.

Fri 20 Feb, 14:00 MST

The DSM at 300m is up and logging data (as of 20:23 UTC).  It has one sonic (pointing SE), and the TRH.

At first the usbdisk (a 60GB pocketec) was not mounting.  I power cycled it:

tusb 1 0
tusb 1 1

But it still wouldn't mount.  I did a soft reboot from FLAB and all is happy.

Bruce has installed the other levels, each with one SE sonic and a TRH, and powered them up, but the network connections are not working.  So we don't really know whether the data is being saved.  I believe all those have flash drives and not pocketecs, so I am pretty optimistic they are logging data.

The flux laptop is also up.  See the internal CABL web page https://internal.eol.ucar.edu/content/cabl for how to login from FLAB.

From: Steven Semmer 

Subject: Baro check

Date: February 6, 2015 4:57:15 PM MST

To: Steven Oncley, Tom Horst

Steve, Tom:

 I tested 4 barometers over the range of 790 mb to 860 mb.

error results (ref - baro):
baro     min      avg     max

005      -.13      -.06     .005

009      -.23      -.15     -.08

003      -.09      -.02      .06

006      -.10       .01     -.04