Blog from October, 2013

Today was a crater day with all ISF and most Utah staff.  ISF equipment is now completely packed and (nearly) ready for helo operations.  Base tower is now shut down as well.

Floor GPS positions:

 Soils depths (approx):
Single Tsoil: 1cm

Quad Tsoil: Nub just below surface

Gsoil: 1cm

Qsoil: 2cm

TP01: 2.5cm

Some of the cables were slightly exposed.  Chris and I wondered if rain/wind removed some of the soil layer.  I took a few photos.

flr boom angles (calibrated DataScope at this location just before making reading):

3m CSAT: 39.8d looking into array

10m RMYoung: 346.8d looking along boom, aligning two transducers, junction box on back side (to the north).

Noon today was the (new) official end of operations.  This morning, Tim, Kurt, John, and I completed tear-down of far.

Since people had asked, these are my GPS locations at far (Sebastian said he took his own readings yesterday).

In the afternoon, Tim again climbed with the other 3 of us on the ground to strip 6 of 8 levels from Rim.

End-of-day status (not <too> bad for a project that ended at noon!):

- Trailer: All computers packed; networking down (but not yet packed).  (Had to disconnect AP24, Hughes, and base DSM since cables ran through space needed for packing.)

- Far: Totally removed (except Utah SODAR)

- Rim: 6/8 levels & 2/2 DSMS removed

- Base: Windsonic & DSM still running (just on battery), data only saved locally

- Flr: Should be completely running, data only saved locally

- Near: Tower should be running, data only saved locally.  (No radiation or soil.)

- 449 site: Tower is up.

- Chris just arrived; Clayton leaves tomorrow.

This morning, John and I removed rads and soils from far, then near.

After lunch, stripped lower sensors from base tower, started packing base trailer.

10/29/13
Summary:

• Partial tear-down starting today
• Still breezy -- 15m/s winds

Actions past 24 hours:

• Continued Qsoil.base plot testing

To dos:

• Tear-down of near/far rad/soil

Sensor status (abbreviated):

QCtables/cockpit show all signals coming in.

P: noticed that NNE deviates from others in the crater -- presumably due to the strong winds

SPN1: Delta-T has suggested some tests, but won't have time to pursue.

Rlw.in: warm event last night at all sites -- must have been a low cloud

kh2o.flr: sharp lower kh2oV event happening right now <possibly a bit of rain, but hard to believe as I overlook the crater>
Labels:

10/28/13
Summary:

• This was supposed to be our down day, so John and I visited Walnut Canyon on our way in
• Preparing for tear-down
• The forecast winds came -- average >20m/s at rim.40m all day; gusts 10m.base also 20m/s
• Open house tonight (despite the blowing)

Actions past 24 hours:

• Continued Qsoil.base plot testing
• flr died just as we came in.  We hesistated due to wind, but went into the crater anyway.  DSM power cycle fixed.  Lost data for ~4 hours.

To dos:

• Start packing up

Sensor status:

T/RH: ok.
P: ok.  NNE clearly higher during these winds
csat u,v: sustained winds over 20m/s at near; 25m/s at rim; 15m/s at base
csat ldiag: ldiag firing at a low level on many of the sonics
csat w, tc: ok
kh2o: typical
motes: ok
Wetness: ok
radiation: typical
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: ok.
2D sonic: ok

John & Steve: 1:30-3:30.  (Wind was wicked on the rim trail, but not bad on the way down.  A fair amount of dust in the crater.)

flr wasn't pinging (and neither was flr lldar) as of 11AM today.  

flr DSM wouldn't respond to console -- cycled power.  Pocketec took a while to fsck, but then booted okay.

lidar generator had stopped (even though the gas tank was only at half).  Sebastian and Eric came into the crater as well to diagnose/fix it.

Over the past 2 hours, starting 20:45, w.3m.near has gone down and up.  Other w's don't show this behavior. ldiag is okay, spd and tc seem reasonable to me at first glance.  The w time series looks okay, just with shifts.  We'll need to see if this behavior persists.

Planning:

- Permitting was difficult. Must be a budget line item.

- Site visit was important for comms testing.

- Need a good relationship with UCAR contracts early in project.

Pre-project testing:

- 2+ weeks of staging area testing was critical for start of ops.

- Need to pay more attention to mote comms during testing.

Training and Safety:

- must do tower training, tower rescue training. Bob Wiley has organized tower training in Dec.

- need training on motes, DSMs, base systems, XML

- AED, first aid

- Helo ops are hazardous, buy our own helicopter

- Must notify Bob Wiley on any injury.

Packing:

- In general a very good job was done in organizing equipment for each station, with color ribbons, rubbermaid tubs

- Replace gas can (cheapo Walmart can's spout broke)

Setup

- Setup was slowed each time anchors needed testing.  Would have been nice to install anchors a week or two before-hand.

- Need to get prior permission from dir office when using non-ISFS staff.  Use temp workers?

- should have a pow-wow of on-site staff each morning

- need additional harness, ascender

- need new winch/capstan, new rope

- need portable generator

- can we improve procedures for plumbing tower, guy wire building and tensioning?  Use come-alongs?

- shorter studs on CSAT electronics boxes; suggest using lock nuts (like TRAM) instead of nylocks. Dedicate some mounting plates for sonics that are small enough to ship in the box. Could one plate work for Rohn and pump-up towers?

Sensors:

- sonics were flawless. Will do post-field zero cal in lab

- two serializers became noisy.  Need to replace or get rid of. Will not filter data for PI (Tom)

- kryptons: generally OK

- barometers: problems were all power related? Was NNE on Oct 9 07:00 MST real?

- One TRH fan wired backwards.  Need to check past data from this sensor and double-check all fans. Chris to do bubble tests.

- One TRH had 0.1C bias.  Pretty good for all this stuff.

- Bad TRH data can be fixed by bit shifting (applies to SCP, SOAS, PCAPS?)

- downlooking PYG at far had short in wire from thermopile

- Schwartzeneggers need refurbishing

- need to check/replace radiometer desiccant prior to project

- Rsw.in.flr died for a while? Needs confirmation

- Vpile.off apparently has -1 LSB = -31uV.  .xml limits should allow this.

- SPN1 isn't perfect. Floor diffuse was sometimes zero for a short period at sunrise. Sent note to manufacturer.  They think it would be useful to run as a serial sensor and have the option for "TEST" mode. Needs leveling system.

- Individual levels of Tsoil at near would fail, but would return by a power cycle.

- Power to all sensors on a mote can be remotely controlled.

- Still need to take soil cores: bring oven/scales/soil install kit (buy a second oven and scales for trailer?)

- Qsoil response not normal in this soil.  Need manual calibration.  What should the strategy be here?  I actually liked having a test plot near the base.  In any case, soil quite dry during most IOPs.

- You can see core sampling in the Qsoil data!  The data jump a bit due to nearby vibration while pounding in the core.

- Forgot to use Trime, need to track down its pieces

- Soil installations:  need more consistency. Update guide for single temperature probe installation.

Power:

- new power monitor motes are great

- ISS was happy with the solar trailer

- Still don't understand mote diodes.  Having a diode between charger board and battery caused failures.  Had no failures with diode removed. 

- redesign small charging systems: lithium-ion?, separate charging module?

- Some small charging systems were flooded during setup.

Mote:

- Disabling radio reset was necessary. Xbees had new firmware since PCAPS.

- Xbee radios worked loose, need to be better secured. Oxidation was present.

- Spare mote 15 wouldn't come up plugged in to console cable.  (Something I've done wrong?)  Labelled.

- Power motes feed power into DSM interface panel.....Not True.  See 'power monitor' log entry update.  Problem not reproduced.

DSM:

- One unexplained failure at flr.  Seems to be related to 75mph gust at rim (though etherant later connected).

Network:

- need to test bulgin/rj45 cables that are used with AP24s.

- Verizon modems worked well, new box is good.

- Netgear WiFi needed to be reset many times.  Changing channel and orientation helped, but still needed at least daily resets.  Toss?

- WIFI worked great!  Etherants are old and slow however.  Need U-bolts that work for various diameters.

Computer:

- APS UPS batteries dead.  Buy bigger UPS for base?

- Trailer AC (and even using vacuum cleaner) caused computer issues -- everyone complained that it was warm during the late afternoon. Check trailer power.

- would have been nice to have check_usb.sh/check_ap24.sh feed a Nagios display.

- Need to change cockpit RIP color from background!

- Can we force a browser to auto-reload plots? (found cache killer, but doesn't seem to work). Need html wrappers around plot images.

- xchat was good to have, though died when 4G went out (can/should we configure a local host as the chat server?)

- Could/should we have set up 4G failure to fall-back to HughesNet?  Requires a better router box.

- R is great!  All known bugs are now fixed.

Infrastructure:

- Bird diverters fell off

- Bigger holes in soil/pressure base plates for T-stakes

- How to keep kinks out of hauling line?  (Had same issues using Jack's frame during xHATS.)

- Need more Bulgin cable splices.  (Could these be printed?)

- New bolts/nuts for newer guy attachment brackets

- Check all cables from near (data, network, power).

Tear-down issues:

- unconnected serial cable to 20m.near is bad

- unconnected RJ45 cable on rim is bad.

- let's get GPS positions for flr rad stand and soils

- blown fuse on base ttyS10(?) caused by plugging in bad mote cable.  (Cable labeled.)

10/27/13
Summary:

• IOP7 ending -- pretty good conditions after 11PM last night; will keep lidars going today (despite forecast for strong winds).
• No more IOPs expected
• 0.1 Ci this morning
• JM arrives today

Actions past 24 hours:

• Continued Qsoil.base plot testing
• Cleaned all rads and kryptons
• Reseated kh2o.flr cables
• Reoriented SPN1 back to original -- dip all the way to 0 came back
• Did GPS surveys of all crater P locations.

To dos:

• Continue measuring Qsoil.base
• Take a day off

Sensor status:

T/RH: ok.
P: ok
csat u,v: ok
csat ldiag: a few more than 0.05 about 10PM last night
csat w, tc: ok
kh2o: flr a bit better after cleaning/cable resetting, but noisy (similar to before we changed the serializer board 2 weeks ago)
motes: ok
Wetness: flr had frost yesterday
radiation: Rsw.in.flr died in the middle of last night, but came back by itself.  Other rads fine.  SPN1 reoriented and 0 came back
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: ok.
2D sonic: ok

Filled generator 25 Oct 2013 Friday will need filled again on 28 Oct 2013 Monday.

GPS positions both from my phone (in green) and from Sebastion's GPS (in orange).  (black are unit conversion of phone data).

Went in with the IOP7 crew to take care of a few tasks:

- ~14:35 cleaned krypton

- also reseated the analog connectors on the krypton electronics box in an attempt to avoid the mess we saw last night

- 14:52 reoriented SPN1 back to original position (analog cable pointing north)

- ~14:54 cleaned radiometers

Then hiked to NNE, SSW1, S, SSW2, SSW3, SSW4 to take GPS positions (Sebastian did SW, WSW, W at the same time)

From 11:45-11:47 cleaned krypton & rads at far.  kh2oV increased from 0.85->1.05.  Not much change on rads.

Cleaned krypton and rads from 1127-1130 at near.  kh2oV went from 1.03->1.13.  Not much change on Rsw, at least.

10/26/13
Summary:

• IOP7 starts tonight; nice drainage flow last night
• totally clear in Flagstaff; frost
• ISFS staff change: SRS left this morning, JM arrives tomorrow

Actions past 24 hours:

• Found that near USB stick had failed 2 days ago, including last few hours of IOP6.  Lost direct archive for about 29 hours.  WiFi data seem robust.
• Started Qsoil.base plot testing
• SRS is taking last (spare) pocketec home
• Derived fix for TRH data that go into "large value" mode (can be applied to past projects).

To dos:

• Continue measuring Qsoil.base
• Clean all rads/krypton
• Get GPS coords for pressure array (do Sunday?)
• Take a day off

Sensor status:

T/RH: ok.
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: flr odd all of yesterday.  Will clean today
motes: ok
Wetness: flr had frost yesterday -- may be associated with krypton issues
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok, doing calibration/test at base (not included in statistics)
Cvsoil: ok.
2D sonic: ok
Labels: