Gary and I did a quick trip to 8 to re-install the Gill sonic.
Everything went as planned including doing the short boat trip both ways.
9/11/16
Summary: Light winds, < 5 m/s, out of the W, partly cloudy.
Actions past 24 hours:
- Spent time at station 8 trying to fix Gill 2D (refer to blog post)
To dos:
- may go to S8 to install Gill
- work on spare power/monitor parts
Sensor Status:
UPDATE: Gary got Mote #2 running again.
T/RH: ok
P: ok
2D Gill: ok, down at S8
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: Mote #2 down since yesterday (no soil at #1 and #3)
Wetness: down
radiation: ok, except Mote #2
Tsoil: down
Gsoil: down
Qsoil: down
Cvsoil: not responding
Vbatt: ok, battery at site 15 getting low at night, will monitor
Left marina at 08:22 local. Low tide at 9:20 am.
When we arrived we discovered the Gill was plugged into ttyS6, and port ttyS6 was jumpered for 422. So the local config must have been set for port ttyS6, since we did not change any wiring or ports when we visited yesterday. When I updated the config yesterday after the visit, that overwrote the local change and set the port in the DSM back to ttyS4. Thus any characters received on ttyS4 since yesterday afternoon must be just noise caused by an unterminated 422 port with no device attached, if that makes sense. (For more context, see Quickie status blog post.)
We moved the Gill back to port 4, but I could not get anything sane from the Gill on either port 4 or 6, neither nt or t, and I tried multiple baud rates in minicom. So I thought this was a flaky Gill. I have since realized that even though there were jumpers for the emerald card serial ports, maybe I also should have been using emode to make sure the Emerald ports we tested with were set to 422. Is that right, we have to switch the jumper as well as run emode to set the mode? If so, then even though we thought we tried everything connected just like it was when it was working yesterday, with the Gill in port 6 and the dsm configured for port 6, port 6 may have been in the wrong mode. However, it should have worked in port 4 just like the Gills work in port 4 of all the other DSMs, so something fishy is going on.
Anyway, we had brought the spare Gill and an extra Gill cable that had been labelled as bad, so we started testing all combinations. We could not get messages out of either Gill over either cable. (We eventually dropped the mast to try the spare Gill through the cable in the mast which we assumed to be good.) The Gill appeared to receive commands, since entering '*' stopped the gibberish, entering 'd3' resulted in a single block of gibberish (I assume that was the response to the d3 command to show the config), and hitting 'q<return>' resumed the continuous gibberish. At one point we could see parts of a message in the gibberish.
At one point I noticed that the other sonic on ttyS5 had stopped recording. The dsm.log reported IOTimeoutException on port 5, even though we had changed nothing there. This sounds suspiciously like the symptoms noted in Site 12 reboot and noted with the tower dsm in JIRA ISFS-125. Maybe the problem was not overloading interrupts like Steve considered when he rebooted s12, but more like the entire Emerald board stops working. That could also explain why he thought that s12 was an older emerald because emode did not work, when in fact both S8 and S12 have Emerald 8P boards and emode does work, at least at the moment while the emerald is behaving.
We finally gave up and removed the Gill from the mast along with it's cable, leaving a serial cable in the mast to be used to pull the Gill cable back through later.
At 11:39 local we stopped at S4 and tested both the Gill sonics on that titan dsm. Both sonics worked on port ttyS4, without changing termination, although the spare is configured at 1 hz. The cable marked as bad did not work, it showed similar results to the garbled bytes seen at s8. But the other cable worked.
Done at s4 by 11:42.
Since the one bad cable caused similar symptoms at both s4 and s8, and s8 had similar symptoms with both cables, I still think this could be some kind of grounding or other signal problem that both the bad cable and the s8 dsm introduce. However, the Gill which was having trouble at s12 was never switched to port 6, and it has been working (fine, I think?) on port 4, in 422 mode using 120 Ohm termination and normal slew.
We arrived back at the marina at 11:53 by way of under the bridge.
Tasks
- Figure out why Gill wind sensors do not work at S8. Maybe try the sensors again at S8 but on ttyS6 and remember to set emode this time, since switching to ttyS6 worked before. Test the senors on the tower DSM since it is also a Titan with an EMM-8p. Maybe we can try to interfere with the ground or signal connection to the Gill to see if we can recreate the gibberish symptoms.
Just looking at typical midday values:
Rsum = 500 W/m2
H = 100 W/m2
LE = 200 W/m2
G = 50 W/m2
Thus, there is an imbalance of 250 W/m2 (which is bigger than any of the measured terms). The above values are before WPL corrections, etc., but I would be surprised if the corrections could double these terms. The logical conclusion is that energy advection by the surface water is huge. I'll try to see if this explanation is validated by the tidal cycle.
P.S. It could be useful to have a dat() function of the tide level...
P.P.S. I'll repeat the suggestion to mount a Wetness sensor close to the surface to detect water.
For my own reference, to first order Pirga is low with a constant offset of 0.97mb, though lsfit() finds that this is really a larger offset combined with a 1% lower slope:
lsfit(dat("P"),dat("Pirga"))$coef
Intercept P
7.9621428 0.9911725
9/9/16
Summary: Light winds, < 5 m/s, out of the SW, partly cloudy.
Morning trip to the Marsh to work on S8, S15, and S4 (rain gauge).
Actions past 24 hours:
- Morning Marsh visit to 4, 8, and 15
To dos:
- installation of rain gauge at S4
- installation of NR01 and mote #1 at S15 and install new TP01 PIC
- repair of power/monitor system at S8
Sensor status:
T/RH: ok
P: ok
2D Gill: ok, down at S8
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok, no soil at #1 and #3
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok, monitor since it is now coming in
Qsoil: ok
Cvsoil: not responding
Vbatt: ok, battery at site 15 getting low at night, will monitor
I filled the boat gas tank yesterday with mid-grade gasoline.
We left earlier this morning than usual, around 8:30 am local, to head into the marsh. Low tide was just before 9:00 am at 1.25 ft above MLLW, so we had no trouble getting under the bridge.
S8
We visited S8 first to replace the charge control monitor board. Steve had mounted the new solar controller and new monitor CPU card he received in yesterday's shipment on an existing board, and that board went into S8. That restored the power monitor messages to the DSM. The readings at the time were 13.3V, .7A load current, 2.7A charge current.
- Still need to update the config on S8 to correct the sample processing for the power monitor messages. Gary Granger
After powering back up, the Gill sonic was not coming in. It looks like that was related to port ttyS6 not getting set to 422 mode, and restart of dsm fixed it.
- Even after updating packages, the Gill did not come back after a reboot, and this time it did not come back after a dsm restart nor a power cycle using 'tio 4'. There is no response from it using rserial to /dev/ttyS4. The Gill appears to be down indefinitely. We can replace it with a spare. Gary Granger
We then realized we forgot to bring the updated PIC processor card for the TP01, also received in yesterday's shipment, so we boated back to the marina so Steve could get it.
S15
At S15, Steve replaced the TP01 PIC processor card on ID2 (m2). We mounted the repaired NR01 and the original mote ID1 (m1), and verified that radiation and power messages came in from that mote. (It took a little while to realize I had left rfcomm bound to that radio when we removed it, so I had to release it to allow the DSM to connect.) The ID2 mote took longer to verify the TP01. We had to try several power cycles, and Steve tried pressing the I2C rescan button also. I connected directly over bluetooth and we tried to look at the output from the mote on boot up and after a rescan. Finally we started seeing all the messages we expected: NR01, Tsoil, Qsoil, Gsoil, TP01, and power.
Steve unplugged and replugged the soil heat flux plate, and that got it reporting "better numbers". He's not sure what good numbers would be for saturated marsh mud.
S4
Installed the tipping bucket rain gauge at 105 cm high on port ttyS6, 9600 8n1. I added to the dsm config and verified the dsm was recording the raw messages, and we simulated some tips also.
- Fill out the rest of the tipping bucket config to process samples. Gary Granger
By the time we were done the tide was too high to fit under the bridge, so we had to return the long way on Old Mill Creek. We returned to the marina around 2:00, pulled the boat out and sprayed it down. The marina seems a little busier now that school is in session. We had to wait for people parked on the ramp both when we launched and when we pulled it.
The signal line from the SWin sensor to the PIC was cut. During assemble the wire was
pinched and eventually broke.
Site 11: TRH26 was replaced with TRH29
Site 5: The TRH shield was replaced due to flaky Ifan values
Results: Both sites looking good
9/8/16
Summary: Light winds, < 5 m/s, out of the W. Mostly clear skies. We did get some rain last night
Plan today is to work in the base on the NR01 and display/data issues.
Site 11 could be a potential location for the rad/soil instruments.
Actions past 24 hours:
- Marsh visit in the alternoon to site 15, 11, and 5
- site 15 - removed NR01 mote #1 to repair SWin
- site 11 - replace questionable TRH probe
- site 5 - replace TRH housing due to flaky Ifan values
- Helped UD get lidar system hooked up to internet
To dos:
- work on NR01 and hardware to extend height of rain gauge mounting stakes
- work on Rsync
Sensor status:
T/RH: ok
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok, mote #1 removed with NR01 (no soil senors at mote #1 and #3)
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: questionable at mote#2
Qsoil: ok
Cvsoil: not responding
Vbatt: ok, battery at site 15 getting low at night, will monitor
In looking at my latest version of the w'w' vs wind direction plot, I see that I didn't site S15 in the best location. For wind directions from the NNE (one of the preferred directions to observe tower wakes), the flow at S15 appears to be influenced by the high ground/clump of trees to its North. I found that this plot was much better behaved when I used S8 as the reference. I'll keep on looking at this issue...
Gary and I went to the Marsh around 16:30 hoping to catch the low tide. Unfortunately
we were to early to go under the bridge so we had to go the long way. We did accomplish a number of tasks.
(1) remove NR01 for repair work, S15, (2) replace the TRH housing at S5 due to flaky I fan, and replace the
TRH at site 11. We were able to get under the bride coming back at about 18:30.
10:00 I just notice the TRH at 5 is not showing on cockpit; however, talking directly to the
DSM shows the sensor is coming in. The fan current is low so it maybe a test in
in cockpit that set TRH to RIP due to the low current. Since this fan current has been acting
funny we will plan to replace the TRH housing.
9/7/16
Summary: Winds out of the N, NW at 5 m/s. Clear skies.
May go into the Marsh today to deal with TRH at 11 and NR01 at 15.
Site 8 power still down due to bad monitor board. John will send a care
package with some spares.
Discovered problem in TP01 firmware. Make fix last night and will work with
Boulder to reprogram spare boards.
Actions past 24 hours:
- Marsh visit yesterday. Did not get done what we wanted due to extreme water levels at sites.
- Work on TP01 issue
To dos:
- reprogram TP01 boards in Boulder
- potential visit to site 11 and 15
Sensor status:
T/RH: ok, s11 T is questionable
P: ok
2D Gill: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: ok (no soil senors at mote #1 and #3)
Wetness: ok
radiation: ok, SWin at mote#1 is bad
Tsoil: ok
Gsoil: questionable at mote#2
Qsoil: ok
Cvsoil: not responding
Vbatt: ok
The trip to the Marsh was not as successful as we had hoped. The water at S15 was the highest
we have seen. It was possible to float the boat to the station, solar panels, and the radiation systems.
The estimated water level was 15" above the foot pads. This visit was about 1.5 to 2 hours after high tide.
Gary did put on the extra plugs to protect the exposed Mote Binder connectors. We did not remove
The bad NR01 at Mote #1.
We then went on to 8 to try and fix the power/monitoring system. No luck.
After 8 we explored the remaining sites, except 1,2, and 3, to determine which offered the best location for the rain guage
and the radiation/soil system. All sites were in water of varying depth but not as bad as 15.
Our next trip to the Marsh will be at low tide to check out our options for the rad/soil and the rain gauge.