Technician from Sunstate arrived at 1 pm to service the generator. He did not bring a separate generator with him, and so power was shut down to the trailer for about 30 minutes for the service.
I did not do anything to shutdown the gaus or the data manager, but left them running on their UPS.
I turned off some monitors, and unmounted and powered off /media/backup and /media/usbdisk, but left the laptops running.
The Sunstate tech noticed a break in a return fuel hose, which was slowly leaking diesel and fixed it.
While it was down I refilled both diesel tanks.
Our UPS battery is shot. When the UPS is turned on, it beeps incessantly until all loads were removed from the battery side. Tim Lim plans to buy a replacement for his unit which he will let us use.
Otherwise all our systems came back, the WIFI antennas associated, and the DSMs reattached on their own.
The only issue was the nameserver on flux. Clients could not resolve names. I did the following, which fixed it:
sudo systemctl restart named.service sudo systemctl restart dhcpd.service
The GAUS data manager (rack mount Dell server) powered back up on its own. I powered up the Dell tower (iss2).
The service interval is 300 hours (12.5 days), so the next service would be around Monday, Oct 20.
10/9/13, twh
Inventory of mote status
rad.far:
started 9/30 10:25 ID16
stopped 10/2 12:15 Tom shorted power input
started 10/2 16:20 Installed ID4, with repaired power cable
stopped 10/4 11:25
started 10/5 15:50 Replaced missing antenna
stopped 10/6 18:50
started 10/6 21:45 outage = 2:55
stopped 10/8 16:40 Removed to reseat antenna
started 10/8 17:50 Replaced
tx power 10/8: pl=4
xb reset change 10/10 9:00 xr=2hours, gps=off
xb-hb crontab 10/14 running every 4hours
still up 10/16 09:00
soil.far:
started 9/29 11:40 ID22
stopped 10/8 09:30
started 10/8 12:25 outage = 2:55
tx power 10/8: pl=4
xb reset change 10/10 9:00 xr=2hours, gps=off
xb-hb crontab 10/14 running every 4hours
stopped 10/16 13:45
started 10/16 15:40 outage = 1:55
rad.near
started 9/29 17:45 ID8
stopped 10/2 03:30
started 10/2 06:25 outage = 2:55
stopped 10/6 06:45
started 10/6 12:45 outage = 3:00
10/6 16:00 steady yellow light on mote, recycled power, but data coming in already
stopped 10/8 15:30
started 10/8 18:15 outage = 2:45
tx power 10/8: pl=4
stopped 10/9 12:20
started 10/9 15:15 outage = 2:45
xb reset change 10/10 9:00 xr=off, gps=off
xb-hb crontab 10/14 running every 4hours
stopped 10/15 01:35
started 10/15 05:20 outage = 3:45
still up 10/16 09:00
soil.near
started 9/30 11:20 ID3
stopped 9/30 20:30
started 9/30 23:25 outage = 2:55
stopped 10/1 02:30
started 10/1 05:25 outage = 2:55
stopped 10/3 17:30
started 10/3 19:20 outage = 2:55
stopped 10/6 01:25
started 10/6 04:20 outage = 2:55
tx power 10/8: pl=3
xb reset change 10/10 9:00 xr=hourly, gps=off
xb-hb crontab 10/14 running every 4hours
xb reset change 10/14 12:00 xr=off, gps=off
still up 10/16 09:00
SPN1.near
started 9/30 16:05 ID18 fried at installation, replaced with ID10
stopped 10/1 04:10
started 10/1 07:05 outage = 2:55
stopped 10/2 07:10
started 10/2 10:05 outage = 2:55
stopped 10/5 21:45
started 10/6 00:40 outage = 2:55
10/6 16:00 steady yellow light on mote, recycled power, but data coming in already
stopped 10/8 04:00
started 10/8 06:55 outage = 2:55
tx power 10/8 pl=4
stopped 10/8 10:00
started 10/8 12:55 outage = 2:55
stopped 10/8 19:00
started 10/8 20:30 outage = 1:30
xb reset change 10/10 9:00 xr=off, gps=off
xb-hb crontab 10/14 running every 4hours
stopped 10/14 05:25 up for 5+ days
started 10/14 09:15 outage = 3:50
still up 10/16 09:00
rad.flr
started 9/30 14:40 ID19
stopped 9/30 21:25
started 9/30 22:45 outage = 1:20
stopped 10/1 08:55
started 10/1 10:50 outage = 1:55
stopped 10/2 18:20
started 10/2 21:15 outage = 2:55
stopped 10/3 20:55
started 10/4 15:10 Replaced with mote ID17; radio of ID19 had worked loose
stopped 10/5 03:15
started 10/5 21:00 outage = 17:45
stopped 10/7 04:20
started 10/8 02:00 outage = 21:40
tx power 10/8 pl=4
stopped 10/9 06:40
started 10/9 09:35 outage = 2:55
stopped 10/9 12:40
started 10/9 15:35 outage = 2:55
stopped 10/10 00:50
started 10/10 10:40 outage = 9:50
stopped 10/10 12:05 then intermittently off and on
xb reset change 10/10 9:00 xr=hourly, gps=off
started 10/10 17:40
stopped 10/12 19:55
started 10/13 16:05 outage = 20:10
stopped 10/14 01:05
started 10/14 05:25 outage = 4:20
xb-hb crontab 10/14 running every 4hours
xb reset change 10/14 12:00 xr=12hours, gps=off
stopped 10/14 21:30
started 10/15 01:00 outage = 3:30
stopped 10/15 08:15
started 10/15 09:05 outage = 00:50
stopped 10/15 15:15
started 10/15 16:15 outage = 1:00
stopped 10/15 21:20
started 10/15 22:25 outage = 1:05
stopped 10/16 00:10
started 10/16 01:10 outage = 1:00
stopped 10/16 03:20
started 10/16 04:30 outage = 1:00
stopped 10/16 06:15
started 10/16 07:15 outage = 1:00
hardwired serial cable about 14:00 10/16
soil.flr
started 9/30 12:40 ID2
stopped 9/30 15:35
started 9/30 17:25 outage = 1:50
stopped 9/30 20:00
started 9/30 22:55 outage = 2:55
stopped 10/1 03:00
started 10/1 05:30 outage = 2:30
stopped 10/1 10:20
started 10/1 15:10 outage = 2:50
stopped 10/3 05:20
started 10/4 02:20 outage = 21:00
stopped 10/5 03:30
started 10/5 06:35 outage = 3:05
stopped 10/6 09:25
started 10/7 01:05 outage = 3:40
stopped 10/8 08:20
started 10/9 04:00 outage = 19:40
tx power 10/11 pl=3
stopped 10/10 15:50
xb reset change 10/10 9:00 xr=hourly, gps=off
started 10/11 04:40 outage = 11:50
stopped 10/11 11:15
started 10/11 13:50 outage = 2:35
stopped 10/12 14:20
started 10/13 07:20 outage = 14:00
stopped 10/14 00:05
started 10/14 01:00
stopped 10/14 09:55
started 10/14 11:55 outage = 2:00
xb-hb crontab 10/14 running every 4hours
xb reset change 10/14 12:00 xr=12hours, gps=off
stopped 10/14 17:10
started 10/14 21:00 outage = 3:50
stopped 10/15 01:30
started 10/15 05:00 outage = 3:30
stopped 10/15 09:35
started 10/15 13:00 outage = 3:25
stopped 10/15 21:20
started 10/15 22:25 outage = 1:05
stopped 10/15 05:10
started 10/15 05:10 outage = 1:00
hardwired serial cable about 14:00 10/16
SPN1.flr
started 10/1 10:50 ID1
stopped 10/1 15:20
started 10/2 03:55 outage = 12:35
stopped 10/4 01:45
started 10/4 16:05 outage = 14:20
stopped 10/5 05:45
started 10/6 00:40 outage = 18:55
stopped 10/7 05:25
started 10/8 03:35 outage = 22:10
tx power 10/8 pl=4
stopped 10/10 00:40
started 10/10 08:30 outage = 7:50
stopped 10/10 09:45 then intermittently off and on
xb reset change 10/10 9:00 xr=hourly, gps=off
started 10/11 13:50
stopped 10/12 07:55
started 10/13 06:10 outage = 22:15
stopped 10/14 05:30
started 10/14 09:00 outage = 3:30
xb-hb crontab 10/14 running every 4hours
xb reset change 10/14 12:00 xr=12hours, gps=off
stopped 10/15 01:45
started 10/15 05:15 outage = 3:30
stopped 10/15 09:10
started 10/15 13:00 outage = 3:50
stopped 10/15 21:20
started 10/15 22:25 outage = 1:05
stopped 10/16 00:10
started 10/16 01:10 outage = 1:00
stopped 10/16 11:45
started 10/16 13:55 outage = 2:10
hardwired serial cable about 14:00 10/16
10/7/13
T/RH: ok, high mr.25m.near; high T.20m.rim is real
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: Sebastian checked FLR connectors 18:15, 10/6, , so we now have a signal but the variance is much higher than near and far.
Wetness: currently missing at FLR; mote is likely down
radiation: sensors ok, motes intermittent
FLR rad mote went down 10/7 04:00
NEAR rad mote restarted 10/6 12:40
FAR rad mote down 10/6 18:50 - 21:45
SPN1.flr mote down 10/7 05:20
Tsoil: sensors ok, motes intermittent
NEAR soil mote down 10/6 01:25 - 04:20
FLR soil mote down 10/6 09:00 to 10/7 01:10
Gsoil: ok
Qsoil: FLR, FAR < 0
Cvsoil: intermittent, but this is calculated, not measured
2D sonic: ok, FLR wind direction not yet corrected
Norbert (German research group) has a scintillometer above the rim tower site powered by two 100 amp deep cycle gell cell batteries, but has no charging system other than hauling a generator up to the remote site. He has asked if the spare (backup) panels are available to be placed up near their site to facilitate charging. Norbert has been made aware that these are ISFS spares, and if any need arises, he must immediately return these to the trailer for ISFS use. Tom and Gordon have agreed to the loan, and I will help Norbert relocate and install these on Tuesday. He is also going to borrow a spare cooler and charge controller. Please be sure to retrieve these at end of project!
At the end of IOP #01, I went to investigate poor repeater performance at the rim, and found the main battery pack voltage at 8.5V (!) I removed all repeater gear and box, relocating them near the base of the rim tower battery bank and solar panels. I replaced the now assumed bad battery with another good battery from the MGAUS truck, (checking before and after voltages) and placed this in parallel with the the western most (closest to tower) battery box. This battery and its load are placed in a blue rubbermaid box. The immediate results were improved repeater performance, which will hopefully be more useful in the coming weeks.
Gordon and I have discussed increased monitoring of battery voltages, and comparing them to previous days to insure that this does no harm to the power bank at the tower. This is a minor load increase, but also a 33% increase in storage capacity, hopefully able to store excess PV energy collected during the day. Please pay increased attention to the morning battery voltages to insure this is the case. If anyone needs to remove the repeater, simply open the cooler and remove the clamps to the battery leads.
Just wanted to write down some thoughts on the FLR and FAR soil probe installations.
FLR:
Soil was very loose. Tried to get probes in their respectable places, but cannot guarantee with such loose soil (sand). I did put in single probe horizontal and did not realize the angle of install on old single probe temperature. Very windy down there with loose soil I was worried about erosion and exposure of probes at the end of this project.
FAR:
All seemed okay on install. More stable sand but was still loose. Single temperature probe was installed horizontally.
This is at start of project.
RIM TOWER
NE (in to out)
300,440,440,400
W
425,475,500,460
S
290,470,470,460
We had a partial network outage in the ISFS trailer that started around 10pm local time on Sunday night. There was a great deal of activity going on in the trailer, that seemed to confuse and complicate the issue, but the initial evidence was that Dave W. lost his wireless connection, and we had to switch him over to a wired connection. Then we noticed that the wired connections in the rear of the trailer that serve the ceilometer, sounding, and data manager computers were not connected to the network... Diagnosis began working from the back to the front, with rebooting the hubs in the rear (linksys) then the middle (d-link), but hesitated at rebooting the white belkin netgear wireless router... Other computers on the d-link (middle) are downstream from the belkin, and DID have outside internet connection. So I was reluctant to risk breaking something that was working, and risk loosing connection to Dave's (critical) laptop during our IOP.
After much diagnosis, I realized there was no other option but to reset the white belkin netgear wireless router, and I finally did about 1:30am, and all seems well after that. I reset the connections to the APG systems in the back, and the backlog of latent connections on Nagios began to clear out. (correction- At 2:15am, still several Nagios services reporting as critical, but are wholeskycamera related to network loss.
Hope I didn't screw things up too much- I blame the lack of sleep!
Thanks, Tim
10/6/13, 6 pm
Sebastian reconnected the krypton connectors, which brought it online. He suspects that it was the "military" connector to the CSAT serializer was not plugged in properly.
10/6/13
Tom is at NEAR to reboot the rad mote, and replace the dessicant in the radiometers.
He arrived at 3:55 pm.
Yellow light in center of radiation and SPN1 mote boards were both on, although both were sampling data (flashing red light). He power cycled both motes, which were not coming in. Rad mote is ID8.
Replacing dessicant at 04:04. Replaced pyranometer dessicants, but did not know how to access pyrgeometers (square dessicant holder).
Left at 4:11.
10/8, twh
Compared 2D sonic 10m wind directions to 3m sonic in order to orient 2D sonics. Only used winds from +/- 20 degrees from CSAT u axis and 10m wind speeds > 3 m/s (at FAR);
2 m/s at FLR. Could recalculate this at the end of the project, but likely will not change much. Used function dir.diff in project S directory.
At FAR, selected the median value of Dir.10m + 230.1 - dir.3m = 3.1 +/- 5.3 degrees. See attached plot. The 2D sonic is aligned with the boom, so need to add the boom direction, 230.1, minus the offset of 3.1 degrees, or 227 degrees to the 2D wind direction data.
10/30, I recalculated the offset using data from Sept 30 through 12:00 Oct 30 and set the minimum speed equal to 10 m/s. The result is Dir.10m - dir.3m = 0.2 +/- 0.8 degree, so need to subtract another 0.2 degrees or add 226.8 degrees to the 2D wind direction data.
. 
10/8, At FLR, the 2D sonic was pointed nominally north, but must be off by 180 degrees. The median value of Dir.10m + 180 - dir.3m = 6.6 +/- 5 deg. Thus need to add 180 - 6.6 = 173.4 deg to 2D wind direction data.
10/30, I recalculated the offset using data from Sept 30 through 12:00 Oct 30 and set the minimum speed equal to 3 m/s. The result is Dir.10m - dir.3m = -0.6 +/- 5.4 degree, so need to add another 0.6 degrees or add 174 degrees to the 2D wind direction data.
10/6/13, twh
Gordon and Tom serviced the RIM tower
Arrived 12:35 pm, left site at 2:00 pm
Taped Gordon hole.
Replaced Verizon modem ethernet cable.
Replaced 20m TRH SN 11 with SN 28 because 20m T was anomolously high. However the new TRH only lowered the anomaly a bit.
Calibrated DataScope; 5m CSAT: 61.9, 61.7, 62.0, 62.1, 61.7, 61.9, 61.7, 62.1 (mean=61.9 magnetic).
10/5/13, twh:
Arrived radiation stand 3:48 pm.
Rad mote missing antenna, replace with antenna from spare; cleaned radiometers. Rad mote has GPS disabled.
Left rad stand at 4:00.
Cleaned kh2o at 4:01.
Calibrated DataScope, CSAT azimuth: 39.6, 38.9, 39.5, 39.1, 39.3, 39.4, 39.2, 39.4 (mean=39.6 magnetic).
2D sonic (Gill Wind Observer) has one path aligned with boom, same as CSAT.
Left site 4:18 pm
10/5/13, twh:
Arrived 3:01 pm. Calibrated DataScope.
3m CSAT azimuth: 39.3, 39.0, 39.3, 39.3, 39.4, 39.3, 39.5, 39.6 (mean = 39.3 magnetic); all sonics visually aligned with 3m sonic
Cleaned kh2o at 3:16, then radiometers. Left radiation stand at 3:22. Note radiometers high to the south ~ 1/2 bubble (Chris said Schwartz screws frozen, which is true),
Left site about 3:30
10/5/13 twh
John was correct. The mote replaced at rad.flr (SN-19) did have the radio disconnected from the socket on the back of the main board. Repaired today.