Daily report for Monday October 21 (early afternoon)
Control burn SSW of ISFS trailer wind from North right now but could change.
Weather: Sunny. Light northerly breeze this morning. IOP to begin with 4pm sounding, end with 10am sounding tomorrow morning.
Profiler power: 0.138mW (at 21:08Z)
Soundings: Usual IOP sequence planned for the overnight. (LT) 4pm, 7pm, 10pm, 1am, 4am, 7am, 10am.
Problems and interesting events:
LIDAR stopped working this morning. Down time was from 11:37 to 18:40 UT (4:37am to 11:40pm). Bill caught this, and after Steve cycled power he was able to restart it from Boulder.
449 profiler: I opened a terminal window. When I closed it the 449 halted, then the 449-pc froze. Rebooted. Down time from approx 21:12 to 21:20Z.
Nagios all green at ISFS Base
Nagios all green (ISS site) except:
- NP time "Unknown"
- Disk space: fs_/scr says "CRIT - 90.2% used (2481.02 of 2750.7 GB), (levels at 80.0/90.0%), trend: +88.52GB / 24 hours"
WILL need to clear disk space within 2 days.
SODAR site: solar trailer battery at 100% (3pm LT); . Sodar chirping.
All sensors appear to be operating well.
Daily report for Sunday October 20
Weather: Sunny. Calm. Mild breeze around sunrise. IOP last night, ending with 10am sounding.
Profiler power: 0.141mW
Soundings: Last night (LT) 4pm, 7pm, 10pm, 1am, 4am, 7am, 10am. All successful.
Problems and interesting events:
Comparing ISS-site and ISFS-site surface winds, both had a direction shift around 0130 UT (from about northerly to about southerly). However, following that shift the ISFS site wind speed dropped to very light (1-2 m/s) and the ISS-site speed accelerated to about 5m/s. Perhaps related, Nihanth noted in the lidar data that the incident wind was spliting around the crater rim.
- Nagios lights red for:
- NP time "Unknown"
- Disk space: fs_/scr says "WARN - 86.6% used (2383.28 of 2750.7 GB), (levels at 80.0/90.0%), trend: +89.64GB / 24 hours"
- (The lidar warnings are gone. Nihanth explained that during IOP's he changes to scans that do not include lidar-dbs or lidar-los.)
SODAR site: solar trailer battery at 91%; . Sodar chirping.
All sensors appear to be operating well.
Filled both tanks on ISS generator today next fill should be 22 Oct 13.
Daily report for Saturday October 19
Weather: Sunny and calm. Clear skies. IOP begins shortly.
Profiler power: 0.138mW
Soundings: (LT) 4pm and 7pm successful. Upcoming scheduled for (LT) 10pm, 1am, 4am, 7am, 10am
Problems and interesting events:
ISFS/GAUS site wind shift from 30 deg to around 240 deg near 6:30pm LT
- Nagios lights red for:
- metcraxii/profiler_latency_lidar/dbs/rtd (this is because the lidar has not been doing dbs scans)
- metcraxii/profiler_latency_lidar/los/rtd (this is because the lidar has not been doing line-of-sight (stare) scans)
- Also:
- NP time "Unknown"
- Disk space: fs_/scr says "WARN - 83.9% used (2306.92 of 2750.7 GB), (levels at 80.0/90.0%), trend: +89.65GB / 24 hours" (so it will be full in a few more days)
SODAR site: solar trailer battery at 100% (!); . Sodar running.
All sensors appear to be operating well.
Phil Chilson will return to Boulder tomorrow. Thanks for helping out Phil, and I hope you got some good RIM data!
|
|
|
|
|
|
Running the radar in RIM mode with 2-MHz frequency separation in the RIM frequencies (started at 19:35 UTC)
- 387.666667 MHz
- 388.333333 MHz
- 389.000000 MHz
- 389.666667 MHz
Switched back to normal mode at 15:00 UTC
Yesterday (Oct 17) the disk on thatcher filled, so the radar products were not being written.
Some files were deleted and the radar was restarted at about 18:30 UTC on Oct 17
Today we realized that the radar was running in the wrong mode. It was set to the correct mode for the Metcrax experiment and restarted at about 21:00 on Oct 18
Consequences:
- No RASS data during this period
- The frequencies written in the NetCDF header file will be wrong: Should be 388.333333, 388.666667, 389.000000, and 389.333333 MHz
- STD processing of the data should be unaffected
- For RIM processing, the code will need to be hacked (hard wired to use the above frequencies) to work during this interval
At 2205 UTC set the radar to run in RIM mode with 2 MHz frequency spread
- 387.666667 MHz
- 388.333333 MHz
- 389.000000 MHz
- 389.666667 MHz
Resume normal 1-MHz RIM at 00:43 UTC
Noticed afterwards that the 389.000000 MHz frequency was actually 390.000000 MHz
The solar trailer was at 96% around 1300 local time
Tim called me and we work out the details of the errors
Sunny and breezy today
.104mW profiler power lower than normal
IOP #4 for today has been moved to tomorrow.
when I came in there were 2 old unknown errors that are still there but we have several new ones.
fs_/scr has a warning issue. I just need to know what this is and where to look and fix if possible
metcraxii/profiler_anyfail_profiler is critical for over 22 hours.
NTP Time just went critical while I was checking the site.
One other went critical now is good. NTP
I will be climbing the near tower for ISFS to replace a WXT
I need info to possibly fix the new problems.
Will go and check the other sites soon.
[ |
|
http://localhost/nagios/cgi-bin//extinfo.cgi?type=2&host=iss1&service=NTP+Time\
] |
|
|
|
|
|
|
|
|
|
|
|
|
|
See my entry in the ISFS logbook. Access to the outside was down from about 0800-1200 this morning because the cradlepoint modem died and wasn't restarted by a check script. Reissuing the script power cycle command manually restarted it.
*** IOP last night - still need to launch the 10am sounding
At ISS1 Trailer
- Weather is clear and very little wind
- Winding down IOP#3
- Visual inspection of site good
Profiler computer and rack
- Spectra and dwell counter updating
- Dashboard shows all green lights
- Amp power at 0.207mW (RASS mode)
Data Manager Computer
- Critical flag on Nagios for metcraxii/profiler_latency_lidar/dbs/rtd
- Critical flag on Nagios for metcraxii/profiler_latency_lidar/los/rtd
- Zeb plots updating
- Connected to WindCube lidar and saw plots updating
- Web plots up to date
IOP today first launch was good
all sites are looking good.
Fueled both generators.
All systems are at the same place since earlier and yesterday just 6 unkowns on the board.
*** IOP scheduled for tonight
At the ISS1 trailer
- weather conditions are clear skies, light winds out of the NNW, temperature about 13C
- Outside check shows all is in order
Profiler Computer and Rack
- spectra and dwell counter good
- Dashboard shows green lights
- Amp power at 0.144mW
Data manager compter
- Nagios flags all green except for those related to png files and shown as "unknown"
- Zeb plots updating
- Web site plots at ISS1 and ISS2/GAUS up to date
- WindCube plots updating as viewed using TeamViewer
At ISS2/GAUS trailer
- outdoor visual inspection good
- All nagios flags green
- Ceilometer seems to be working and updating
When I arrived we had an issue with metcraxii/profiler_latency_lidar/los/rtd and metcraxii/profiler_latency_lidar/dbs/rtd these issues went good while I was in the trailer earlier and now are still good.
At 12:30 the solar battery charge was at 90%.
Everything looks good around all the sites.
IOP tomorrow so our checks will be after 2 pm.
wind has picked up since earlier with full sun not a cloud in the sky.
Profiler power is .142mW
At the ISS trailer
Visual inspection of site good: All RASS speakers working, cables seem in tact, tower guy lines good, fan working
449: operating, dashboard good (green lights and spectra counts updating), amp power @ 0.151 mW
Nagios:
** metcraxii/profiler_latency_lidar/dbs/rtd ** CRITICAL
** metcraxii/profiler_latency_lidar/los/rtd ** CRITICAL
Also red plags pertaining to the png files from the lidar not being updated
ZEB Plots:
okay
Web site plots
ISS1 site plots up to date
ISS2/GAUS site plots up to date except maybe the sodar. Will check again and check the site
WindCube:
plots are updating as viewed through TeamViewer
At ISS1/GAUS site
Visited sodar site - sodar running. There was an error message (warning?) on the monitor in the solar trailer stating that another computer was using the IP address. Batteries in the trailer charged to 87%
Nagios flags for ISS2/GAUS all green
Site inspection good
At approximately 12:30pm local time Phil and Tim launched a radiosonde from the barTbar road, just NE of the 449 trailer location. Phil noted an interesting wind shift difference over the profiler, with lower level winds shifting from SW to NE at higher elevations. The balloon was intentionally under inflated to keep the sonde ascending as slowly as possible throughout the lower boundary layer area before and after the wind shift.