summary plots good
Outside check good
Sunny and slight breeze.
Profiler power is .142mW
Direction wind shear around 500-700m level thinking about doing a sounding launch for training and science.
Same 6 issues all involve the lidar png issues from the last few days.
Team going into the crator today for supplies and repairs.
Checked the other trailers and the data feeds associated with the instruments at that site.
Nagios shows all green lights
Visual check of instruments - all okay
Inspection of the sodar - okay. Battery show charged to 97%
Visited the windcube lidar near the visitor center. The data cable from the WXT was disconnected. Tim reconnected and secured the cable. This fixed three errors that were showing up on the Nagios page: namely those related to the latency of updates from the sensor, and those pertaining to the nidas_1_10 and nidas_1_11
Generator was at 60% in lower tank,
We refilled both upper and lower tanks.
Should be good until Wednesday the 16th.
This morning's session is a training and orientation session for Clayton and Phil.
***Gary, can you add Clayton's login enabling him to edit or post to METCRAX-ISS Blog? He is currently not on the list. Thanks!
449:
Dashboard checks are all ok
power amp reading is .132mW
Nagios status:
Previous post by Kate on oct 12 noted NIDAS 1_10 and 1_11 critical errors, these still remain. I don't know the source of these errors, Gary and Bill, Can you explain?
The remaining 9 errors pertain to Lidar data (images) that aren't being placed in the right locations for the field catalog, I believe. Again, Gary, can you look into this? I believe this was caused when we swapped out the computer on the lidar, and we aren't processing the images correctly. I'm telling Clayton and Phil not to worry about these, there is little we can do about these at this time.
All plots for 449 site have been reviewed for current status, all are up to date, except LIDAR plots, (see notes above)
All outside visual checks completed, all ok.
Generator inspected and tanks were dipped and found to be at 60% full, we refilled both lower and upper tanks today at 11:30am. Should be ok till Wednesday afternoon the 16th.
inside cleaning, vacuumed all surfaces, cleaned A/C filters.
Clear and warm, slight breeze from south, seems to be building strength. Dave W, says it will be gusty winds today.
IOP STILL (tenatively) scheduled for tomorrow night!
Log notes actually made at 8 am LST, but report entry was delayed by work being done on network back in Boulder.
Weather today was clear and sunny, with light winds.
Generator was filled by Tim on 10/11/2013. Should be checked/refilled in a few days.
Visual inspection of the site showed nothing of concern. Rass speakers are all working, as is the fan on the tower.
Power amp was as .148 mW
Profiler display looked good, with spectral profiles were all in alignment.
Nagios flags indicated critical warnings for the following, indicating that the samples were late:
Nidas_1_10
Nidas_1_11
WXT/standard/nc
Additionally, most of the lidar flags were still flagged as critical.
Phil Chilson arrived last night to relieve me. Clayton is scheduled to arrive today to relieve Tim. Phil went to the 449 profiler site this afternoon and was working on analysis code and checking on the past performance of the the radar. Tim will be training both Phil and Clayton tomorrow on site operations. There is possibility of another IOP tomorrow evening.
IOP #2 begins tonight with a balloon launch at 5 pm LST.
Weather is cool and crisp with clear skies.
Inspection of 449 site went well. Rass speakers all working, and fan on tower is working as well.
Profiler display shows no problems, Spectral profiles are all in agreement and dwell counter is working.
Nagios screen shows red flags for lidar, however earlier entry by Tim indicates system is up and running. Both profiler_anyfail_prof and profiler_anyfail_surface are flagged as red as well. Same problem occurred yesterday and problem seemed to fix itself after some time. I will keep an eye on it.
Surface met data looks good.
Daily amp power is at .139 mW
Damien and I will be working on the Lidar this morning, cleaning the optics, checking power supply(potential for effects on range) and various other fixes.
Bill and Gary, we have not installed the additional software that was on the previous PC. There was not enough time to accomplish this while we were installing the system, and so we need you to access the system remotely and institute these updates. It may appear that the system is not online, (and having some direct effects on Nagios checks) but the system is online.
TEAMVIEWER is active, but we will be making some modifications. The current "foolproof" way of accessing the system is with the TEAMVIEWER credentials available from Tim, Bill, or Gary.
We will be working at the lidar platform as well, so let me know (call my cell # 303 818 9906) if you need us to do anything, or to hold off on rebooting the system, etc...
WE DO HAVE AN IOP scheduled to start at 4pm this afternoon. I hope we can get all systems back online by then.
449 profiler seems to be running fine, no alerts or issues. All products seem to be normal, I had to restart the Display icon bar to get local graphs to respond again. They seem to stop plotting (locally) around 5am UTC. (I've seen this stop several times before) but restarting the (ISS display manager program only) seems to fix this problem.
Thanks
Tim
[edited to remove password and re-posted by Gary]
I refueled the upper and lower belly tanks on the generator today at 6pm. I'll plan to refuel again on Sunday evening while training for Clayton and Phil.
Damien Ceus, Field Engineer for Leosphere arrived in Flagstaff late Wednesday night, with plans to replace the troubled computer in the Lidar. The computer arrived via FedEx around 10am, and we departed to the site. As soon as we stopped at the 449 trailer, the problems with the 449 became apparent, and it appeared the profiler PCM had rebooted, and no causes could be identified. The Display manager PC was running just as I had left it on Wednesday night, so power had not been interrupted. After restarting the dashboard, all systems appeared to return to normal, but the backlog of Nagios errors would take time to clear.
We proceeded to the Lidar to commence repairs, and removed the PC after saving the configuration and calibration data to the portable HD. The bulk of the work was done at the ISFS trailer, where we were able to scavenge and utilize two 12v batteries and power supplies to create a 24vdc power source for the pc. We have installed both hard drives- the new and the old in the same PC, but they will only be bootable one at a time, no master or slave jumper settings could be identified. Contact me for further details, but rest assured, we retain the entire HD from the previous PC.
Reinstallation took some time, and many complications were uncovered, but the Lidar is operational - to a degree- there is a problem with the RHI scans, but hopefully the Leosphere team in France will be working on it overnight tonight. PPI scans, and DBS scans appear normal.
The PC is online for the Leosphere team in France tonight, but let them work on it first. Contact me for login info, it's all changed from the previous PC.
more info to come tomorrow.
tim
Weather today is cold and partly cloudy. We've been having light snow off and on all day, but there has been no accumulation.
Power amp is at .143 mW
Nagios screen shows critical error with the profiler_any_profiler (checks 3 of 5 have failed). After checking again at 0500 LST, problem corrected itself and no flag appeared.
Profiler appears to be working and power spectral plots look good. Spoke with Bill and he suspects the error has to do possibly with some work that was done earlier. I'll keep an eye on it.
Nagios also show critical errors with the Lidar. System Tech arrived today and is working with Tim on replacing the computer. I imagine that is why the lidar is flagged red.
In looking the profiler display the dashboard buttons are all green, and the dwell counter is working properly.
An IOP has been called for tomorrow evening.
The generator is half full. I was going to refill, but couldn't figure out where how to hook up the tank to my rental car engine. Will ask Tim for help with this when I see him.
Internet connection is out again at main site, but just for the GAUS and ISS computers (same problem as we saw on Sunday night, cause unknown but data is all coming in.
Windy Day!!!
High gusty windstorm from the South has this trailer rocking like I was going down the interstate.
Critical error and stopped 449 was identified by Bill, and I came to investigate. The Dashboard showed the DRX program had stopped, and all it took was a stop and restart of that program to resolve the error. See 449 stopped for reference, including copied output text of normal and abnormal output.
Additional problems were found in the display of the surface data graphing. The graph as shown by the ISS Icon bar ceased to update around 09:00UTC 09oct, even though current time was around 19:00UTC 09oct. Blanking and restarting the display did not fix the error, but stopping and restarting the ISS Icon bar did result in an updated graph. The data continues to appear updated. Apparently the webplots were not affected, according to Bill.
NAGIOS service monitor continues to show 4 critical errors at this time:
metcraxii/profiler_anyfail_surface
metcraxii/profiler_latency_raw/nidas_1_10
metcraxii/profiler_latency_raw/nidas_1_11
metcraxii/profiler_latency_surface/wxt/standard/nc
Nagios shows WXT ping OK, however..... I'll not be going up to the rim to investigate at this time. The winds are howling at 22 M/Sec, and I don't think they would let me outside, nor do I think I could do much good in this wind!
All other systems appear ok, dust is settling on everything, I vacuumed everything I could, including the dust atop the 449 amplifier electronics.
(wow- motion sickness/queasiness from trailer rocking becoming an issue!- I am going outside!)
The 449 profiler had a critical error on Nagios this morning, and the onsite investigation revealed the DRX program had stopped. The Dashboard showed the DRX program had stopped (red "STOP") but showed the products and status as "running".
The pertinent text in the output of the DRX is as follows.
This first block of output is NORMAL. This was the last section of normal data prior to the error, ending at 07:55:00.
Scroll down below to find ABNORMAL output block, This was the first section of abnormal data, starting at 07:55:06.
sd3c revision: 10007
data rate: 0.805 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 5.73e-06s
blanker delay: 0s
blanker width: 7.81e-06s
nsum: 60
comp code len: 4
schedule name: Metcrax
paramset name: Mecrax_RIM
gates: 40
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
384 DAC memory locations being used
Signals blocked in main are:
TTLIn(1:0):11 2013-Oct-09 07:45:22 (37,48 C) 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:32 (37,48 C) 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:42 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:52 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:43 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:23 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:43 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:53 (38,46 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.187 MB/s nopub:0 tags:0 sync:0 0.184 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:03 (37,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.215 MB/s nopub:0 tags:0 sync:0 0.218 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:13 (38,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:23 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:43 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.205 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:03 (38,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:13 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:33 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:43 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:53 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:03 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:13 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:23 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:33 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.208 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:13 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:23 (38,49 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:43 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:03 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:13 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:33 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:53 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:23 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:33 (38,50 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:55:00 (37,48 C) 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0
Shutting down...
Stopping thread for channel 0
Stopping thread for channel 1
Stopping thread for channel 2
This is the next output with the first set of ABNORMAL output of the DRX program.
sd3c revision: 10007
data rate: 7.17 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 2.92e-06s
blanker delay: 0s
blanker width: 4.58e-06s
nsum: 4
comp code len: 1
schedule name: Metcrax
paramset name: rass
gates: 10
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
96 DAC memory locations being used
Signals blocked in main are:
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
This output continued for quite some time before I stopped and restarted the DRX program at roughly 19:00 UTC.
Hope this is useful in debug.
Thanks,
Tim
Gary has reported that the WXT520 installed at the LIDAR on North Rim was down, and not reporting. I investigated, at found no obvious issues. The unit has power, and is constantly making it's wind measurements, and the cable was inspected for (potential) rodent gnawing, none found. I checked and reinstalled the RS232 connector to the digiport, finding no break or abnormality in the connection. I unplugged and repowered the digiport to see if that fixes the problem. There is no Nagios status for the WXT to check, and I cannot find the location of the WXT520 files in the file structures as I know them, so I believe Gary or Bill will have to point me in the right direction to verify status. Hope it's resolved, tell me if it isn't.
I also spent much of the afternoon installing a solar power charging system for the German research group on the rim scintillometer. It should be working now. Installed outdoor lighting at GAUS trailer, and prepared for next potential IOP, could be Thursday night or Friday.
449 running in Metcrax_RIM mode, ok
All other systems, ok
nagios status reports, ok
Gnite, Tim
Daily Report for Monday October 7
Fine today, southerly breeze last night turning to light easterlies, then southerly again by evening. Scattered high cloud, cirrus. Apparently conditions are unfavorable for another IOP for the next few days.
IOP-1 last night went well. Seven soundings were launched, and a tethersonde was operated at the ISFS base, and two other tethers were in the crater. The NCAR windcube scanned into the crater (operated by Nihanth) and the UU Halo lidar was in the crater doing coordinated scans. There are a very sharp inversion seen by the soundings and by RASS, 5C near the ground with the southerly, then 15C up a few hundred meters where there were easterlies (seen by the lidar and 449 wind profiler). Soundings at ISFS were carried out by Tim, Dave and Joanas.
449 Profiler running well. Using "Metcrax" mode which is now 10 minute winds followed by 5 minutes RASS. RASS sound now appears to be stable. Amplifier power is 141W in winds, 199W in RASS.
Did a bunch of 449 tests this evening to check the bandwidth (for freqs see data recorded in the files). Started around 01:54 UT (Oct 8) and continued various tests until 4:06.
From 4;30 to 4:59 used delay line to calibrate range - in standard mode appears to be correct (to better than 50m). Will analyze later to check RIM. Back to regular Metcrax schedule from 5UT.
Training Kate on ISS operations, visiting ISS, lidar, sodar and ISFS base (soundings and ceilometer).
Various data backup activities: DVD on ISS DM; completed disk SHUTTLE449_2 (has data from Oct 1 to Oct 8 5UT) and installed fresh disk (SHUTTLE449_3); installed USB drive on lidar and backed up all data so far.
We met Judy Prossier yesterday and she said that cattle are being moved into our area, so installed an electric fence around the 449 profiler. Tested and no apparent affect on 449 data, but turned off for now until the cattle actually appear.
Generator was serviced today around 21UT. Contrary to expectations, the generator wasn't switched out, but instead was shutdown and serviced in place (apparently they had a problem with their replacement units). Fortunately we were here so we shut down the computers and Kate brought them back up (as a training exercise). The shutdown lasted around 15 minutes so we could have remained operating on the large UPS in the trailer.
I am headed back to Boulder. Kate and Tim are now the operators. Good luck for the rest of the project.
We had a partial network outage in the ISFS trailer that started around 10pm local time on Sunday night. There was a great deal of activity going on in the trailer, that seemed to confuse and complicate the issue, but the initial evidence was that Dave W. lost his wireless connection, and we had to switch him over to a wired connection. Then we noticed that the wired connections in the rear of the trailer that serve the ceilometer, sounding, and data manager computers were not connected to the network... Diagnosis began working from the back to the front, with rebooting the hubs in the rear (linksys) then the middle (d-link), but hesitated at rebooting the white belkin netgear wireless router... Other computers on the d-link (middle) are downstream from the belkin, and DID have outside internet connection. So I was reluctant to risk breaking something that was working, and risk loosing connection to Dave's (critical) laptop during our IOP.
After much diagnosis, I realized there was no other option but to reset the white belkin netgear wireless router, and I finally did about 1:30am, and all seems well after that. I reset the connections to the APG systems in the back, and the backlog of latent connections on Nagios began to clear out. (correction- At 2:15am, still several Nagios services reporting as critical, but are wholeskycamera related to network loss.
Hope I didn't screw things up too much- I blame the lack of sleep!
Thanks, Tim