The 449 profiler had a critical error on Nagios this morning, and the onsite investigation revealed the DRX program had stopped. The Dashboard showed the DRX program had stopped (red "STOP") but showed the products and status as "running".
The pertinent text in the output of the DRX is as follows.
This first block of output is NORMAL. This was the last section of normal data prior to the error, ending at 07:55:00.
Scroll down below to find ABNORMAL output block, This was the first section of abnormal data, starting at 07:55:06.
sd3c revision: 10007
data rate: 0.805 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 5.73e-06s
blanker delay: 0s
blanker width: 7.81e-06s
nsum: 60
comp code len: 4
schedule name: Metcrax
paramset name: Mecrax_RIM
gates: 40
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
384 DAC memory locations being used
Signals blocked in main are:
TTLIn(1:0):11 2013-Oct-09 07:45:22 (37,48 C) 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:32 (37,48 C) 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:42 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:52 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:43 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:23 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:43 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:53 (38,46 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.187 MB/s nopub:0 tags:0 sync:0 0.184 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:03 (37,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.215 MB/s nopub:0 tags:0 sync:0 0.218 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:13 (38,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:23 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:43 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.205 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:03 (38,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:13 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:33 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:43 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:53 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:03 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:13 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:23 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:33 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.208 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:13 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:23 (38,49 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:43 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:03 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:13 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:33 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:53 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:23 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:33 (38,50 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:55:00 (37,48 C) 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0
Shutting down...
Stopping thread for channel 0
Stopping thread for channel 1
Stopping thread for channel 2
This is the next output with the first set of ABNORMAL output of the DRX program.
sd3c revision: 10007
data rate: 7.17 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 2.92e-06s
blanker delay: 0s
blanker width: 4.58e-06s
nsum: 4
comp code len: 1
schedule name: Metcrax
paramset name: rass
gates: 10
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
96 DAC memory locations being used
Signals blocked in main are:
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
This output continued for quite some time before I stopped and restarted the DRX program at roughly 19:00 UTC.
Hope this is useful in debug.
Thanks,
Tim