rsw02 is the PC104 Titan DSM which is only samples a Setra analog pressure sensor.  It went down about 9 May 05:30 UTC, so I cycled power on it using 'pio aux' on rsw02x.  It's back up now and sampling pressure at 20 Hz.

...Actually, it only stayed up for a few minutes and then stopped again, so I cycled power a second time.  The problem appears to be related to Diamond A/D interrupts (same as in ISFS-149?), but I have not investigated any further.  rsw02 seemed to stable for a long time, so I wonder what changed.

> less /var/log/messages
...
May  7 10:05:42 localhost kernel: pc104_irq_watchdog bark! #4107243, quiet IRQs=  5
May  7 10:10:42 localhost kernel: pc104_irq_watchdog bark! #4109043, quiet IRQs=  5
May  7 10:15:42 localhost kernel: pc104_irq_watchdog bark! #4110844, quiet IRQs=  5
May  7 10:20:42 localhost kernel: pc104_irq_watchdog bark! #4112644, quiet IRQs=  5

> irqs
IRQ      Interrupt Type            Total Int  Int/sec   
------------------------------------------------------
5:       PC104 dmd_mmat:           50         10        
19:      SC ohci_hcd:usb1:         26         5.2       
42:      SC ost0:                  237        47.4      
126:     GPIO eth0:                67         13.4      
129:     GPIO GPIO_17-PC104:       20         4

2 Comments

  1. Gordon Maclean

    Those watchdog barks are probably not the problem.  I'll bet they have been happening since the system started.  It logs the count every 5 minutes, it looks like a consistent 1800 barks in 300 seconds, or 6 barks/sec.

    I think the watchdog wakes up every 0.1 second to check if activated PC104 interrupts are quiet. This is more frequent than necessary for the dmd_mmat (but I did that to lessen possible problems in the aircraft).  IRQ 129 (GPIO_17) is happening at 4 HZ, which are the actual interrupts from the diamond.  The 10 Hz irqs on IRQ 5 (dmd_mmat) are the result of the watchdog barking unnecessarily an extra 6 times/sec.

    Does Vdsm look OK?  Was the pressure data good (20 Hz and believable values) right up to the crash?

     

  2. Gary Granger AUTHOR

    This DSM (rsw02) is powered by the auxiliary port of a Pi DSM directly above it (rsw02x), but the power mote on rsw02x is reading 13 Volts, so I think that rules out power as a problem.  The pressure values also have appeared reasonable, 944.9 for the most recent reading available at the moment, at 11 May 13:21 UTC.