The tt DSM went down last night right before 11 PM local, and it has not come back.  The radio ttstation has been up the whole time, so this does not appear to be related to spol.  I'll try to get out there this morning to look at it.


  • No labels

1 Comment

  1. Gary Granger AUTHOR

    I visited the site yesterday afternoon and restarted the DSM.  Details follow.

    Obviously there was power at the site, I could hear fans, I could plug into ttstation and ping it. No response from tt console, solid green and red lights on Pi next to SD card.

    Cycled power to Pi using Pi's power plug, leaving power on everything else. Then I could log into the console, but for some reason I did not see boot messages on console on power-up. I could ssh into it also.

    No messages in {/var/log/messages} for the past four days, until the reboot:

    Sep 19 00:00:17 tt rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="408" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Sep 20 00:00:05 tt rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="408" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Sep 21 00:00:05 tt rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="408" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Sep 22 00:00:05 tt rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="408" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Feb  1 00:00:06 tt kernel: [    0.000000] Booting Linux on physical CPU 0x0
Feb  1 00:00:06 tt kernel: [    0.000000] Linux version 5.10.52-v7+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1440 SMP Tue Jul 27 09:54:13 BST 2021
Feb  1 00:00:06 tt kernel: [    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
Feb  1 00:00:06 tt kernel: [    0.000000] CPU: div instructions available: patching division code
...
Feb  1 00:00:39 tt gpsd[1192]: gpsd:WARN: date more than a year in the future!
Sep 22 20:32:05 tt rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="412" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Sep 22 20:32:27 tt kernel: [   69.336512] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com
Sep 22 20:32:32 tt kernel: [   75.014225] ftdi_sio ttyUSB10: FTDI USB Serial Device converter now disconnected from ttyUSB10
Sep 22 20:32:32 tt kernel: [   75.014301] ftdi_sio 1-1.5.3:1.1: device disconnected
Sep 22 20:32:33 tt kernel: [   75.231290] ftdi_sio ttyUSB11: FTDI USB Serial Device converter now disconnected from ttyUSB11
Sep 22 20:32:33 tt kernel: [   75.231367] ftdi_sio 1-1.5.3:1.2: device disconnected
Sep 22 20:33:30 tt kernel: [  132.261769] ftdi_sio ttyUSB12: FTDI USB Serial Device converter now disconnected from ttyUSB12
Sep 22 20:33:30 tt kernel: [  132.261861] ftdi_sio 1-1.5.3:1.3: device disconnected

    Restarted dashboard data, and then dashboard was reporting all green again:

    systemctl --user restart json_data_stats.service

    I could find no evidence as to why the Pi hung.

    While there, I heard the fan drop out and then start up, then noticed that SPol was scanning again; could have been some kind of intermittent surveillance scan. It locked up my laptop too. So I suppose the Pi lockup the previous night could have been caused by a spol scan, except usually the radio goes out too, and usually the Pi reboots rather than hangs.

    Anyway, the DSM seems to be working again, except for some strange behavior in the archive stream which has been added to this issue: ISFS-306.

    The SPOL scans also caused my laptop to lock up twice, and the second time resulted in some filesystem corruption which took some time to fix. No more working at the site while SPOL is running.