...
I had to install xinetd on the DSM to get check_mk to work from snoopy, and then nagios noticed that the USB filesystem was missing. So I decided to upgrade the whole DSM before rebooting it, and it turns out there were quite a few updates. The USB came back and has not failed all night, but it's good that it's getting checked now.
I also modified the chrony.conf to add the iburst option to the tardis server line. That makes chrony sync much faster to the network NTP server, and it makes sense to rely on that since the PPS is not working.
Finally, I discovered a bug with the data_stats and dashbaoard, where the data_stats will just keep counting up from 2010 if the system time was wrong when data_stats started. The quick workaround was to restart the json_data_stats service after the system time was correct.
The DSM went down again at 8:20 this morning. Dan is at Marshall this morning and I asked him to reboot, which is presumably why it came back up again around 11:30. It looks like now it's down again, thoughrebooted the DSM around 11:30, which brought it back on the net. I looked through the logs and it seems like this time the DSM was collecting data while it was off the net, so it may have just been an ethernet issue. DSM is back off the net now though, about 10 minutes after Dan rebooted it.
Since the DSM clearly still isn't reliable, I'm thinking the next step would be setting up a whole new dsm + SD card as tt when I'm in the office tomorrow...
...