From the logs of the check_trh process on flux I see these entries since it was started on April 9. For some reason the higher TRHs had some issues yesterday.
TRH problems
Times in MDT: fgrep cycling /var/log/messages* Apr 18 18:49:09 flux check_trh.sh: 300m temperature is 137.88 . Power cycling port 5 Apr 23 13:03:58 flux check_trh.sh: 300m temperature is 174.1 . Power cycling port 5 Apr 23 13:06:58 flux check_trh.sh: 300m temperature is 174.28 . Power cycling port 5 Apr 23 13:08:18 flux check_trh.sh: 200m temperature is 181.61 . Power cycling port 5 Apr 23 13:08:38 flux check_trh.sh: 300m temperature is 174.28 . Power cycling port 5 Apr 23 13:08:58 flux check_trh.sh: 200m temperature is 181.53 . Power cycling port 5 Apr 23 13:09:48 flux check_trh.sh: 300m temperature is 174.06 . Power cycling port 5 Apr 23 13:16:48 flux check_trh.sh: 200m temperature is 179.15 . Power cycling port 5 Apr 23 13:19:18 flux check_trh.sh: 250m temperature is 173.33 . Power cycling port 5 Apr 23 13:30:38 flux check_trh.sh: 200m temperature is 177.04 . Power cycling port 5 Apr 23 13:48:38 flux check_trh.sh: 250m temperature is 171.63 . Power cycling port 5 Apr 23 13:50:48 flux check_trh.sh: 250m temperature is 173.08 . Power cycling port 5
Yesterday I reworked things so that the check script is run on each DSM. The only entries after that are at 300m
Times in UTC ssh 300m fgrep cycling /var/log/isfs/dsm.log Apr 23 19:27:33 300m root: temperature is -62.52 . Power cycling port 5 Apr 23 19:28:25 300m root: temperature is -62.52 . Power cycling port 5 Apr 23 19:29:41 300m root: temperature is -62.52 . Power cycling port 5