Dan and I spent the morning trying to fix as many csat/ec150/ec100 issues as we can with the spares we have.

MW: ec100 was reporting yesterday, but winds and gas were both all nans. By the time we got there this morning, the ec100 was not reporting at all, with red status lights for both gas and sonic. After replacing the ec100 we did receive data, but the winds were still all nans, so we replaced the sonic head as well. Now the whole system is working.

PRS:

  • 1m sonic: ec100 not reporting anything, and looking like it never had reported anything since it got set up. In ec100 box gas light was red, power and sonic lights were green. After replacing ec100 the entire system is working.
  • 7m sonic: was reporting data but all bad winds, diag value of 16 ("re-acquiring signal"). After tilting the tower down it appears that the heating wire on the sonic shorted:
  • We replaced the sonic head and now the whole system is working.
  • 17m sonic: ec100 had been unresponsive, but was reporting when we arrived at site, but then became completely unresponsive again before we got around to replacing it. data_stats output from the current nidas file showed a start time of 01 03 22:22:42.056 (which I think must be wrong, this file's begin time was 20220104_120000 and the previous file didn't show any data from port 2), and an end time of 01 04 18:23:29.970, shortly after we checked it on site. Anyhow, in the ec100 box the power light was blinking red and the sonic + gas lights were blinking yellow. When we plugged in a spare ec100 it had the same light situation, so we eventually tracked the problem down to port 2 on the prst dsm, which is not properly powering the ec100. There are no spare ports on the dsm so we will need to swap the dsm itself to get eight working ports. Tentatively planning to do this tomorrow since we also need to move radiometers at prs tomorrow. When we plugged the 17m ec100 into a different port it looks like it is reporting data, some nans for winds but some good wind data.
  • 32m sonic: was reporting data but all bad winds, diag value of 16ish. Replaced the sonic head, after that entire system is working.

Dan and I have been wondering about the effect of the deicing setup and if that's causing our problems. Before we left this morning we saw that the ldiag value at SH has been consistently 1 since 12/21/21 12:07MST/19:07UTC, and from the csat_heat log the first time deicing ran was 2021-12-21 19:00:05,078, which could mean that running the heating system killed the sonic. We have yet to visit SH so don't know yet whether it shorted like the 7m prst sonic did. This afternoon I looked into other sonics to see if I found this pattern again and mostly didn't. Ldiag was consistently 1 at prst 7m after  13:02 MST 12/21, which is around the time I was deploying the deicing script, but I can't check the heater log right now because prst is down.

Since we were worried about deicing, this morning I logged in and commented out the crontab entry for the csat_heat script in all dsms except dcst (which was and is offline–originally serial card issues, now according to Steve modem issues). So at least now the sonics that we replaced should stay operational, if it's the heating that's interfering with that.

After our rounds SH still has a sonic that's giving bad winds but we don't have any more known good spares. Dan has been experimenting with the ec100 windows utility and the bad ec100s and csats and did manage to get a working csat/ec100 combo with the prst 7m sonic and the prst 1m ec100, but since that's the csat that got shorted we don't feel great about re-deploying it as a spare yet. Dan has also called Campbell and gotten some troubleshooting advice and assistance.


  • No labels

1 Comment

  1. Isabel Suhr AUTHOR

    Looking at the heater log on prst, the the heater first runs for 20 seconds while I was debugging at 1908 UTC, then runs for 5 minutes for the first time at 2000 UTC/1300 MST, which is the same time that ldiag went consistently bad. Since this is the sonic that got shorted, I am very interested to see what the sonic at SH that also had this pattern looks like...

    2021-12-21 19:08:32,966: Relay 8 state 0
2021-12-21 19:08:33,053: Relay 8 state 1
2021-12-21 19:08:53,593: Relay 8 state 0
2021-12-21 19:10:03,276: Relay 8 state 0
2021-12-21 19:15:03,787: Relay 8 state 0
2021-12-21 19:20:03,039: Relay 8 state 0
2021-12-21 19:25:03,392: Relay 8 state 0
2021-12-21 19:30:03,596: Relay 8 state 0
2021-12-21 19:35:03,795: Relay 8 state 0
2021-12-21 19:40:03,148: Relay 8 state 0
2021-12-21 19:45:03,372: Relay 8 state 0
2021-12-21 19:50:03,703: Relay 8 state 0
2021-12-21 19:55:03,902: Relay 8 state 0
2021-12-21 20:00:03,650: Relay 8 state 0
2021-12-21 20:00:03,707: Relay 8 state 1
2021-12-21 20:05:03,804: Relay 8 state 0
2021-12-21 20:05:03,942: Relay 8 state 0