Sites 4, 9, 10, and 13 online, but 11, 12, and 14 offline
I just talked with John. They setup 4 and 14 today, but 14 never came online. Both 14 and 11 are Vipers, so John wonders if that's part of the problem. John said he had poor cell coverage for his third-party cell phone, but Matt has Verizon and his signal was adequate. It seems unlikely that this is a signal strength problem. Both 11 and 14 had some trouble during the outside testing, as noted in JIRA issues ISFS-99 and ISFS-103.
Site 12 went offline today around 11:13 am MDT, and at the moment it has not come back. Site 12 is a Titan.
Possible Workarounds
Maybe we'll have to implement something to restart the cell modem interface if it appears to be down... However, in the meanwhile, here are a couple things to try. The first is to unplug the USB modem and then plug it back in. The second achieves something similar but without unplugging anything, and it has worked once before. Log into the DSM and enter these commands:
sudo su
ifdown eth1
rmmod cdc_ether
rmmod usbnet
modprobe cdc_ether
exit
Finally, John will check on the space on the media disk on each DSM. Probably there is plenty of space, but we don't need any of the pre-project data, and we should make sure to give ourselves as much time as possible to record offline until we can visit the site and get the Internet connection up.
1 Comment
Steve Oncley
In the past, Gordon has put a crontab script on the DSMs to cycle power to our old cradlepoint modems if the station detects that the network is down. A version of it is at:
/net/isf/isfs/svn/admin/data_systems/dpkgs/isff-scripts/usr/local/scripts/router_check.sh
I note that this won't work for VERTEX, since the DSMs can't simply power cycle the USB modems, however this could be a template for inserting Gary's module commands above.