ncar4 stopped reporting data around 15:30 CDT on June 23rd. This was around the time that I was hacking on ncar3 and ncar4 from the motel in Ames.
On 6/24 around 10:55 Dan and I visited the site. The system was spewing messages to the console port, the primary error was "ntpd page allocation error", indicating that the system had run out of memory. All voltages look OK.
Around the time it died, I was testing a "sv" command to show the system voltage from the power mote. For some weird reason I couldn't get a
mote_dump | fgrep | awk
pipeline to work, it would never show data. What finally worked was
mote_dump | awk
. Seems like a bug in the busybox linux. Perhaps in all the hacking I started an awk command which somehow ate up the memory.
So we just power cycled the system, and monitored it.
free -m
shows 52 Mbytes free.
Later I realized we had seen this once at playa1 during PCAPS: https://wiki.ucar.edu/x/nJhCB Wouild be interesting to check if it is the same box. The box was not replaced during PCAPS and ran fine the rest of the project.