The turbulence tower data system clock was behind by 1 second from May 4 to Jun 11. If sub-second absolute time-tag accuracy of the data samples is a not a concern, then you can ignore this log message. Otherwise, here are the gory details...
The data system was replaced on the turbulence tower on May 4.
The new system is running a new version (4.2.6p1) of the NTP (network time protocol) service. NTP on the data system is configured to use the attached GPS (Garmin 25-HVS) as a reference clock. NTP reads the NMEA messages from the GPS, which contain the absolute time, at a precision of a second, and it monitors the PPS (pulse-per-second) signal from the GPS. The leading edge of the PPS has approximate 1 microsecond absolute accuracy. Using the two inputs, NTP can condition the embedded data system clock to an absolute accuracy which is typically better than 300 microseconds.
The NMEA messages from the 25-HVS have a roughly 0.4 - 0.8 second lag, meaning the NMEA message for 00:00:00 arrives at the data system between 0.4 and 0.8 seconds late, at sometime after 00:00:00.4
The new version of NTP must be passed a parameter, known as "time2", specifying the approximate lag in seconds of the NMEA messages. If the value is left as the default, 0.0, then NTP assigns the time of a PPS with the time of the most recently received NMEA message, which results in a Linux system clock that is exactly 1 second behind absolute time.
On Jun 11 I detected this issue when fiddling with a data system in the FLAB parking lot, where I also had access to a NTP server on the network.
Here's the last 2 points of data from the GPS on the old system on May 4 at 18:33:34 UTC. There is a gap of 3848 seconds while the data system was swapped out, and the first 2 GPS data points from the new system at 19:37:42.
data_dump -i 1,30 -A manitou_20100504_120000.bz2 | more ... 2010 05 04 18:33:34.7163 1.127 73 $GPRMC,183334,A,3906.0321,N,10506.3310,W,000.0,289.2,040510,010.5,E*60\r\n 2010 05 04 18:33:34.8746 0.1583 72 $GPGGA,183334,3906.0321,N,10506.3310,W,1,06,1.7,2396.4,M,-21.3,M,,*46\r\n 2010 05 04 19:37:42.3816 3848 73 $GPRMC,193743,A,3906.0352,N,10506.3241,W,000.0,000.0,040510,010.5,E*65\r\n 2010 05 04 19:37:42.5430 0.1614 72 $GPGGA,193743,3906.0352,N,10506.3241,W,1,06,4.2,2391.6,M,-21.3,M,,*47\r\n
The absolute time in the NMEA messages is the HHMMSS field between the commas after $GPRMC or $GPGGA. The $GPRMC NMEA message for 183334 was received and timetagged with a system clock value of 18:33:34.7163, which is a typical lag of .7163 seconds. After the gap, the first NMEA message with a value 193743 was assigned a timetag of 19:37:42.3816, indicating that the system clock was 1 second early.
On Jun 11 the value of the "time2" parameter was changed to 0.8 seconds and ntp restarted on the data system. By looking at the data_dump it appears that NTP corrected the system clock around 20:53:43 UTC.
2010 06 11 20:53:42.1074 0.9465 73 $GPRMC,205343,A,3906.0359,N,10506.3337,W,000.0,206.3,110610,010.5,E*66\r\n 2010 06 11 20:53:42.2698 0.1624 72 $GPGGA,205343,3906.0359,N,10506.3337,W,1,06,1.6,2398.5,M,-21.3,M,,*4F\r\n 2010 06 11 20:53:44.7440 2.474 73 $GPRMC,205344,A,3906.0358,N,10506.3338,W,000.0,206.3,110610,010.5,E*6F\r\n 2010 06 11 20:53:44.9072 0.1632 72 $GPGGA,205344,3906.0358,N,10506.3338,W,1,06,1.6,2398.7,M,-21.3,M,,*44\r\n
NTP was adjusting the Linux system clock for a minute or two prior to 20:53:43 and so the accuracy of those timetags is unknown. For example, the 20 Hz sonic data shows a jump forward of 1.598 seconds at this time. It looks like NTP and the Linux kernel first slowed the the clock down for a period of time and then made a ~1.55 second jump forward.
data_dump -i 1,100 -H ../raw_data/manitou_20100611_120000.dat | more ... 2010 06 11 20:53:42.7157 0.05111 12 48 00 26 02 fc 01 d7 09 d0 0f 55 aa 2010 06 11 20:53:42.7647 0.04894 12 55 ff 9a 01 64 02 e8 09 d1 0f 55 aa 2010 06 11 20:53:44.3625 1.598 12 8a fe f0 01 45 02 53 0a d2 0f 55 aa 2010 06 11 20:53:44.4117 0.04916 12 d9 ff 35 02 0d 02 8e 0a d3 0f 55 aa 2010 06 11 20:53:44.4631 0.0514 12 37 01 42 02 d9 01 b9 0a d4 0f 55 aa
After the correction on Jun 11 the absolute accuracy of the data system clock should be generally better than 300 microseconds.
Much ado about nothing...