-------- Forwarded Message --------
Subject: warning: invalid Kaena "kna" radial data 2013 to 11/30/2019
Date: Sun, 1 Dec 2019 17:59:57 -1000
From: Pierre Flament <pflament@hawaii.edu>
To: hfrnet administrators <hfrnet.administrators@sio.ucsd.edu>
1. Synopsis
2. Actions
3. Q&A
__
1. Synopsis:
- erroneous insertion of a spurious byte between the ascii meta-data header and the raw antenna time series was discovered
- extra byte was probably inserted when configuring the site, by using vi editor instead of bvi editor on the header file, automatically appending a new-line "\n" upon write
- each integer*16 time series sample consists then of 8 of the least significant bits from the previous sample, the sign bit and 7 of the most significant bit of the current sample, with the new sign bit being random, and the old sign bit falling in the middle
- spurious clipping and random sign changes resulted in corrupt Fourier transforms and corrupt phases, preventing beamforming (see attached maps before and after repair, which was performed on 20193341500).
- all Kaena radial data provided are invalid, including derivative data partly based on them (vector currents, assimilating model output)
What was done:
char*19 char*478 int16 int16 int16
"time" "header" ("\n" sample1.byte1) (sample1.byte2 sample2.byte1) (sample2.byte2 sample3.byte1) ...
What should have been done:
char*19 char*478 int16 int16 int16
"time" "header" (sample1.byte1 sample1.byte2) (sample2.byte1 sample2.byte2) (sample3.byte1 sample3.byte2) ...
If the good data was:
int16 int16 int16
(sabcdefg hijklnop) (sabcdefg hijklnop) (sabcdefg hijklnop) ....
It was erroneously written as:
int16 int16 int16
("\n" sabcdefg) (hijklnop sabcdefg) (hijklnop sabcdefg) ....
(where letters denote 0's or 1's with a the most significant and where s is the sign bit)
2. Actions:
- all Kaena data and derivative until 11/30/2019 should be expunged immediately from all servers
- all users registered to have downloaded the data should be warned not to use them
3. Q&A:
- is the current real-time data for Kaena good? YES, the bug was fixed at 15:00 UTC on 11/30/2019.
- can the archived data be repaired? NO; because we use truncated Fourier transforms, we cannot reconstitute the original time series and byte-shift them
- are other Hawaii radars affected? NO; it has been verified that only Kaena is affected and of course combined Kaena-Kapolei vector currents
- is there a mean of recovering valid data? may be YES; until September 2016, some if not most of the raw time series were archived at 6 kHz on external disks, which can be slowly reloaded and reprocessed with the correct header; since October 2016, the raw time series were decimated in matlab to 750 Hz, and archived as compressed .mat files on external disks, easier and faster to reload and reprocess
- what is the schedule for recovering/reprocessing the available radial data? data since October 2016 will be reloaded and reprocessed by the end of April 2020 at the latest; data prior to September 2016 will only be reloaded and reprocessed depending on staff availability
- what is the schedule for regenerating combined vector currents? radlab does not generate vector currents, nor has software to do so; once the radial data is reprocessed and validated, it will be made available to the HFRnet server and to the PacIOOS server, which have their own policy/procedures for reprocessing older data
- how could such problems be avoided or detected earlier? they are currently unavoidable because we lack reasonable support staff for real time operations; we currently only have 1 technical staff for maintenance of the radars, but no staff for programming, archiving and quality control, thereby left to chance
