--------Message --------
Subject: quick fixes for removing extra byte
Date: Sun, 1 Dec 2019 23:00:31 -1000
From: Pierre Flament <pflament@hawaii.edu>
To: radlab
1. through shell script:
sunrise:/home/sunset0/data/kna/realtime> cat cleanraw
#!/bin/csh -f
dd if=$1 bs=512 count=1 >! $2
dd if=$1 bs=512 skip=513 iflag=skip_bytes >> $2
sunrise:/home/sunset0/data/kna/realtime> cleanraw 20130932330_kna.RAW.old 20130932330_kna.RAW
sunrise:/home/sunset0/data/kna/realtime> ls -l 20130932330_kna*RAW*
-rw-rw-r-- 1 wera users 251658752 Dec 1 22:44 20130932330_kna.RAW
-rw-rw-r-- 1 wera users 251658753 Apr 3 2013 20130932330_kna.RAW.old
2. through matlab m-file
sunrise:/home/sunset0/data/kna/realtime> cat cleanraw.m
% cleanraw: remove extra header byte
% define filein and filout before calling
IQ=2; % number of channels to make pair
NANT=16; % the WERA number of antennas
MT=1920; % the number of WERA samples/chirp
NCHIRP=2048; % the number of chirps
% read binary wera data in .RAW format of short int16
f=fopen(filein,'r','ieee-le')
header=uint8(zeros(512,1));
skip=uint8(zeros(1,1));
raw=int16(zeros(IQ*MT*NANT*NCHIRP,1));
header=fread(f,512,'uint8=>uint8');
skip=fread(f,1,'uint8=>uint8');
raw=fread(f,IQ*MT*NANT*NCHIRP,'int16=>int16');
fclose(f);
% output the wera .RAW file
f=fopen(fileout,'w','ieee-le');
fwrite(f,header,'uint8');
fwrite(f,raw,'int16');
fclose(f);
-------- Forwarded Message --------
Subject: warning: invalid Kaena "kna" radial data 2013 to 11/30/2019
Date: Sun, 1 Dec 2019 17:59:57 -1000
From: Pierre Flament <pflament@hawaii.edu>
To: hfrnet administrators <hfrnet.administrators@sio.ucsd.edu>
1. Synopsis
2. Actions
3. Q&A
__
1. Synopsis:
- erroneous insertion of a spurious byte between the ascii meta-data header and the raw antenna time series was discovered
- extra byte was probably inserted when configuring the site, by using vi editor instead of bvi editor on the header file, automatically appending a new-line "\n" upon write
- each integer*16 time series sample consists then of 8 of the least significant bits from the previous sample, the sign bit and 7 of the most significant bit of the current sample, with the new sign bit being random, and the old sign bit falling in the middle
- spurious clipping and random sign changes resulted in corrupt Fourier transforms and corrupt phases, preventing beamforming (see attached maps before and after repair, which was performed on 20193341500).
- all Kaena radial data provided are invalid, including derivative data partly based on them (vector currents, assimilating model output)
What was done:
char*19 char*478 int16 int16 int16
"time" "header" ("\n" sample1.byte1) (sample1.byte2 sample2.byte1) (sample2.byte2 sample3.byte1) ...
What should have been done:
char*19 char*478 int16 int16 int16
"time" "header" (sample1.byte1 sample1.byte2) (sample2.byte1 sample2.byte2) (sample3.byte1 sample3.byte2) ...
If the good data was:
int16 int16 int16
(sabcdefg hijklnop) (sabcdefg hijklnop) (sabcdefg hijklnop) ....
It was erroneously written as:
int16 int16 int16
("\n" sabcdefg) (hijklnop sabcdefg) (hijklnop sabcdefg) ....
(where letters denote 0's or 1's with a the most significant and where s is the sign bit)
2. Actions:
- all Kaena data and derivative until 11/30/2019 should be expunged immediately from all servers
- all users registered to have downloaded the data should be warned not to use them
3. Q&A:
- is the current real-time data for Kaena good? YES, the bug was fixed at 15:00 UTC on 11/30/2019.
- can the archived data be repaired? NO; because we use truncated Fourier transforms, we cannot reconstitute the original time series and byte-shift them
- are other Hawaii radars affected? NO; it has been verified that only Kaena is affected and of course combined Kaena-Kapolei vector currents
- is there a mean of recovering valid data? may be YES; until September 2016, some if not most of the raw time series were archived at 6 kHz on external disks, which can be slowly reloaded and reprocessed with the correct header; since October 2016, the raw time series were decimated in matlab to 750 Hz, and archived as compressed .mat files on external disks, easier and faster to reload and reprocess
- what is the schedule for recovering/reprocessing the available radial data? data since October 2016 will be reloaded and reprocessed by the end of April 2020 at the latest; data prior to September 2016 will only be reloaded and reprocessed depending on staff availability
- what is the schedule for regenerating combined vector currents? radlab does not generate vector currents, nor has software to do so; once the radial data is reprocessed and validated, it will be made available to the HFRnet server and to the PacIOOS server, which have their own policy/procedures for reprocessing older data
- how could such problems be avoided or detected earlier? they are currently unavoidable because we lack reasonable support staff for real time operations; we currently only have 1 technical staff for maintenance of the radars, but no staff for programming, archiving and quality control, thereby left to chance