• Print
• .

Software decimation

lrb 10-jun-2020

pf 17-jun-2020

This work was done with two files: 20182451400_lpwr.dta, a calibration file with sine waves and 80 Hz offset, and 20201540600_lpwr.dta, a regular chirping file.

Decimate

Matlab documents for decimate say that when the compression factor is greater than 13, it is better to divide that into several factors and call the function multiple times. Here we need a compression factor of 32, so we test whether splitting the factor makes a difference in the final result. We use an fir filter of order 30 for this compression, and there is no windowing or chirp extension with tails.

The three different scenarios are:

1. Decimate once with a factor of 32
2. Decimate twice, first by 2 and then by 16
3. Decimate twice, first by 4 and then by 8

Plotted below are the timeseries for all channels for the last chirp of the acquisition. For the calibration data (Figure 1), all three compressed timeseries are similar to the raw timeseries, though with choppy peaks rather than smooth peaks; additionally, the three compressed timeseries appear identical to one another. For the chirping data (Figure 2), the three compressed timeseries are very similar to the raw timeseries, and a zoom on the tails of the chirp return (Figure 3) shows that they are very close if not truly identical.

Looking at the same chirp but only 1 channel and only the first 20 samples of that chirp show that the three compressed timeseries are not actual identical: the peak amplitudes for the calibration data (Figure 4) and the exact values through the curve of the chirp return for the chirping data (Figure 5) are slightly different.

The spectra of the timeseries for all channels for the last chirp of the acquisition for the raw data and all three compressed timeseries is below. For the calibration data (Figure 6), there are peaks at 0 Hz, 80 Hz, and ~160 Hz in all four cases, but there are some differences otherwise. For example, all three compressed timeseries have peaks at 130 Hz which are lacking in the raw data. For the chirping data (Figure 7), the general shape and the peak at 120 Hz are the same for the compressed timeseries and the raw data, but the details of the channels differ.

Looking at the differences between the raw spectrum and the compressed spectra, none of the decimation compression tests seem to represent the calibration raw data better than the others (Figure 8). For the chirping data (Figure 9), however, the decimate once by 32 scenario is better in the lower frequencies.

Looking at the differences between the three compressed spectra shows that while the single decimation by 32 is different from the two cases with two rounds of compression, the latter two are identical for both the calibration data (Figure 10) and the chirping data (Figure 11). All three compression timeseries spectra are identical at the offset frequency of 80 Hz in the calibration data and near 0 Hz for the chirping data as well.

In short, there seems to be no advantage to splitting the decimation into two steps, and it may actually be a hindrance.

Windowing

Pierre noted that the way we have been applying and unapplying a window to every chirp seems to be causing some amplitude issues. The amplitude of a calibration signal passing through the compression software should be constant.

The two scenarios here, which do a single 32-factor decimation and do not extend the chirp with tails, are:

1. No windowing
2. Apply and unapply the window

Plotted below are timeseries of the raw data, the two compression scenarios, and the difference between those latter two. In the calibration data (Figure 12), the amplitude of the no window case is constant for the entire chirp like the raw data, but for the windowed case the amplitude is noticeably larger at the end of the chirp. The difference between the tests shows that there is a slight difference at the beginning of the chirp but a large difference at the end. In the chirping data (Figure 13), there does not appear to be an amplitude change at the end of the chirp from flat; however, the difference between the tests shows that there is a large difference at the beginning and end of the chirp.

The spectra for the calibration data (Figure 14) show that the there are small differences between the frequency content of the two scenarios, though they are identical at the 80 Hz offset. The no window case better preserves the drops in spectral amplitude immediately outside the the 80 Hz peak. Both compressed scenarios have a peak at ~130 Hz that is not in the raw data. For the chirping data (Figure 15), there are no noticeable differences in the two compressed spectra, and the difference between them is very small.

The windowing does not seem to add anything to the process but makes it more complicated, though the spectral impacts appear to the quite low.

Tails on the chirp

Because decimation and filtering can have effects at the edges of a timeseries, we normally have tails on both sides of the chirp that we chop off later to remove those effects.

There are two scenarios here, both of which do not use windows but do a single 32-factor decimation:

1. No tails
2. Tails

Plotted below are the timeseries of the raw data, compressed data without using tails, compressed data with using tails, and the difference between the tails and the no-tails cases. For the calibration data (Figure 16), the two compressed timeseries are a little different from one another, which can only really be seen at the very beginning of the timeseries. This is also true in the chirping data (Figure 17).

The spectra are similar for the calibration data (Figure 18), but the no tails case is smoother but lacking the 160 Hz peak, while the tails case is rough like the raw data but has a spurious peak at 130 Hz. For the chirping data (Figure 19), the tails cause a ripple effect to the spectrum (seen in the raw data and in the tails timeseries). This rippling is definitely due to the tails, as increasing them to twice their size actually increases the rippling (Figure 20).

The tails smooth out the spectrum, but may also add in a ripple which needs further investigation.

Windows and tails together

Because windows and tails can both have impacts on the edges of the data, a test was done with both together.

The four situations plotted below are:

1. No window, no tails, a single 32-factor decimation
2. No tails, a window, a single 32-factor decimation
3. No window, both tails, a single 32-factor decimation
4. A window, both tails, a single 32-factor decimation

All four scenarios have data that looks similar, though the increase in amplitude towards the end of the chirp is obvious in the two that use windowing for the calibration data (Figure 21). A slight increase in amplitude is visible at the very end of the chirp for the scenario with just windowing and no tails for the chirping data (Figure 22).

Looking at the differences between the plain scenario with no tails or windowing and the other three for the calibration data (Figure 23) shows similar, though not identical, increases in amplitude due to the windowing for the two applicable scenarios, while the other shows a convergence towards the plain scenario through the period. This convergence may be present in the tails/window combination scenario, but it is difficult to determine because of the small amplitudes of the differences before the flaring at the end. The chirping data (Figure 24) show similar trends with flaring at the end of the chirp due to windowing and a spread that converges towards the plain scenario visible in the tails scenarios.

The presence of tails cuts the drop in spectral amplitude outside the main 80 Hz peak in the calibration data (Figure 25). The impact of windowing is not clear from this plot. For the chirping data (Figure 26), using tails once again leads to a ripple effect in the spectrum, while the impact of windowing is unclear.

The decrease in off-peak drop in the calibration spectra is visible in the difference plots (Figure 27), and the ripple is visible in the difference spectra for the chirping data (Figure 28).

Bit shifting

This is a test of whether when we cut out the least significant bits makes any difference in the final result. Because we are compressing while keeping spurious bits, things may be different. On the other hand, the spurious bits are the least significant ones, so it may make no difference at all.

The two scenarios are:

1. Shifting the bits right after reading the data in but before compression
2. Shifting the bits right before writing the data out, so after the compression

Timeseries and spectral comparisons show that the timing of the shifting makes no difference at all.

Conclusions

1. Decimating once versus twice does change things, though the specific split of the factors is irrelevant. Also it doesn't seem inherently better to do it twice, and may in fact look worse for chirped signals.
2. Windowing and unwindowing changes the amplitudes at the end of the chirp and should be avoided.
3. Adding tails on the chirp does cause a change in the first few and last few samples, but otherwise not so much. Spectrally, however, using tails makes the spectrum ripple, and increasing the size of the tails increases the frequency of the rippling.
4. The timing of the bit shifting is irrelevant.