FitACF
From Sharewiki
What is FITACF?
FITACF is a set of computer routines that provide the pathway from data sampled by Super Dual Auroral Radar Network (SuperDARN) radars to the derived data products. These data products are the Doppler velocity, spectral width, received power and elevation angle. We understand from Kile Baker that the basics of FITACF were developed by Christian Hanuise around 1984. Since then, the algorithms have been modified and adjusted to provide the data products now used to conduct the science using the SuperDARN data. Until recently, the data from the radars were stored in 3 file sets: *.inx, *.fit and *.dat. The *.fit files are the *.dat files processed by FITACF.
Who is Responsible for the integrity of FITACF?
FITACF is the main pathway for deriving the final radar data products. Reports of large spectral width values (>200 m/s) from the radars appeared in the literature around the year 2000. These observations could not be explained by any known physical mechanism. At the 2004 SuperDARN meeting, the presentation Possible causes of large spectral width in SuperDARN echoes from high latitudes, P.V. Ponomarenko, C.L. Waters, L. Rogers, suggested that FITACF was over estimating the spectral width.
At the 2004 meeting the SuperDARN PIs appointed the FITACF Working Group to look into the performance of FITACF. At present, this WG is responsible for assessing the FITACF code and bringing forward to the SuperDARN community any recommendations for change. A number of recommendations for changing some routines in FITACF were presented at the 2006 SuperDARN meeting by the University of Newcastle group. These were debated by the FITACF WG and a number of changes (not all) were implemented. At the 2008 SuperDARN meeting, the Software WG suggested a FITACF Wiki page on this site.
This page has been written by C.L. Waters and P.V. Ponomarenko. Comments and suggestions are welcome.
What is on this page?
A description of the routines that make up the FITACF codes. Scientific claims based on data from any measuring instrument must be developed within the limits of the instrument performance. The radars are no exception. It is in the best interest of everyone to understand how the radar data products are derived. We provide an introduction to radar operation, FITACF software flowchart, descriptions of each routine and a list of recommendations for changing the code.
Basic Principles of Radar
Radars emit short pulses of EM waves which travel to the target, reflect and are received some time later by the radar antennae. If the target is moving, the received signal contains a Doppler shift. Assume the speed is c (speed of light). This ignores changes in refractive index near the reflection region, which may alter the derived Doppler velocity (presently under investigation). For a maximum range of 3,500 km the total round trip distance is 7,000 km. The time it takes for one pulse to traverse this distance is 0.0233 s. The maximum Doppler shift that can be detected for a single pulse system with this range is 1/(2*0.0233) or ~20 Hz. However, the convection velocity of the ionosphere plasma at auroral latitudes can produce Doppler shifts up to 200 Hz. These requirements for range and Doppler shift call for a multi-pulse radar sequence.
Multi-Pulse Sequence
In order for SuperDARN to measure the required Doppler velocities, a seven-pulse coded sequence was implemented (Greenwald et al., 1985; Hanuise et al., 1993; Baker et al., 1995; Barthes et al., 1998). While this sequence is not the only possibility, it appears to have become the 'standard'. We will assume this sequence and common mode operational parameters (scanning on 16 beams at 2 min intervals).
For each radar beam the transmit sequence produces ACFs at each range gate with an elementary lag (smallest time difference) of 2.4 ms, giving a maximum Doppler shift of 200 Hz. The maximum lag value is 43.2 ms (between 9-27 in the timing combinations figure). The transmit pulse length is 300μs which gives a spatial resolution of 45 km over 70–75 range gates with a maximum range of 3550 km. The receiver data are sampled at 300μs intervals for a time that allows the last pulse from the furthest range to arrive (~0.1 s).
The ACF
Various time lags are available by comparing the arrival times of combinations of pulse pairs. An auto correlation function (ACF) is computed from the 17 possible time lags. FITACF assumes a single component ACF as shown in this ideal ACF figure. The ACF magnitude with time curve shows the decay time, τc which is used to compute the spectral width. For each beam, the dwell or integration time is usually between 3 to 7 sec. This sets the number of ACFs that are averaged (usually around 70) to produce the ACFs for each range.
From the pulse sequence and ACF figures, it is clear that values at certain lags may be unusable. For example, the receiver must be 'blanked' when the transmitter is operating. A large proportion of the FITACF code is concerned with identifying badlags and the various sources for these, including estimates of noise and uncertainties (errors) in the ACF values.
While the multi-pulse method allows for large Doppler velocities, there are disadvantages. The main problem is cross-range interference.
(The pulse sequence figures in this section are from: SuperDARN Pulse Sequences - Optimization and Testing, K. McWilliams, D. Andre, R. Greenwald, A. Schiffler, G. Sofko, T. Yeoman, a tutorial presented at the SuperDARN 2003 meeting.)
What is CRI?
Cross-range interference (CRI) (see Ponomarenko and Waters, 2006) occurs when returns from different pulses in the sequence arrive at the receiver from different ranges at the same time. The CRI figure shows the simple case involving 2 pulses. Why is it important to understand CRI? It turns out that an examination and modification of the CRI identification routine allows us to ignore a large section of the FITACF code, as explained later.
FITACF Flowchart
A flowchart of the routines in FITACF was provided by Ponomarenko and Waters, Spectral width of SuperDARN echoes: Measurement, use and physical interpretation, Annales Geophys., 24, 115, 2006. The flowchart shown on this site is an updated version that no longer includes the noise_acf, fit_noise and remove_noise routines. FITACF treats the magnitude (power) and phase of the ACF estimates separately. The ACF magnitude (power) is used to estimate the spectral width while the ACF phase gives the Doppler velocity. In summary, the FITACF processes are:
Create data structures and populate them (rawdata)
Identify ACF lags contaminated by CRI and Tx pulse overlap
Determine noise parameters
Identify badlags based on the shape of the ACF power curve
Fit model functions to ACF power and phase
Estimate output data products and respective errors
Identify and mark ground scatter ACFs
Calculate elevation from interferometer data (XCF)
Write data to the *.fit file.
Badlags.c
Code in the badlags.c file deals with range and pulse overlap. The defined functions are:
(a) badlags identifies bad receiver samples that overlap transmitter (Tx) pulses (for a given pulse sequence). For non-stereo radars, the pulse times are calculated from
where pn is obtained from the pulse pattern table (rawdata.pulse_pattern), t1 is the pulse start time (μs), t2 is the pulse end time (μs), mpinc is the basic lag separation (μs) stored as rawdata.parms.mpinc and txpl is the pulse length (rawdata.parms.txpl). A receiver sample time is:
where lagfr is the lag to the first range (rawdata.parms.lagfr), smsep is the sample period (rawdata.parms.smsep). Bad samples are values of m where 
.
The algorithm is slightly different for stereo radars. Another quirk is that if the year is less than 1993 and the number of lags is 17, then lag 13 is always bad. We have found that improvements in the ACF arise if the Tx blanking time is increased. This should be checked for each radar (change the value of 100 above and reprocess the *.dat data).
(b) ckrng calls r_overlap to provide the CRI table and determine badlags that might be affected by cross-range interference. The functions lag_overlap and r_overlap are in the lag_overlap.c file.
For all pulses, the range check algorithm is:
where pn is from the pulse pattern (rawdata.pulse_pattern) and range is the range number under consideration. The original code had the condition for CRI as
where nave is the number of pulse sequences transmitted over the dwell time for that beam and pwr0 are the lag zero power estimates. Ponomarenko and Waters (2006) questioned the 0.3 x nave factor and argued for it to be removed.
fit_acf.c
With the CRI and Tx affected lags taken care of, the processing moves on to estimating the uncertainties in the remaining ACF data. Recall that for a given radar beam, each range has an associated ACF that is averaged over the beam dwell time. The number of ACFs in the average is nave.
The main parameter used in estimating the errors in an (averaged) ACF is the statistical fluctuation, 
. This is calculated using the lag zero power estimates, R(0). The noise level is the average of the σR from the 10 weakest ACFs for that beam. In the code, the fluctuation level is the P0n variable and the noise level is the noise_lev variable. Pre-2006 the fitacf.c code modified the ACFs using P0n as follows:
.
249 w[k] = w[k] - P0n;
250 if (w[k] <= 0.0) w[k] = 0.1;
.
The fluctuation level was treated as a positive offset and subtracted from the ACF (see line 249). This over estimated the spectral width as discussed in Ponomarenko and Waters (2006) and illustrated here from their Figure 4. Notice how the decay time is reduced after σR is subtracted from R(τ). This increases the estimate for W, the spectral width. A similar argument applies for the noise level. The 2007 release of FITACF no longer subtracts this value.
Now for the lag zero noise estimate, Rn(0). This appears as a delta function type spike at zero lag. Figure 9 (Ponomarenko and Waters, 2006) illustrates the associated decrease in the ACF decay time and increase in spectral width. The 2007 release of FITACF subtracts the lag zero noise value from the zero lag power estimate.
Velocity Estimation
The Doppler velocity is derived from the ACF phase. The two main challenges are phase skips on a modulo 2*pi basis and estimating the velocity errors.
Where can I get more information ?
1. Spectral characteristics paper (Baker, Greenwald, Villain, Wing)
2. Kile Baker : White paper on FITACF
3. Kathryn McWilliams FITACf tutorial presented at 2003 SDarn meeting
4. Ponomarenko and Waters, 2006 fitACF paper
Appendix
Newcastle recommended changes to FITACF:
1. Do not subtract the statistical fluctuation level from the ACF power. [Ver 2.0]
2. Mark ACF lags below the statistical fluctuation level as bad [Ver 2.0].
3. Subtract the lag zero noise, Rn(0), from each lag zero estimate, R(0) [Ver 2.0].
4. Use the total (signal + noise) at lag zero when calculating the fluctuation level [Ver 2.0].
5. Reduce the CRI threshold from 0.3NAR(0) to R(0) [Ver 2.0].
6. Lags below the statistical fluctuation are used to determine 2*pi phase flips in the velocity calculation. To fix this, use the same set of badlags for both power and phase.
7. Do not include the default omega_err=9999, value in the Doppler velocity error estimate. Only use the data fit errors for the Doppler velocity errors.
8. Check the Tx pulse blanking length for all radars and for stereo modes.
9. Badlag labels 5, 7, and 9 in the more_badlags.c code are unnecessary. They appear to be caused by 5. above.
10. A number of coefficients appear in the FITACF code that do not have any clear physical justification. Examples are:
(a) PLIM=1.6 in noise_stat.c
(b) ROOT_3 in noise_stat.c
(c) Factor of 1/2 in If (ave_noise_pwr > noise_pwr/2.0) in do_fit.c
(d) Factor of pi/64 in While(fabs(omega_old - omega_loc) > fabs(omega_loc*pi/64)) in fit_acf.c
(e) Minimum allowed number of good lags is actually only one (fit_acf.c)
(f) Minimum signal/noise (p_0) is unity (0 dB)
(g) Lag zero noise power is calculated from the 10 weakest ACFs. What is the role of the clear sky noise. Should we use this instead?
FITACF FAQ
email questions to colin.waters@newcastle.edu.au
