ASAP v2 Data Container

Author:	Malte Marquarding, Vince McIntyre
Serial No:	20051017
Revision:	$Date$
Circulation:	MM, CP, VJM, MRC

Abstract

ASAP is a package for processing single-dish radio telescope spectra. Its application within ATNF covers the Mopra, Parkes and Tidbinbilla telescopes. The initial ASAP code was built making some assumptions about the form of the data it would be expected to deal with, largely stemming from an unintentional omission in the original requirements. Those assumptions turned out to be incorrect. This note documents the problem as it stood in ASAP-1.0 (2005 Feb) and the adopted solution which will enter the code tree during the development cycle for -2.0.

The Problem

In the normal course of events, data flows through ASAP as follows:

At the telescope, observations are made by pointing the telescope at an object and integrating for some number of observing-system cycles. The spectrum and some metadata is recorded to disk, usually after every cycle, in a FITS-like format (usually RPFITS at ATNF, often SDFITS at other observatories). One of the key pieces of metadata is the timestamp, usually corresponding to the start of an observing-system cycle.

At each timestamp a number of spectra may be written, depending on the instrumental setup:

There may be more than one receiver horn on the telescope (and thus more than one beam projected on the sky).

For each beam there may be more than one frequency of interest sampled (presented as a different intermediate frequency or IF).

For each beam, at each frequency of interest, the correlator may produce multiple polarisation (also presented as separate IFs ?FIXME?).

Each IF covers a limited range of observing frequency (e.g. 1.0-1.2 Ghz), called the bandpass. The bandpass is sampled into some number of channels. A single spectrum is a collection of complex numbers (usually represented as an amplitude and phase), with one complex value per channel.

Spectra are read in from an RPFITS or SDFITS file and placed in a data structure.

With the functionality in ASAP, the user manipulates the data (interactively or in a Python script) and saves the resulting data structures to FITS or some other format.

The ASAP package is largely python code, but does much of the processing via calls to AIPS++ libraries. It works with the AIPS++ data structures directly.

The data structure use for storing the spectra is an AIPS++ table, with the spectra contained in multi-dimensional arrays in the rows. Each row in the table correspnds to a timestamp in the input FITS file. There is one multi-dimensional array per row. The other columns of the row contain – what ?FIXME?.

In ASAPv1, the multi-dimensional array has the shape [nBeam, nIF, nPol, nChan]. For a particular data table, the length of each axis is fixed and must be the same for all rows. This is because ASAP needs to perform mathematical operations on the spectra, such as forming a quotient. It does this by making calls from the Python level to the AIPS++ libraries, and those libraries assume the data are on the same pixel grid.

The difficulty this design presents is that there are cases in which one wants to allow the number of channels to be different in different IFs. This is not allowed for in the design above – the number of channels must be the same for each IF. Two cases where this affects users of ASAP are as follows:

The new spectrometer for Mopra may have “zoom” modes. This implies having multiple IFs with possibly changing bandwidths. ASAP and the PKSreader cannot handle this. ?FIXME? – explain PKSreader.

ASAP is under evaluation by the ALMA single dish group, and may also be used by NRAO at the Greenbank telescope. Such a feature is essential for ALMA, as they will be covering a large number of frequencies and will almost certainly have different numbers of channels in adjacent IFs. At the GBT, it is common practice for the observer to change setup such that the same IF number has a different number of channels. While neither of these are a driver for ASAP (our remit is to serve ATNF telescopes) they do illustrate the shortcoming of the current code.

Other telescopes will most likely add this kind of system in the future, so it was requested by the Project Scientist to enable varying bandwidths in ASAP.

?FIXME? – quote the requirement from Chris.

New Design

The current version of ASAP stores all its data as an array of shape [nBeam, nIF, nPol, nChan], which is fixed. For each time stamp one of these arrays is stored in the underlying table. However Array/Vector shapes can vary from one row to the next, which is the feature to be exploited in this new design.

Each row will just contain one spectrum (vector), plus an index for the beam, IF and polarization. Doing this means that one timestamp will now correspond to many rows, and thus the same set of metadata will copied into [nBeam * nIF * nPol] rows. To avoid duplication of this possibly large set of metadata, the main table will only carry integer indices that point into a subtable. This design is similar to relational databases and mimics some behaviour of MeasurementSets.

FIXME: is it going to be hard to automatically create a properly normalized set of tables?

As all mathematical operations within ASAP work with the data structures directly, most functions will need to be changed. This is due to the fact that ASAP v1 works with multi-dimensional arrays, where the new design is based around table rows.

FIXME: can we give an example function call.

FIXME: how many functions will need to be changed?

Data selection is done on these rows - data processing only on the vector.

FIXME What does that sentence mean?

Figure 1 FIXME - needs caption text. Does this show the old or new data structure? The row based design allows for very flexible data selection via table operations such as row iterations and SQL-like statements (TaQL), which has been a major requirement for ASAP v2.

Migration

Translation Object

Initially we thought that all functions could be kept as in v1 by implementing a container translation object, which feeds the new structure into the existing functions. However this is estimated to be nearly as much work as rewriting the existing algorithms as the paradigm behind the two containers is quite different (array vs. row indexing).

FIXME: example code would help a lot here.

Some of the old code, especially around data averaging, is quite convoluted ?FIXME? as a result of the multidimensional array design. This makes it hard to maintain. In contrast, the new design makes the processing code much more transparent. This should increase the lifetime and maintainability of the code.

?FIXME? - how many functions contain this kind of convolutedness?

Rewriting for new containers

The alternative is to rewrite the code to use the new data container directly. This would have to be done at some point anyway. Some bootstrap coding has been done to explore how difficult this will be, by recoding the most complicated function in ASAP (average_time) using the new data container.

?FIXME? was the new code shorter (& thus simpler)?

?FIXME? how long did the recoding take? (most functions should be less)

?FIXME? How many functions will need to be changed?

?FIXME? Is it an all-or-nothing refactoring or can we have a mix of the two interfaces?

Besides addressing the inflexibility of the old container, the new structure should be more efficient. This is a consideration because some instruments now being built have the potential to emit very large amounts of data – for example the new Mopra spectrometer may have a “megachannel” mode, in which 106 channels in two polarisations will be written at each time stamp. ASAP v1 didn't focus specifically on optimization for large data sets. Some of the code is not efficient at this stage and would also need to be re-worked.

Impact on Project Schedule

Improved data selection (?FIXME? cite the requirement numbers) was set up as a target for the first 3-month development cycle of ASAP v2. Choosing to refactor the data containers (which will change how the data selection is implemented) means delaying the implementation of these data selection requirements.

The complete evaluation of the re-design as addressed above shows that delaying the data selection implementation until after the data container re-design to be the most logical and possibly efficient solution. The data containers are too different to re-use any code. The old time line would mean that the code for data selection would not be present until the next development cycle, Dec 2005 - Feb 2006.

The rescheduling does not affect the project as a whole. By the end of the second stage all targets up to that point would be met. It only affects the order of major development milestones.

FIXME: this would be better described with a table showing the old and new ordering.

ASAP v2 Data Container

2005-10-17T12:25:38