ASAP v2 Data Container Design

Author:	Malte Marquarding, Vince McIntyre
Serial No:	20051017
Revision:	20051122
Circulation:	CP, VJM, MRC
Filename:	$CVSROOT/asap/doc/admin/asapv2_design.sxw

Abstract

ASAP is a package for processing single-dish radio telescope spectra. Its application within ATNF covers the Mopra, Parkes and Tidbinbilla telescopes. The initial data container used throughout ASAP was designed under some assumptions about the form of the data it would be expected to deal with. Some of those assumptions turned out to be incorrect, largely stemming from an unintentional omission in the original requirements.

This note documents the problem as it stood in ASAP-1.0 (2005 Feb) and the adopted solution which will enter the code tree during the development cycle for -2.0. We also consider the impact of the change on the project schedule and note areas of the project that need further investigation.

Background

ASAP is a package for processing observations from a single-dish radio telescope. In the normal course of events, data flows through ASAP as follows:

At the telescope, observations are made by pointing the telescope at an object and integrating for some number of observing-system cycles. The spectrum and some metadata is recorded to disk, usually after every cycle, in a FITS-like format (usually RPFITS at ATNF, often SDFITS at other observatories). One of the key pieces of metadata is the timestamp, usually corresponding to the start of an observing-system cycle.

At each timestamp a number of spectra may be written, depending on the instrumental setup:

There may be more than one receiver horn on the telescope (and thus more than one beam projected on the sky).

For each beam there may be more than one frequency of interest sampled (presented as a different intermediate frequency or IF).

For each beam, at each frequency of interest, the correlator may produce multiple polarisations.

Each IF covers a limited range of observing frequency (e.g. 1.0-1.2 Ghz), called the bandpass. The bandpass is sampled into some number of channels. A single spectrum is a collection of complex numbers (usually represented as an amplitude and phase), with one complex value per channel.

Spectra are read into ASAP from an RPFITS file, SDFITS file or AIPS++ MS2 directory and placed in a data structure. At present, the entire data file must fit into the memory of the computer running ASAP, but this restriction could be lifted in future versions.

With the functionality in ASAP, the user manipulates the data (interactively or in a Python script).

At the end of processing, the user can save the resulting data structures to SDFITS or some other format. A common format to leave the data in is “ASAP format”, i.e. an AIPS++ table.

The ASAP package is largely C++ code that makes calls to AIPS++ libraries, with a user interface in Python code. It works with the AIPS++ data structures directly.

The data structure used for storing the spectra is an AIPS++ table, with the spectra contained in multi-dimensional arrays in the rows. Each row in the table correspnds to a timestamp in the input FITS file. There is one multi-dimensional array per row. The other columns of the row contain the various metadata fields associated with that observation timestamp.

The Problem

In ASAPv1, the multi-dimensional array has the shape [nBeam, nIF, nPol, nChan].

Arrays can vary in shape from row to row within a table of observations. However for a given table row, the number of channels in an array may not change as one iterates over the nBeam, nIF or nPol indices. This is because ASAP needs to perform mathematical operations on the spectra, such as forming a quotient.The Array libraries assume the data are on the same pixel grid.

The difficulty this design presents is that there are cases in which one wants to allow any of the IFs to have a different number of channels to the others, at the same timestamp. This is not allowed for in the design above.

Two cases where this affects users of ASAP are as follows:

The new spectrometer planned for Mopra (MOPS) may provide “zoom” modes, in which as many as 8 subsections of a very wide bandpass are selected and sampled at high resolution. This implies the output RPFITS files will have multiple IFs, possibly with different numbers of channels and bandwidths from one IF to the next. In its present form, ASAP v1 cannot handle this sort of data.

ASAP is under evaluation by the ALMA single dish group, and may also be used by NRAO at the Green Bank Telescope (GBT). Such a feature is essential for ALMA, as they will be covering a large number of frequencies and will almost certainly have different numbers of channels in adjacent IFs.

On ATNF instruments, anything that affects the correlator setup forces a file closure. Thus, we may assume nBeam, nIF, nPol do not change within an RPFITS file, so do not change within a given row-filling read operation. Most users of single-dish telescopes assume this behaviour as well. There is an exception to this rule in that the user may concatenate some RPFITS files with different correllator setups and then attempt to read the result into ASAP.

However at the GBT it is common practice for the observer to change setup such that the same IF number has a different number of channels, but recording continues to the same output data file.

While neither of the ALMA or GBT needs are a driver for ASAP (our remit is to serve ATNF telescopes) they do illustrate the shortcoming of the current code.

Other telescopes will most likely add this kind of system in the future, so it was requested by the Project Scientist to enable varying bandwidths in ASAP.

(See Requirements Document, 8.2-2)

New Design

The current version of ASAP stores all its data as an array of shape [nBeam, nIF, nPol, nChan]. For each time stamp one of these arrays is stored in a table row. However Array/Vector shapes may vary from one table row to the next, which is the feature to be exploited in the new design.

In the new design, we change the structure of the container so that each table row will hold just one spectrum (represented as a column holding a 1-dimensional vector, plus additional columns containing index values for the beam, IF and polarization). One timestamp will now correspond to many rows; the same set of metadata will be copied into [nBeam * nIF * nPol] rows. To avoid duplication of this possibly large set of metadata, the main table will only carry integer indices that point into subtables. Figure 1 shows the details of this.

Figure 1 The new ASAP data container

Benefits

Besides addressing the inflexibility of the old container with respect to the kinds of correlator setups it can accept, there are some additional benefits to this design.

Better handling of corrupt data files – a single bad record will not make the data invalid, as happens occasionally at present.

Data selection via the robust, SQL-like Table Query Language (TaQL) mechanism in the AIPS++ table package. This is proven, optimized code that will give more flexible selection than that currently available in ASAPv1. For example, to make a selection of rows with (Tsys < threshold_value) takes one TaQL statement. This would be considerably more complex to code in ASAPv1.

Simpler code – some of the old code, especially around data averaging, is quite convoluted as a result of using multidimensional arrays. This makes it hard to maintain. The new design makes the processing code much easier to follow, which should increase the lifetime and maintainability of the code.

Better memory utilisation – this can be important in the case of the MOPS spectrometer where there may be a “megachannel” mode emitting 106 channels in each spectrum (with two polarizations), or MultiBeam receivers (which have of order 10 beams, each with thousands of channels). The improvement comes in accumulation operations like averaging and addition which will need to hold less data in memory (with a more precise selection mechanism only the data actually needed is looked at). ASAP v1 made no effort to optimize throughput when dealing with large data sets; some of the code is not efficient enough and if we did not change data containers that code would need to be re-worked.

Migration Options

Translation Object

Initially we thought that all functions could be kept as in ASAPv1 by implementing a container translation object, which would feed the new data container into the existing functions. However upon investigation this appears to be nearly as much work as rewriting the existing algorithms, since the paradigm behind the two containers is quite different (array vs. table row indexing, see below).

Rewriting for new container

The alternative is to rewrite the code to use the new data container directly, which we had planned to do later in the project anyway.

The new design places the individual spectra in (1-dimensional) Vector objects rather than (n-dimensional) Arrays. This forces some recoding in most of ASAP's processing functions, since a) selecting data is now done via the _ID values (see Figure 1) rather than iterating over the IF and Polarization indices and picking out the relevant array elements and b) iterating over IFs or Polarizations is now done by stepping over table rows rather than array elements.

Without actually doing the rewrite, it is difficult to say how much of the code will need to be rewritten, and how much effort that will require. To explore this, we recoded the most complicated C++ function in ASAP (SDMath::averageScan), using the new data container. The resulting code was no shorter (about 100 lines), but much better structured and simpler to read. Once the concept for the data container was defined, the recoding took about 1 day. We expect other, simpler, functions will be faster to recode.

The change of data container means that the format used for on-disk storage of spectra (“ASAP format”) is incompatible between v1 and v2. That is, ASAPv2 code should be able to read v1 files (simply by leaving the old reader code in the package), but existing installations of ASAP will not be able to read v2 files. This is acceptable since the on-disk format was never intended for long-term storage. It is also easily worked around, by converting spectra held in the on-disk format into SDFITS, and then reading those files into the new version of the software.

Impact on Project Schedule

In our project plan [1], improved data selection was set as a target for the first 3-month development cycle of ASAP v2 (Sep-Dec 2005). Choosing to refactor the data containers now means we should delay implementing these data selection requirements, to avoid having to do it twice. This is an all-or-nothing refactoring: the change affects a large fraction of the project code and the data containers are too different to allow much re-use of affected code.

Nevertheless, the rescheduling should not affect the project as a whole. By the time release 2.0 is made, all level-1 functionality targets in the plan up to that point would be met. This change only affects the ordering of major development milestones.

As measured with the sloccount tool, the ASAP code base is about 6400 lines of C++ and about 2400 of Python. We estimate that the modules which will be affected total about 3000 lines of C++, but not all of that will need recoding. We do not expect to have to change more than a few lines of the Python code.

Based on the exploration work noted above, a conservative estimate of the recoding time is 20 working days (1 calendar month). Testing the code for obvious regressions will take an additional 10 working days initially, with some additional effort from time to time later in the project.

One other area this change impacts is the handling of polarization. We believe that the change will make implementation of the necessary functionality easier. However it may require dropping the current “virtual columns” approach to calculating Stokes parameters. This requires further investigation.

References

[1] McIntyre, V. 2005. Project plan document, Single Dish Analysis Software 2 http://www.atnf.csiro.au/internal/management/pm/086/plan20050825.htm

Mesurement Set version 2, see (FIXME – needs a reference)

ASAP v2 Data Container

2005-10-17T12:25:38