ASAP v2 Data Container

Malte Marquarding

Monday, October 17, 2005

Motivation:

The new spectrometer for Mopra will have “zoom” modes. This implies having multiple IFs with possibly changing bandwidths. ASAP and the PKSreader cannot handle this.

The current design of the ASAP data container makes support of this feature almost impossible. Other telescopes will most likely add this kind of system in the future, so it was requested by the Project Scientist to enable varying bandwidths in ASAP.

(note: ASAP is under evaluation by the ALMA single dish group, such a feature is essential for ALMA to consider ASAP)

Design:

The current version of ASAP stores all its data as an array of shape [nBeam, nIF, nPol, nChan], which is fixed. For each time stamp one of these arrays is stored in the underlying table. Array/Vector shapes can vary by row, which is the feature to be exploited in this new design.

Each row will just contain one spectrum (vector), plus an index for the beam, IF and polarization.

One issue in this design is that one set of metadata will copied into [nBeam * nIF * nPol] rows. To avoid duplication of this possibly large set of metadata, the main table will only carry integer indeces for pointing in to a subtable. This design is similar to relational databases and mimics some behaviour of MeasurementSets.

As all mathematical operations within ASAP work with the data structures directly, most functions will need to be changed. This is due to the fact that ASAP v1 works with multi-dimensional arrays, where the new design is based around table rows. Data selection is done on these rows - data processing only on the vector.

The row based design allows for very flexible data selection via table operations such as row iterations and SQL-like statements (TaQL), which has been a major requirement for ASAP v2.

Migration:

Initially it was thought that all functions could be kept as in version one by implementing a container translation object, which feeds the new structure into the existing functions. This is estimated to be nearly as much work as rewriting the existing algorithms as the paradigm behind the two containers is quite different (array vs. row indexing). Also some of the old code, especially around data averaging is quite convoluted and unmaintainable and will become much more transparent in the new design. This should increase the lifetime and maintainability of the code. Some bootstrap coding has been done to prove this. The most complicated function in asap was taken (average_time) and re-coded using the new data container. The new spectrometer will also write out very large spectra. ASAP v1 didn't focus specifically on optimization for large data sets. Some of the code is not efficient at this stage and would also need to be re-worked.

Alerts:

Data selection was set up as a target for the first development cycle of ASAP v2. The complete evaluation of the re-design as addressed above shows that delaying the data selection implementation until after the data container re-design to be the most logical and possibly efficient solution. The data containers are too different to re-use any code. The old time line would mean that the code for data selection would only be present until the next development stage. The rescheduling does not affect the project as a whole. By the end of the second stage all targets up to that point would be met. It only affects the order of major development milestones.

ASAP v2 Data Container

Malte Marquarding

2005-10-17T12:25:38