= Scantable Redesign Proposal =

== Summary ==

The current Scantable schema is biased towards rpfits style data (or the structure of {{{atnf/PKSIO/PKSrecord.h}}}). Using the PKSreader and the current Scantable format a lot of metadata from the !MeasurementSets get lost. This proposal tries to rectify this without as major impact on the current use of the Scantable.

The proposal in summary:

 * Keep Scantable as the asap internal format
 * extend the schema to include all relevant (meta)data in the !MeasurementSet (MS)
 * implement a direct MS->Scantable Filler avoiding (meta)data loss by going through the PKSreader
 * use MS as the persistent asap storage container
  * implement a direct Scantable->MS Writer avoiding (meta)data loss by going through the PKSwriter

optional performance/usability changes:

 * implement a direct ASDM->Scantable Filler (performance)
 * implement an upgrade task for the old Scantable to the new Scantable schema ( not required for standard as Scantable is not assumed to be persistent)

= Scantable vs MS =

Even though both containers are based on {{{casa::Table}}} they use different philosophies in terms of access. The main differences are:

== Frequency handling ==

 * MS has the '''SPECTRAL_WINDOW_ID''' for identifying similar frequency set-ups
 * Scantable has a combination of '''IFNO''' and '''FREQUENCY_ID''' to group similar frequencies.

The concept of an ''IF'' for grouping doesn't really exist in the MS (there are entries which can be used for this in the '''SPECTRAL_WINDOW''' table
An ''IF'' for example can have multiple '''FREQUENCY_ID'''s e.g. in case of doppler tracked data. Frequency alignment then unifies these.

== Handling of spectral channels ==

 * MS has two columns '''DATA''' and '''FLOAT_DATA''' containing {{{n_pol x n_channel}}} matrices of complex and float values respectively. In case of single-dish data the '''FLOAT_DATA''' column contains the spectra.

 * Scantable has only one column '''SPECTRA''' and one row per polarisation containing a float vector of {{{n_channel}}}. Cross-correlations are stored as real and imaginary parts in separate rows. For example for polarimetry data. we have four rows containing '''XX, YY, Real(XY), Imag(XY)'''

asap relies on these structures in almost all access methods. As such, changing the internal format to MS would mean almost a complete reimplementation of asap.


 
= Relevant Documentation =

 * ''attach spreadsheet for impact on all asap functions/methods/tasks''
 * ''attach image of Scantable schema''