wiki:ScantableRedesignProposal

Scantable Redesign Proposal

Summary

The current Scantable schema is biased towards rpfits style data (or the structure of atnf/PKSIO/PKSrecord.h). Using the PKSreader and the current Scantable format a lot of metadata from the MeasurementSets get lost. This proposal tries to rectify this without as major impact on the current use of the Scantable and proposes and incremental addition of the features/changes. It is planned to usesMS as the persistent storage container for asap data.

The proposal in summary:

  • Keep Scantable as the asap internal format
  • extend the schema to include all relevant (meta)data in the MeasurementSet (MS)
  • implement a direct MS->Scantable Filler avoiding (meta)data loss by going through the PKSreader
  • use MS as the persistent asap storage container
    • implement a direct Scantable->MS Writer avoiding (meta)data loss by going through the PKSwriter

optional performance/usability changes:

  • implement a direct ASDM->Scantable Filler (performance)
  • implement an upgrade task for the old Scantable to the new Scantable schema ( not required for standard as Scantable is not assumed to be persistent)

Scantable vs MS

Even though both containers are based on casa::Table they use different philosophies in terms of access. The main differences are:

Frequency handling

  • MS has the SPECTRAL_WINDOW_ID for identifying similar frequency set-ups
  • Scantable has a combination of IFNO and FREQUENCY_ID to group similar frequencies.

The concept of an IF for grouping doesn't really exist in the MS (there are entries which can be used for this in the SPECTRAL_WINDOW table An IF for example can have multiple FREQUENCY_IDs e.g. in case of doppler tracked data. Frequency alignment then unifies these.

Handling of spectral channels

  • MS has two columns DATA and FLOAT_DATA containing n_pol x n_channel matrices of complex and float values respectively. In case of single-dish data the FLOAT_DATA column contains the spectra.
  • Scantable has only one column SPECTRA and one row per polarisation containing a float vector of n_channel. Cross-correlations are stored as real and imaginary parts in separate rows. For example for polarimetry data. we have four rows containing XX, YY, Real(XY), Imag(XY)

asap relies on these structures in almost all access methods. As such, changing the internal format to MS would mean almost a complete reimplementation of asap.

Referencing

  • Scantable relates to all sub-tables via an ID in the main table
  • MS relates to sub-tables via a time stamp.

Proposed changes

Structure

  • add columns in MS::WEATHER table to Scantable::WEATHER, including Time
    • MS is time based, so MS->Scantable Filler needs to generate WEATHER_ID in main table
  • Add ANTENNA sub-table to Scantable, referencing in main table keyword.
    • Copy from MS.
    • MS for single-dish should always be one row only.
  • add columns from MS::SYSCAL to Scantable::TCAL
  • add FIELD sub-table to Scantable and move existing FIELD related columns from main table into it.
  • Add OBSERVATION sub-table to Scantable, referencing in main table keyword
    • Copy from MS.
    • should always be one row only.
  • Use MS::HISTORY in Scantable::HISTORY (message column)
  • Copy MS::STATE to new sub-table and add STATE_ID in main Scantable
  • Copy MS::POINTING table. No referencing in main table. Plain copy
  • Copy MS::POLARIZATION table
    • add POL_ID
    • add getPolType method
  • move SRCxyz columns into new SOURCE sub-table.
    • add SOURCE_ID to main table
    • copy MS::SOURCE columns
    • note that RESTFREQUENCY etc. are in different sub-tables.
    • not a direct copy from MS
  • add extra columns from MS::SPECTRAL_WINDOW to Scantable::FREQUENCIES
    • note: use IF_CONV_CHAIN or FREQ_GROUP as IFNO equivalent in export to MS

API

  • make polarisation handling row-based (POL_ID) and remove hard-coding of feed type etc.
  • make sure all methods accessing moved/new columns get updated

Effort

  • Design/proposal - Malte Marquarding
  • Scantable changes - Malte Marquarding/Takeshi Nakazato
  • asap changes to methods/tasks - all developers
  • MS->Scantable Filler - Takeshi Nakazato
  • Scantable->MS Writer - Takeshi Nakazato

Relevant Documentation

  • attach spreadsheet for impact on all asap functions/methods/tasks
  • attach image of Scantable schema

References

Last modified 14 years ago Last modified on 10/12/10 09:42:55
Note: See TracWiki for help on using the wiki.