= Scantable Redesign Proposal = == Summary == The current Scantable schema is biased towards rpfits style data (or the structure of {{{atnf/PKSIO/PKSrecord.h}}}). Using the PKSreader and the current Scantable format a lot of metadata from the !MeasurementSets get lost. This proposal tries to rectify this without as major impact on the current use of the Scantable and proposes and incremental addition of the features/changes. It is planned to uses''MS'' as the persistent storage container for asap data. The proposal in summary: * Keep Scantable as the asap internal format * extend the schema to include all relevant (meta)data in the !MeasurementSet (MS) * implement a direct MS->Scantable Filler avoiding (meta)data loss by going through the PKSreader * use MS as the persistent asap storage container * implement a direct Scantable->MS Writer avoiding (meta)data loss by going through the PKSwriter optional performance/usability changes: * implement a direct ASDM->Scantable Filler (performance) * implement an upgrade task for the old Scantable to the new Scantable schema ( not required for standard as Scantable is not assumed to be persistent) = Scantable vs MS = Even though both containers are based on {{{casa::Table}}} they use different philosophies in terms of access. The main differences are: == Frequency handling == * MS has the '''SPECTRAL_WINDOW_ID''' for identifying similar frequency set-ups * Scantable has a combination of '''IFNO''' and '''FREQUENCY_ID''' to group similar frequencies. The concept of an ''IF'' for grouping doesn't really exist in the MS (there are entries which can be used for this in the '''SPECTRAL_WINDOW''' table An ''IF'' for example can have multiple '''FREQUENCY_ID'''s e.g. in case of doppler tracked data. Frequency alignment then unifies these. == Handling of spectral channels == * MS has two columns '''DATA''' and '''FLOAT_DATA''' containing {{{n_pol x n_channel}}} matrices of complex and float values respectively. In case of single-dish data the '''FLOAT_DATA''' column contains the spectra. * Scantable has only one column '''SPECTRA''' and one row per polarisation containing a float vector of {{{n_channel}}}. Cross-correlations are stored as real and imaginary parts in separate rows. For example for polarimetry data. we have four rows containing '''XX, YY, Real(XY), Imag(XY)''' asap relies on these structures in almost all access methods. As such, changing the internal format to MS would mean almost a complete reimplementation of asap. == Referencing == * Scantable relates to all sub-tables via an '''ID''' in the main table * MS relates to sub-tables via a time stamp. = Proposed changes = == Structure == * add columns in '''MS::WEATHER''' table to '''Scantable::WEATHER''', including '''Time''' * '''MS''' is time based, so MS->Scantable Filler needs to generate '''WEATHER_ID''' in main table * Add '''ANTENNA''' sub-table to Scantable, referencing in main table keyword. * Copy from '''MS'''. * MS for single-dish should always be one row only. * add columns from '''MS::SYSCAL''' to '''Scantable::TCAL''' * add '''FIELD''' sub-table to '''Scantable''' and move existing '''FIELD''' related columns from main table into it. * Add '''OBSERVATION''' sub-table to Scantable, referencing in main table keyword * Copy from '''MS'''. * should always be one row only. * Use '''MS::HISTORY''' in '''Scantable::HISTORY''' (message column) * Copy '''MS::STATE''' to new sub-table and add '''STATE_ID''' in main Scantable * Copy '''MS::POINTING''' table. No referencing in main table. Plain copy * Copy '''MS::POLARIZATION''' table * add '''POL_ID''' * add {{{getPolType}}} method * move '''SRCxyz''' columns into new '''SOURCE''' sub-table. * add '''SOURCE_ID''' to main table * copy '''MS::SOURCE''' columns * note that '''RESTFREQUENCY''' etc. are in different sub-tables. * not a direct copy from '''MS''' * add extra columns from '''MS::SPECTRAL_WINDOW''' to '''Scantable::FREQUENCIES''' * note: use '''IF_CONV_CHAIN''' or '''FREQ_GROUP''' as '''IFNO''' equivalent in export to '''MS''' == API == * make polarisation handling row-based ('''POL_ID''') and remove hard-coding of feed type etc. * make sure all methods accessing moved/new columns get updated = Effort = * Design/proposal - ''Malte Marquarding'' * Scantable changes - ''Malte Marquarding/Takeshi Nakazato'' * asap changes to methods/tasks - ''all developers'' * MS->Scantable Filler - ''Takeshi Nakazato'' * Scantable->MS Writer - ''Takeshi Nakazato'' = Relevant Documentation = * ''attach spreadsheet for impact on all asap functions/methods/tasks'' * ''attach image of Scantable schema'' = References = * http://www.atnf.csiro.au/people/tim.cornwell/software/MSV2.jpg * http://www.atnf.csiro.au/computing/software/casacore/casacore-1.2.0/doc/notes/229.html