source: trunk/admin/requirements2.tex @ 644

\documentclass[11pt]{article}
\usepackage{a4}
\usepackage{calc}
\usepackage{ifthen}
\usepackage{smartref}
\usepackage{longtable}
%\usepackage{arrayjob}
%\usepackage{multido}
\usepackage[dvips]{graphicx}

\def\complete{no}

% Adjust the page size
\addtolength{\oddsidemargin}{-0.4in}
\addtolength{\evensidemargin}{+0.4in}
\addtolength{\textwidth}{+0.8in}

\setlength{\parindent}{0mm}
\setlength{\parskip}{1ex}


\title{ASAP - ATNF Spectral Analysis Package\\
  Software Requirements - Development Cycle 2 }
 \author{Chris Phillips \& Malte Marquarding}

\newcounter{requirement}
\newcounter{subrequirement}

\addtoreflist{requirement}
\newcommand{\reqref}[1]{R\ref{#1}-\requirementref{#1}}

\newcommand{\makenote}[1]{{\bf \tt \em#1}}

\newcommand{\anitem}[2]{\smallskip \parbox[t]{2cm}{#1}%
  \parbox[t]{\textwidth-2cm}{#2}}

\newcommand{\showreqcounter}{
  R\arabic{section}.\arabic{subsection}-\arabic{requirement}
}

\newcommand{\showsubreqcounter}{
  R\arabic{section}.\arabic{subsection}-\arabic{requirement}.\arabic{subrequirement}
}

\newcommand{\status}[2]{
  \ifthenelse{\equal{#1}{Started}}{Started \hspace*{1cm} {\em Priority #2}}
 {\ifthenelse{\equal{#1}{Not started}}{Not Started\hspace*{1cm} {\em Priority #2}}
 {\ifthenelse{\equal{#1}{Done1}}{Completed}
 { \ifthenelse{\equal{#1}{Duplicate}}{Duplicate?}
 {#1}
 }}}
}

% Also Deferred, Obsolete

% Requirement comment
%  Summary
%  Status
%  Priority
%  Text

%\newarray\Requirements

%\newcounter{numreq}

\newcommand{\requirement}[4]{
  \ifthenelse{\equal{\complete}{yes}}
             {\dorequirement{#1}{#2}{#3}{#4}}
             {\ifthenelse{\equal{#2}{Done1} \or \equal{#2}{Deferred}
                 \or \equal{#2}{Obsolete}}{}
               {\dorequirement{#1}{#2}{#3}{#4}}}
}


\newcommand{\dorequirement}[4]{

  \setcounter{subrequirement}{0}

  \hspace*{2mm}\begin{minipage}{\textwidth-2mm}
    \setlength{\parindent}{-2mm}
    \stepcounter{requirement}
    {\bf \showreqcounter\ \bf #1} \\ 
    #4 \\
    \hspace*{1cm} \status{#2}{#3}
  \end{minipage}

  \typeout{REQUIREMENT: \showreqcounter & #1 & #2 & #3 :ENDREQ}
}

\newcommand{\extendedrequirement}[4]{
  \setcounter{subrequirement}{0}

  \hspace*{2mm}\begin{minipage}{\textwidth-2mm}
    \setlength{\parindent}{-2mm}
    \stepcounter{requirement}
    {\bf \showreqcounter\  #1}
    #4 
    \hspace*{1cm} \status{#2}{#3}
  \end{minipage}

  \typeout{REQUIREMENT: \showreqcounter & #1 & #2 & #3 :ENDREQ}
}

\newcommand{\subrequirement}[4]{
  \hspace*{2mm}\begin{minipage}{\textwidth-2mm}
    \setlength{\parindent}{-2mm}
    \stepcounter{subrequirement}
    {\bf \showsubreqcounter\ \bf #1} \\ 
    #4 \\
    \hspace*{1cm} \status{#2}{#3}
  \end{minipage}

  \typeout{REQUIREMENT: \showsubreqcounter & #1 & #2 & #3 :ENDREQ}
}

\newcommand{\oldrequirement}[2]{
  \hspace*{2mm}\begin{minipage}{\textwidth-2mm}
    \setlength{\parindent}{-2mm}
    \showreqcounter\ #1 \\ 
    \hspace*{1cm} {\em Priority #2}
  \end{minipage}
}

\newcommand{\oldextendedrequirement}[2]{
  \hspace*{2mm}\begin{minipage}{\textwidth-2mm}
    \setlength{\parindent}{-2mm}
    \showreqcounter\ #1 
    \hspace*{1cm} {\em Priority #2}
  \end{minipage}
}

\newcommand{\reqeqn}[1]{\\\hspace*{1cm} $#1$}

\let\oldsection\section
\renewcommand{\section}[1]{\setcounter{requirement}{0}\oldsection{#1}}

\let\oldsubsection\subsection
\renewcommand{\subsection}[1]{\setcounter{requirement}{0}\oldsubsection{#1}}

\begin{document}

\maketitle

\section{Introduction}

ASAP has been written to replace the venerable single-dish software
{\tt spc} for processing of single dish spectral line data from all
ATNF observatories. Version 1.0 of ASAP was released in ****. This
document reflects an update of the initial requirements document. Some
new requirements have been added and the requirement priorities have
been reassessed for the next development cycle.

\section{Scope}

ASAP should be able to process all spectral line single-dish
observations from ATNF telescopes (Parkes, Mopra \& Tidbinbilla). This
includes reading the data produced by the telescope, calibration and
reduction of the data and basic analysis of the data such as fitting
line profiles etc.

It has been assumed that the following processing is out of the scope
of ASAP.
\begin{itemize}
\item Raster or ``on-the-fly'' mapping (This is handled by
``livedata'' and gridzilla).
\item Very complex or specific data processing. (A route into
  Class\footnote{Part of the GLIDAS software package, produced by
  Institut de Radio Astronomie Millime\'trique http://www.iram.fr}
  should be available for advanced processing).
%%TODO%% give example
\item Continuum data.
\item Pulsar timing observations.
\end{itemize}

\section{Priorities}

Requirements have been given a value of 1 to 3. The other requirements
will be implemented mainly depending on priority, with ``1'' the
highest. Priority 3 and some priority 2 requirements will probably not
get implemented in the duration of the second development cycle.

\section{User Interface}

The user interface (UI) is the most important part of a single dish
processing package, but probably the most difficult to get right. Long
term the UI for this software will consist of three parts.
\begin{itemize}
  \item A graphical user interface (GUI).
  \item An interactive command line interface (CLI).
  \item A scriptable interface for batch processing.
\end{itemize}

The CLI and scriptable interface are essentially be the same.

The software does not {\em need} to be able to run solely from a
``vt100'' style terminal. It can be assumed that the user is running
the software from within a windowed (i.e. X11) environment. This will
mean it will not necessarily be possible to run the software remotely
over a slow network connection (e.g. internationally or from home).
Where possible, operations on the data should be possible from all
three aspects of the user interface. 

The user interface needs to be implemented so that the user can easily
and transparently work on spectra either one at a time or by
processing multiple spectra in parallel. This means there must be an
easy way to select specific or multiple spectra to display or process.

At this stage the development of a GUI has been deferred until the
basic package has stabilised and most features have been
implemented. At that stage a decision will be made on how to best
implement a GUI. On a shorter timescale specific purpose GUIs (such as
a simple Wizard for processing standard Mopra data) may be produced on
an as-needed basic.

\subsection{Graphical User Interface}

At this stage the ASAP GUI has been deferred to a later date.

\smallskip

\requirement{Simple interface}{Deferred}{}{It should be simple,
intuitive and uncluttered. Specifically, use of many windows
simultaneously should be discouraged, as should hiding functionality
behind layers of dialog boxes.}

\requirement{Integrated plotter}{Deferred}{}{The plotting window
should be a major component of the GUI control, not a separate
isolated window.}

\requirement{Minimal controls}{Deferred}{}{The interface should use
  minimal ``always visible'' controls, with use of pull down menus and
  maybe a toolbar for frequency used functions. }

\requirement{Keyboard shortcuts}{Deferred}{}{Keyboard shortcuts should
be available.}

\requirement{GUI user preferences}{Deferred}{}{Most user preferences
(i.e. keywords in the CLI) should be presented in a popup, tabbed,
dialog box.}

\requirement{GUI line fitting}{Deferred}{}{When performing line
profile fitting, a spreadsheet type window should be viewable which
shows the current parameter values (amplitude, velocity etc) for each
line fitted and allow the user to change these parameters or set the
current value as fixed. This GUI should stay synchronised with any CLI
changes to these values.}

\subsection{Command Line Interface}

The command line interface is the main user interface to ASAP. It is
implemented in ipython using a objected oriented command approach.

\requirement{Virtual CLI}{Obsolete}{}{While the GUI should be the main
interface for new users and for basic manipulation, some tasks can be
more efficiently performed using a CLI. A virtual CLI could be
integrated as part of the GUI.}

\requirement{CLI keyword/argument}{Obsolete}{}{The CLI should have a
keyword/argument form and never prompt the user for specific values
(the user should be able to change values which are retained until
they wants to change them again).}

\requirement{CLI case insensitive}{Obsolete}{}{The CLI should be case
insensitive and accept minimum matching and short forms of
keywords.}

\requirement{CLI available routines}{Done1}{}{The user must be able to
quickly and easily see from the command line the available routines
and keywords which affect it, so they can see which parameters may
need changing.}

\subsection{Scripting}

\requirement{Scripting}{Done1}{1}{It must be possible to run the
software in a scripting mode. This would be to process large amounts
of data in a routine manner and also to automatically reproduce
specific plots etc (So the scripting must have full control of the
plotter). Preferably the scripting ``language'' and the CLI would be
the same.}

%\requirement{Scripts from History}{Duplicate}{}{It would be worthwhile
%having a method to auto-generate scripts (for reduction or plotting)
%from current spectra history, or some similar method.}

\section{Plotter}

The plotter should be fully interactive and be an integral part of the
GUI and software interface.

\requirement{High quality plots}{Done1}{}{It must be able to
produce plots of publishable quality.}

\subrequirement{Histogram plots}{Not started}{1} {As well as line
plots, there needs to be an option to plot spectra in ``Histogram''
mode}


The user must be able to specify:

\subrequirement{Line Thickness}{Started}{1}{}

\subrequirement{Character size}{Not started}{1}{}

\subrequirement{Colours}{Started}{1}{}

\subrequirement{Line styles}{Not started}{1}{}

\subrequirement{Position of axis ticks}{Done1}{2}{}

\subrequirement{Hard Copies}{Done1}{1}{Producing hard copies
in postscript and .png format. Other formats may be added on an as
need basic.}

\subrequirement{Non-interactive hard copies}{Not started}{1} 
{It must be possible to produce hard copiers without an interactive
(i.e X11) plotter starting}.

\subrequirement{Scriptable plotting}{Done1}{1} {All aspects of
the plotter (zooming etc) must be setable from the command line for
scripting}

\requirement{Arbitrary plots}{Not started}{3}
{It must be possible to flexibly select the data to plot (e.g. Tsys vs
time etc as well as plots such as amplitude vs channel number or
velocity). Preferably any of the header values for a selection of
scans could be plotted on a scatter plot (e.g. Tsys vs elevation)}

\requirement{Overlay spectra}{Done1}{}{It must be possible to overlay
  multiple spectra on a single plot using different colours and/or
  different line styles. (Including multiple stokes data and multiple
  IFs).[[CHECK]]}

\requirement{Plot individual spectra}{Done1}{}{It must be possible to
  plot either the individual integrations (in either a stacked
  fashion, or using a new subplot per integration)}

\subrequirement{Auto-average integrations for plotting}{Not started}{2}
{It should be possible to optionally auto-average integrations of a
scan for plotting (for data thats has not already been scan averaged)}

\requirement{Plotter multi-panelling}{Done1}{1}
{It must be possible to multi-panel spectra in an n$\times$m size
grid. It must be possible to easily change the number of plots per
page, ie define the ``n'' and ``m'' values.}

\subrequirement{Step between plots}{Not started}{1}
{If more spectra than can fit on the plot matrix are to be plotted,
then it must be possible to step back and forth between the viewable
spectra (i.e. ``multi-page'' plots). This includes stepping through a
single plot on the pages at a time.}

\requirement{Multi-panel: change \# panels}{Not started}{2} 
{When using multi-panelling, the plotter should automatically update
the plot when the plot matrix dimensions (``n'' and ``m'' are changed)}

\requirement{Plotter interactive zooming}{Done1}{}{It must be possible
  to interactively zoom the plot (channel range selection and
  amplitude of the spectra etc.)  This includes both GUI control of
  the zooming as well as command line control of either the zoom
  factor or directly specifying the zoom bounds. }

\requirement{Zoomed subplot}{Not started}{2}
{On a single plot, it should be possible to plot the full spectrum and
a zoomed copy of the data (using a different lie style) to see weak
features. The user must be able to specify the zoom factor.}

\requirement{Offset plots}{Not started}{2}{Optionally when stacking
  multiple spectral plots in one subwindow, a (user definable) offset
  in the ``y'' direction should be added to each subsequent
  spectra.}

\requirement{Plotter auto-update}{Not started}{3} 
{The plotter should automatically update to reflect user processing,
either from the CLI or GUI. The user should have to option to turn
this feature off if they so wish.}

\requirement{Waterfall plot}{Not started}{3}
{It should be possible to plot individual integrations (possibly from
multiple scans) in a ``waterfall'' plot. This is an image based
display, where spectral channel is along the x-axis of the plot, time
(or integration number) along the y-axis and greyscale or colour
represent the amplitude of spectra. Interactive zooming and panning of
this image should be supported. }

\requirement{Waterfall editing}{Not started}{3} 
{When plotting ``waterfall'' plots, it should be possible to
interactively select regions or points and mark them as invalid
(i.e. to remove RFI affected data). The plotter should also show the
time/velocity of the pixel beneath the cursor.}

\requirement{Export waterfall to FITS}{Not started}{3} 
{It should be possible to export the ``waterfall'' plot images as a
FITs file, for user specific analysis.}

\requirement{Plot overlays}{Not started}{1} {Line markers overlays,
read from a catalogue should be optionally available. This would
include the full Lovas catalogue, the JPL line catalogue and radio
recombination lines. The lines would be Doppler corrected to a
specified velocity. The user must be able to plot just a sub-section
of the lines in any specific catalogue (to avoid clutter).}

\subrequirement{Plot overlays}{Not started}{2}
{Simple user definable catalogue should be definable for plot overlays}

\requirement{Plot fitted functions}{Done1}{}
{Optionally plot fitted functions (e.g line profiles or baseline
fit). If multiple components (e.g. Gaussian) have been fit, it should
be possible to show the individual functions or the sum of the
components}

\requirement{Plot residual data}{Started}{1}
{It should be possible to plot the residual data with or without
subtraction of fit functions. This includes plotting the spectra with
or without baseline removal and the residual after subtracting
Gaussian fits. The default should be to plot the data with baseline
subtracted but profile fits not subtracted.}

\requirement{Plot header data}{Not started}{2} {Basic header data
(source name, molecule, observation time, Tsys, elevation, parallactic
angle etc) should be optionally shown, either on the plot or next to
it. This may either consist of a set of values, or only one or two
values the user specifically wants to see (source name and molecule,
for example).}

\subrequirement{User define header plot positions}{Not started}{3}
{The user should be able to define where on the plot the header info
would appear.}

\requirement{Realtime cursor position}{Done1}{} 
{Optionally, relevant data such as the current mouse position should
be displayed (maybe with a mode to display an extended cross,
horizontal or vertical line at the current cursor position).}

\requirement{Plot annotations}{Not started}{2}{The user should be able
  to define simple annotations. This would include text overlay and
  probably simple graphics (lines, arrows etc).}

The user must be able to use the plotter window to interactively set
initial values and ranges used for fitting functions etc. The use of
keyboard ``shortcuts'' or other similar ``power user'' features should
be available to save the time of experienced users. 

{\em The following requirements are deferred as they require a version
of Matplotlib not yet available.}

The plotter should be used to set the following values:

\requirement{Interactive channel selection}{Deferred}{1}{Range of
 spectral channels needed for specific tasks (ie the channel mask)
 (See requirement \reqref{ref:chansel})}

\requirement{Interactive line fitting}{Deferred}{1}{Initial
Gaussian parameters (velocity, width, amplitude) for profile
fitting.}

\requirement{Plotter change fit values}{Deferred}{1}
{Change the parameter values of existing line profile
fits, or channel ranges used for baseline fits.}

\section{Functionality}

\subsection{Import/export}

The software needs a set of import/export functions to deal with a
variety of data formats and to be able to exchange data with other
popular packages. These functions should be flexible enough to allow
the user to perform analysis functions in an different package and
re-import the data (or vice versa). The import function must be
modular enough to easily add new file formats when the need arises.
To properly import data, extra information may have to be read from
secondary calibration files (such as GTP, Gated Total Power, for 3~mm
wavelength data taken with Mopra). The import functions should be
flexible enough to gracefully handle data files with missing headers
etc. They should also be able to make telescope and date specific
corrections to the data (for ATNF observatories).

The software must be able to read (import) the following file formats.

\requirement{Read rpfits}{Done1}{}{The rpfits file format produced by
  all current ATNF correlators.}

\requirement{Read sdfits}{Done1}{}{SDFITS (currently written by {\tt SPC}).}

\requirement{Read simple FITS}{Not started}{2}{Simple ``image'' FITS
(used by CLASS}

\requirement{Read historic formats}{Not started}{3}
{Historic ATNF single dish formats (Spectra, SPC, SLAP). Possibly a
set of routines to translate these formats to SDFITs would suffice.}

\requirement{Read PSRFITS}{Deferred}{}{PSRFIT for pulsar spectroscopy.}

\requirement{Read online data}{Not started}{1}
{For online analysis, the software should be able to read an rpfits
file which is is still currently open for writing by the telescope
backend processor.}

\requirement{Handle Doppler data}{Done1}{1}{Data which has been
observed in either a fixed frequency or Doppler tracked fashion needs
to be handled.}

The software should be able to export the data in the following formats.

\requirement{Write SDFITS}{Done1}{}{Single Dish FITS.}

\requirement{Write simple FITS}{Done1}{}
{Simple ``image'' FITS (as used by CLASS). It must be possible to to
export multiple spectra simultaneously, using default file name and
postfix.}

\requirement{}{Removed}{}{In a format which can be imported by other popular
    packages such as Class. }

\requirement{Write ASCIIs}{Done1}{}
{Simple ASCIIs format, suitable for use with programs such as Perl,
Python, SuperMongo etc.}

\requirement{Header Writing}{Not started}{1}
{The exported data should retain as much header data as possible. It
should also be possible to request specific data be written in the
desired form (B1950 coordinates, optical velocity definition etc).}

\requirement{Import corrections}{Done1}{}
{The import function should apply relevant corrections (especially
those which are time dependent) to specific telescopes. See
$\S$\ref{sec:issues} for a list of currently known issues.}

\requirement{Append output files}{Not started}{1} {It must be possible
to append spectra to existing output files, specifically sdfits and
asap output files.}

\subsection{Sky subtraction}
\label{sec:skysubtraction}
To remove the effects of the passband filter shape and atmospheric
fluctuations across the band, sky subtraction must be performed on the
data. The software must be able to do sky subtraction using both
position switching (quotient spectra) and frequency switching
techniques.

\requirement{Quotient Spectra}{Done1}{}
{\label{ref:skysub} Position switched sky subtraction should be
implemented using the algorithm \medskip\reqeqn{T_{ref} \times
\frac{S}{R} - T_{sig}} -- removes continuum\bigskip \reqeqn{T_{ref}
\times \frac{S}{R} - T_{ref}} -- preserves continuum\medskip}

\requirement{Arbitrary reference}{Not started}{2}
{The user should be able to specify an arbitrarily complex
reference/source order (which repeats), which can then be used to make
perform multiple sky subtractions in parallel.}

\requirement{Frequency switching}{Not started}{2}
{Frequency switched sky subtraction should be supported. (Ref. Liszt,
1997, A\&AS, 124, 183) }

%\requirement{For wideband multibit sampled data it may be desirable or
%even required to assume Tsys has a frequency dependency. Appropriate
%sky subtraction algorithms will need to be investigated.}{3}

\requirement{Pulsar off pulse quotient}{Deferred}{3}
{For pulsar binned data, the (user specified) off pulse bins can be
used as the reference spectra. Due to potentially rapid amplitude
fluctuations, sky subtractions may need to be done on a 
integration basis.}

Multibeam systems can observe in a nodding fashion (called MX mode at
Parkes), where the telescope position is nodded between scans so that
the source is observed in turn by two beams and a reference spectra
for one beam is obtained while the other is observing the target source.

\requirement{Multibeam MX mode}{Not started}{2}
{For multibeam systems, it must be possible to perform sky subtraction
with the source and reference in an alternate pair of beams}

\subsection{Baseline removal}

Baseline removal is needed to correct for imperfections in sky
subtraction. Depending on the stability of the system, the residual
spectral baseline errors can be small or quite large. Baseline removal
is usually done by fitting a function to the (user specified) line
free channels.

\requirement{Baseline removal}{Done1}{}
{The software must be able to do baseline removal by fitting a n'th
order polynomials to the line free channels using a least squares
method.}

\requirement{Standing wave ripples}{Not started}{3}
{Removal of standing wave ripples should be done by fitting a Sine
function to the line free channels.}

\requirement{Robust fitting}{Not started}{3} 
{``Robust'' fitting functions should be available, which are more
tolerant to RFI.}

\requirement{Auto-baseline}{Done1}{}
{Automatic techniques for baselining should be investigated.}

\subsection{Line Profile Fitting}

The user will want to fit multicomponent line profiles to the data in
a simple manner and be able to manipulate the exact fitting
parameters.

\requirement{Gaussian fitting}{Done1}{}
{The software must be able to do multi-component Gaussian fitting of
the spectra. The initial amplitude, width and velocity of each
component should be able to be set by the user and specific values to
be fit should be easily set.}

\requirement{Chi squared}{Done1}{}
{The reduce Chi squared (or similar statistic) of the fit should given
to the user, so that they can easily see if adding extra components
give a statistically significant improvement to the fit.}

%\requirement{The fit parameters should be stored with the data so that
%the user can work on multiple data sets simultaneously and experiment
%with different fitting values. These values should be saved to disk
%along with the data.}{1}

\requirement{Fit multipol data}{Done1}{}
{For multiple polarisation data, the individual stokes parameters or
polarisation products should be fit independently.}

\requirement{Export fits}{Not started}{1}
{There should be an easy way of exporting the fit parameter from
multiple spectra, e.g. as an ASCII table.}

\requirement{Constrained fitting}{Not started}{1}
{It should be also possible to do constrained fitting of multiple
hyperfine components, e.g. the NH$_3$ hyperfine components. (The
constraints may be either the frequency separation of the individual
components or the amplitude ratio etc.)}

\requirement{Edit fits parameters}{Done1}{}
{It must be possible to alter the line profile fit parameter values by
hand at any stage.}

\requirement{Fix fit parameters}{Done1}{}
{It must be possible to ``fix'' particular values of the line profile
parameters, so that only subset of lines or (say) the width of a line
is fit.}

\requirement{Arbitrary line fitting}{Done1}{}
{The software should allow hooks for line profile shapes other than
Gaussian to be added in the future, possible user specified.}

\requirement{Save fit parameters}{Done1}{}
{The fitting parameters for functions which have been fit to the data
(e.g. for baseline removal or Gaussian fits) should be retained as an
integral part of the data and stored permanently on disk.}

\requirement{Undo subtracted fits}{Not started}{3}
{It should be possible to ``undo'' functions which have been
subtracted from the data (e.g. baseline polynomials).}

\requirement{Gaussian line area}{Not started}{1} 
{Optionally the area under a fitted Gaussian should be calculated for
the user.}

%\makenote{Should it be possible to attach multiple sets of fits to the
%data (similar to CL tables in classic AIPS), so the user can
%experiment with different ways of fitting the data?}

%\makenote{Should calculations of rotational temperatures etc be
%handled when fitting hyperfine components, or should the user be doing
%this themselves?}

\subsection{Calibration}

The software should handle all basic system temperature (Tsys) and
gain calibration as well as opacity corrections where relevant. The
Tsys value should be contained in the rpfits files. The actual
application of the T$_{\mbox{sys}}$ factor will be applied as part of
the sky subtraction ($\S$\ref{sec:skysubtraction}). The units of Tsys
recorded in the data may be either in Jy or Kelvin, which will affect
how the data is calibrated. The rpfits file does {\em not} distinguish
if the flux units are Kelvin or Janskys. 

\requirement{Gain-elevation}{Done1}{}
{Gain elevation corrections should be implemented using a elevation
dependent polynomial. The polynomial coefficients will be telescope
and frequency dependent. They will also have a (long term) time
dependence.}

\requirement{User gain polynomial}{Done1}{}
{The user may wish to supply their own gain polynomial.}

\requirement{K-Jy conversion}{Done1}{}
{When required by the user, the spectral units must be converted from
Kelvin to Jansky. At higher (3mm) frequencies this conversion is often
not applied. The conversion factor is\medskip \reqeqn{\mbox{Flux (Jy)}
= \frac{T \times 2 k_b \times 10^{26}}{\eta A}},\medskip\\where $k_b$
is Boltzmann's constant, A is the illuminated area of the telescope
and $\eta$ is the efficiency of the telescope (frequency, telescope
and time dependent)}

\requirement{Scale Tsys}{Done1}{}
{In some cases the recorded Tsys values will be wrong. There needs to
be a mechanism to scale the Tsys value and the spectrum if the Tsys
value has already been applied (i.e. a simple and consistent rescaling
factor).}

\requirement{Opacity}{Done1}{}
{The data may need to be corrected for opacity effects, particularly
at frequencies of 20~GHz and higher. The opacity factor to apply is
given by\medskip\reqeqn{C_o = e^{\tau/cos(z)}}\medskip\\ where $\tau$
is the opacity and z is the zenith angle (90-El). These corrections
will generally be derived from periodic ``skydip'' measurements. These
values will not be contained in the rpfits files, so there should be a
simple way of the software obtaining them and interpolating in time
(the user should not {\em have} to type them in, but may want
to). Reading in an ASCIIs file which contains the skydip data along
with a time-stamp would be one possibility.}

\requirement{Tsys variation with freq}{Not started}{3}
{For wideband, multibit observations, the software should have the
option to handle Tsys which varies across the band. The exact
implementation will have to be decided once experience is gained with
the new Mopra digital filterbank. This will affect the sky subtraction
algorithms (requirement \reqref{ref:skysub}).}

%\makenote{Is the dependence of gain on frequency weak enough for one
%set of coefficients for each receiver, or is a full frequency dependent
%set of values needed?}

%\makenote{Should it be possible to read ``correct'' Tsys values from
%an external ascii file?} 

\subsection{Editing \& RFI robustness}

In a data set with many observations, individual spectra may be
corrupted or the data may be affected by RFI and ``birdies''. The user
needs to be able to easily flag individual spectra or channels. This
may affect other routines such as sky-subtraction, as this will
disrupt the reference/source sequence.

\requirement{Spectra flagging}{Started}{1}
{The user must be able to set an entire spectra or part thereof
(individual polarisation, IF etc) as being invalid.}

\requirement{Channel flagging}{Started}{1}
{The user must be able to indicate an individual spectral point or
range of spectral points are invalid. This should be applied to an
individual spectra, or set of spectra.}

\subrequirement{Flagged channel plotting}{???}{1} {When plotting data
with flagged spectral channels, the plotting should left a gap (blank)
in the plotted data|}.

\subrequirement{Flagged channel interpolatin}{Not started}{2} 
{When plotting or processing data (e.g. quotient spectra), the users
should be able to request that the values for flagged data are
obtained by interpolation from good data either side of the flagged
points.}

\requirement{Plot average flux vs time}{Not started}{3}
{The user should be able to plot the average spectral flux across the
band, or part of the band, as a function of time and interactively
select sections of data which should be marked as invalid (similar to
IBLED in classic aips).}

\requirement{Robust Fitting}{Duplicate}{2}
{Where relevant, fitting routines etc should have the option of
selecting RFI tolerant (``robust'') algorithms. This will require
investigating alternate fitting routines other than the least-squares
approach.}

\requirement{Birdie finder}{Not started}{2}
{A routine to automatically find birdies or RFI corrupted data and
indicate the data as invalid would be useful.}

\requirement{Handle flagged data}{Done1}{}
{Other routines must be able to cope with portions of data which are
marked as invalid.}

\subsection{Spectra mathematics and manipulation}

A flexible suite of mathematical operations on the spectra should be
possible. This should include options such as adding, subtracting,
averaging and scaling the data. For common operations such as
averaging and smoothing, it must be simple for the user to invoke the
function (i.e. not to have to start up a complex spectral
calculator). Where it makes sense, it should be possible to manipulate
multiple spectra simultaneously.

The spectral manipulations which should be available are:

\requirement{Add spectra}{Done1}{}{Add or subtract multiple spectra.}

\requirement{Average spectra}{Done1}{1}
{Averaging multiple spectra, with optional weighting based on Tsys,
integration or rms.}

\subrequirement{Average spectra with velocity shift}{Not started}{1}
{If the velocity of the spectra to be averaged is different, the data
should be aligned in velocity. The user should be able to turn this
feature on or off.}

\requirement{Robust averaging}{Not started}{2}
{Various robust averaging possibilities (e.g. median averaging,
clipped means etc) should be possible.}

\requirement{Data re-binning}{Done1}{}
{Re-sampling or re-binning of the data to a lower (or higher) spectral
resolution (i.e. change the number of spectral points). The
re-sampling factor may not necessarily be an integer.}

\requirement{Velocity shift}{Done1}{} 
{It must be possible to shift the data in ``frequency/velocity''. This
should include channel, frequency and velocity shifts of an arbitrary
amount.}

\requirement{Spectra smoothing}{Done1}{}
{Spectral smoothing of the data. Hanning, Tukey, boxcar and Gaussian
smoothing of variable widths should be possible.}

\requirement{Spectra scaling}{Done1}{}{Scaling of the spectra.}

\requirement{Spectra statistics}{Done1}{}
{Calculate basic statistical values (maximum, minimum, rms, mean) on a
range of spectral points. The range may not be contiguous. The
calculated rms value should be retained with the spectra so it can be
optionally used for weighted averaging of spectra.}

\requirement{Line flux}{Not started}{2}
{It must be possible to calculate the flux integral over a range of
channels. The units should be Jy.km/s (or Kelvin.km/s). The channel
range for the calculation should be specific via the GUI or CLI.}

\requirement{Line width}{Not started}{2}
{It must be possible to calculate the numerical ``width'' of a line
(full width at half maximum type measurement). This should be
calculated by specifying a channel range and finding the maximum value
in this range and then finding the interpolated crossing point of the
data as a user defined fraction of the maximum (default 50\%). The
profile width and velocity mid-point should then be computed. If the
profile shape is complex (e.g. double arch) with multiple crossing
points of the fraction value, the minimum and maximum width values
should be calculated. There should be the option of using a user
specified ``maximum value''.}

\requirement{Change rest frequency}{Done1}{}
{The user must be able to easily change the rest-frequency to which
the velocity is referenced.}

\requirement{FFT filtering}{Not started}{3}
{FFT filtering for high- and lowpass filtering and tapering.}

\requirement{FFT to/from autocorrelation function}{Not started}{3}
{It should be possible to FFT the data to and from power spectra to
the autocorrelation function.}

\requirement{Cross correlation}{Not started}{3}
{The user may wish to compute the cross correlation function of two
spectra. The result should be a standard ``spectra'', which can be
displayed and analysed using other functions (max, rms etc).}

\requirement{Spectral calculator}{Started}{1}
{Complex experiment specific processing can often be done using a
series of the simple of basic functions. A spectral calculator options
should be added to the CLI to perform a series of manipulations on a
set of spectra.}

The user may want to perform specific analysis on the data using the
functionality above, but wish to do the manipulation between two
polarisations or IFs. Allowing the functions to also, optionally,
specify specific polarisations or IF would be an implementation and
interface nightmare. The simplest solution is to allow the data to be
``split'' into separate spectra.

\requirement{Splice data}{Not started}{1}
{It must be possible to take multi IF, multibeam or polarisation data
and split out the individual spectral portions to form self contained
spectra.}

\requirement{Splice spectral channels}{Not started}{1} {It must be
possible to select a range of spectral channels to form self contained
spectra. The channel selection may be different for different IFs.}

\requirement{Merge scantables}{Not started}{1}
{It must be possible to append rows from one scantable onto another}

\subsection{Polarimetry}

The software must fully support polarmetric analysis. This includes
calibration and basic conversions. Observations may be made with
linear or circular feed and the backend may or may not compute the
cross polarisation products. As such the software must cope with a
variety of conversions. The software should be able to calculate
stokes parameters with or without solving for leakage terms.

%\makenote{It is debatable whether stokes I is the sum or average or
%two dual polarisation measurements.}

\requirement{Support polarimetry}{Done1}{}
{All functions on the data (calibration, sky subtraction spectral
mathematics) must support arbitrary, multiple, polarisation (linear,
circular \& stokes and single, dual \& cross polarisations.}

\requirement{Calculate stokes I}{Done1}{}
{It must be possible to calculate stokes I from single or dual
polarisation observations.}

\requirement{Average mixed pol data}{Not started}{2}
{Average a mixture of dual polarisation and single polarisation data
and form average stokes I (e.g. for a long observation of a source, in
which one polarisation is missing for some time.}

\requirement{Calculate stokes}{Done1}{}
{Full stokes parameters should be obtained from dual pol (linear or
circular) observations where the cross polarisation products have been
calculated.}

%\requirement{If the observations used linear polarisations and the
%cross polarisations were not computed, the source needs to be
%observed with the feeds set at least 3 different parallactic angles
%(note that if dual linear feeds are available, 2 orthogonal
%parallactic angles are obtained at once). The Stokes parameters can be
%solved using a least squares fit to the equation:
%\reqeqn{Iu/2 + Ip * cos^2 (PA + p)},\\ 
%where PA is the linear feed position angle, p is the polarisation
%angle, Iu and Ip and the unpolarised and linearly polarised
%intensity. {\em Stolen from SPC. I need to write this in more useful
%language. Is this technique likely to be used anymore?.}}{3}

\requirement{Compute stokes V without crosspol}{Not started}{2}
{If dual circular polarisation measurements are taken, without
computing the cross products, the software should still be able to
compute stokes I and V.}

\requirement{Polarisation leakages}{Not started}{3}
{The software should be able to calculate leakage terms from a
calibrator source and correct the data either before or after
conversion to Stokes. (ref. Johnston, 2002, PASA, 19, 277)}

\requirement{Calibrate position angle}{Not started}{3}
{The software should be able to determine absolute position angle from
a calibrator source and correct the data either before or after
conversion to Stokes.}

\requirement{Zeeman splitting}{Not started}{3}
{Zeeman splitting factors should be derived from (previous) profile
fitting and the left and right circular polarisations. The velocity
shift varies linearly with the magnetic field, but the scaling factor
depends on the molecule and transition. Scaling factor for common
transitions should be known by the software and the user able to enter
factors for less common transitions. Correctly identifying Zeeman
pairs is crucial in getting the correct result. The software should
attempt to make an initial guess of pairs (based on component velocity
and width) but make the user confirm and override the pairing if
required.}

\subsection{Data Selection}
While the software is running the user will usually have loaded
multiple (possibly many) spectra each of which may have multiple IFs,
data from multiple beams and multiple polarisations. The user will
want to be able to quickly flip from considering one spectra to
another and, where relevant, want to perform parallel processing on
multiple spectra at once (e.g. baselining a sequence of on/off
observations of the same source which will later be averaged
together).

\requirement{Spectra selection}{Started}{1}
{The software needs an easy-to-use mechanism to select either
individual or multiple spectra for viewing, parallel processing
etc.}

\requirement{Beam/IF selection}{Started}{1}
{An easy-to-use mechanism to select individual IFs, beams or
polarisations is needed.}

\requirement{Interactive channel selection}{Duplicate}{1}
{\label{ref:chansel} The range of spectral points to use for baseline
removal, statistical calculations, RFI editing, analysis etc must be
easily set by the user from both the CLI and GUI. From the CLI there
must be the option of setting the range using a variety of units
(channel number, velocity, frequency). The selection range will
probably not be a contiguous set of channels, but many sets of
disjoint channel ranges. For some tasks (such as baseline subtraction
and statistical values), the channel range should be retained and be
available as a plot overlay.}

\requirement{Auto-identify reference spectra}{Not started}{1}
{When performing sky subtraction on many spectra simultaneously, the
software should have a mechanism for identifying ``on'' and ``off''
spectra and automatically selecting the signal and quotient
spectra. The algorithm needs to cope with on/off/on/off sequences as
well as off/on/on/off. If an individual quotient spectra has been
marked as invalid, an alternative should be found. User specified
preference such as ``closest in time'' to ``first reference before
source'' should be accommodated.}

\requirement{Select source via header values}{Started}{1}
{The software should be able to select sets of sources based on simple
regular expression type filtering (wild cards) on a range of header
values. }

\subrequirement{Select on source name}{Done1}{1}
{The use should be able to select dana on source name, e.g G309$*$ or
G309$*$w}

\subrequirement{Select on molecule}{Done1}{2}
{The use should be able to select data on molecule name, e.g. NH3$*$.}

\subsection{Plugins}

\requirement{Plugins}{Started}{1}
{The package should support ``plugins'', user definable
functions for specific processing. The plugin code must have full
access (read/write) to the spectra data and headers.}

\requirement{Plugins can reduce dimensions}{Not started}{2}
{Plugins need to be able to create ``derived'' spectra with reduced
  dimensions (i.e.. less beams, IFs, polarisations or spectral
  channels)}

\requirement{Simulated data}{Not stated}{3} 
{The user should be able to create new spectra which the software
treats the same as the original data. This includes full specification
of the header items.}

\subsection{Pipelining}

\requirement{Pipelining}{Done1}{}
{Some sort of pipelining mode is required. This would involve doing a
quotient spectra, applying appropriate calibration and possibly
fitting a Gaussian to any lines present.}

\subsection{Methanol Multibeam Survey}

The software may need to support reduction of data from the methanol
multibeam project. If so the pipelining will need to be flexible and
powerful enough to support this.

\subsection{Miscellaneous functionality}

\requirement{Position fitting}{Not started}{2}
{The software should be able to take a simple ``grid'' of observations
(normally a set of observations in a cross pattern on the sky) and,
for a subset of channels, fit the position of the emission. The fit
positions should be either plotted on the screen or exported in a
simple ASCIIs form.}

\requirement{Kinematic distance}{Not started}{3}
{The kinematic distance of a source should be calculated using basic
Galactic rotation models. Multiple Galactic rotation models must be
supported and a mechanism for easily adding more.}

\requirement{Plot sigma errors on spectra}{Not started}{3}
{For 1420 MHz observations of HI, the rms (Tsys) values vary
significantly across the band. The software should be able to compute
the rms as a function of frequency across the spectra from the
off-pulse data and then be able to plot n-sigma error bars on the
spectra.}

\requirement{Simple Mapping}{Not started}{3}
{It should be possible to take a selection of calibrated spectra which
are then passed to the ``Gridzilla'' program to produce an image
cube. Analysis of this cube would be done using external programs
(e.g. Miriad, aips++)}

\section{Help}

\requirement{Built in help}{Done1}{} 
{There should be built-in and web-based documentation, which can be
easily kept up-to-date}

\requirement{Cookbook}{Done1}{}
{A short and simple end-to-end cookbook for basic data analysis should
be available.}

\requirement{Programmers Documentation}{Not started}{2}
{There should be documentation aimed at astronomers wishing to write
there own scripts, detailing the methods needed and how to get low
level access to the data.}

\section{Data and meta-data}

\requirement{Handle multi dimensional data}{Done1}{}
{The software must be capable of handling multi-IF (potentially dozens
of IFs) and multi-beam data with arbitrary polarisation (e.g. single
pol, dual pol, full stokes etc).}

\requirement{Handle pulsar data}{Deferred}{}
{The software should handle pulsar binned data for pulsar absorption
experiments.}

\subsection{History}

\requirement{History}{Done1}{}
{A user viewable history of data processing steps should be kept as
part of the data. Where possible this should be retained when data is
imported from other packages.}{

\requirement{Convert history to script}{Not started}{2}
{It should be possible to use the history information to create
template pipeline scripts for batch processing.}

\subsection{Multiple IFs}

\requirement{Transparently handle multi-IF data}{Done1}{}
{If multiple IFs are present (currently Tidbinbilla can produce two
IFs and the new wideband spectrometer for Mopra may have dozens of
IFs) the software should handle the data transparently. Potentially
each IF may have a significantly different sky frequency and be
observing a different molecule or transition with a different rest
frequency. From the users point of view, simultaneously obtained IFs
should be kept within the same ``container'' (not split into a myriad
of separate ``container'').}

\requirement{IFs with different number of spectral channels}{Not started}{2}
{Separate IFs may have a different number of spectral channels.}

\subsection{Multibeam}

\requirement{Handle multibeam data}{Done1}{}
{Basic handling of multibeam data should be possible (ie in general
each beam will be treated as a separate observation, but all within
the same container). The user should be able to view or process either
individual beams or all beams in parallel.}

\requirement{Multibeam simultaneous reference/signal}{Not started}{3}
{The use of a single beam observing a source and the rest of the beams
as reference beams for sky-subtraction should be investigated.}

\subsection{Robust fitting}

\requirement{Retain raw correlator integrations}{Done1}{}
{If robust fitting using median filtering is used, then the individual
integrations from the observations should {\em not} be averaged when
the data is imported, but retained within a single
container. Inspection of this data should be optionally of the
averaged or individual data.}

\subsection{Coordinate frames and units}

\requirement{Flexible coordinate frames}{Done1}{}
{Coordinate frames and unit selection and handling needs to be
flexible and relatively transparent to the user (i.e. if the users
preference is for LSRK velocities, they do not need to worry about the
reference frame in which the data was observed).}

\requirement{Specific reference frames}{Done1}{}
{At a minimum the following reference frames and conventions should be
handled: \setlength{\parindent}{0pt}

\smallskip
\anitem{Position}{(RA,Dec) in J2000 \& B1950 (as well as other
  arbitrary epochs), Galactic, (Az,El).}

\anitem{Frequency}{Velocity (Topocentric, Geocentric, Barycentric,
    Heliocentric, kinematical LSR, dynamical LSR, Rest), Frequency
    (MHz, GHz), channel number.}

\anitem{Velocity}{ Optical, Radio, Relativistic.}

\anitem{Flux}{ Jansky, Kelvin (mJy etc).}}

\requirement{Data units and frames properly labelled}{Done1}{}
{All data should be internally labelled with the appropriate
 coordinate frame and units. If this information is ambiguous for some
 reason, it should be set when the data is imported and the user
 should not have to worry about it again.}

\requirement{Current reference frames clear to user}{Done1}{}
{It should be clear to the user what coordinate frame (velocity,
position etc) the data is being presented as.}

\requirement{Positional Reference Frame}{Not started}{1} {The user
should be able to specify the reference frame (Epoch,Equinox etc) for
which is used for exporting data, simple mapping output etc. J2000,
B1950 and Galactic should be supported. The default should be the frame
in what the data was recorded.}

\requirement{Non-conformist Positional Reference Frame}{Not Started}{2} 
{Non-confirmist positional frames such as Az-El should be supported.}

\subsection{Meta-data}

A comprehensive set of header data should be read from the input data
files. In general all meta-data available in the rpfits file should be
retained. The user may wish to enter some specific values by hand.

\requirement{View and edit header data}{Started}{1}
{All header data should be viewable and editable by the user. This
includes changes such as scaling the given Tsys values.}

\requirement{Missing header data}{Done1}{}
{Missing header data should be handled gracefully, i.e. the software
should fill the values with ``blanks'' and be able to continue to
process the data if possible.}

\requirement{User add missing header data}{Not started}{2}
{The user must be able to add missing header data, which is not
present in the RPFITs file. It must be possible to add the same header
data to multiple scans simultaneously.}

\extendedrequirement{Itemised header items}{Started}{1}
{The following header data would be required per scan:
\begin{itemize}
  \item Source name
  \item Scan type (signal or reference)
  \item Integration time
  \item Scan length (actual time of observation, $\ge$ integration time)
  \item Telescope
  \item UT time and date of observation
  \item Telescope elevation of observation
  \item Parallactic angle
  \item Beam size
  \item Scan ID
  \item Observer
  \item Project
  \item Polarisation
  \item Receiver
  \item Telescope coordinates
  \item Weather info (temperature, pressure, humidity)
  \item User axis display preference (LSR velocity, frequency etc).
\end{itemize}
}

\extendedrequirement{IF header items}{Started}{1}
{\label{req:if}
The following header data is required for each IF, beam etc:
\begin{itemize}
\item Source coordinates and coordinate frame
\item Frequency/velocity axis definition and type
\item System Temperature
\item Beam number
\item Molecule rest frequency$^\dagger$
\item Molecular name$^\dagger$
\item Molecular transition$^\dagger$
\item Molecular formula$^\dagger$
\end{itemize}
}

\requirement{Pretty print formula}{Not started}{3}
{The molecular formula could be stored with embedded superscripted and
subscripted symbols for ``pretty'' printing on the plotted, but
printed in plain text on the CLI or in ASCIIs output}

Some molecular line rest-frequencies are close enough that two or more
molecules or transitions may be observed in a single IF. Typical
examples include the 1665/1667~MHz OH maser pair, NH$_3$ transitions,
and many observations in the 3~mm band.
\vspace{\parskip}

\requirement{Multiple rest frequencies per IF}{Not started}{2}
{The software should optionally support multiple lines per IF, by
storing a set of rest frequencies per IF, rather than a single
value. The header values in requirement \reqref{req:if} marked with a
$\dagger$ would all have to be stored as an array of values rather
than a scalar. A simple mechanism must be possible to change the
currently ``active'' rest-frequency.}

\section{Installation}

\requirement{Easy installation}{Started}{1}
{It must be possible for astronomers to install the software at their
own institute with either a moderate amount of OS experience or some
help from the local system administrators. This includes installation
on a central ``NFS'' server as well as local desk-tops.}

\requirement{Linux Support}{Started}{1}
{The software must run on major flavours of Linux
(Fedora/Redhat, Debian, etc).}

\subrequirement{Solaris Support}{Started}{1}
{The software must run on Solaris}

\requirement{Run on laptop}{Done1}{}
{It must be possible for users to install the software on their
laptops and run with no network connection.}

\requirement{Easy upgrade}{Done1}{}
{It should be relatively easy to upgrade to the latest version of the
software.}

\requirement{MacOS/X support}{Not started}{1}
{The software should run on MacOS/X}

\requirement{Windows support}{Not started}{3}
{It would be desirable for the software to run on Windows.}

\section{Known Issues}
\label{sec:issues}
The following issue are known problems with the data from ATNF
telescopes, which probably should be automatically corrected for if at
all possible. The best place to do this is while loading the data.

\subsection{General}

\begin{itemize}
  \item All polarisations in the RPFITS files are labelled as
  XX/YY. These need to be relabelled as LL/RR when appropriate.
\end{itemize}

\subsection{Mopra}

\begin{itemize}
\item Data obtained in 2002 \& 2003 (and probably before) have an
  error in the frequency headers (this may be corrected by an external
  program). \makenote{Nedd Ladd}

\item The (RA,Dec) positions in the data file are in date coordinates
  not J2000. This causes problems for packages like Class when
  averaging the data. \makenote{Maria Hunt}

\item It is possible Tsys calibration is inconsistent currently.
  \makenote{Cormac Purcell??}

\end{itemize}

\subsection{Parkes}

\begin{itemize}
\item For pulsar data the automatic gain control is disabled. This
means the nominal Tsys measurement does not change and Tsys per
integration is encoded in a non-standard way. \makenote{Simon
Johnston}
\end{itemize}

\subsection{Tidbinbilla}

\begin{itemize}
\item All 20-GHz data is calibrated in flux units of Kelvin.
\end{itemize}


\section{Requirements Matrix}

\begin{longtable}{|l|l|l|c|}

\input{reqsum.tex}

\end{longtable}


\end{document}