source: branches/Release-2-fixes/doc/cookbook.tex@ 1845

\documentclass[11pt]{article}
\usepackage{a4}
\usepackage[dvips]{graphicx}

% Adjust the page size
\addtolength{\oddsidemargin}{-0.4in}
\addtolength{\evensidemargin}{+0.4in}
\addtolength{\textwidth}{+0.8in}

\setlength{\parindent}{0mm}
\setlength{\parskip}{1ex}


\title{ATNF Spectral Analysis Package\\Cookbook }
\author{Chris Phillips}


\newcommand{\cmd}[1]{{\tt #1}}

\begin{document}

\maketitle

\section{Introduction}

ASAP is a single dish spectral line processing package currently being
developed by the ATNF. It is intended to process data from all ATNF
antennas, and can probably be used for other antnnas if they can
produce ``Single Dish FITS'' format. It is based on the AIPS++
package.

%\section{Documentation Standards}

%In most of the examples in this document, it has been assumed that the 

\section{Installation and Running}

Currently there are installations running on Linux machines at 

\begin{itemize}
\item Epping - use hosts {\tt draco} or {\tt hydra}
\item Narrabri - use host {\tt kaputar}
\item Parkes - use host {\tt bourbon}
\item Mopra - use host {\tt minos}
\end{itemize}

To start asap log onto one of these Linux hosts and  enter

\begin{verbatim}
  > cd /my/data/directory
  > asap
\end{verbatim}

This starts the ASAP. To quit, you need to type \verb+^+-d
(control-d).

\section{Interface}

ASAP is written in C++ and python. The user interface uses the
``ipython'' interactive shell, which is a simple interactive interface
to python. The user does not need to understand python to use this,
but certain aspects python affect what the user can do.  The current
interface is object oriented.  In the future, we will build a
functional (non object oriented) shell on top of this to ease
interactive use.

\subsection {Integer Indices are 0-relative}

Please note, all integer indices in ASAP and iPython are {\bf 0-relative}.

\subsection{Objects}

The ASAP interface is based around a number of ``objects'' which the
user deals with. Objects range from the data which have been read from
disk, to tools used for fitting functions to the data. The following
main objects are used :

\begin{itemize}
  \item[\cmd{scantable}] The data container (actual spectra and header
    information)
  \item[\cmd{fitter}] A tool used to fit functions to the spectral data
  \item[\cmd{plotter}] A tool used to plot the spectral line data
  \item[\cmd{reader}] A tool which can be used to read data from disks
    into a scantable object.
\end{itemize}

There can be many objects of the same type. Each object is referred to 
by a variable name made by the user. The name of this variable is not
important and can be set to whatever the user prefers (ie ``s'' and
``ParkesHOH-20052002'' are equivalent).  However, having a simple and
consistent naming convention will help you a lot.

\subsection{Member Functions (functions)}

Following the object oriented approach, objects have associated
``member functions'' which can either be used to modify the data in
some way or change global properties of the object. In this document
member functions will be referred to simply as functions. From the
command line, the user can execute these functions using the syntax:
\begin{verbatim}
  ASAP> out = object.function(arguments)
\end{verbatim}

Where \cmd{out} is the name of the returned variable (could be a new
scantable object, or a vector of data, or a status return),
\cmd{object} is the object variable name (set by the user),
\cmd{function} is the name of the member function and \cmd{arguments}
is a list of arguments to the function. The arguments can be provided
either though position or \cmd{name=}.  A mix of the two can be used.
E.g.

\begin{verbatim}
  ASAP> av = scans(msk,weight='tsys')
  ASAP> av = scans(mask=msk,weight='tsys')
  ASAP> av = scans(msk,True)
  ASAP> scans.polybaseline(mask=msk, order=0, insitu=True)
  ASAP> scans.polybaseline(msk,0,True)
  ASAP> scans.polybaseline(mask, insitu=True)
\end{verbatim}

\subsection{Global Functions}

It does not make sense to implement some functions as member
functions, typically functions which operate on more than one
scantable (e.g. time averaging of many scans). These functions will
always be referred to as global functions.

\subsection{Interactive environment}

ipython has a number of useful interactive features and a few things to be aware
of for the new user.

\subsubsection{String completion}

Tab completion is enabled for all function names. If you type the
first few letters of a function name, then type {\tt <TAB>} the
function name will be auto completed if it is un-ambiguous, or a list
of possibilities will be given. Auto-completion works for the user
object names as well as function names. It does not work for
filenames, nor for function arguments.

Example
\begin{verbatim}
  ASAP> scans = scantable('MyData.rpf')
  ASAP> scans.se<TAB>
scans.set_cursor      scans.set_freqframe   scans.set_unit        scans.setpol
scans.set_doppler     scans.set_instrument  scans.setbeam         
scans.set_fluxunit    scans.set_restfreqs   scans.setif     
  ASAP> scans.set_in<TAB>
  ASAP> scans.set_instrument
\end{verbatim}

\subsubsection{Leading Spaces}

Python uses leading space to mark blocks of code. This means that it
you start a command line with a space, the command generally will
fail with an syntax error.

\subsubsection{Unix Interaction}

Basic unix shell commands (\cmd{pwd}, \cmd{ls}, \cmd{cd} etc) can be
issued from within ASAP. This allows the user to do things like look
at files in the current directory. The shell command ``\cmd{cd}''
works within ASAP, allowing the user to change between data
directories. Unix programs cannot be run this way, but the shell
escape ``$!$'' can be used to run arbitrary programs. E.g.

\begin{verbatim}
  ASAP> pwd
  ASAP> ls
  ASAP> ! mozilla&
\end{verbatim}

\subsection{Help}

ASAP has built in help for all functions. To get a list of functions type:

\begin{verbatim}
  ASAP> commands
\end{verbatim}

To get help on specific functions, the built in help needs to be given
the object and function name. E.g.

\begin{verbatim}
  ASAP> help scantable.get_scan
  ASAP> help scantable.stats
  ASAP> help plotter.plot
  ASAP> help fitter.plot

  ASAP> scans = scantable('mydata.asap')
  ASAP> help scans.get_scan # Same as above
\end{verbatim}

Global functions just need their name

\begin{verbatim}
  ASAP> help average_time 
\end{verbatim}

Note that if you just type \cmd{help} the internal ipython help is
invoked, which is probably {\em not} what you want. Type \verb+^+-d
(control-d) to escape from this.

\subsection{.asaprc}

ASAP use an \cmd{.asaprc} file to control the user's preference of
default values for various functions arguments. This includes the
defaults for arguments such as \cmd{insitu}, scantable \cmd{freqframe}
and the plotters \cmd{set\_mode} values. The help on individual
functions says which arguments can be set default values from the
\cmd{.asaprc} file. To get a sample contents for the \cmd{.asaprc}
file use then command \cmd{list\_rcparameters}.

Common values include:
\begin{verbatim}
  # apply operations on the input scantable or return new one
  insitu                     : False

  # default ouput format when saving scantable
  scantable.save             : 'ASAP'


  # default frequency frame to set when function
  # scantable.set_freqframe is called
  scantable.freqframe        : 'LSRK'

  # auto averaging on read
  scantable.autoaverage      : True
\end{verbatim}

\section{Scantables}

\subsection {Description}

\subsubsection {Basic Structure}

ASAP data handling works on objects called scantables.  A scantable
holds your data, and also provides functions to operate
upon it.

The building block of a scantable is an integration, which is a single
row of a scantable. Each row contains spectra for each beam, IF and
polarisation. For example Parkes multibeam data would contain many
beams, one IF and 2-4 polarisations, while the new Mopra 8-GHz
filterbank will eventually produce one beam, many IFs, and 2-4
polarisations.

A collection of sequential integrations (rows) for one source is termed
a scan (and each scan has a unique numeric identifier, the ScanID). A
scantable is then a collection  of one or more scans. If you have
scan-averaged your data in time, then each scan would  hold just one
(averaged) integration.

Many of the functions which work on scantables can either return a
new scantable with modified data or change the scantable insitu. Which
method is used depends on the users preference. The default can be
changed via the {\tt .asaprc} resource file.

\subsubsection {Contents}

A scantable has header information and data (a scantable is actually an AIPS++
Table and it is stored in Memory when you are manipulating it with ASAP.
You can store it to disk and then browse it with the AIPS++
Table browser if you know how to do that !).

The data are stored in columns (the length of a column is the number of
rows/integrations of course).  

Two important columns are those that describe the frequency setup.  We mention
them explicitly here because you need to be able to understand the presentation
of the frequency information and possibly how to manipulate it.

These columns are called FreqID and RestFreqID.  They contain indices, for
each IF, pointing into tables with all of the frequency information for that
integration.   More on these below when we discuss the \cmd{summary} function
in the next subsection.

There are of course many other columns which contain the actual spectra,
the flags, the Tsys, the source names and so on, but those are a little
more transparently handled.

\subsection{Management}

During processing it is possible to create a large number of scan
tables. These all consume memory, so it is best to periodically remove
unneeded scan tables. Use \cmd{list\_scans} to print a list of all
scantables and \cmd{del} to remove unneeded ones.

Example:

\begin{verbatim}
  ASAP> list_scans
  The user created scantables are:
  ['s', 'scans', 'av', 's2', 'ss']

  ASAP> del s2   
  ASAP> del ss
\end{verbatim}

There is also a function \cmd{summary} to list a summary of the scantable.
You will find this very useful.

Example:

\begin{verbatim}
  ASAP> scans = scantable('MyData.rpf')
  ASAP> scans.summary()                # Brief listing
  ASAP> scans.summary(verbose=True)    # Include frequency information

  # Equivalent to brief summary function call
  ASAP> print scan                     
\end{verbatim}

Most of what the \cmd{summary} function  prints out is obvious. However,
it also prints out the FreqIDs and RestFreqIDs to which we alluded above. 
These are the last column of the listing.

The summary function gives you a scan-based summary.  So it lists all of
the FreqIDs and RestFreqIDs that it encountered for each scan.  If you'd
like to see what each FreqID actually means, then set the verbose
argument to True and the frequency table will be listed at the end. 
FreqID of 3 say, refers to the fourth row of the frequency table (ASAP
is 0-relative). The list of rest frequencies, to which the RestFreqIDs
refer, is always listed.

%You can copy one scantable to another with the \cmd{copy} function.

%Example:

%\begin{verbatim}
%  ASAP> scans = scantable('MyData.rpf')
%  ASAP> scan2 = scans.copy()
%\end{verbatim}

\subsection{State}

Each scantable contains "state"; these are properties  applying to all
of the data in the scantable.  

Examples are the selection of beam, IF and polarisation,  spectral unit
(e.g. km/s) frequency reference frame (e.g. BARY) and velocity Doppler
type (e.g. RADIO).

\subsubsection{Units, Doppler and Frequency Reference Frame}

The information describing the frequency setup for each integration
is stored fundamentally in frequency in the reference frame
of observation (E.g. TOPO).   

When required, this is converted to the desired reference frame
(e.g. LSRK), Doppler (e.g. OPTICAL) and unit (e.g. km/s) on-the-fly.
This is important, for example, when you are displaying the data or
fitting to it.

For units, the user has the choice of frequency, velocity or channel.
The \cmd{set\_unit} function is used to set the current unit for a
scantable. All functions will (where relevant) work with the selected
unit until this changes. This is mainly important for fitting (the fits
can be computed in any of these units), plotting and mask creation. 

The velocity definition can be changed with the \cmd{set\_doppler}
function, and the frequency reference frame can be changed with the
\cmd{set\_freqframe} function.

Example usage:

\begin{verbatim}
  ASAP> scans = scantable('2004-11-23_1841-P484.rpf') # Read in the data
  ASAP> scans.set_freqframe('LSRK')  # Use the LSR velocity frame
  ASAP> scans.set_unit('km/s')        # Use velocity for plots etc from now on
  ASAP> scans.set_doppler('OPTICAL')  # Use the optical velocity convention
  ASAP> scans.set_unit('MHz')         # Use frequency in MHz from now on
\end{verbatim}


\subsubsection{Rest Frequency}

ASAP reads the line rest frequency from the RPFITS file when reading
the data. The values stored in the RPFITS file are not always correct
and so there is a function \cmd{set\_restfreq} to set the rest frequencies.

For each integration, there is a rest-frequency per IF (the rest
frequencies are just stored as a list with an index into them).
There are a few ways to set the rest frequencies with this function.

If you specify just one rest frequency, then it is selected for the
specified source and IF and added to the list of rest frequencies.

\begin{verbatim}
  # Select for specified source/IF
  ASAP> scans.set_restfreqs(freqs=1.667359e9, source='NGC253', theif=0)   

  # Select for all sources and IFs
  ASAP> scans.set_restfreqs(freqs=1.667359e9)                             
\end{verbatim}


If you specify a list of frequencies, then it must be of length the
number of IFs.  Regardless of the source, the rest frequency will be set
for each IF to the corresponding value in the provided list.  The
internally stored list of rest frequencies will be replaced by this
list.


\begin{verbatim}
  # Select for specified source/IF
  ASAP> scans.set_restfreqs(freqs=1.667359e9, source='NGC253', theif=0)   

  # Select for all sources and IFs
  ASAP> scans.set_restfreqs(freqs=1.667359e9)                             
\end{verbatim}


In both of the above modes, you can also specify the rest frequencies via
names in a known list rather than by their values.

Examples:

\begin{verbatim}
  ASAP> scans.lines()                 # Print list of known lines
  ASAP> scans.set_restfreqs(lines=['OH1665','OH1667'])
\end{verbatim}

  

\subsection{Data Selection}

Data selection is currently fairly limited. This will be improved in
the future. 


\subsubsection{Cursor}

Generally the user will want to run functions on all rows in a
scantable. This allows very fast reduction of data. There are situations
when functions should only operate on specific elements of the spectra. This
is handled by the scantable cursor, which allows the user to select a
single beam, IF and polarisation combination.

Example :

\begin{verbatim}
  ASAP> scans.set_cursor(0,2,1)      # beam, IF, pol
  ASAP> scans.smooth(allaxes=F)      # in situ by default or .aipsrc
\end{verbatim}

\subsubsection{Row number}

Most functions work on all rows of a scan table. Exceptions are the
fitter and plotter. If you wish to only operate on a selected set of
scantable rows, use the \cmd{get\_scan} function to copy the rows into
a new scantable.

\subsubsection{Allaxes}

Many functions have an \cmd{allaxes} option which controls whether the
function will operate on all elements within a scantable row, or just
those selected with the current cursor. The default is taken from the
users {\tt .asaprc} file.

\subsubsection{Masks}

Many tasks (fitting, baseline subtraction, statistics etc) should only
be run on range of channels. Depending on the current ``unit'' setting
this range is set directly as channels, velocity or frequency
ranges. Internally these are converted into a simple boolean mask for
each channel of the abscissa. This means that if the unit setting is
later changed, previously created mask are still valid. (This is not
true for functions which change the shape or shift the frequency axis).
You create masks with the function \cmd{create\_mask} and this specified
the channels to be included in the selection.

When setting the mask in velocity, the conversion from velocity
to channels is based on the current cursor setting, selected row and
selected frequency reference frame.

Example :
\begin{verbatim}

  # Select channel range for baselining
  ASAP> scans.set_unit('channels')
  ASAP> msk = scans.create_mask([100,400],[600,800])
 
  # Select velocity range for fitting
  ASAP> scans.set_unit('km/s')
  ASAP> msk = scans.create_mask([-30,-10])
\end{verbatim}

Sometimes it is more convenient to specify the channels to be
excluded, rather included.  You can do this with the ``invert''
argument.

Example :
\begin{verbatim}
  ASAP> scans.set_unit('channels')
  ASAP> msk = scans.create_mask([0,100],[900-1023], invert=True)   
\end{verbatim}

By default \cmd{create\_mask} uses the frequency setup of the first row
to convert velocities into a channel mask. If the rows in the data
cover different velocity ranges, the scantable row to use should be
specified:

\begin{verbatim}
  ASAP> scans.set_unit('km/s')
  ASAP> msk = q.create_mask([-30,-10], row=5)
\end{verbatim}

Because the mask is stored in a simple python variable, the users is
able to combine masks using simple arithmetic. To create a mask
excluding the edge channels, a strong maser feature and a birdie in
the middle of the band:

\begin{verbatim}
  ASAP> scans.set_unit('channels')
  ASAP> msk1 = q.create_mask([0,100],[511,511],[900,1023],invert=True)
  ASAP> scans.set_unit('km/s')
  ASAP> msk2 = q.create_mask([-20,-10],invert=True)

  ASAP> mask = msk1 and msk2
\end{verbatim}


\section{Data Input}

Data can be loaded in one of two ways; using the reader object or via
the scantable constructor. The scantable method is simpler but the
reader allow the user more control on what is read.

\subsection{Scantable constructor}

This loads all of the data from filename into the scantable object scans
and averages all the data within a scan (i.e.  the resulting scantable
will have one row per scan).  The recognised input file formats are
RPFITS, SDFITS (singledish fits), ASAP's scantable format and aips++
MeasurementSet2 format. 


Example usage:

\begin{verbatim}
  ASAP> scan = scantable('2004-11-23_1841-P484.rpf')

  # Don't scan average the data
  ASAP> scan = scantable('2004-11-23_1841-P484.rpf', average=False)
\end{verbatim}


\subsection{Reader object}

For more control when reading data into ASAP, the reader object should
be used.  This has the option of only reading in a range of integrations
and does not perform any scan averaging of the data, allowing analysis
of the individual integrations.  Note that due to limitation of the
RPFITS library, only one reader object can be open at one time reading
RPFITS files.  To read multiple RPFITS files, the old reader must be
destroyed before the new file is opened.  However, multiple readers can
be created and attached to SDFITS files. 


Example usage:

\begin{verbatim}
  ASAP> r = reader('2003-03-16_082048_t0002.rpf')
  ASAP> r.summary 
  ASAP> scan = r.read()
  ASAP> s = r.read(range(100)) # To read in the first 100 integrations
  ASAP> del r
\end{verbatim}

\section{Basic Processing}

In the following section, a simple data reduction to form a quotient
spectrum of a single source is followed.  It has been assume that the
\cmd{.asaprc} file has {\em not} been used to change the \cmd{insitu}
default value from \cmd{True}.

%\subsection{Editing}

%How and when?

\subsection{Separate reference and source observations}

Most data from ATNF observatories distinguishes on and off source data
using the file name. This makes it easy to create two scantables with
the source and reference data. As long as there was exactly one
reference observation for each on source observation for following
method will work.

For Mopra and Parkes data:
\begin{verbatim}
  ASAP> r = scans.get_scan('*_R')
  ASAP> s = scans.get_scan('*_S')
\end{verbatim}

For Tidbinbilla data
\begin{verbatim}
  ASAP> r = scans.get_scan('*_[ew]')
  ASAP> s = scans.get_scan('*_[^ew]')
\end{verbatim}

\subsection{Make the quotient spectra}

Use the quotient function

\begin{verbatim}
  ASAP> q = s.quotient(r)
\end{verbatim}

This uses the rows in scantable \cmd{r} as reference spectra for the
rows in scantable \cmd{s}. Scantable \cmd{r} must have either 1 row
(which is applied to all rows in \cmd{s}) or both scantables must have
the same number of rows. By default the quotient spectra is calculated
to preserve continuum emission. If you wish to remove the continuum
contribution, use the \cmd{preserve} argument:

\begin{verbatim}
  ASAP> q = s.quotient(r, preserve=True)
\end{verbatim}

\subsection{Time average separate scans}

If you have observed the source with multiple source/reference cycles you
will want to scan-average the quotient spectra together.

\begin{verbatim}
 ASAP> av = average_time(q)
\end{verbatim}

If for some you want to average multiple sets of scantables together
you can:

\begin{verbatim}
 ASAP> av = average_time(q1, q2, q3)
\end{verbatim}

The default is to use integration time weighting. The alternative is
to use none, variance , Tsys weighting or Tsys \& integration time.

\begin{verbatim}
 ASAP> av = average_time(q, weight='tintsys')
\end{verbatim}

To use variance based weighting, you need to supply a mask saying which
channel range you want it to calculate the variance from.

\begin{verbatim}
 ASAP> msk = scans.create_mask([200,400],[600,800])
 ASAP> av = average_time(scans, mask=msk, weight='var')
\end{verbatim}

\subsection{Baseline fitting}

To make a baseline fit, you must first create a mask of channels to
use in the baseline fit. 

\begin{verbatim}
 ASAP> msk = scans.create_mask([100,400],[600,900])
 ASAP> scans.poly_baseline(msk, 1) 
\end{verbatim}

This will fit a first order polynomial to the selected channels and subtract
this polynomial from the full spectra.

\subsubsection{Auto-baselining}

The function \cmd{auto\_poly\_baseline} can be used to automatically
baseline your data with out having to specify channel ranges for
the line free data. It automatically figures out the line-free
emission and fits a polynomial baseline to that data. The user can use
masks to fix the range of channels or velocity range for the fit as
well as mark the band edge as invalid. 

Simple example

\begin{verbatim}
  ASAP> scans.auto_poly_baseline(order=2,threshold=5)
\end{verbatim}

\cmd{order} is the polynomial order for the fit. \cmd{threshold} is
the SNR threshold to use to deliminate line emission from
signal. Generally the value of threshold is not too critical, however
making this too large will compromise the fit (as it will include
strong line features) and making it too small will mean it cannot find
enough line free channels.


Other examples:

\begin{verbatim}
  # Don't try and fit the edge of the bandpass which is noisier
  ASAP> scans.auto_poly_baseline(edge=(500,450),order=3,threshold=3)

  # Only fit a given region around the line
  ASAP> scans.set_unit('km/s')
  ASAP> msk = scans.create_mask((-60,-20))
  ASAP> scans.auto_poly_baseline(mask=msk,order=3,threshold=3)

\end{verbatim}

\subsection{Average the polarisations}

If you are just interested in the highest SNR for total intensity you
will want to average the parallel polarisations together.

\begin{verbatim}
 ASAP> scans.average_pol()
\end{verbatim}

\subsection{Calibration}

For most uses, calibration happens transparently as the input data
contains the Tsys measurements taken during observations. The nominal
``Tsys'' values may be in Kelvin or Jansky. The user may wish to
supply a Tsys correction or apply gain-elevation and opacity
corrections.

\subsubsection{Brightness Units}

RPFITS files to not contain any information as to whether the telescope
calibration was in units of Kelvin or Janskys.  On reading the data a
default value is set depending on the telescope and frequency of
observation.  If this default is incorrect (you can see it in the
listing from the \cmd{summary} function) the user can either override
this value on reading the data or later.  E.g:

\begin{verbatim}
  ASAP> scans = scantable(('2004-11-23_1841-P484.rpf', unit='Jy')
  # Or in two steps
  ASAP> scans = scantable(('2004-11-23_1841-P484.rpf')
  ASAP> scans.set_fluxunit('Jy)
\end{verbatim}

\subsubsection{Tsys scaling}

Sometime the nominal Tsys measurement at the telescope is wrong due to
an incorrect noise diode calibration. This can easily be corrected for
with the scale function. By default, \cmd{scale} only scans the
spectra and not the corresponding Tsys.

\begin{verbatim}
  ASAP> scans.scale(1.05, tsys=True) 
\end{verbatim}

\subsubsection{Unit Conversion}

To convert measurements in Kelvin to Jy (and vice versa) the global
function \cmd{convert\_flux} is needed. This converts and scales the data
from K to Jy or vice-versa depending on what the current brightness unit is
set to. The function knows the basic parameters for some frequencies
and telescopes, but the user may need to supply the aperture
efficiency, telescope diameter or the Jy/K factor.

\begin{verbatim}
  ASAP> scans.convert_flux                 # If efficency known
  ASAP> scans.convert_flux(eta=0.48)       # If telescope diameter known
  ASAP> scans.convert_flux(eta=0.48,d=35)  # Unknown telescope
  ASAP> scans.convert_flux(jypk=15)        # Alternative
\end{verbatim}

\subsubsection{Gain-Elevation and Opacity Corrections}

As higher frequencies (particularly $>$20~GHz) it is important to make
corrections for atmospheric opacity and gain-elevation effects. 

Gain-elevation curves for some telescopes and frequencies are known to
ASAP (currently only for Tidbinbilla at 20~GHz).  In these cases making
gain-corrections is simple.  If the gain curve for your data is not
known, the user can supply either a gain polynomial or text file
tabulating gain factors at a range of elevations (see \cmd{help
scantable.gain\_el}).

Examples:

\begin{verbatim}
  ASAP> scans.gain_el()   # If gain table known
  ASAP> scans.gain_el(poly=[3.58788e-1,2.87243e-2,-3.219093e-4])
\end{verbatim}

Opacity corrections can be made with the global function
\cmd{opacity}. This should work on all telescopes as long as a
measurement of the opacity factor was made during the observation.

\begin{verbatim}
  ASAP> scans.opacity(0.083)
\end{verbatim}

Note that at 3~mm Mopra uses a paddle wheel for Tsys calibration,
which takes opacity effects into account (to first order). ASAP
opacity corrections should not be used for Mopra 3-mm data.

\subsection{Frequency Frame Alignment}

When time averaging a series of scans together, it is possible that
the velocity scales are not exactly aligned.  This may be for many
reasons such as not Doppler tracking the observations, errors in the
Doppler tracking etc.  This mostly affects very long integrations or
integrations averaged together from different days.  Before averaging
such data together, they should be frequency aligned using
\cmd{freq\_align}.

E.g.:

\begin{verbatim}
  ASAP> scans.freq_align()
  ASAP> av = average_time(scans)
\end{verbatim}

\cmd{freq\_align} has two modes of operations controlled by the
\cmd{perif} argument. By default it will align each source and freqid
separately. This is needed for scan tables containing multiple
sources. However if scan-based Doppler tracking has been made at the observatory,
each row will have a different freqid. In these cases run with
\cmd{perif=True} and all rows of a source will be aligned to the same
frame. In general \cmd{perif=True} will be needed for most
observations as Doppler tracking of some form is made at Parkes, Tid
and Mopra.

\begin{verbatim}
  ASAP> scans.freq_align(perif=True)
\end{verbatim}

To average together data taken on different days, which are in
different scantables, each scantable must aligned to a common
reference time then the scantables averaged. The simplest way of
doing this is to allow ASAP to choose the reference time for the first
scantable then using this time for the subsequent scantables. 

\begin{verbatim}
  ASAP> scans1.freq_align() # Copy the refeference Epoch from the output
  ASAP> scans2.freq_align(reftime='2004/11/23/18:43:35')
  ASAP> scans3.freq_align(reftime='2004/11/23/18:43:35')
  ASAP> av = average_time(scans1, scans2, scans3)
\end{verbatim}

\section{Scantable manipulation}

While it is very useful to have many independent sources within one
scantable, it is often inconvenient for data processing. The
\cmd{get\_scan} function can be used to create a new scantable with a
selection of scans from a scantable. The selection can either be on
the source name, with simple wildcard matching or set of scan ids.

For example:

\begin{verbatim}
  ASAP> ss = scans.get_scan(10) # Get the 11th scan (zero based)
  ASAP> ss = scans.get_scan(range(10)) # Get the first 10 scans
  ASAP> ss = scans.get_scan(range(10,20)) # Get the next 10 scans
  ASAP> ss = scans.get_scan([2,4,6,8,10]) # Get a selection of scans

  ASAP> ss = scans.get_scan('345p407') # Get a specific source
  ASAP> ss = scans.get_scan('345*')    # Get a few sources

  ASAP> r = scans.get_scan('*_R') # Get all reference sources (Parkes/Mopra)
  ASAP> s = scans.get_scan('*_S') # Get all program sources (Parkes/Mopra)
  ASAP> r = scans.get_scan('*_[ew]')  # Get all reference sources (Tid)
  ASAP> s = scans.get_scan('*_[^ew]') # Get all program sources (Tid)

\end{verbatim}

To copy a scantable the following does not work:

\begin{verbatim}
  ASAP> ss = scans
\end{verbatim}

as this just creates a reference to the original scantable. Any
changes made to \cmd{ss} are also seen in \cmd{scans}. To duplicate a
scantable, use the copy function.

\begin{verbatim}
  ASAP> ss = scans.copy
\end{verbatim}

\section{Data Output}

ASAP can save scantables in a variety of formats, suitable for reading
into other packages. The formats are:

\begin{itemize}
\item[ASAP] This is the internal format used for ASAP. It is the only
  format that allows the user to restore the data, fits etc. without
  loosing any information.  As mentioned before, the ASAP scantable is
  an AIPS++ Table (a memory-based table).  This function just converts
  it to a disk-based Table.  You can the access that Table with the
  AIPS++ Table browser or any other AIPS++ tool.

\item[SDFITS] The Single Dish FITS format. This format was designed to
  for interchange between packages, but few packages actually can read
  it.

\item[FITS] This uses simple ``image'' fits to save the data, each row
  being written to a separate fits file. This format is suitable for
  importing the data into CLASS.

\item[ASCII] A simple text based format suitable for the user to
processing using Perl or, Python, gnuplot etc.

\item[MS2] Saves the data in an aips++ MeasurementSet V2 format.
You can also access this with the Table browser and other AIPS++
tools.

\end{itemize}

The default output format can be set in the users {\tt .asaprc} file. 
Typical usages are:

\begin{verbatim}
  ASAP> scans.save('myscans') # Save in default format
  ASAP> scans.save('myscans', 'FITS') # Save as FITS for exporting into CLASS

  ASAP> scans.save('myscans', stokes=True) # Convert raw polarisations into Stokes
  ASAP> scans.save('myscans', overwrite=True) # Overwrite an existing file
\end{verbatim}


\section{Plotter}

Scantable spectra can be plotter at any time. An asapplotter object is
used for plotting, meaning multiple plot windows can be active at the
same time. On start up a default asapplotter object is created called
``plotter''. This would normally be used for standard plotting.

The plotter, optionally, will run in a multipanel mode and contain
multiple plots per panel. The user must tell the plotter how they want
the data distributed. This is done using the set\_mode function. The
default can be set in the users {\tt .asaprc} file. The units (and frame
etc) of the abscissa will be whatever has previously been set by
\cmd{set\_unit}, \cmd{set\_freqframe} etc.

Typical plotter usage would be:

\begin{verbatim}
  ASAP> scans.set_unit('km/s')
  ASAP> plotter.set_mode(stacking='p',panelling='t')
  ASAP> plotter.plot(scans)
\end{verbatim}

This will plot multiple polarisation within each plot panel and each
scan row in a separate panel.

Other possibilities include:

\begin{verbatim}
  # Plot multiple IFs per panel
  ASAP> plotter.set_mode(stacking='i',panelling='t')

  # Plot multiple beams per panel
  ASAP> plotter.set_mode(stacking='b',panelling='t')

  # Plot one IF per panel, time stacked
  ASAP> plotter.set_mode('t', 'i')

  # Plot each scan in a seperate panel
  ASAP> plotter.set_mode('t', 's')

\end{verbatim}

\subsection{Plot Selection}
\label{sec:plotter_cursor}

The plotter can plot up to 25 panels and stacked spectra per
panel. If you have data larger than this (or for your own sanity) you
need to select a subset of this data. This is particularly true for
multibeam or multi IF data. The plotter \cmd{set\_cursor} function is
used to select a subset of the data. The arguments \cmd{row},
\cmd{beam} and \cmd{IF} all accept a vector of indices corresponding
to row, beam or IF selection. Only the selected data will be plotted.
To select on polarisation, see section~\ref{sec:polplot}.

Examples:

\begin{verbatim}
  # Select second IF
  ASAP> plotter.set_cursor(IF=[1])

  # Select first 4 beams
  ASAP> plotter.set_cursor(beam=[0,1,2,3])

  # Select a few rows
  ASAP> plotter.set_cursor(row=[2,4,6,10])

  # Multiple selection
  ASAP> plotter.set_cursor(IF=[1], beam=[0,2], row=range(10))
\end{verbatim}

Note that the plotter cursor selection is independent of the scantable
cursor.

\subsection{Plot Control}

The plotter window has a row of buttons on the lower left. These can
be used to control the plotter (mostly for zooming the individual
plots). From left to right:

\begin{itemize}

\item[Home] This will unzoom the plots to the original zoom factor

\item[Plot history] (left and right arrow). The plotter keeps a
history of zoom settings. The left arrow sets the plot zoom to the
previous value. The right arrow returns back again. This allows you,
for example, to zoom in on one feature then return the plot to how it
was previously.

\item[Pan] (The Cross) This sets the cursor to pan, or scroll mode
       allowing you to shift the plot within the window. Useful when
       zoomed in on a feature.

\item[Zoom] (the letter with the magnifying glass) lets you draw a
       rectangle around a region of interest then zooms in on that
       region. Use the plot history to unzoom again.  

\item[Save] (floppy disk). Save the plot as a postscript or .png file

\end{itemize}

\subsection{Other control}

The plotter has a number of functions to describe the layout of the
plot. These include \cmd{set\_legend}, \cmd{set\_layout} and \cmd{set\_title}.

To set the exact velocity or channel range to be plotted use the
\cmd{set\_range} function. To reset to the default value, call
\cmd{set\_range} with no arguments. E.g.

\begin{verbatim}
  ASAP> scans.set_unit('km/s')
  ASAP> plotter.plot(scans)
  ASAP> plotter.set_range(-150,-50)
  ASAP> plotter.set_range() # To reset
\end{verbatim}

Both the range of the ``x'' and ``y'' axis can be set at once, if desired:

\begin{verbatim}
  ASAP> plotter.set_range(-10,30,-1,6.6)
\end{verbatim}

To save a hardcopy of the current plot, use the save function, e.g. 

\begin{verbatim}
  ASAP> plotter.save('myplot.ps')
\end{verbatim}

\section{Fitting}

Currently multicomponent Gaussian function is available. This is done
by creating a fitting object, setting up the fit and actually fitting
the data. Fitting can either be done on a single scantable row/cursor
selection or on an entire scantable using the \cmd{auto\_fit} function.

\begin{verbatim}
 ASAP> f = fitter()
 ASAP> f.set_function(gauss=2) # Fit two Gaussians
 ASAP> f.set_scan(scans)
 ASAP> scans.set_cursor(0,0,1) # Fit the second polarisation
 ASAP> scans.set_unit('km/s')  # Make fit in velocity units
 ASAP> f.fit(1)                # Run the fit on the second row in the table
 ASAP> f.plot()                # Show fit in a plot window
 ASAP> f.get_parameters()      # Return the fit paramaters
\end{verbatim}

This auto-guesses the initial values of the fit and works well for data
without extra confusing features. Note that the fit is performed in
whatever unit the abscissa is set to.

If you want to confine the fitting to a smaller range (e.g. to avoid
band edge effects or RFI you must set a mask.

\begin{verbatim}
  ASAP> f = fitter()
  ASAP> f.set_function(gauss=2)
  ASAP> scans.set_unit('km/s')  # Set the mask in channel units
  ASAP> msk = s.create_mask([1800,2200])
  ASAP> scans.set_unit('km/s')  # Make fit in velocity units
  ASAP> f.set_scan(s,msk)
  ASAP> f.fit()
  ASAP> f.plot()
  ASAP> f.get_parameters()
\end{verbatim}

If you wish, the initial parameter guesses can be specified and
specific parameters can be fixed:

\begin{verbatim}
  ASAP> f = fitter()
  ASAP> f.set_function(gauss=2)
  ASAP> f.set_scan(s,msk)
  ASAP> f.fit() # Fit using auto-estimates
  # Set Peak, centre and fwhm for the second gaussian. 
  # Force the centre to be fixed
  ASAP> f.set_gauss_parameters(0.4,450,150,0,1,0,component=1)
  ASAP> f.fit() # Re-run the fit
\end{verbatim}

The fitter \cmd{plot} function has a number of options to either view
the fit residuals or the individual components (by default it plots
the sum of the model components).

Examples:

\begin{verbatim}
  # Plot the residual
  ASAP> f.plot(residual=True) 

  # Plot the first 2 componentsa
  ASAP> f.plot(components=[0,1]) 

  # Plot the first and third component plus the model sum
  ASAP> f.plot(components=[-1,0,2])  # -1 means the compoment sum
\end{verbatim}

\subsection{Fit saving}

One you are happy with your fit, it is possible to store it as part of
the scantable.

\begin{verbatim}
  ASAP> f.storefit()
\end{verbatim}

This will be saved to disk with the data, if the ``ASAP'' file format
is selected. Multiple fits to the same data can be stored in the
scantable. 

The scantable function \cmd{get\_fit} can be used to retrieve the
stored fits. Currently the fit parameters are just printed to the
screen.

\begin{verbatim}
  ASAP> scans.get_fit(4) # Print fits for row 4
\end{verbatim}

\section{Polarisation}

Currently ASAP only supports polarmetric analysis on linearly
polarised feeds and the cross polarisation products measured. Other
cases will be added on an as needed basic.

Conversions of linears to Stokes or Circular polarisations are done
``on-the-fly''. Leakage cannot be corrected for nor are these routines
able to calibrate position angle offsets. 

\subsection{Simple Calibration}

{\em Currently the receiver position angle is not stored in the RPFITS
file. This severely hampers correct handling of polarimetry. In the future
we aim to define a general framework and populate the RPFITS files
with the data required for transparent polarimetric calibration.}

It is possible that there is a phase offset between polarisation which
will effect the phase of the cross polarisation correlation, and so give
rise to spurious polarisation. \cmd{rotate\_xyphase} can be used to
correct for this error. At this point, the user must know how to
determine the size of the phase offset themselves.

\begin{verbatim}
  ASAP> scans.rotate_xyphase(10.5)            # Degrees
\end{verbatim}

Note that if this function is run twice, the sum of the two values is
applied because it is done in-situ.

A correction for the receiver parallactic angle may need to be made,
either because of how it is mounted or if parallactifiying had to track
at 90 degrees rather than 0. Use \cmd{rotate\_linpolphase} to correct
the position angle. Running this function twice results in the sum of
the corrections being applied because it is applied in-situ.

\begin{verbatim}
  ASAP> scans.rotate_linpolphase(-20) # Degrees; correct for receiver mounting

  # Receiver was tracking 90 degrees rather than 0
  ASAP> scans.rotate_linpolphase(90)  
\end{verbatim}

\subsection{Plotting}
\label{sec:polplot}

To plot Stokes values, the plotter \cmd{set\_cursor} function should
be called first using the \cmd{pol} argument. The values which can be
plotted include a selection of [I,Q,U,V], [I, Plinear, Pangle, V],
[RR, LL] or [XX, YY, Real(XY), Imaginary(XY)]. (Plinear and Pangle are
the percentage and position angle of linear polarisation). Conversion
to circular polarisations are currently not available.

Example:

\begin{verbatim}
  ASAP> plotter.set_cursor(pol=[``I'',''Q'']
  ASAP> plotter.set_cursor(pol=[``RR'',''LL'']
  ASAP> plotter.set_cursor(pol=[``XX'',''YY'']
  ASAP> plotter.set_cursor(pol=[``I'',''Plinear'']
\end{verbatim}

Row, beam and IF selection are also available in \cmd{set\_cursor} as
describe in section~\ref{sec:plotter_cursor}.

\subsection{Saving}

When saving data using the \cmd{save} function, the \cmd{stokes}
argument can be used to save the data as Stoke values when saving in
FITS format.

Example:

\begin{verbatim}
  ASAP> scans.save('myscan.sdfits', 'SDFITS', stokes=True)
\end{verbatim}

\section{Function Summary}

\begin{verbatim}
        scantable           - a container for integrations/scans
                              (can open asap/rpfits/sdfits and ms files)
            copy            - returns a copy of a scan
            get_scan        - gets a specific scan out of a scantable
            summary         - print info about the scantable contents
            set_cursor      - set a specific Beam/IF/Pol 'cursor' for
                              further use
            get_cursor      - print out the current cursor position
            stats           - get specified statistic of the spectra in
                              the scantable
            stddev          - get the standard deviation of the spectra
                              in the scantable
            get_tsys        - get the TSys
            get_time        - get the timestamps of the integrations
            get_unit        - get the currnt unit
            set_unit        - set the abcissa unit to be used from this
                              point on
            get_abcissa     - get the abcissa values and name for a given
                              row (time)
            set_freqframe   - set the frame info for the Spectral Axis
                              (e.g. 'LSRK')
            set_doppler     - set the doppler to be used from this point on
            set_instrument  - set the instrument name
            get_fluxunit    - get the brightness flux unit
            set_fluxunit    - set the brightness flux unit
            create_mask     - return an mask in the current unit
                              for the given region. The specified regions
                              are NOT masked
            get_restfreqs   - get the current list of rest frequencies
            set_restfreqs   - set a list of rest frequencies
            lines           - print list of known spectral lines
            flag_spectrum   - flag a whole Beam/IF/Pol
            save            - save the scantable to disk as either 'ASAP'
                              or 'SDFITS'
            nbeam,nif,nchan,npol - the number of beams/IFs/Pols/Chans
            history         - print the history of the scantable
            get_fit         - get a fit which has been stored witnh the data
            average_time    - return the (weighted) time average of a scan 
                              or a list of scans
            average_pol     - average the polarisations together.
                              The dimension won't be reduced and
                              all polarisations will contain the
                              averaged spectrum.
            quotient        - return the on/off quotient
            scale           - return a scan scaled by a given factor
            add             - return a scan with given value added 
            bin             - return a scan with binned channels
            resample        - return a scan with resampled channels
            smooth          - return the spectrally smoothed scan
            poly_baseline   - fit a polynomial baseline to all Beams/IFs/Pols
            auto_poly_baseline - automatically fit a polynomial baseline 
            gain_el         - apply gain-elevation correction
            opacity         - apply opacity correction
            convert_flux    - convert to and from Jy and Kelvin brightness
                              units
            freq_align      - align spectra in frequency frame
            rotate_xyphase  - rotate XY phase of cross correlation
            rotate_linpolphase - rotate the phase of the complex
                                 polarization O=Q+iU correlation
     [Math] Mainly functions which operate on more than one scantable

            average_time    - return the (weighted) time average 
                              of a list of scans
            quotient        - return the on/off quotient
            simple_math     - simple mathematical operations on two scantables,
                              'add', 'sub', 'mul', 'div'
     [Fitting]
        fitter
            auto_fit        - return a scan where the function is
                              applied to all Beams/IFs/Pols.
            commit          - return a new scan where the fits have been
                              commited.
            fit             - execute the actual fitting process
            store_fit       - store the fit paramaters in the data (scantable)
            get_chi2        - get the Chi^2
            set_scan        - set the scantable to be fit
            set_function    - set the fitting function
            set_parameters  - set the parameters for the function(s), and
                              set if they should be held fixed during fitting
            set_gauss_parameters - same as above but specialised for individual
                                   gaussian components
            get_parameters  - get the fitted parameters
            plot            - plot the resulting fit and/or components and
                              residual
    [Plotter]
        asapplotter         - a plotter for asap, default plotter is
                              called 'plotter'
            plot            - plot a (list of) scantable
            save            - save the plot to a file ('png' ,'ps' or 'eps')
            set_mode        - set the state of the plotter, i.e.
                              what is to be plotted 'colour stacked'
                              and what 'panelled'
            set_cursor      - only plot a selected part of the data
            set_range       - set a 'zoom' window
            set_legend      - specify user labels for the legend indeces
            set_title       - specify user labels for the panel indeces
            set_ordinate    - specify a user label for the ordinate
            set_abcissa     - specify a user label for the abcissa
            set_layout      - specify the multi-panel layout (rows,cols)
            
    [Reading files]
        reader              - access rpfits/sdfits files
            read            - read in integrations
            summary         - list info about all integrations

    [General]
        commands            - this command
        print               - print details about a variable
        list_scans          - list all scantables created bt the user
        del                 - delete the given variable from memory
        range               - create a list of values, e.g.
                              range(3) = [0,1,2], range(2,5) = [2,3,4]
        help                - print help for one of the listed functions
        execfile            - execute an asap script, e.g. execfile('myscript')
        list_rcparameters   - print out a list of possible values to be
                              put into .asaprc file
        mask_and,mask_or,
        mask_not            - boolean operations on masks created with
                              scantable.create_mask
        
    Note:
        How to use this with help:
                                         # function 'summary'
        [xxx] is just a category
        Every 'sub-level' in this list should be replaces by a '.' Period when
        using help 
        Example:
            ASAP> help scantable # to get info on ths scantable
            ASAP> help scantable.summary # to get help on the scantable's
            ASAP> help average_time

\end{verbatim}

\section{Scantable Mathematics}

It is possible to to simple mathematics directly on scantables from
the command line using the \cmd{+, -, *, /} operators as well as their
cousins \cmd{+=, -= *=, /=}. This works between two scantables or a
scantable and a float. (Note that it does not work for integers).

\begin{verbatim}
  ASAP> sum = scan1+scan2
  ASAP> scan2 = scan1+2.0
  ASAP> scan *= 1.05
\end{verbatim}

%\section{Scripting}

%Malte to add something

\section{Appendix}

\subsection{Installation}

ASAP depends on a number of third-party libraries which you must
have installed before attempting to build ASAP. These are:

\begin{itemize}
\item AIPS++
\item Boost
\item Matplotlib
\item ipython/python
\end{itemize}

Debian Linux is currently supported and we intend also
to support other popular Linux flavours, Solaris and Mac.

Of the dependencies, AIPS++ is the most complex to install.

\subsection{ASCII output format}

\subsection{.asaprc settings}

\end{document}