source: trunk/docs/outputs.tex @ 964

% -----------------------------------------------------------------------
% outputs.tex: Section detailing the different forms of text- and
%              plot-based output.
% -----------------------------------------------------------------------
% Copyright (C) 2006, Matthew Whiting, ATNF
%
% This program is free software; you can redistribute it and/or modify it
% under the terms of the GNU General Public License as published by the
% Free Software Foundation; either version 2 of the License, or (at your
% option) any later version.
%
% Duchamp is distributed in the hope that it will be useful, but WITHOUT
% ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
% FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
% for more details.
%
% You should have received a copy of the GNU General Public License
% along with Duchamp; if not, write to the Free Software Foundation,
% Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA
%
% Correspondence concerning Duchamp may be directed to:
%    Internet email: Matthew.Whiting [at] atnf.csiro.au
%    Postal address: Dr. Matthew Whiting
%                    Australia Telescope National Facility, CSIRO
%                    PO Box 76
%                    Epping NSW 1710
%                    AUSTRALIA
% -----------------------------------------------------------------------
\secA{Outputs}
\label{sec-output}

\secB{During execution}

\duchamp provides the user with feedback whilst it is running, to
keep the user informed on the progress of the analysis. Most of this
consists of self-explanatory messages about the particular stage the
program is up to. The relevant parameters are printed to the screen at
the start (once the file has been successfully read in), so the user
is able to make a quick check that the setup is correct (see
Appendix~\ref{app-input} for an example).

The extent of memory allocation made at the start is indicated. This
will include the arrays needed for the pixel array, the reconstruction
or smoothed array, and the 2D detection map, but \emph{not} additional
space needed for working within individual algorithms, nor storage
needed for the detected objects.

\duchamp will report the amount of memory that is allocated when the
image is read in. This includes the storage for the array as well as
additional storage for the reconstructed/smoothed array and/or the
baseline arrays (if these are needed).

If the cube is being trimmed (\S\ref{sec-modify}), the resulting
dimensions are printed to indicate how much has been trimmed. If a
reconstruction is being done, a continually updating message shows
either the current iteration and scale, compared to the maximum scale
(when \texttt{reconDim = 3}), or a progress bar showing the amount of
the cube that has been reconstructed (for smaller values of
\texttt{reconDim}).

During the searching algorithms, the progress through the search is
shown. When completed, the number of objects found is reported (this
is the total number found, before any merging is done).

In the merging process (where multiple detections of the same object
are combined -- see \S\ref{sec-merger}), two stages of output
occur. The first is when each object in the list is compared with all
others. The output shows two numbers: the first being how far through
the list the current object is, and the second being the length of the
list. As the algorithm proceeds, the first number should increase and
the second should decrease (as objects are combined). When the numbers
meet, the whole list has been compared. If the objects are being
grown, a similar output is shown, indicating the progress through the
list. In the rejection stage, in which objects not meeting the minimum
pixels/channels requirements are removed, the total number of objects
remaining in the list is shown, which should steadily decrease with
each rejection until all have been examined. Note that these steps can
be very quick for small numbers of detections.

Since this continual printing to screen has some overhead of time and
CPU involved, the user can elect to not print this information by
setting the parameter \texttt{verbose = false}. In this case, the user
is still informed as to the steps being undertaken, but the details of
the progress are not shown.

There may also be Warning or Error messages printed to screen. The
Warning messages occur when something happens that is unexpected (for
instance, a desired keyword is not present in the FITS header), but
not detrimental to the execution. An Error message is something more
serious, and indicates some part of the program was not able to
complete its task. This is not necessary fatal, but it may mean the
full functionality requested will not be achieved. The message will
also indicate which function or subroutine generated it -- this is
primarily a tool for debugging, but can be useful in determining what
went wrong.

\secB{Text-based output files}

\secC{Table of results}
\label{sec-results}

Finally, we get to the results -- the reason for running \duchamp in
the first place. Once the detection list is finalised, it is sorted
according to the value of the \texttt{sortingParam}. This can take the
value ``xvalue'', ``yvalue'', ``zvalue'', ``ra'', ``dec'', ``vel'',
``w50'', ``iflux'' (for integrated flux), or ``pflux'' (for peak
flux), or ``snr''. The default value is ``vel'' (which means the
spectral WCS value -- this could be frequency or wavelength depending
on the nature of the FITS file). If no good WCS exists, the mean pixel
position equivalent is used (``ra'' is replaced by ``xvalue'', ``dec''
by ``yvalue'', ``vel'' and ``w50'' by ``zvalue''). The sense of the
sorting will be increasing value with position in the list. To sort in
the opposite sense, prepend the parameter name with a '-' (\eg
``-vel'' instead of ``vel''). The results are then printed to the
screen and to the output file, given by the \texttt{OutFile}
parameter.

The output consists of two sections. First, a list of the parameters
are printed to the output file, for future reference. Next, the
detection threshold that was used is given, so comparison can be made
with other searches. The statistics estimating the noise parameters
are given (see \S\ref{sec-stats}). Thirdly, the number of detections
are reported.

All this information, known as the ``header'', can either be written
to the start of the output file (denoted by the parameter
\texttt{OutFile}), or written to a separate file from the list of
detections. This second option is activated by the parameter
\texttt{flagSeparateHeader}, and the information is written to the
file given by \texttt{HeaderFile}.

The most interesting part, however, is the list of detected
objects. This list, an example of which can be seen in
Appendix~\ref{app-output}, contains the following columns (note that
the title of the columns depending on WCS information will depend on
the details of the WCS projection: they are shown below for the
Equatorial and Galactic projection cases).

\begin{Lentry}
\item[{Obj\#}] The ID number of the detection (simply the
  sequential count for the list, which is ordered by increasing
  velocity, or channel number, if the WCS is not good enough to find
  the velocity).
\item[{Name}] The ``IAU''-format name of the detection (derived from the
  WCS position -- see below for a description of the format).
\item[{X,Y,Z}] The ``centre'' pixel position, determined by the input
  parameter \texttt{pixelCentre}.
\item[{RA/GLON}] The Right Ascension or Galactic Longitude of the centre
  of the object.
\item[{DEC/GLAT}] The Declination or Galactic Latitude of the centre of
  the object.
\item[{VEL}] The mean spectral coordinate of the object [units given by the
  \texttt{spectralUnits} parameter]. The actual values here will
  depend on the WCS of the FITS file, and the name of the column will
  come from the CTYPE of the spectral axis (specifically, the
  4-character S-type code).
\item[{w\_RA/w\_GLON}] The width of the object in Right Ascension or
  Galactic Longitude (depending on FITS coordinates) [arcmin].
\item[{w\_DEC/w\_GLAT}] The width of the object in Declination Galactic
  Latitude [arcmin].
\item[{w\_50}] The spectral width of the detection at 50\% of the peak
  flux (the measured full-width at half-maximum, FWHM), in the
  appropriate spectral units [see note below].
\item[{F\_int}] The integrated flux over the object, in the units of
  flux times velocity, corrected for the beam if necessary. See below
  for a discussion on this correction.
\item[{F\_peak}] The peak flux over the object, in the units of flux.
\item[{S/Nmax}] The signal-to-noise ratio at the peak pixel.
\item[{X1, X2}] The minimum and maximum X-pixel coordinates.
\item[{Y1, Y2}] The minimum and maximum Y-pixel coordinates.
\item[{Z1, Z2}] The minimum and maximum Z-pixel coordinates.
\item[{Npix}] The number of voxels (\ie distinct $(x,y,z)$ coordinates)
  in the detection.
\item[{Flag}] Whether the detection has any warning flags (see below).
\end{Lentry}

These parameters are written to the screen during execution. There are
alternative ways of calculating the total flux, the position and
velocity width, however, and so additional parameters are written to
the output file:
\begin{Lentry}
\item[{w\_20}] The spectral width of the detection at 20\% of the peak
  flux, in spectral units [see note below].
\item[{w\_VEL}] The full spectral width of the detection (max channel
  $-$ min channel, in the appropriate spectral units).
\item[{F\_tot}] The sum of the flux values of all detected voxels.
\item[{X\_av, Y\_av, Z\_av}] The average pixel value in each
  axis direction \ie X\_av is the average of the $x$-values of all
  pixels in the detection.
\item[{X\_cent, Y\_cent, Z\_cent}] The centroid position, being
  the flux-weighted average of the pixels.
\item[{X\_peak, Y\_peak, Z\_peak}] The location of the pixel
  containing the peak flux value.
\end{Lentry}

The velocity width of the detection is calculated at 50\% and 20\% of
the peak flux, as well as the full detected width (if the detection
threshold is greater than 20\% or 50\% of the peak, then these values
will be the same as \texttt{w\_VEL}. The type of position value given
in the \texttt{X, Y, Z} columns in the screen output is determined by
the \texttt{pixelCentre} parameter. All three alternatives are shown
in the output file.

The user can specify the precision used to display the flux, velocity
and S/Nmax values, by using the input parameters \texttt{precFlux},
\texttt{precVel} and \texttt{precSNR} respectively. These values apply
to the tables written to the screen and to the output file, as well as
for the VOTable (see below).

The \texttt{Name} is derived from the WCS position. For instance, a
source that is centred on the RA,Dec position 12$^h$53$^m$45$^s$,
-36$^\circ$24$'$12$''$ will be given the name J125345$-$362412, if the
epoch is J2000, or the name B125345$-$362412 if it is B1950. The
precision of the RA and Dec values is determined by the pixel size,
such that sufficient precision is used to uniquely define a
position. The RA value will have one figure greater precision than
Dec. An alternative form is used for Galactic coordinates: a source
centred on the position ($l$,$b$) = (323.1245, 5.4567) will be called
G323.124$+$05.457. If the WCS is not valid (\ie is not present or does
not have all the necessary information), the \texttt{Name, RA, DEC,
  VEL} and related columns are not printed, but the pixel coordinates
are still provided.

The velocity units can be specified by the user, using the parameter
\texttt{spectralUnits} (enter it as a single string with no
spaces). The default value comes from the FITS header. These units are
also used to give the units of integrated flux. 

When the cube brightness units are quoted per beam (\eg Jy/beam), then
the integrated flux \texttt{F\_int} includes a correction for
this. This involves dividing by the integral over the beam. This is
calculated using the BMAJ, BMIN \& BPA header keywords from the FITS
file. BMAJ and BMIN are assumed to be the full-width at half maximum
(FWHM) in the major and minor axis directions of a Gaussian beam. The
integral is calculated as follows: the functional form of a 2D
elliptical Gaussian can be written as
$\exp(-((x^2/2\sigma_x^2)+(y^2/2\sigma_y^2)))$, and the FWHM in a
given direction is then $f=2\sqrt{2\ln2}\sigma$. Then, 
\[
\int
\exp\left(-\left(\frac{x^2}{2\sigma_x^2}+\frac{y^2}{2\sigma_y^2}\right)\right)
= 2\pi\sigma_x\sigma_y
=\frac{\pi f_x f_y}{4\ln2}
\]
provides the correction factor to go from units of Jy/beam to Jy. 

If the FITS file does not have the beam information, the user can
either:
\begin{enumerate}
\item Specify the FWHM of the beam in pixels (assuming a circular
  beam) via the parameter \texttt{beamFWHM}.
\item Specify the area of the beam, again in pixels, via the parameter
  \texttt{beamArea}\footnote{Note that this is equivalent to the old
    parameter \texttt{beamSize}, which was highlighted as being
    ambiguous.}. This should be the value given by the equation above.
\end{enumerate}
If both are given, \texttt{beamFWHM} takes precendence. If neither are
given, and there is no beam information in the header, then no
correction to the integrated flux is made (and so it will stay in
units of Jy/beam or equivalent).

If the WCS is absent or not sufficiently specified, then all columns
from \texttt{RA/GLON} to \texttt{w\_VEL} will be left blank. Also,
\texttt{F\_int} will be replaced with the more simple \texttt{F\_tot}.

The \texttt{Flag} column contains any warning flags, such as:
\begin{itemize}
\item \textbf{E} -- The detection is next to the spatial edge of the image,
meaning either the limit of the pixels, or the limit of the non-BLANK
pixel region.
\item \textbf{S} -- The detection lies at the edge of the spectral region.
\item \textbf{M} -- The detection is adjacent to, or overlaps the
  ``Milky Way'' range of channels (see Sec.~\ref{sec-MW}).
\item \textbf{N} -- The total flux, summed over all the (non-BLANK)
pixels in the smallest box that completely encloses the detection, is
negative. Note that this sum is likely to include non-detected
pixels. It is of use in pointing out detections that lie next to
strongly negative pixels, such as might arise due to interference --
the detected pixels might then also be due to the interference, so
caution is advised.
\end{itemize}
In the absence of any of these flags, a \textbf{-} will be recorded in
this column.

\secC{Other results lists}

Three additional results files can also be requested. One option is a
VOTable-format XML file, containing just the RA, Dec, Velocity and the
corresponding widths of the detections, as well as the fluxes. The
user should set \texttt{flagVOT = true}, and put the desired filename
in the parameter \texttt{votFile} -- note that the default is for it
not to be produced. This file should be compatible with all Virtual
Observatory tools (such as Aladin%
\footnote{%Aladin can be found on the web at
\href{http://aladin.u-strasbg.fr/}{http://aladin.u-strasbg.fr/}}
or TOPCAT\footnote{%Tool for OPerations on Catalogues And Tables:
\href{http://www.star.bristol.ac.uk/~mbt/topcat/}%
{http://www.star.bristol.ac.uk/~mbt/topcat/}}). 

A second option is an annotation file for use with the Karma toolkit
of visualisation tools (in particular, with \texttt{kvis}). There are
two options on how objects are represented, governed by the
\texttt{annotationType} parameter. Setting this to \texttt{borders}
results in a border being drawn around the spatial pixels of the
object, in a manner similar to that seen in Fig.~\ref{fig-spect}. Note
that Karma/\texttt{kvis} does not always do this perfectly, so the
lines may not be directly lined up with pixel borders. The other
option is \texttt{annotationType = circles}. This will draw a circle
at the position of each detection, scaled by the spatial size of the
detection, and number it according to the Obj\# given above. To make
use of this option, the user should set \texttt{flagKarma = true}, and
put the desired filename in the parameter \texttt{karmaFile} -- again,
the default is for it not to be produced.

The final optional results file produced is a simple text file that
contains the spectra for each detected object. The format of the file
is as follows: the first column has the spectral coordinate, over the
full range of values; the remaining columns represent the flux values
for each object at the corresponding spectral position. The flux value
used is the same as that plotted in the spectral plot detailed below,
and governed by the \texttt{spectralMethod} parameter. An example of
what a spectral text file might look like is given below:

\begin{quote}
  {\footnotesize
    \begin{tabular}{lllll}
      1405.00219727  &0.01323344  &0.23648241  &0.04202826  &-0.00506790  \\
      1405.06469727  &0.01302835  &0.27393046  &0.04686056  &-0.00471084  \\
      1405.12719727  &0.01583361  &0.27760920  &0.04114933  &-0.01168737  \\
      1405.18969727  &0.01271889  &0.31489247  &0.03307962  &-0.00300790  \\
      1405.25219727  &0.01597644  &0.30401203  &0.05356426  &-0.00551653  \\
      1405.31469727  &0.00773827  &0.30031312  &0.04074831  &-0.00570147  \\
      1405.37719727  &0.00738304  &0.27921870  &0.05272378  &-0.00504959  \\
      1405.43969727  &0.01353923  &0.26132512  &0.03667958  &-0.00151006  \\  
      1405.50219727  &0.01119724  &0.28987029  &0.03497849  &-0.00645589  \\  
      1405.56469727  &0.00813379  &0.29839963  &0.04711142  &0.00536576   \\  
      1405.62719727  &0.00774377  &0.26530230  &0.04620502  &0.00724631   \\  
      1405.68969727  &0.00576067  &0.23215000  &0.04995513  &0.00290841   \\ 
      1405.75219727  &0.00452834  &0.16484940  &0.04261605  &-0.00612812  \\  
      1405.81469727  &0.01406293  &0.15989439  &0.03817926  &-0.00758385  \\ 
      1405.87719727  &0.01116611  &0.11890682  &0.05499069  &-0.00626362  \\  
      1405.93969727  &0.00687582  &0.10620256  &0.04743370  &0.00055177   \\
      $\vdots$       &$\vdots$    &$\vdots$    &$\vdots$    &$\vdots$     \\
    \end{tabular}
  }
\end{quote}

In addition to these three files, a log file can also be produced. As
the program is running, it also (optionally) records the detections
made in each individual spectrum or channel (see \S\ref{sec-detection}
for details on this process). This is recorded in the file given by
the parameter \texttt{LogFile}. This file does not include the columns
\texttt{Name, RA, DEC, w\_RA, w\_DEC, VEL, w\_VEL}. This file is
designed primarily for diagnostic purposes: \eg to see if a given set
of pixels is detected in, say, one channel image, but does not survive
the merging process. The list of pixels (and their fluxes) in the
final detection list are also printed to this file, again for
diagnostic purposes. The file also records the execution time, as well
as the command-line statement used to run \duchamp. The creation of
this log file can be prevented by setting \texttt{flagLog = false}
(which is the default).

\secB{Graphical output}

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=\textwidth]{example_spectrum}
  \end{center}
  \caption{\footnotesize An example of the spectral output. Note several
    of the features discussed in the text: the red lines indicating the
    reconstructed spectrum; the blue dashed lines indicating the
    spectral extent of the detection; the green hashed area indicating
    the Milky Way channels that are ignored by the searching algorithm;
    the blue border showing its spatial extent on the 0th moment map;
    and the 15~arcmin-long scale bar.}
  \label{fig-spect}
\end{figure}

\begin{figure}[!t]
  \begin{center}
    \includegraphics[width=\textwidth]{example_moment_map}
  \end{center}
  \caption{\footnotesize An example of the moment map created by
    \duchamp. The full extent of the cube is covered, and the 0th moment
    of each object is shown (integrated individually over all the
    detected channels). The purple line indicates the limit of the
    non-BLANK pixels.}
  \label{fig-moment}
\end{figure}

\secC{Mask image}
\label{sec-maskOut}

It is possible to create a FITS file containing a mask array. This
array is designed to indicate the location of detected objects. The
value of the detected pixels is determined by the
\texttt{flagMaskWithObjectNum} parameter: if \texttt{true}, the value
of the pixels is given by the corresponding object ID number; if
\texttt{false}, they take the value 1 for all objects. Pixels not in a
detected object have the value 0. To create this FITS file, set the
input parameter \texttt{flagOutputMask=true}. The file will be named
according to the \texttt{fileOutputMask} parameter, or, if this is not
given, \texttt{image.MASK.fits} (where the input image is called
\texttt{image.fits}).

\secC{Spectral plots}

As well as the output data file, a postscript file (with the filename
given by the \texttt{spectralFile} parameter) is created that shows
the spectrum for each detection, together with a small cutout image
(the 0th moment) and basic information about the detection (note that
any flags are printed after the name of the detection, in the format
\texttt{[E]}). If the cube was reconstructed, the spectrum from the
reconstruction is shown in red, over the top of the original
spectrum. The spectral extent of the detected object is indicated by
two dashed blue lines, and the region covered by the ``Milky Way''
channels is shown by a green hashed box. An example detection can be
seen in Fig.~\ref{fig-spect}.

The spectrum that is plotted is governed by the
\texttt{spectralMethod} parameter. It can be either \texttt{peak} (the
default), where the spectrum is from the spatial pixel containing the
detection's peak flux; or \texttt{sum}, where the spectrum is summed
over all spatial pixels, and then corrected for the beam size.  The
spectral extent of the detection is indicated with blue lines, and a
zoom is shown in a separate window.

The cutout image can optionally include a border around the spatial
pixels that are in the detection (turned on and off by the
\texttt{drawBorders} parameter -- the default is \texttt{true}). It
includes a scale bar in the bottom left corner to indicate size -- its
length is indicated next to it (the choice of length depends on the
size of the image).

There may also be one or two extra lines on the image. A yellow line
shows the limits of the cube's spatial region: when this is shown, the
detected object will lie close to the edge, and the image box will
extend outside the region covered by the data. A purple line, however,
shows the dividing line between BLANK and non-BLANK pixels. The BLANK
pixels will always be shown in black. The first type of line is always
drawn, while the second is governed by the parameter
\texttt{drawBlankEdges} (whose default is \texttt{true}), and
obviously whether there are any BLANK pixel present.

Note that the creation of the spectral plots can be prevented by
setting \texttt{flagPlotSpectra = false}.

\secC{Output for 2-dimensional images}

When the input image is two-dimensional, with no spectral dimension,
this spectral plot would not make much sense. Instead, \duchamp
creates a similar postscript file that simply includes the text
headers as well as the 0th-moment map of the detection. As for the
normal spectral case, this file will be written to the filename given
by the \texttt{spectralFile} parameter.

\secC{Spatial maps}

Finally, a couple of images are optionally produced: a 0th moment map
of the cube, combining just the detected channels in each object,
showing the integrated flux in grey-scale; and a ``detection image'',
a grey-scale image where the pixel values are the number of channels
that spatial pixel is detected in. In both cases, if
\texttt{drawBorders = true}, a border is drawn around the spatial
extent of each detection, and if \texttt{drawBlankEdges = true}, the
purple line dividing BLANK and non-BLANK pixels (as described above)
is drawn. An example moment map is shown in Fig.~\ref{fig-moment}.
The production or otherwise of these images is governed by the
\texttt{flagMaps} parameter.

The moment map is also displayed in a PGPlot XWindow (with the
\texttt{/xs} display option). This feature can be turned off by
setting \texttt{flagXOutput = false} -- this might be useful if
running \duchamp on a terminal with no window display capability, or
if you have set up a script to run it in a batch mode.

The moment map can also be written to a FITS file, so that it can be
examined more closely, and have annotation files overlaid. This works
in the same way as for the mask image. To create the FITS file, set the
input parameter \texttt{flagOutputMomentMap=true}. The file will be named
according to the \texttt{fileOutputMomentMap} parameter, or, if this is not
given, \texttt{image.MOM0.fits} (where the input image is called
\texttt{image.fits}).

The purpose of these images are to provide a visual guide to where the
detections have been made, and, particularly in the case of the moment
map, to provide an indication of the strength of the source. In both
cases, the detections are numbered (in the same sense as the output
list and as the spectral plots), and the spatial borders are marked
out as for the cutout images in the spectra file. Both these images
are saved as postscript files (given by the parameters
\texttt{momentMap} and \texttt{detectionMap} respectively), with the
latter also displayed in a \textsc{pgplot} window (regardless of the
state of \texttt{flagMaps}).



\secB{Re-using previous detections}
\label{sec-reuse}

It may be the case that, once you have run \duchamp with a set of
parameters, you are unsatisfied with the output spectra -- perhaps you
would have preferred integrated rather than peak flux to be
plotted. However, the searching might have taken a while to run, and
the thought of doing it again just for different plots may be a bit
off-putting.

Well, provided you have made a log file when running \duchamp (with
the \texttt{flagLog=true} setting), it is possible to do this easily
without having to go through the process of detecting your sources a
second time. By using the same input file, with the additional
parameter \texttt{usePrevious=true}, the log file that was created
with a previous \duchamp run can be read to extract each of the
individual detections. The output stage is then run again, with the
parameters (in particular \texttt{pixelCentre} and
\texttt{spectralMethod}) as given in the input file. 

Perhaps you would also like to extract a single source's
spectral plot (\eg for use in a journal paper). The use-previous
method allows you to specify particular sources to re-plot. Only these
sources will be plotted in the \texttt{SpectraFile} file, and
individual files will be created for each of the listed sources. Their
filenames will follow the format of \texttt{SpectraFile}: if,
\texttt{SpectraFile=file.ps}, source \#3 will appear in
\texttt{file-03.ps}. To give a list of sources, use the
\texttt{objectList} parameter, and provide a string with individual
object numbers or object ranges: \eg 1,2,4-7,8,11.


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "Guide"
%%% End: