source: trunk/CHANGES @ 1249

Duchamp version 1.5 (2013/??/??)
--------------------------------

Input parameter changes:

* There is now a more flexible way to specify channels that should be
  ignored by the source-finding. Use flaggedChannels to provide a list
  of single channels or channel ranges that should be ignored. This
  replaces the flagMW/minMW/maxMW parameters, extending their
  functionality and allowing more than one set of channels to be
  specified. 


Code changes:

* The plotting code has been refactored, improving its class structure
  and inheritance. No functional change from this should be evident,
  but it improves the maintainability of the code.


Duchamp version 1.4 (2013/06/20)
--------------------------------

Installation:

* The "make lib" command will now create a shared library
  libduchamp.1.4.so as well as the static library libduchamp.1.4.a
  (formerly just called libduchamp.a).

Bug Fixes:

* The orientation of ellipses in the DS9 region files are now correct
  (they were wrong by 90 degrees).

* Corrected the output FITS filenames so that they work when the input
  filename does *not* end in ".fits".

Code changes:

* The code that manages the output columns for the ASCII and VOTable
  output has been overhauled and streamlined. Little functional change
  should be evident, but it should be much more robust. And easier to
  use in derived software.

Duchamp version 1.3.2 (2013/05/02)
----------------------------------

Bug fixes:

* The units for the integrated flux (for F_int) were not being
  calculated for cases of a FITS file without a spectral
  axis. Additionally, a vestigial default value meant this column had
  units of "counts". This has been fixed so that a FITS file with
  Jy/beam units will have F_int values in Jy.

* The ellipse fitting to find the MAJ/MIN/PA was not working correctly
  in the case of negative searches. This has been fixed.

* The scaling of MAJ/MIN values was not being done correctly in all
  cases, so that the values were out by a factor of 60 or 3600. This
  was particularly the case when the ellipse fitting failed (due to
  small numbers of pixels, for instance).

* The principle axes calculations have been simplified slightly, and
  now are done only on pixels above half the peak in the moment-0 map
  (to estimate the FWHM).

Duchamp version 1.3.1 (2013/04/18)
----------------------------------

Bug fixes:

* The kernel used in the 2D smoothing was not normalised properly,
  which meant the fluxes in the smoothed array were scaled
  incorrectly. The kernel is now normalised, so that the fluxes in the
  smoothed array decrease (and to an increasing degree with larger
  kernels, as expected).

* The way the position angle of the smoothing kernel was interpreted
  was wrong. It now conforms to the typical astronomical position
  angle definition of degrees east of north.

* The Z_PEAK column was being left out of the results file for
  Duchamp-1.3. It has been put back in.

* The calculation of S/Nmax has changed so that when smoothing or
  reconstruction is done, the peak flux used is measured in the
  reconstructed/smoothed array (where the noise is measured).

* The flux errors are now not reported in the case of smoothing, since
  we don't measure the noise in the original input image array.

Other fixes:

* The vertical range of the spectral plots now properly takes into
  account all spectra (not just the input array), and quirks with a MW
  range lying outside the plotted range have been fixed.

* The flux scaling for the moment-0 map plot now ignores negative
  values, meaning the scale should be sensible.

* The spelling of the Object2D::getPrincipalAxes() has been corrected.

* Minor code optimisations have been made to the smoothing code.

* The subsection class is now initialised with a blank string.

Duchamp version 1.3 (2013/03/08)
--------------------------------

New features:

* New source parameters:

  # Each source now has a measurement of the major & minor axis sizes,
    along with the position angle of the major axis. The sizes are the
    full-width at half maximum, measured from the moment-0 map. These
    now appear in the catalogue and in the spectral plots, with the
    ellipse drawn on the moment map cutout in the spectral plots. The
    old measurements of w_RA & w_DEC are still made and included in
    the full catalogue, but are no longer reported in the smaller
    catalogue printed to screen (nor in the XML/VOTable output).

  # Each source also now has a measurement of the error on the
    integrated and total fluxes. This is calculated from summing in
    quadrature the statistical noise on each detected voxel. This is
    from a request by Sara Shakouri.

* Output files - new types and changes to the way they are written:

  # Ability to create CASA region format files showing the location of
    the detected sources. This is analogous to the Karma annotation
    files or DS9 region files, for use with the casa viewer, but can
    also be imported into casapy.

  # Ability to save baseline values to a FITS file (in the same way as
    for the reconstructed or smoothed array).

  # Ability to save the moment-0 mask to a FITS file (that is, a mask
    indicating which spatial pixels form part of the moment-0 map).

  # There is an improved way to store the detected pixels for later
    re-use. While previously the detected pixels were written as scans
    to the logfile, and then read in when the usePrevious flag was
    set, we now write the scans and other metadata to a binary
    catalogue that can be read in again. Writing as a binary file
    leads to much better compression of the data, particularly when
    combined with the run-length encoding. It is anticipated future
    tools will be provided to allow additional interaction with these
    binary catalogues.

  # There are slight changes to the format of the ASCII catalogues,
    where lines not containing sources are prefixed with a comment
    character '#'. This will facilitate their use in other software.

* There is a new command-line parameter, -t THRESH, which specifies
  the flux threshold for searching. It is designed to be used with the
  -f option, as a quick way of searching a FITS image to a specified
  flux level.

Bug fixes:

* A bug in the spatial smoothing has been fixed. The mask was being
  interpreted incorrectly, so that no valid pixels were getting
  smoothed. 

* The WCS for the moment-0 FITS cube has been fixed (specifically, the
  axis names).

* When usePrevious was used, the behaviour in previous versions was to
  always write individual spectra. This is now controlled by a new
  parameter flagPlotIndividualSpectra.

* Minor improvements to the output when verbose=false.

* Improvements to the Makefile, so that header files from a
  previously-installed version of Duchamp do not get used instead of
  those from the current version.

* The vertical range for the spectral plots now take into account the
  baseline and reconstructed/smoothed arrays as well (outside of any
  MW range, of course).

Other

* The code-base for reading from and writing to FITS files has been
  overhauled and streamlined. It can now be found in the FitsIO
  directory. The interface for the writing of FITS files via the Cube
  class has not changed.

* The User Guide has been slightly re-ordered to provide more explicit
  information about the source parameters.


Duchamp version 1.2.2 (2012/10/29)
----------------------------------

Bug fixes:

* The reading of reconstructed & smoothed arrays in the case of large
  (>2GB) cubes has been improved. While the most recent versions of
  cfitsio were able to handle these, earlier versions would result in
  a segementation fault using the previous interface. An alternative
  interface has been implemented that avoids these faults, even for
  the older cfitsio libraries.

* A bug affecting the writing of FITS mask files (apparently for the
  case of flagging by the object number) has been fixed.

* The moment map occasionally failed if sources lay spatially within
  the blank pixel region of the first channel. This check has been
  removed and the moment map should be produced successfully in all
  situations.

Other:

* The UCDs used in the VOTable catalogue output have been
  updated. Several of these were shared between different columns, and
  so they have been changed so that each UCD is unique to one
  column. The column ID strings have been updated for clarity.

* The determination of default output image filenames has been
  improved -- it previously assumed the input filename ended in
  '.fits', but this is no longer the case.

Duchamp version 1.2.1 (2012/10/09)
----------------------------------

New feature:

* Duchamp can now produce region files to annotate images displayed in
  SAOImage DS9. There are new parameters flagDS9 and ds9File that work
  in the same way as the Karma annotation file parameters. The region
  files can also be displayed in (at least some versions of)
  casaviewer. 

* There is a new script to fix occasional problems with the
  automatically-detected Makefile, and the verification script has
  been updated and improved.

Bug fixes:

* The main bug fix for this release corrects a problem with the
  velocity width measurements made when searching for negative
  detections. It was possible to get very small (and incorrect) widths
  in some cases - this was due to the integrated spectrum not being
  inverted correctly.

* In the case of smoothing, the stats of the full cube that were being
  reported in the output files were actually those of the un-smoothed
  (input cube). This only affected the output files, not the detection
  process itself.

* The units were not being reported in the VOTable. This has been
  fixed.

* The UCDs for the spectral parameters in the VOTable were not being
  adapted to the user requests, so didn't always match the types of
  the parameters. All should be consistent now.

Other changes:

* The codebase relating to the creation of and writing to output files
  (results, logs, annotation, VOTables) has been overhauled, with a
  cleaner interface. Most of this code is in the new subdirectory
  src/Outputs, to where the columns code has been moved as well.

  This was instigated by ASKAP/Selavy development, but also allowed
  the expansion of output formats to include DS9 region files. No
  change to the user should be visible, but anyone using the software
  will notice differences. Please get in touch if changes are
  difficult to follow.

Duchamp version 1.2 (2012/08/08)
--------------------------------

Key changes:

* The way the wavelet reconstruction is done in the presence of BLANK
  voxels is slightly different, specifically when there were BLANK
  voxels on the edge. While BLANK voxels are always ignored by the
  reconstruction (and left in place), the previous approach dealt with
  BLANKS on the edge by moving the effective edge of the image inwards
  until the first non-blank voxel. This meant that the size of the
  dimensions used to determine the number of scales, and the location
  where boundary reflections took place where different to the input
  image size.

  However, this meant that BLANK voxels on the edge were treated
  differently to those not on the edge, so this inconsistency was
  removed. The full image dimensions are always used to calculate the
  number of scales, and reflections (which is how the reconstruction
  deals with the boundaries) will occur at the actual image edges.

  If your data has this property (BLANK pixels at the image
  boundaries), then any reconstruction will be affected (most likely
  only in a small way) and potentially any subsequent searching. You
  are advised to check this (for instance by saving the results of the
  resonstruction from 1.1.13 and 1.2 and comparing).

* The way the spectral world coordinate system is dealt with has been
  improved. Previously Duchamp tried to convert to a velocity axis,
  and got a bit grumpy if that didn't work well. Now it will express
  the spectral units in the axis type indicated by the FITS
  header. Duchamp will also allow you to change this type by providing
  an appropriate WCS type as an input parameter. You may also provide
  a rest frequency in the parameter file and this will be used to
  calculate velocities. And, as before, you can provide specific units
  for the spectral quantities to be converted into.

* Since v1.1.13 was released, the journal paper describing Duchamp has
  been published. It can be found at MNRAS 421, 3242 (2012), or
  http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2966.2012.20548.x/full. It
  includes descriptions of the various algorithms used for Duchamp, as
  well as some indicative comparisons of the relative completeness &
  reliability. Please cite this paper when describing Duchamp.

Other new features:

* The default value of snrCut has changed from 3 to 5. This is
  espeically important if you use the -f option, but also if you don't
  provide snrCut in your parameter file.

* The XML VOTable output now includes a more complete list of
  parameters, matching the output given in text form in the results &
  log output files. Only those parameters affecting the results are
  included - things like the output file names and so on are left out.

* The minVoxels parameter is now adjusted to be at least (minPix +
  minChannels - 1). If it is changed, a warning message is triggered.

* The warning/error reporting has been overhauled, with an improved
  interface for developers. Also, some ERROR instances that previously
  resulted in the program terminating now allow it to continue
  (provided that is possible given the error).

* It is now possible to give the convergence criterion for the wavelet
  reconstruction via the parameter reconConvergence. When the relative
  change in standard deviation of the residuals from the
  reconstruction falls below this value, the a trous algorithm stops
  iterating.

* The way one-dimensional arrays were dealt with was unsatisfactory,
  both with the way positions were registered and the format of the
  graphical output. Improvements have been made such that a 1D data
  structure is assumed to be a spectrum, and so positions in the
  spectrum are reported as z-pixels (or the equivalent world
  coordinate). Instead of a 2D moment map, the X-windows display will
  show the spectrum with all detected pixels plus the final list of
  detections highlighted, along with the detection and growth
  thresholds. This plot will also be written to the "detectionMap"
  image.

* All spectral plots will now show the detection (and growth, if
  applicable) thresholds when plotting the peak flux spectrum.

* Accessing FITS files when a subsection string is provided has been
  streamlined. The cfitsio package, which handles the FITS access,
  creates a temporary file containing this image section, and this
  meant the previous approach was slow, particularly for large
  files/sections. Now the file is only opened once and the overhead of
  creating the temporary file is only encountered once.

* Several aspects of the processing & output did not handle a
  non-existent WCS well. This has been improved, so that all stages of
  execution will proceed without errors.

Bug fixes:

* The absence of a BLANK keyword when trimming will now not trigger a
  failure, only a warning message.

* The flux unit conversion (eg. from Jy/beam to mJy/beam) was not
  being applied correctly when an existing reconstructed or smoothed
  array was being read in. This is now applied properly.

* When saving reconstructed or smoothed arrays, the data type is now
  set to FLOAT, so that the correct values are written (and
  subsequently read in). The previous version was potentially saving
  to a FITS file with integer data type (this depended on the type of
  the input array). In this case, the full precision of data values
  would not be preserved, so reading back in and doing an identical
  search could yield different sources.

* For certain cases, the reconstruction attempted to use a very large
  number of scales. This has been fixed.

* The maximum scale for the reconstruction was not always being
  applied. The scale ranges are now enforced for all types of
  reconstructions.

* When saving a mask image while using a subsection of a cube, the
  offsets for the subsection were not being accounted for
  correctly. This is now fixed.

* The detection map plots were not recording pixels added from the
  growing stage. This has been fixed.

* The RA & Dec now have appropriate precision, rather than an
  arbitrary 2 decimal places on the seconds value. The RA will be one
  decimal place more than the Dec, and they will reflect the degree of
  precision in the image pixels.

* A number of data type conversions (typically ints to floats) led to
  errors in some calculations, such as WCS conversions. Care has been
  taken to make sure all indexing is done appropriately with no
  overflow, and that all conversions (where necessary) are done
  appropriately.

* It was possible to get world positions looking like 11:59:60.0! Not
  any more.

* Reading previous detections from the log file (via the use of the
  usePrevious parameter) did result in error messages due to slightly
  incompatible formats. These have been fixed, and all graphical
  output should work properly.

* When using a spectral search, having a blank pixel in the first
  channel could lead to that entire spectrum being ignored, leading to
  inconsistent results between spectral and spatial searches. This
  issue has been fixed, with both types of searches returning the same
  catalogue.


Code base development notes:

* Most instances of integer overflow due to insufficient precision
  have been tracked down - many of these were indexing variables of
  type int, which have been changed to size_t.

* Importantly, for code interfacing with the duchamp::Cube and related
  classes, axisDim is now an array of size_t elements, rather than
  long.

* Object3D now carries its spatial map with it, rather than
  calculating it on the fly when requested. This saves a lot of
  unnecessary computational effort.

* The warning/error interface has improved, making use of C++
  exception handling. This should be compatible with external logging
  infrastructure (such as log4cxx).

* The interface to the FitsHeader & WCS structures for external code
  has been improved, so that the WCS can be set up via function calls.



Duchamp version 1.1.13 (2011/04/10)
-------------------------------------

Bug fixes:

* The median statistics calculations had a bug in the test for whether
  the array under consideration was of even size. The correction for
  this can mean small differences in the resulting values. This can
  flow through to calculated detection thresholds - please check this
  with your data if you use this mode!

* The reconstruction code had a bug where the incorrect number of
  scales was being used. This occurred when the blank pixel flag was
  set, and when the spectral dimension was smaller than both the
  spatial dimensions.

Other:

* The default value of the blank pixel flag has been set to false,
  rather than true. For Duchamp, this should not matter, as it is set
  when the FITS file is read in. For external code using, for
  instance, the reconstruction algorithms, this default may be
  important. Having it true was contributing to the second bug listed
  above. It also makes more sense for it to be false unless explicitly
  set in some way.

* The reconstruction code has been streamlined a little, reducing its
  memory impact and using more appropriate variable types
  (eg. unsigned ints instead of ints and so forth).

* For those who care about the actual code base, the statistics
  functions have been expanded, and separated from getStats.cc into
  "normal" and "robust" versions.


Duchamp version 1.1.12 (2011/03/18)
-------------------------------------

Bug fixes:

* The velocity width calculations had a bug that occasionally led to
  very large values. The widths are calculated using the integrated
  spectrum, using a threshold relative to its peak. In v.1.1.10 and
  1.1.11, however, the peak was searched for over the entire spectral
  range, rather than just the range for the current object. This led
  to spurious values. This has been fixed, so that the peak is only
  searched for within the detected ranage of the object.

* Both the pixel and statistics subsections are now more carefully
  examined, with errors reported if there are no valid pixels in the
  pixel subsection, or if the statistics subsection does not intersect
  with the pixel subsection. 

Other:

* The configure script has been slightly improved so that the pgsbox
  library is only added if PGPlot has been enabled - this is
  particularly necessary with the slightly changed build procedure for
  recent wcslib versions. 

* In a related note, one file - momentMap.cc - had some #include
  statements that failed when PGPlot (and hence pgsbox in WCSLIB) was
  not enabled. These were actually unnecesssary and have been removed.

* There is a new input parameter - flagPlotSpectra - that allows the
  user to turn off the production of the spectral plots. It defaults
  to true (meaning the plots are produced).


Duchamp version 1.1.11 (2011/03/01)
-------------------------------------

Improvements:

* The largest changes in this release involved improving the
  efficiency of the code, resulting in further speed-ups. The code
  affected was the low-level object-handling code, with the merging
  stages in particular showing good speed-ups.

* The default value for the beam size, when the beam parameters are
  not in the FITS header, has been dropped. Now the default behaviour
  is to simply not make any beam corrections if the beam is not
  specified by either the FITS file or the input parameters. Thanks to
  Tobias Westemeier for the suggestion. 

* The reporting of the beam size has changed slightly to provide a bit
  more information.

Bug Fixes:

* The statistics section (statSec) could be larger than the image
  subsection (if provided), yet the pixels required would never be
  read in from the FITS file. The behaviour has been changed so that
  the statSec, if requested, is trimmed to be wholly within the image
  section.

* It was possible to request, for instance, spectral smoothing on a 2D
  image. This resulted in a blank smoothed array, which led to
  problems in the calculations of statistics (and hence the
  threshold), as well as subsequent calculations. This has been fixed,
  so that in cases like this the smoothing is turned off (and a
  warning message is displayed).

* There was a problem with using a MW range and subsectioning when
  growing objects, as a result of changes in 1.1.10. This has been
  fixed.

* Inversion was failing when no array was allocated (not a problem for
  standard Duchamp, but affected its use as a library by other code).

* Writing text spectra would only occur when PGPlot output was enabled
  in the build (it was combined with the plotting of the spectral
  graphs). This has been changed so that this output can be requested
  without requiring PGPlot.

* The equation for the beam size in Sec.4.2 of the User Guide was
  incorrect and has been fixed.

* Minor change to the configure script to improve the detection of the
  PGPlot library.

Code development notes:

* The main work has been additions and improvements to the interfaces
  for the Param and the Section classes, as well as new classes to
  manage the beam information. The efficiency changes mentioned above
  also affected the internals of some of the PixelInfo classes, and
  provided additional functions for judging the closeness of
  neighbouring objects.

Duchamp version 1.1.10 (2010/11/15)
-------------------------------------

Improvements:

* The major change for this release is a greatly improved growing
  algorithm. The previous incarnation was very inefficient (testing
  all pixels in an object, most of which only have neighbours that are
  already in the object), and could take an unnecessarily large amount
  of time in the case of extended sources. The improved algorithm
  makes use of voxel flags to quickly identify those voxels not yet in
  an object and only examines them. For a certain test case
  (observations of extended Galactic HI), the time required reduced
  from about a day to about a minute! (Note that compact sources, such
  as HI galaxies, will not show this degree of speedup...) Many thanks
  to Tara Murphy for pointing out this bug and her help in
  diagnosing/fixing the problem.

* There have been a number of further improvements to the code to
  speed it up, based on some profiling. The most important was the
  function to add pixels to a 2D object, but there were several other
  minor changes as well.

* The minimum size of the detections can now also be specified in
  terms of the total number of voxels (in addition or instead of
  specifying the minimum spatial pixels or channels).

* The way memory is allocated when reading the image has been
  streamlined. The amount of memory being allocated is also now
  reported - this is in an effort to allow users to identify problems
  with running out of memory.

* The version number is now reported in the results and log files.

* Only the filename of the input image (ie. not the leading
  directories) is now shown on the detection and moment-0 maps. This
  is to avoid problems with large directory paths being too wide for
  the plot.

* The verification script was improved for readability (although no
  functional change was made).

Bug fixes:

* Growing was not being done when a single source was found. Now
  fixed.

* For particular WCS setups, the RA can be interpreted as being
  negative, which corrupted the Source ID (ie. the IAU-format
  string). This has been fixed.

* When the flux units were changed, problems had been seen when
  writing a reconstructed cube to a FITS file. Any unit change is now
  undone prior to writing and there should now be no problem.

* Filenames with spaces in them were not handled well. New functions
  were included and moved to a more obvious location. These filenames
  now work as expected.

* Spectral axes of type FREQ that had a rest frequency were not being
  put into velocity units. Now fixed.

* Improvements to the output to screen, making sure everything is
  covered by the verbose flag.

* Better management of the allocation of arrays.

* The minSize parameter in Image was never initialised, leading to
  unpredictable behaviour. Now fixed.

Code development notes:

* New functionality includes: a function to return a vector list of
  voxels in an Object3D or Detection; a new generic Search function
  (basically a front-end to each of the individual functions); more
  complete copy constructors for the DataArray classes; new methods
  for finding the velocity widths.

Duchamp version 1.1.9 (2010/03/26)
-------------------------------------

Improvements: 

* There is now the ability to save the moment-0 map as a FITS
  file. This is controlled by the parameters flagOutputMomentMap and
  fileOutputMomentMap, which work in the same way for the mask file.

* There is a new parameter flagRejectBeforeMerge that, when true,
  rejects sources according to the minPix/minChannels requirements
  before the merging and growing is done. This defaults to false (so
  default behaviour is as before). Thanks to Tobias Westmeier for
  suggesting this and the above improvement.

* There is also a new parameter flagTwoStageMerging. When true (the
  default), newly-detected objects are merged with the first close
  object in the list. When false, they are simply added to the end of
  the list. In both cases, a second round of merging is done to pick
  up all close matches.

* The way the beam is specified in the absence of the BMAJ/BMIN/BPA
  parameters has been slightly changed. The use of "beamSize" was
  somewhat ambiguous (it meant the number of pixels covered by the
  beam, not the FWHM), so new parameters have been introduced. The
  beam can be specified by beamArea (which replaces beamSize), or
  beamFWHM (the FWHM of the beam in pixels). If beamFWHM is provided,
  it will be used, else the beamArea will be.

* There is a new parameter searchType that allows the user to choose
  how the searching is done. The behaviour up till now has been to
  search the 2D channel maps one at a time to build up the 3D
  detections. This is the default, and can be obtained by setting
  searchType=spatial. Alternatively, you can set searchType=spectral
  to do the searching in the 1D spectra.

* Profiling of the code has led to a few changes that have sped up
  some calculations, particularly those involving the median. Rather
  than using std::sort, the code now simply uses std::nth_element,
  which shows a substantial speedup factor.

* When using the objectList parameter, the maps will now only show the
  selected objects, rather than all detected objects.

* There is a new verification test that looks at the "growing"
  functionality.

Bug fixes:

* When saving a mask FITS file, there was an occasional memory leak
  that caused Duchamp to crash (only when the cube was suitably
  large). This has been fixed.

* The median calculation had a bug that led to an incorrect value
  being reported. The difference will not be great (the element of an
  array being reported as the median was out by one), but could be
  noticeable depending on the dataset.

* The MW range parameters did not respond to the use of a
  subsection. They should refer to channel ranges in the original
  cube, and changes have been made to the code to reflect this.

* There was an occasional error in writing out FITS keywords, due to
  the way comment strings were intialised. This has been improved and
  made more robust.

* Testing with gcc-4.3.2 showed a number of compilation warnings &
  errors, which have been fixed.

* The Gaussian smoothing kernel was too small - it is now made big
  enough to encompass all possible values. It was also being
  incorrectly scaled. This had no functional effect on the detection
  of objects, but for other uses was incorrect. This has been
  rectified.

* Some outputs that weren't caught by the verbose flag now are.


Code development notes:

  The following notes only apply if you want to use the code as a
  library, rather than the stand-alone package. These have generally
  come out of the ASKAP software development. If you just use the
  stand-alone package, they won't be a concern.

* The API for storing detected objects has changed, making it more
  robust from a programming point of view.

* Additional versions of some functions (and the Gaussian smoothing
  class) have been added - this has come out of the ASKAP development
  work.

* A number of instances of "unsigned int" being used for loops have
  been changed to iterators or "size_t". There shouldn't be any
  functional change from this -- the code should just build more
  cleanly.

* The Detection class now records whether the parameters have been
  set, and unsets that flag when objects are merged. This allows
  external programs to track which objects have been merged and deal
  with them appropriately.


Duchamp version 1.1.8 (2009/07/20)
----------------------------------

Improvements:

* The final source list can now be sorted on one of a range of object
  parameters: x-value, y-value, z-value, ra, dec, vel, w50, iflux,
  pflux (integrated and peak flux respectively), or snr. Simply use
  the new input parameter sortingParam. Its default value is vel, so
  the default behaviour stays the same.

* The FDR method involves calculating how many pixels are
  correlated. This previously involved multiplying the beam area by 2,
  implying that 2 neighbouring channels were correlated. This factor
  is now a user-selectable parameter, so that if you know how many
  spectral channels should be correlated, you can set the parameter
  FDRnumCorChan and the FDR calculation will be done correctly.

* Screen output now has more information on the growing threshold when
  the cube statistics are printed.


Bug fixes:

* When reading in previous detections, there was a bug causing a
  segmentation fault. This has been fixed and made more robust.

* There was an apparent bug when writing out the reconstructed cubes
  to a FITS file that caused the new file to be very large. Changing
  the cfitsio function used has fixed this.

* Using NCP projections near the equator could result in a large
  stream of error messages, due to coordinate transformations being
  done in blank regions outside of the acceptable range of world
  coordinates. This has been fixed.

* The function that grows objects had a major bug, that stopped it
  working when flagAdjacent=false. This has been fixed and should work
  fine.

* The input parameter spectraTextFile was not being read
  properly. This has been remedied.

* The units for the integrated flux were wrong for some types of pixel
  flux units. This has been fixed.

* A minor bug caused incorrect calculation of the 20% widths.

* All calls to uint were changed to unsigned int, as some compilers
  had problems recognising that they were the same thing.

* src/fitsHeader.cc was missing an include statement for stdlib.h,
  again causing some compilers to fail.


Duchamp version 1.1.7 (2009/01/16)
----------------------------------

Improvements:

* The mask file that can be created with flagMask can now specify
  which object a given detected pixel belongs to, by taking the object
  ID as its value. This applies if you set flagMaskWithObjectNum=true
  (it defaults to false, which means each detected pixel has the value
  1). Non-detected pixels still have the value 0.

* The names of the output FITS files are now able to be specified
  directly by input parameters. These will override the default names
  that Duchamp provides.

* The Karma annotation file now specifies a font - previously it did
  not and so was at the mercy of whatever was specified in other
  annotation files being viewed at the same time. Please let me know
  if the font I've chosen (lucidasans-12) is not suitable.

* The default flag, used in the absence of 'E', 'S' or 'N', is now '-'
  rather than blank. This will ensure that each column in the results
  file has something in it.

* The configure script is somewhat more robust to non-standard
  systems, particularly when the X11 libraries are in unusual
  places. It can also deal with the slightly different way that
  wcslib-4.3 installs itself.

* The installation part of the Makefile was altered so that the
  symbolic link Duchamp that points to Duchamp-1.1.7 does not have the
  absolute path encoded in it. This will enable the files to be moved
  around at a later date.

Bug Fixes:

* Fixed a bug that caused problems when the flux units were less than
  5 characters long (e.g. 'K')

* The new velocity width values were not getting proper headers in the
  VOTable output. This has been remedied.

* Minor improvements to help non-pgplot use

* Minor fix in WCS functionality.

* Minor fixes to the FitsHeader class.

* Improved FITS data reading, that doesn't require a nullarray
  array. This will reduce memory usage.

* Improved subsection parsing, particularly for non-standard WCS
  systems.

* Improving the memory allocation of the Cube class.

* Fixed a bug that meant the growthThreshold values were sometimes
  being lost.

* Improving the way the growing and merging is done, particularly in
  the case of non-adjacent merging.

* Fixed a scaling bug in the GaussSmooth function.

Other:

* Numerous changes to the code that do not affect how Duchamp itself
  runs, but were necessary for code development with the Duchamp
  libraries (primarily as a part of the ASKAP code development).


Duchamp version 1.1.6 (2008/09/08)
----------------------------------

New features:

* The velocity widths can now be quoted in two additional ways: the
  full width at 50% of the peak flux (ie. the FWHM) and the full width
  at 20% of the peak flux (w_50 and w_20 respectively). The w_50
  parameter is now quoted in the results printed to the screen
  (instead of w_VEL), but all three are written to the results file.

* The facility now exists to bypass the searching step and go straight
  to the output stage using a list of previously detected objects. The
  list is extracted from a previously-created log file. The user can
  change the way the spectra are plotted (e.g. choose to plot
  integrated rather than peak spectra), and select a subset of the
  detected objects to be printed to individual postscript files. Use
  the input parameter usePrevious (set to true) to use this facility,
  and provide a list of objects using the objectList parameter
  (eg. 1,2,4,7-10,15)

* A new parameter "growthThreshold" has been created, that goes with
  "threshold". If you manually specify a threshold, and you wish to
  grow detections, you need to manually specify a growthThreshold (in
  flux units).

Bug fixes:

* Cubes that do not have a spectral axis (such as Stokes cubes) were
  causing memory leaks and crashing, due to some assumptions of the
  existence of the spectral axis. These bugs have been fixed, so that
  Duchamp will run on such cubes, although the WCS transforms may not
  always work appropriately. Please let me know via bug reports of
  difficulties you encounter!

* The growing function has had a few bugs fixed and is now more
  efficient.

* The FDR method was previously overriding any manually-specified
  threshold. This has been fixed so that if the user specifies
  threshold=X, then the FDR method is not called.

* Some undefined things were happening with textual output when no
  objects were detected. This has been fixed.

* Memory leaks caused by FITS WCS irregularities have been fixed
  (specifically, if a FITS header had NAXIS=2 but more than two WCS
  axes were defined).



Duchamp version 1.1.5 (2008/03/20)
----------------------------------

Bug fixes:
* A major bug was discovered in the calculation of integrated
  fluxes. The flux was incorrectly being weighted by the spatial size
  of the detection, resulting in a (often considerable) overestimation
  of the integrated flux. This has been corrected.

* The beam size calculation was also incorrect, being out by a factor
  of 1/ln(2). 

* The "make install" command previously failed to work properly if the
  library had not been created. This has been fixed so that the
  installation works even if there is no library file.

New Features:
* You can now provide a new flux unit string, so that the pixel fluxes
  will be converted to these new fluxes before any calculations are
  done. This is done with the newFluxUnits input parameter. The new
  units should be directly convertible from the old ones (given by the
  BUNIT header -- e.g. if BUNIT = Jy/beam, you can give newFluxUnits
  as mJy/beam, but not mJy).

* You can now specify the precision used in the output files for flux,
  velocity and S/N ratio columns, via new input parameters precFlux,
  precVel and precSNR. These will apply to the VOTable output as well
  as the text table and information on the spectral plots.

* The Karma annotation files previously provided just a circle at the
  spatial location of the detected object. You now have the option to
  draw a border around the detected pixels (in a similar manner to the
  PGPLOT moment maps). Karma does not do this perfectly, so the lines
  may not line up exactly with the pixel borders, so the option
  remains to keep using the circles. This is controlled by the
  parameter "annotationType", with possible options being "borders"
  (the default) or "circles".

* There is a new function getMetadata() that loads the header and WCS
  data, but no pixel information. This is only of interest if you are
  doing code development with the Duchamp library -- it does not
  affect the running of Duchamp itself.

Duchamp version 1.1.4 (2008/02/15)
----------------------------------

Most of the changes incorporated in this update result from ASKAP code
development (that makes use of some of the Duchamp functionality),
but do not actually affect the way Duchamp runs (other than, in a
couple of cases, making it a bit faster). Changes that are
relevant for regular users of Duchamp are as follows:

* A new option has been made available (thanks to Mary Putman et al
  for the suggestion!) to save the spectra of all detected objects in
  a text file. This has the form of one column for the spectral
  coordinates and one column of fluxes for each object. This is
  enabled by the input parameters flagTextSpectra and spectraTextFile.

* Slight changes to the output include he formatting of the way the
  detected objects are written to the log file. This meant the
  verification files were also updated. Also the cube statistics are
  not printed to the results file if a user threshold is specified (as
  they are not calculated!).

* The determination of beam size has been made more reliable, as has
  the BUNIT determination.

* Fixed a bug that meant source merging was not done for 2D images
  (not usually a problem, as the source-finding takes care of this,
  but came up in parallelisation development).

Duchamp version 1.1.3 (2008/01/08)
----------------------------------

A minor update that just improves the build process. The location of
the wcslib include files was not specified correctly, so that some
machines were unable to find them. The configure script has been fixed
so that the compilation should work in all cases. 

Some minor changes have also been made to the code in the PixelMap
directory, and to param.cc, to fix warnings or errors that may appear
in compilation.

However, there are no changes to the actual workings of the code in
this minor version, so if you have a working version of Duchamp 1.1.2
there is probably no reason to update.

Duchamp version 1.1.2 (2007/12/07)
----------------------------------

Compilation:

* The compilation had problems working correctly when the cfitsio and
  wcs libraries were not in standard locations. The configure script
  has been tweaked to make sure the libraries are found.

* Problems with compiling in the absence of pgplot have been fixed,
  and it should now compile fine.

* Improved the configure script for use with Mac OS X and gfortran.


Bugfixing:

* A major problem with the text-based output was fixed. The widths of
  the columns were not being calculated correctly, and were running
  together. This should no longer be the case.

* There was a bug in the way Duchamp interpreted spectral axes,
  particularly AIPS types such as FELO-HEL, leading to incorrect
  scaling of the velocity axis. This has been fixed, so that
  non-standard types are now dealt with appropriately and the velocity
  scaling should be accurate.

* Another bug meant the beam correction was always applied to the
  integrated spectra, regardless of whether the flux units involved
  the beam (e.g. Jy/beam). This has been fixed. Note that the values
  of the integrated flux presented in the results table were always
  calculated correctly.

* Other minor changes to improve the speed and efficiency, especially
  of the merging code.

New user parameters:

* There is now the possibility to save a FITS file that contains a
  mask array, indicating where the detected objects are. This is
  governed by the flagOutputMask parameter.

* The header information that is printed at the start of the results
  file -- that is, the list of parameters, the statistics of the cube,
  and the number of detections -- can now be written to a separate
  file. Set the flagSeparateHeader parameter to true or 1 and it will
  go to the file indicated by the headerFile parameter (the default is
  duchamp-Results.hdr).

* A "scaleMax" parameter has been added, so that the wavelet
  reconstruction can include only a specified range of scales (for
  instance, if one wants to search just for narrow features such as
  absorption lines).

* A "flagRobustStats" parameter has been added. This allows the user
  to elect to use robust statistics (median and median absolute
  deviation from the median) to estimate the noise parameters. If set
  to false, the mean and standard deviation will be used. The default
  value is true.

New Feature:

* When Duchamp is run on a 2D image (i.e. with no spectral output), it
  now produces a postscript file that shows information each
  individual source plus the image cutout of the source. This is
  similar to the spectral output file, but without the spectra. The
  filename is given by the spectraFile parameter.

* Ability to save a FITS file containing a mask array, with pixel
  values of 1 for pixels in a detected object, and 0 elsewhere.


Duchamp version 1.1.1 (2007/07/20)
------------------------------------

Minor update to the release, with just a few changes:

* The verification script was generating diffs due to differences in
  the precision of some calculations when done on different machines. 
  One of these was the integrated flux, and this has been corrected so
  that the same value should appear on all machines.
  The verification script has also changed so that just the number of
  sources and their positions are tested, as well as the
  logfiles. This should avoid the reporting of spurious differences.

* The grey-scale used in the image cutouts (in the spectral plots) has
  been inverted. This makes it consistent with the scale used in the
  maps, and hopefully easier to see when printing.

* The configure/make system has been altered to make it easier to
  install, and to specify different external libraries. In particular,
  it is now not necessary to have PGPLOT installed on your system to
  run Duchamp. In the absence of PGPLOT, Duchamp will run as normal,
  just without any graphical output. Some source code needed to be
  restructured to make this work. Please read the README file, or the
  Installation appendix in the User's Guide, for complete details.

* The time of completion is now written to the log file, so that the
  user can have some idea of how long the program took.


Duchamp version 1.1 (2007/05/17)
------------------------------------

Changes made since the last numbered release:

EFFICIENCY OF EXECUTION:

* Major change to the way detections are stored. They are now recorded
  via run-length encoding in the x-y plane, and stored as collections
  of channel-maps. This improves the memory usage and the speed of
  execution (particularly for routines such as the merging
  routines). The memory usage has also been improved by changing the
  way the list of detected objects is stored.

SMOOTHING:

* Addition of ability to spatially smooth the cube before
  searching. This is done with a Gaussian kernel, the size and shape
  of which can be specified by the user, using the parameters kernMaj
  & kernMin (which are the FWHM for the major & minor axes -- if
  kernMin is not given it is assumed to match kernMaj) and kernPA (the
  position angle of the major axis).  There is a new parameter
  smoothType that distinguishes this method from the spectral (Hanning)
  smoothing already in use.

DETECTIONS:

* Changed the way detection is done -- now only search in 2D images,
  keeping all detections (ie. no minimum-pixel requirement). This
  should be the same as doing both the 1D and 2D searches with the
  minimum pixel requirement in, but simpler. The rejection is done at
  the end after all the merging and growing has been done. This leads
  to a larger number of intermediate detections (ie.  the number
  before merging), which may increase the run-time slightly, although
  this is offset by the changes to the efficiency (see above).

* Increased the range over which objects can be grown to match the
  threshSpatial & threshVelocity parameters, as well as rejecting new
  pixels based on the BLANK and MW tests.

INPUTS:

* There has been a subtle change in the way BLANK pixels are dealt
  with. To enable trimming of BLANK pixels, users should use the new 
  parameter flagTrim. The parameters flagBlankPix and blankPixVal are
  no longer accessible via the parameter file -- warning messages are
  given if they are provided in the parameter file.

* Added ability to specify a statsec, so that the statistics are only
  calculated from a given subsection of the cube, rather than the full
  cube.

* The subsections can also specify the ability to remove a border of n
  pixels from the edges of an axis (ie. n from the start and from the
  end) -- as requested in Trac ticket #5.

* Changed the default value of the reconDim parameter from 3 to 1, and
  the default value of the flagATrous parameter to false.

OUTPUTS:

* The output now includes three different estimates of the "centre" of
  the detection: peak location, centroid (flux-weighted), or average. 

* Some slight improvements to the user feedback during execution.

* Spectral output now includes the baseline, if one was fitted. The
  format of the plot headers is slightly changed, to stop information
  falling off the edge of the page.

* Added command-line option -x to directly disable the PGPLOT X-window
  map plotting. This overrides the setting of flagXOutput in the
  parameter file.

* Fallback spectral units (in case the FITS header fails to record
  them) changed to SPC, but code has been fixed so that any problem
  should be caught before this becomes an issue.

BUGFIXES:

* Fixed bug (Trac ticket #4) that meant the velocity units were not
  correctly dealt with. Duchamp should be more robust for different
  velocity axis setups now.

* Fixed bug (ticket #9) with the VOTable output, where the tags were
  not closing.

* Fixed a bug that meant the integrated fluxes were not calculated
  properly -- no allowance was made for the number of spatial
  pixels. This should also work properly for 2D images now as well.

* Fixed minor bug in calculating the flux threshold from the p-value
  threshold when the FDR method is used. This did not affect the
  actual thresholding, merely the quoted flux threshold in the results
  file. Also, the MW channels are kept out of the p-value threshold
  determination if they are being flagged.



Duchamp version 1.0.7 (2006/11/22)
----------------------------------

One new feature, and some improvements and bug-fixes.

* The user can now elect to Hanning-smooth the spectral direction of
  the cube before the searching is done. This can be done instead of the
  wavelet reconstruction (although if both are selected, the
  reconstruction takes precendence). This functionality is switched on
  and off by the flagSmooth parameter, and the width of the Hanning
  kernel is governed by the hanningWidth parameter.

* The smoothed array can also be written to a FITS file and read in at
  the start in the same manner as the reconstructed array.

* Several improvements have been made to the memory
  management. Several memory leaks were fixed, and the memory usage of
  Duchamp is considerably less, which will be a help for running it on
  large cubes. The speed of some routines has also been improved, by
  removing unnecessarily repetitive loops.

* The sorting routines have been changed, and should now be slightly quicker
  (and a bit more robust).

* Enabled the scale bar (plotted on the 0th moment map in the spectral
  output file) to be able to deal with much finer spatial scales (as
  may be expected from VLBI data).

* Other minor changes to code and presentation.


Duchamp version 1.0.6 (2006/11/01)
----------------------------------

Some incremental improvements and changes to the code:

* The statistics used for the detection are now calculated once for
  the entire cube, rather than being calculated separately for each
  channel map and spectrum. This results in a uniform threshold value
  across the entire cube. The way the statistics calculations are
  implemented in the code was changed, using a new statistics class.

* This means we can report a peak signal-to-noise value for each
  detection. This is given in the text-based output, as well as in the
  summary information above the spectral output for each source.

* The user can now elect not to have the moment map displayed in a
  PGPlot window, through the use of the new flagXOutput
  parameter. This should help in cases of running Duchamp on a dumb
  terminal (with no X-display capabilities), or as part of a batch
  script where the continual popping-up of the PGPlot window might
  become annoying. 

* There are different Flags reported for the case of an object lying on
  the spatial edge and spectral edge of the cube (E and S respectively).

* A few minor bug fixes:
  - When providing a subsection, any axes with the range given by *
    had the pixel values offset incorrectly. This has been fixed.
  - Negative searches now correctly invert the reconstructed array
    when it is read in from a file.

* Other minor changes to the code, to improve memory handling and
  remove redundant arrays and variables.


Duchamp version 1.0.5 (2006/09/06)
----------------------------------

Since version 1.0, there have been a range of mostly minor
changes. These are generally bug-fixes, thanks to feedback from a
number of users, as well as a few extra features. 

Here is a summary of the changes since version 1.0:

USER INTERFACE:

* The user is now able to enter the value of flag parameters in the
  parameter file as strings, as well as integers (so one can enter
  true or 1 to give the same result).

GRAPHICAL OUTPUT:

* Added a new function that draws the edge of the blank region on the
  maps and image cutouts. There is a new flag parameter --
  drawBlankEdges -- that controls the use of this function (the
  default is true).
  Also, the edge of the field is plotted on the image cutouts as a
  yellow line. This distinguishes it from the purple blank pixel
  dividing line.

* The size of the tick mark in the image cutout (indicating the
  spatial scale of the image) is now adaptable, so that it shows the
  scale that best fits the image (ranging from 1 arcsec to 15
  degrees). (This is a change from the previous standard length of 15
  arcmin.)

* The zoomed spectrum had its flux scale determined by all points in
  the zoom box,including those flagged by the Milky Way (MW)
  range. This has been fixed so that any MW channels will not affect
  the flux scaling, potentially improving the appearance of objects
  detected close in velocity to strong MW emission or absorption.

* Improved Karma Annotation file output, so that files without good 
  WCS are dealt with appropriately.

TEXT-BASED OUTPUT:

* The position resolution in the IAU-format names has been increased,
  so that distinct objects close to one another are more readily
  distinguished.  Overlapping sources at different velocities may
  still have the same name, but this is a more general problem.

* The presentation of the results in tabular format has been improved,
  so that the precision of faint sources is sufficient to show their
  flux, for instance, is not 0. This also applies to the information 
  presented above the spectral plots.

* Improved error and warning reporting, so that the formatting and
  presentation of information is clearer.

FITS I/O RELATED:

* The previous version was did not deal well with 4 (or greater)
  dimensions in the FITS file -- the memory allocation was badly done,
  and the axes needed to be in a particular order. This has been
  fixed, so that the FITS I/O routines can now handle any sized FITS
  file, with axes in any order (as long as there are two spatial and one
  spectral axes).

* When the FITS file does not have a rest frequency defined, there is
  no way to transform the frequency axis into velocity. In this case,
  the spectral axis is left as frequency, and put into units of MHz.

* If the beam size is not indicated by the FITS header (through the
  BMAJ and BMIN keywords), the user can manually specify the size
  through the parameter file (using parameter beamSize).

* The FITS I/O routines make use of the command fits_file_exists. This
  was introduced into the CFITSIO package from version 2.5, so Duchamp
  will only work with this version or higher. Version 3+ is preferred.

OTHER BUG-FIXING:

* Several bugs related to the blank-pixels were dealt with.
  The correct values were not propagating through correctly, meaning
  that the default values were being used. Also, there was the
  potential for blank pixels to be detected when the FDR method was
  used. Finally, the image cutout displays occasionally did not
  obscure the blank pixels. All these problems should be fixed now.

* The FDR setup routine was occasionally failing with a segmentation
  fault, due to it accessing unallocated memory. This has been fixed. 

* Fixed bugs that affected the way 2D images (rather than 3D cubes)
  were dealt with. The minChannels test failed when minChannels=0.

* If there is no beam information in the FITS header, the user can now
  enter a beam size as a parameter.

* Other minor fixes to the code, to improve its structure and
  robustness.


Duchamp version 1.0 (2006/06/30)
--------------------------------

The first full release version.

Several major changes compared to previous version:

* Incorporated a FitsHeader class, to keep track of header information
  (including WCS) from the FITS file. All units-related calculations
  are done with this class, such as conversion to velocity, and
  pixel-to-WCS conversion.  The main advantage of this is improved
  speed of execution, more adaptable to different units, and improved
  code (easier for me!).

* Able to select the dimensionality of the reconstruction, rather than
  just doing it in 3-D.

* Robust text output, so that the columns are sized correctly given
  the data being written, and the relevant units (derived from the
  FITS header) are included in the output table.

* Improved the way the reconstructed image is saved and read back in,
  with a better naming scheme and a better set of new FITS header
  keywords. The MW headers are now not written.

Other, more minor changes include:

* MW channels now not set to 0, but just ignored for searching
  purposes (and for the scaling of the spectral output -- they are
  plotted though and highlighted with a hashed box).

* The scale bar in the moment map in the spectral output file has its
  length indicated.

* Improved VOTable output, with more information about the way Duchamp
  was run.

* Made sure all statistics calculations in the reconstruction are
  based on robust (ie. median) methods.

Also moved to an autoconf based configure/make compilation, for ease of use.


Duchamp version 0.9.2 (2006/06/27)
--------------------------------

* Added the use of warning flags for detections: for either edge
  location or negative enclosed flux.

* A new command line option added so that the user can specify a FITS
  file to be searched with the default parameters, rather than giving
  a full parameter file.

* The scale bar on the moment cutout now works in general case, using
  an iterative process to get to the correct length.

* RA axes are now done correctly on the full-scale maps.

* Various coding fixes, improving the readability of the code,
  particularly regarding the declaration of temporary variables.

* Changed the way the time of execution is called : now using time.h
  functions, rather than a system call.

* Changed output in Detection/outputDetection.cc for the pixel
  information of detected objects (the info printed above the spectral
  plots) -- now print out number of voxels in object, and have
  improved the formatting a little.

* Improved the functionality of saving the FITS file containing the
  reconstructed array. Added ability to save and read back in the MW
  parameters when saving a reconstructed cube (if a reconstructed cube
  has different MW range to that being considered, it would cause
  problems). ReadRecon now just returns a FAILURE and will reconstruct
  the cube anew.

* Changed default value of flagGrowth to 0. 



Duchamp version 0.9 (2006/05/05)
--------------------------------

First public release version.