source: trunk/CHANGES @ 1107

Last change on this file since 1107 was 1107, checked in by MatthewWhiting, 12 years ago

Updating the version number to 1.2.2, and getting the documentation all up to date for the next release.

File size: 51.9 KB
Line 
1Duchamp version 1.2.2 (2012/10/29)
2----------------------------------
3
4Bug fixes:
5
6* The reading of reconstructed & smoothed arrays in the case of large
7  (>2GB) cubes has been improved. While the most recent versions of
8  cfitsio were able to handle these, earlier versions would result in
9  a segementation fault using the previous interface. An alternative
10  interface has been implemented that avoids these faults, even for
11  the older cfitsio libraries.
12
13* A bug affecting the writing of FITS mask files (apparently for the
14  case of flagging by the object number) has been fixed.
15
16* The moment map occasionally failed if sources lay spatially within
17  the blank pixel region of the first channel. This check has been
18  removed and the moment map should be produced successfully in all
19  situations.
20
21Other:
22
23* The UCDs used in the VOTable catalogue output have been
24  updated. Several of these were shared between different columns, and
25  so they have been changed so that each UCD is unique to one
26  column. The column ID strings have been updated for clarity.
27
28* The determination of default output image filenames has been
29  improved -- it previously assumed the input filename ended in
30  '.fits', but this is no longer the case.
31
32Duchamp version 1.2.1 (2012/10/09)
33----------------------------------
34
35New feature:
36
37* Duchamp can now produce region files to annotate images displayed in
38  SAOImage DS9. There are new parameters flagDS9 and ds9File that work
39  in the same way as the Karma annotation file parameters. The region
40  files can also be displayed in (at least some versions of)
41  casaviewer.
42
43* There is a new script to fix occasional problems with the
44  automatically-detected Makefile, and the verification script has
45  been updated and improved.
46
47Bug fixes:
48
49* The main bug fix for this release corrects a problem with the
50  velocity width measurements made when searching for negative
51  detections. It was possible to get very small (and incorrect) widths
52  in some cases - this was due to the integrated spectrum not being
53  inverted correctly.
54
55* In the case of smoothing, the stats of the full cube that were being
56  reported in the output files were actually those of the un-smoothed
57  (input cube). This only affected the output files, not the detection
58  process itself.
59
60* The units were not being reported in the VOTable. This has been
61  fixed.
62
63* The UCDs for the spectral parameters in the VOTable were not being
64  adapted to the user requests, so didn't always match the types of
65  the parameters. All should be consistent now.
66
67Other changes:
68
69* The codebase relating to the creation of and writing to output files
70  (results, logs, annotation, VOTables) has been overhauled, with a
71  cleaner interface. Most of this code is in the new subdirectory
72  src/Outputs, to where the columns code has been moved as well.
73
74  This was instigated by ASKAP/Selavy development, but also allowed
75  the expansion of output formats to include DS9 region files. No
76  change to the user should be visible, but anyone using the software
77  will notice differences. Please get in touch if changes are
78  difficult to follow.
79
80Duchamp version 1.2 (2012/08/08)
81--------------------------------
82
83Key changes:
84
85* The way the wavelet reconstruction is done in the presence of BLANK
86  voxels is slightly different, specifically when there were BLANK
87  voxels on the edge. While BLANK voxels are always ignored by the
88  reconstruction (and left in place), the previous approach dealt with
89  BLANKS on the edge by moving the effective edge of the image inwards
90  until the first non-blank voxel. This meant that the size of the
91  dimensions used to determine the number of scales, and the location
92  where boundary reflections took place where different to the input
93  image size.
94
95  However, this meant that BLANK voxels on the edge were treated
96  differently to those not on the edge, so this inconsistency was
97  removed. The full image dimensions are always used to calculate the
98  number of scales, and reflections (which is how the reconstruction
99  deals with the boundaries) will occur at the actual image edges.
100
101  If your data has this property (BLANK pixels at the image
102  boundaries), then any reconstruction will be affected (most likely
103  only in a small way) and potentially any subsequent searching. You
104  are advised to check this (for instance by saving the results of the
105  resonstruction from 1.1.13 and 1.2 and comparing).
106
107* The way the spectral world coordinate system is dealt with has been
108  improved. Previously Duchamp tried to convert to a velocity axis,
109  and got a bit grumpy if that didn't work well. Now it will express
110  the spectral units in the axis type indicated by the FITS
111  header. Duchamp will also allow you to change this type by providing
112  an appropriate WCS type as an input parameter. You may also provide
113  a rest frequency in the parameter file and this will be used to
114  calculate velocities. And, as before, you can provide specific units
115  for the spectral quantities to be converted into.
116
117* Since v1.1.13 was released, the journal paper describing Duchamp has
118  been published. It can be found at MNRAS 421, 3242 (2012), or
119  http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2966.2012.20548.x/full. It
120  includes descriptions of the various algorithms used for Duchamp, as
121  well as some indicative comparisons of the relative completeness &
122  reliability. Please cite this paper when describing Duchamp.
123
124Other new features:
125
126* The default value of snrCut has changed from 3 to 5. This is
127  espeically important if you use the -f option, but also if you don't
128  provide snrCut in your parameter file.
129
130* The XML VOTable output now includes a more complete list of
131  parameters, matching the output given in text form in the results &
132  log output files. Only those parameters affecting the results are
133  included - things like the output file names and so on are left out.
134
135* The minVoxels parameter is now adjusted to be at least (minPix +
136  minChannels - 1). If it is changed, a warning message is triggered.
137
138* The warning/error reporting has been overhauled, with an improved
139  interface for developers. Also, some ERROR instances that previously
140  resulted in the program terminating now allow it to continue
141  (provided that is possible given the error).
142
143* It is now possible to give the convergence criterion for the wavelet
144  reconstruction via the parameter reconConvergence. When the relative
145  change in standard deviation of the residuals from the
146  reconstruction falls below this value, the a trous algorithm stops
147  iterating.
148
149* The way one-dimensional arrays were dealt with was unsatisfactory,
150  both with the way positions were registered and the format of the
151  graphical output. Improvements have been made such that a 1D data
152  structure is assumed to be a spectrum, and so positions in the
153  spectrum are reported as z-pixels (or the equivalent world
154  coordinate). Instead of a 2D moment map, the X-windows display will
155  show the spectrum with all detected pixels plus the final list of
156  detections highlighted, along with the detection and growth
157  thresholds. This plot will also be written to the "detectionMap"
158  image.
159
160* All spectral plots will now show the detection (and growth, if
161  applicable) thresholds when plotting the peak flux spectrum.
162
163* Accessing FITS files when a subsection string is provided has been
164  streamlined. The cfitsio package, which handles the FITS access,
165  creates a temporary file containing this image section, and this
166  meant the previous approach was slow, particularly for large
167  files/sections. Now the file is only opened once and the overhead of
168  creating the temporary file is only encountered once.
169
170* Several aspects of the processing & output did not handle a
171  non-existent WCS well. This has been improved, so that all stages of
172  execution will proceed without errors.
173
174Bug fixes:
175
176* The absence of a BLANK keyword when trimming will now not trigger a
177  failure, only a warning message.
178
179* The flux unit conversion (eg. from Jy/beam to mJy/beam) was not
180  being applied correctly when an existing reconstructed or smoothed
181  array was being read in. This is now applied properly.
182
183* When saving reconstructed or smoothed arrays, the data type is now
184  set to FLOAT, so that the correct values are written (and
185  subsequently read in). The previous version was potentially saving
186  to a FITS file with integer data type (this depended on the type of
187  the input array). In this case, the full precision of data values
188  would not be preserved, so reading back in and doing an identical
189  search could yield different sources.
190
191* For certain cases, the reconstruction attempted to use a very large
192  number of scales. This has been fixed.
193
194* The maximum scale for the reconstruction was not always being
195  applied. The scale ranges are now enforced for all types of
196  reconstructions.
197
198* When saving a mask image while using a subsection of a cube, the
199  offsets for the subsection were not being accounted for
200  correctly. This is now fixed.
201
202* The detection map plots were not recording pixels added from the
203  growing stage. This has been fixed.
204
205* The RA & Dec now have appropriate precision, rather than an
206  arbitrary 2 decimal places on the seconds value. The RA will be one
207  decimal place more than the Dec, and they will reflect the degree of
208  precision in the image pixels.
209
210* A number of data type conversions (typically ints to floats) led to
211  errors in some calculations, such as WCS conversions. Care has been
212  taken to make sure all indexing is done appropriately with no
213  overflow, and that all conversions (where necessary) are done
214  appropriately.
215
216* It was possible to get world positions looking like 11:59:60.0! Not
217  any more.
218
219* Reading previous detections from the log file (via the use of the
220  usePrevious parameter) did result in error messages due to slightly
221  incompatible formats. These have been fixed, and all graphical
222  output should work properly.
223
224* When using a spectral search, having a blank pixel in the first
225  channel could lead to that entire spectrum being ignored, leading to
226  inconsistent results between spectral and spatial searches. This
227  issue has been fixed, with both types of searches returning the same
228  catalogue.
229
230
231Code base development notes:
232
233* Most instances of integer overflow due to insufficient precision
234  have been tracked down - many of these were indexing variables of
235  type int, which have been changed to size_t.
236
237* Importantly, for code interfacing with the duchamp::Cube and related
238  classes, axisDim is now an array of size_t elements, rather than
239  long.
240
241* Object3D now carries its spatial map with it, rather than
242  calculating it on the fly when requested. This saves a lot of
243  unnecessary computational effort.
244
245* The warning/error interface has improved, making use of C++
246  exception handling. This should be compatible with external logging
247  infrastructure (such as log4cxx).
248
249* The interface to the FitsHeader & WCS structures for external code
250  has been improved, so that the WCS can be set up via function calls.
251
252
253
254Duchamp version 1.1.13 (2011/04/10)
255-------------------------------------
256
257Bug fixes:
258
259* The median statistics calculations had a bug in the test for whether
260  the array under consideration was of even size. The correction for
261  this can mean small differences in the resulting values. This can
262  flow through to calculated detection thresholds - please check this
263  with your data if you use this mode!
264
265* The reconstruction code had a bug where the incorrect number of
266  scales was being used. This occurred when the blank pixel flag was
267  set, and when the spectral dimension was smaller than both the
268  spatial dimensions.
269
270Other:
271
272* The default value of the blank pixel flag has been set to false,
273  rather than true. For Duchamp, this should not matter, as it is set
274  when the FITS file is read in. For external code using, for
275  instance, the reconstruction algorithms, this default may be
276  important. Having it true was contributing to the second bug listed
277  above. It also makes more sense for it to be false unless explicitly
278  set in some way.
279
280* The reconstruction code has been streamlined a little, reducing its
281  memory impact and using more appropriate variable types
282  (eg. unsigned ints instead of ints and so forth).
283
284* For those who care about the actual code base, the statistics
285  functions have been expanded, and separated from getStats.cc into
286  "normal" and "robust" versions.
287
288
289Duchamp version 1.1.12 (2011/03/18)
290-------------------------------------
291
292Bug fixes:
293
294* The velocity width calculations had a bug that occasionally led to
295  very large values. The widths are calculated using the integrated
296  spectrum, using a threshold relative to its peak. In v.1.1.10 and
297  1.1.11, however, the peak was searched for over the entire spectral
298  range, rather than just the range for the current object. This led
299  to spurious values. This has been fixed, so that the peak is only
300  searched for within the detected ranage of the object.
301
302* Both the pixel and statistics subsections are now more carefully
303  examined, with errors reported if there are no valid pixels in the
304  pixel subsection, or if the statistics subsection does not intersect
305  with the pixel subsection.
306
307Other:
308
309* The configure script has been slightly improved so that the pgsbox
310  library is only added if PGPlot has been enabled - this is
311  particularly necessary with the slightly changed build procedure for
312  recent wcslib versions.
313
314* In a related note, one file - momentMap.cc - had some #include
315  statements that failed when PGPlot (and hence pgsbox in WCSLIB) was
316  not enabled. These were actually unnecesssary and have been removed.
317
318* There is a new input parameter - flagPlotSpectra - that allows the
319  user to turn off the production of the spectral plots. It defaults
320  to true (meaning the plots are produced).
321
322
323Duchamp version 1.1.11 (2011/03/01)
324-------------------------------------
325
326Improvements:
327
328* The largest changes in this release involved improving the
329  efficiency of the code, resulting in further speed-ups. The code
330  affected was the low-level object-handling code, with the merging
331  stages in particular showing good speed-ups.
332
333* The default value for the beam size, when the beam parameters are
334  not in the FITS header, has been dropped. Now the default behaviour
335  is to simply not make any beam corrections if the beam is not
336  specified by either the FITS file or the input parameters. Thanks to
337  Tobias Westemeier for the suggestion.
338
339* The reporting of the beam size has changed slightly to provide a bit
340  more information.
341
342Bug Fixes:
343
344* The statistics section (statSec) could be larger than the image
345  subsection (if provided), yet the pixels required would never be
346  read in from the FITS file. The behaviour has been changed so that
347  the statSec, if requested, is trimmed to be wholly within the image
348  section.
349
350* It was possible to request, for instance, spectral smoothing on a 2D
351  image. This resulted in a blank smoothed array, which led to
352  problems in the calculations of statistics (and hence the
353  threshold), as well as subsequent calculations. This has been fixed,
354  so that in cases like this the smoothing is turned off (and a
355  warning message is displayed).
356
357* There was a problem with using a MW range and subsectioning when
358  growing objects, as a result of changes in 1.1.10. This has been
359  fixed.
360
361* Inversion was failing when no array was allocated (not a problem for
362  standard Duchamp, but affected its use as a library by other code).
363
364* Writing text spectra would only occur when PGPlot output was enabled
365  in the build (it was combined with the plotting of the spectral
366  graphs). This has been changed so that this output can be requested
367  without requiring PGPlot.
368
369* The equation for the beam size in Sec.4.2 of the User Guide was
370  incorrect and has been fixed.
371
372* Minor change to the configure script to improve the detection of the
373  PGPlot library.
374
375Code development notes:
376
377* The main work has been additions and improvements to the interfaces
378  for the Param and the Section classes, as well as new classes to
379  manage the beam information. The efficiency changes mentioned above
380  also affected the internals of some of the PixelInfo classes, and
381  provided additional functions for judging the closeness of
382  neighbouring objects.
383
384Duchamp version 1.1.10 (2010/11/15)
385-------------------------------------
386
387Improvements:
388
389* The major change for this release is a greatly improved growing
390  algorithm. The previous incarnation was very inefficient (testing
391  all pixels in an object, most of which only have neighbours that are
392  already in the object), and could take an unnecessarily large amount
393  of time in the case of extended sources. The improved algorithm
394  makes use of voxel flags to quickly identify those voxels not yet in
395  an object and only examines them. For a certain test case
396  (observations of extended Galactic HI), the time required reduced
397  from about a day to about a minute! (Note that compact sources, such
398  as HI galaxies, will not show this degree of speedup...) Many thanks
399  to Tara Murphy for pointing out this bug and her help in
400  diagnosing/fixing the problem.
401
402* There have been a number of further improvements to the code to
403  speed it up, based on some profiling. The most important was the
404  function to add pixels to a 2D object, but there were several other
405  minor changes as well.
406
407* The minimum size of the detections can now also be specified in
408  terms of the total number of voxels (in addition or instead of
409  specifying the minimum spatial pixels or channels).
410
411* The way memory is allocated when reading the image has been
412  streamlined. The amount of memory being allocated is also now
413  reported - this is in an effort to allow users to identify problems
414  with running out of memory.
415
416* The version number is now reported in the results and log files.
417
418* Only the filename of the input image (ie. not the leading
419  directories) is now shown on the detection and moment-0 maps. This
420  is to avoid problems with large directory paths being too wide for
421  the plot.
422
423* The verification script was improved for readability (although no
424  functional change was made).
425
426Bug fixes:
427
428* Growing was not being done when a single source was found. Now
429  fixed.
430
431* For particular WCS setups, the RA can be interpreted as being
432  negative, which corrupted the Source ID (ie. the IAU-format
433  string). This has been fixed.
434
435* When the flux units were changed, problems had been seen when
436  writing a reconstructed cube to a FITS file. Any unit change is now
437  undone prior to writing and there should now be no problem.
438
439* Filenames with spaces in them were not handled well. New functions
440  were included and moved to a more obvious location. These filenames
441  now work as expected.
442
443* Spectral axes of type FREQ that had a rest frequency were not being
444  put into velocity units. Now fixed.
445
446* Improvements to the output to screen, making sure everything is
447  covered by the verbose flag.
448
449* Better management of the allocation of arrays.
450
451* The minSize parameter in Image was never initialised, leading to
452  unpredictable behaviour. Now fixed.
453
454Code development notes:
455
456* New functionality includes: a function to return a vector list of
457  voxels in an Object3D or Detection; a new generic Search function
458  (basically a front-end to each of the individual functions); more
459  complete copy constructors for the DataArray classes; new methods
460  for finding the velocity widths.
461
462Duchamp version 1.1.9 (2010/03/26)
463-------------------------------------
464
465Improvements:
466
467* There is now the ability to save the moment-0 map as a FITS
468  file. This is controlled by the parameters flagOutputMomentMap and
469  fileOutputMomentMap, which work in the same way for the mask file.
470
471* There is a new parameter flagRejectBeforeMerge that, when true,
472  rejects sources according to the minPix/minChannels requirements
473  before the merging and growing is done. This defaults to false (so
474  default behaviour is as before). Thanks to Tobias Westmeier for
475  suggesting this and the above improvement.
476
477* There is also a new parameter flagTwoStageMerging. When true (the
478  default), newly-detected objects are merged with the first close
479  object in the list. When false, they are simply added to the end of
480  the list. In both cases, a second round of merging is done to pick
481  up all close matches.
482
483* The way the beam is specified in the absence of the BMAJ/BMIN/BPA
484  parameters has been slightly changed. The use of "beamSize" was
485  somewhat ambiguous (it meant the number of pixels covered by the
486  beam, not the FWHM), so new parameters have been introduced. The
487  beam can be specified by beamArea (which replaces beamSize), or
488  beamFWHM (the FWHM of the beam in pixels). If beamFWHM is provided,
489  it will be used, else the beamArea will be.
490
491* There is a new parameter searchType that allows the user to choose
492  how the searching is done. The behaviour up till now has been to
493  search the 2D channel maps one at a time to build up the 3D
494  detections. This is the default, and can be obtained by setting
495  searchType=spatial. Alternatively, you can set searchType=spectral
496  to do the searching in the 1D spectra.
497
498* Profiling of the code has led to a few changes that have sped up
499  some calculations, particularly those involving the median. Rather
500  than using std::sort, the code now simply uses std::nth_element,
501  which shows a substantial speedup factor.
502
503* When using the objectList parameter, the maps will now only show the
504  selected objects, rather than all detected objects.
505
506* There is a new verification test that looks at the "growing"
507  functionality.
508
509Bug fixes:
510
511* When saving a mask FITS file, there was an occasional memory leak
512  that caused Duchamp to crash (only when the cube was suitably
513  large). This has been fixed.
514
515* The median calculation had a bug that led to an incorrect value
516  being reported. The difference will not be great (the element of an
517  array being reported as the median was out by one), but could be
518  noticeable depending on the dataset.
519
520* The MW range parameters did not respond to the use of a
521  subsection. They should refer to channel ranges in the original
522  cube, and changes have been made to the code to reflect this.
523
524* There was an occasional error in writing out FITS keywords, due to
525  the way comment strings were intialised. This has been improved and
526  made more robust.
527
528* Testing with gcc-4.3.2 showed a number of compilation warnings &
529  errors, which have been fixed.
530
531* The Gaussian smoothing kernel was too small - it is now made big
532  enough to encompass all possible values. It was also being
533  incorrectly scaled. This had no functional effect on the detection
534  of objects, but for other uses was incorrect. This has been
535  rectified.
536
537* Some outputs that weren't caught by the verbose flag now are.
538
539
540Code development notes:
541
542  The following notes only apply if you want to use the code as a
543  library, rather than the stand-alone package. These have generally
544  come out of the ASKAP software development. If you just use the
545  stand-alone package, they won't be a concern.
546
547* The API for storing detected objects has changed, making it more
548  robust from a programming point of view.
549
550* Additional versions of some functions (and the Gaussian smoothing
551  class) have been added - this has come out of the ASKAP development
552  work.
553
554* A number of instances of "unsigned int" being used for loops have
555  been changed to iterators or "size_t". There shouldn't be any
556  functional change from this -- the code should just build more
557  cleanly.
558
559* The Detection class now records whether the parameters have been
560  set, and unsets that flag when objects are merged. This allows
561  external programs to track which objects have been merged and deal
562  with them appropriately.
563
564
565Duchamp version 1.1.8 (2009/07/20)
566----------------------------------
567
568Improvements:
569
570* The final source list can now be sorted on one of a range of object
571  parameters: x-value, y-value, z-value, ra, dec, vel, w50, iflux,
572  pflux (integrated and peak flux respectively), or snr. Simply use
573  the new input parameter sortingParam. Its default value is vel, so
574  the default behaviour stays the same.
575
576* The FDR method involves calculating how many pixels are
577  correlated. This previously involved multiplying the beam area by 2,
578  implying that 2 neighbouring channels were correlated. This factor
579  is now a user-selectable parameter, so that if you know how many
580  spectral channels should be correlated, you can set the parameter
581  FDRnumCorChan and the FDR calculation will be done correctly.
582
583* Screen output now has more information on the growing threshold when
584  the cube statistics are printed.
585
586
587Bug fixes:
588
589* When reading in previous detections, there was a bug causing a
590  segmentation fault. This has been fixed and made more robust.
591
592* There was an apparent bug when writing out the reconstructed cubes
593  to a FITS file that caused the new file to be very large. Changing
594  the cfitsio function used has fixed this.
595
596* Using NCP projections near the equator could result in a large
597  stream of error messages, due to coordinate transformations being
598  done in blank regions outside of the acceptable range of world
599  coordinates. This has been fixed.
600
601* The function that grows objects had a major bug, that stopped it
602  working when flagAdjacent=false. This has been fixed and should work
603  fine.
604
605* The input parameter spectraTextFile was not being read
606  properly. This has been remedied.
607
608* The units for the integrated flux were wrong for some types of pixel
609  flux units. This has been fixed.
610
611* A minor bug caused incorrect calculation of the 20% widths.
612
613* All calls to uint were changed to unsigned int, as some compilers
614  had problems recognising that they were the same thing.
615
616* src/fitsHeader.cc was missing an include statement for stdlib.h,
617  again causing some compilers to fail.
618
619
620Duchamp version 1.1.7 (2009/01/16)
621----------------------------------
622
623Improvements:
624
625* The mask file that can be created with flagMask can now specify
626  which object a given detected pixel belongs to, by taking the object
627  ID as its value. This applies if you set flagMaskWithObjectNum=true
628  (it defaults to false, which means each detected pixel has the value
629  1). Non-detected pixels still have the value 0.
630
631* The names of the output FITS files are now able to be specified
632  directly by input parameters. These will override the default names
633  that Duchamp provides.
634
635* The Karma annotation file now specifies a font - previously it did
636  not and so was at the mercy of whatever was specified in other
637  annotation files being viewed at the same time. Please let me know
638  if the font I've chosen (lucidasans-12) is not suitable.
639
640* The default flag, used in the absence of 'E', 'S' or 'N', is now '-'
641  rather than blank. This will ensure that each column in the results
642  file has something in it.
643
644* The configure script is somewhat more robust to non-standard
645  systems, particularly when the X11 libraries are in unusual
646  places. It can also deal with the slightly different way that
647  wcslib-4.3 installs itself.
648
649* The installation part of the Makefile was altered so that the
650  symbolic link Duchamp that points to Duchamp-1.1.7 does not have the
651  absolute path encoded in it. This will enable the files to be moved
652  around at a later date.
653
654Bug Fixes:
655
656* Fixed a bug that caused problems when the flux units were less than
657  5 characters long (e.g. 'K')
658
659* The new velocity width values were not getting proper headers in the
660  VOTable output. This has been remedied.
661
662* Minor improvements to help non-pgplot use
663
664* Minor fix in WCS functionality.
665
666* Minor fixes to the FitsHeader class.
667
668* Improved FITS data reading, that doesn't require a nullarray
669  array. This will reduce memory usage.
670
671* Improved subsection parsing, particularly for non-standard WCS
672  systems.
673
674* Improving the memory allocation of the Cube class.
675
676* Fixed a bug that meant the growthThreshold values were sometimes
677  being lost.
678
679* Improving the way the growing and merging is done, particularly in
680  the case of non-adjacent merging.
681
682* Fixed a scaling bug in the GaussSmooth function.
683
684Other:
685
686* Numerous changes to the code that do not affect how Duchamp itself
687  runs, but were necessary for code development with the Duchamp
688  libraries (primarily as a part of the ASKAP code development).
689
690
691Duchamp version 1.1.6 (2008/09/08)
692----------------------------------
693
694New features:
695
696* The velocity widths can now be quoted in two additional ways: the
697  full width at 50% of the peak flux (ie. the FWHM) and the full width
698  at 20% of the peak flux (w_50 and w_20 respectively). The w_50
699  parameter is now quoted in the results printed to the screen
700  (instead of w_VEL), but all three are written to the results file.
701
702* The facility now exists to bypass the searching step and go straight
703  to the output stage using a list of previously detected objects. The
704  list is extracted from a previously-created log file. The user can
705  change the way the spectra are plotted (e.g. choose to plot
706  integrated rather than peak spectra), and select a subset of the
707  detected objects to be printed to individual postscript files. Use
708  the input parameter usePrevious (set to true) to use this facility,
709  and provide a list of objects using the objectList parameter
710  (eg. 1,2,4,7-10,15)
711
712* A new parameter "growthThreshold" has been created, that goes with
713  "threshold". If you manually specify a threshold, and you wish to
714  grow detections, you need to manually specify a growthThreshold (in
715  flux units).
716
717Bug fixes:
718
719* Cubes that do not have a spectral axis (such as Stokes cubes) were
720  causing memory leaks and crashing, due to some assumptions of the
721  existence of the spectral axis. These bugs have been fixed, so that
722  Duchamp will run on such cubes, although the WCS transforms may not
723  always work appropriately. Please let me know via bug reports of
724  difficulties you encounter!
725
726* The growing function has had a few bugs fixed and is now more
727  efficient.
728
729* The FDR method was previously overriding any manually-specified
730  threshold. This has been fixed so that if the user specifies
731  threshold=X, then the FDR method is not called.
732
733* Some undefined things were happening with textual output when no
734  objects were detected. This has been fixed.
735
736* Memory leaks caused by FITS WCS irregularities have been fixed
737  (specifically, if a FITS header had NAXIS=2 but more than two WCS
738  axes were defined).
739
740
741
742Duchamp version 1.1.5 (2008/03/20)
743----------------------------------
744
745Bug fixes:
746* A major bug was discovered in the calculation of integrated
747  fluxes. The flux was incorrectly being weighted by the spatial size
748  of the detection, resulting in a (often considerable) overestimation
749  of the integrated flux. This has been corrected.
750
751* The beam size calculation was also incorrect, being out by a factor
752  of 1/ln(2).
753
754* The "make install" command previously failed to work properly if the
755  library had not been created. This has been fixed so that the
756  installation works even if there is no library file.
757
758New Features:
759* You can now provide a new flux unit string, so that the pixel fluxes
760  will be converted to these new fluxes before any calculations are
761  done. This is done with the newFluxUnits input parameter. The new
762  units should be directly convertible from the old ones (given by the
763  BUNIT header -- e.g. if BUNIT = Jy/beam, you can give newFluxUnits
764  as mJy/beam, but not mJy).
765
766* You can now specify the precision used in the output files for flux,
767  velocity and S/N ratio columns, via new input parameters precFlux,
768  precVel and precSNR. These will apply to the VOTable output as well
769  as the text table and information on the spectral plots.
770
771* The Karma annotation files previously provided just a circle at the
772  spatial location of the detected object. You now have the option to
773  draw a border around the detected pixels (in a similar manner to the
774  PGPLOT moment maps). Karma does not do this perfectly, so the lines
775  may not line up exactly with the pixel borders, so the option
776  remains to keep using the circles. This is controlled by the
777  parameter "annotationType", with possible options being "borders"
778  (the default) or "circles".
779
780* There is a new function getMetadata() that loads the header and WCS
781  data, but no pixel information. This is only of interest if you are
782  doing code development with the Duchamp library -- it does not
783  affect the running of Duchamp itself.
784
785Duchamp version 1.1.4 (2008/02/15)
786----------------------------------
787
788Most of the changes incorporated in this update result from ASKAP code
789development (that makes use of some of the Duchamp functionality),
790but do not actually affect the way Duchamp runs (other than, in a
791couple of cases, making it a bit faster). Changes that are
792relevant for regular users of Duchamp are as follows:
793
794* A new option has been made available (thanks to Mary Putman et al
795  for the suggestion!) to save the spectra of all detected objects in
796  a text file. This has the form of one column for the spectral
797  coordinates and one column of fluxes for each object. This is
798  enabled by the input parameters flagTextSpectra and spectraTextFile.
799
800* Slight changes to the output include he formatting of the way the
801  detected objects are written to the log file. This meant the
802  verification files were also updated. Also the cube statistics are
803  not printed to the results file if a user threshold is specified (as
804  they are not calculated!).
805
806* The determination of beam size has been made more reliable, as has
807  the BUNIT determination.
808
809* Fixed a bug that meant source merging was not done for 2D images
810  (not usually a problem, as the source-finding takes care of this,
811  but came up in parallelisation development).
812
813Duchamp version 1.1.3 (2008/01/08)
814----------------------------------
815
816A minor update that just improves the build process. The location of
817the wcslib include files was not specified correctly, so that some
818machines were unable to find them. The configure script has been fixed
819so that the compilation should work in all cases.
820
821Some minor changes have also been made to the code in the PixelMap
822directory, and to param.cc, to fix warnings or errors that may appear
823in compilation.
824
825However, there are no changes to the actual workings of the code in
826this minor version, so if you have a working version of Duchamp 1.1.2
827there is probably no reason to update.
828
829Duchamp version 1.1.2 (2007/12/07)
830----------------------------------
831
832Compilation:
833
834* The compilation had problems working correctly when the cfitsio and
835  wcs libraries were not in standard locations. The configure script
836  has been tweaked to make sure the libraries are found.
837
838* Problems with compiling in the absence of pgplot have been fixed,
839  and it should now compile fine.
840
841* Improved the configure script for use with Mac OS X and gfortran.
842
843
844Bugfixing:
845
846* A major problem with the text-based output was fixed. The widths of
847  the columns were not being calculated correctly, and were running
848  together. This should no longer be the case.
849
850* There was a bug in the way Duchamp interpreted spectral axes,
851  particularly AIPS types such as FELO-HEL, leading to incorrect
852  scaling of the velocity axis. This has been fixed, so that
853  non-standard types are now dealt with appropriately and the velocity
854  scaling should be accurate.
855
856* Another bug meant the beam correction was always applied to the
857  integrated spectra, regardless of whether the flux units involved
858  the beam (e.g. Jy/beam). This has been fixed. Note that the values
859  of the integrated flux presented in the results table were always
860  calculated correctly.
861
862* Other minor changes to improve the speed and efficiency, especially
863  of the merging code.
864
865New user parameters:
866
867* There is now the possibility to save a FITS file that contains a
868  mask array, indicating where the detected objects are. This is
869  governed by the flagOutputMask parameter.
870
871* The header information that is printed at the start of the results
872  file -- that is, the list of parameters, the statistics of the cube,
873  and the number of detections -- can now be written to a separate
874  file. Set the flagSeparateHeader parameter to true or 1 and it will
875  go to the file indicated by the headerFile parameter (the default is
876  duchamp-Results.hdr).
877
878* A "scaleMax" parameter has been added, so that the wavelet
879  reconstruction can include only a specified range of scales (for
880  instance, if one wants to search just for narrow features such as
881  absorption lines).
882
883* A "flagRobustStats" parameter has been added. This allows the user
884  to elect to use robust statistics (median and median absolute
885  deviation from the median) to estimate the noise parameters. If set
886  to false, the mean and standard deviation will be used. The default
887  value is true.
888
889New Feature:
890
891* When Duchamp is run on a 2D image (i.e. with no spectral output), it
892  now produces a postscript file that shows information each
893  individual source plus the image cutout of the source. This is
894  similar to the spectral output file, but without the spectra. The
895  filename is given by the spectraFile parameter.
896
897* Ability to save a FITS file containing a mask array, with pixel
898  values of 1 for pixels in a detected object, and 0 elsewhere.
899
900
901Duchamp version 1.1.1 (2007/07/20)
902------------------------------------
903
904Minor update to the release, with just a few changes:
905
906* The verification script was generating diffs due to differences in
907  the precision of some calculations when done on different machines.
908  One of these was the integrated flux, and this has been corrected so
909  that the same value should appear on all machines.
910  The verification script has also changed so that just the number of
911  sources and their positions are tested, as well as the
912  logfiles. This should avoid the reporting of spurious differences.
913
914* The grey-scale used in the image cutouts (in the spectral plots) has
915  been inverted. This makes it consistent with the scale used in the
916  maps, and hopefully easier to see when printing.
917
918* The configure/make system has been altered to make it easier to
919  install, and to specify different external libraries. In particular,
920  it is now not necessary to have PGPLOT installed on your system to
921  run Duchamp. In the absence of PGPLOT, Duchamp will run as normal,
922  just without any graphical output. Some source code needed to be
923  restructured to make this work. Please read the README file, or the
924  Installation appendix in the User's Guide, for complete details.
925
926* The time of completion is now written to the log file, so that the
927  user can have some idea of how long the program took.
928
929
930Duchamp version 1.1 (2007/05/17)
931------------------------------------
932
933Changes made since the last numbered release:
934
935EFFICIENCY OF EXECUTION:
936
937* Major change to the way detections are stored. They are now recorded
938  via run-length encoding in the x-y plane, and stored as collections
939  of channel-maps. This improves the memory usage and the speed of
940  execution (particularly for routines such as the merging
941  routines). The memory usage has also been improved by changing the
942  way the list of detected objects is stored.
943
944SMOOTHING:
945
946* Addition of ability to spatially smooth the cube before
947  searching. This is done with a Gaussian kernel, the size and shape
948  of which can be specified by the user, using the parameters kernMaj
949  & kernMin (which are the FWHM for the major & minor axes -- if
950  kernMin is not given it is assumed to match kernMaj) and kernPA (the
951  position angle of the major axis).  There is a new parameter
952  smoothType that distinguishes this method from the spectral (Hanning)
953  smoothing already in use.
954
955DETECTIONS:
956
957* Changed the way detection is done -- now only search in 2D images,
958  keeping all detections (ie. no minimum-pixel requirement). This
959  should be the same as doing both the 1D and 2D searches with the
960  minimum pixel requirement in, but simpler. The rejection is done at
961  the end after all the merging and growing has been done. This leads
962  to a larger number of intermediate detections (ie.  the number
963  before merging), which may increase the run-time slightly, although
964  this is offset by the changes to the efficiency (see above).
965
966* Increased the range over which objects can be grown to match the
967  threshSpatial & threshVelocity parameters, as well as rejecting new
968  pixels based on the BLANK and MW tests.
969
970INPUTS:
971
972* There has been a subtle change in the way BLANK pixels are dealt
973  with. To enable trimming of BLANK pixels, users should use the new
974  parameter flagTrim. The parameters flagBlankPix and blankPixVal are
975  no longer accessible via the parameter file -- warning messages are
976  given if they are provided in the parameter file.
977
978* Added ability to specify a statsec, so that the statistics are only
979  calculated from a given subsection of the cube, rather than the full
980  cube.
981
982* The subsections can also specify the ability to remove a border of n
983  pixels from the edges of an axis (ie. n from the start and from the
984  end) -- as requested in Trac ticket #5.
985
986* Changed the default value of the reconDim parameter from 3 to 1, and
987  the default value of the flagATrous parameter to false.
988
989OUTPUTS:
990
991* The output now includes three different estimates of the "centre" of
992  the detection: peak location, centroid (flux-weighted), or average.
993
994* Some slight improvements to the user feedback during execution.
995
996* Spectral output now includes the baseline, if one was fitted. The
997  format of the plot headers is slightly changed, to stop information
998  falling off the edge of the page.
999
1000* Added command-line option -x to directly disable the PGPLOT X-window
1001  map plotting. This overrides the setting of flagXOutput in the
1002  parameter file.
1003
1004* Fallback spectral units (in case the FITS header fails to record
1005  them) changed to SPC, but code has been fixed so that any problem
1006  should be caught before this becomes an issue.
1007
1008BUGFIXES:
1009
1010* Fixed bug (Trac ticket #4) that meant the velocity units were not
1011  correctly dealt with. Duchamp should be more robust for different
1012  velocity axis setups now.
1013
1014* Fixed bug (ticket #9) with the VOTable output, where the tags were
1015  not closing.
1016
1017* Fixed a bug that meant the integrated fluxes were not calculated
1018  properly -- no allowance was made for the number of spatial
1019  pixels. This should also work properly for 2D images now as well.
1020
1021* Fixed minor bug in calculating the flux threshold from the p-value
1022  threshold when the FDR method is used. This did not affect the
1023  actual thresholding, merely the quoted flux threshold in the results
1024  file. Also, the MW channels are kept out of the p-value threshold
1025  determination if they are being flagged.
1026
1027
1028
1029Duchamp version 1.0.7 (2006/11/22)
1030----------------------------------
1031
1032One new feature, and some improvements and bug-fixes.
1033
1034* The user can now elect to Hanning-smooth the spectral direction of
1035  the cube before the searching is done. This can be done instead of the
1036  wavelet reconstruction (although if both are selected, the
1037  reconstruction takes precendence). This functionality is switched on
1038  and off by the flagSmooth parameter, and the width of the Hanning
1039  kernel is governed by the hanningWidth parameter.
1040
1041* The smoothed array can also be written to a FITS file and read in at
1042  the start in the same manner as the reconstructed array.
1043
1044* Several improvements have been made to the memory
1045  management. Several memory leaks were fixed, and the memory usage of
1046  Duchamp is considerably less, which will be a help for running it on
1047  large cubes. The speed of some routines has also been improved, by
1048  removing unnecessarily repetitive loops.
1049
1050* The sorting routines have been changed, and should now be slightly quicker
1051  (and a bit more robust).
1052
1053* Enabled the scale bar (plotted on the 0th moment map in the spectral
1054  output file) to be able to deal with much finer spatial scales (as
1055  may be expected from VLBI data).
1056
1057* Other minor changes to code and presentation.
1058
1059
1060Duchamp version 1.0.6 (2006/11/01)
1061----------------------------------
1062
1063Some incremental improvements and changes to the code:
1064
1065* The statistics used for the detection are now calculated once for
1066  the entire cube, rather than being calculated separately for each
1067  channel map and spectrum. This results in a uniform threshold value
1068  across the entire cube. The way the statistics calculations are
1069  implemented in the code was changed, using a new statistics class.
1070
1071* This means we can report a peak signal-to-noise value for each
1072  detection. This is given in the text-based output, as well as in the
1073  summary information above the spectral output for each source.
1074
1075* The user can now elect not to have the moment map displayed in a
1076  PGPlot window, through the use of the new flagXOutput
1077  parameter. This should help in cases of running Duchamp on a dumb
1078  terminal (with no X-display capabilities), or as part of a batch
1079  script where the continual popping-up of the PGPlot window might
1080  become annoying.
1081
1082* There are different Flags reported for the case of an object lying on
1083  the spatial edge and spectral edge of the cube (E and S respectively).
1084
1085* A few minor bug fixes:
1086  - When providing a subsection, any axes with the range given by *
1087    had the pixel values offset incorrectly. This has been fixed.
1088  - Negative searches now correctly invert the reconstructed array
1089    when it is read in from a file.
1090
1091* Other minor changes to the code, to improve memory handling and
1092  remove redundant arrays and variables.
1093
1094
1095Duchamp version 1.0.5 (2006/09/06)
1096----------------------------------
1097
1098Since version 1.0, there have been a range of mostly minor
1099changes. These are generally bug-fixes, thanks to feedback from a
1100number of users, as well as a few extra features.
1101
1102Here is a summary of the changes since version 1.0:
1103
1104USER INTERFACE:
1105
1106* The user is now able to enter the value of flag parameters in the
1107  parameter file as strings, as well as integers (so one can enter
1108  true or 1 to give the same result).
1109
1110GRAPHICAL OUTPUT:
1111
1112* Added a new function that draws the edge of the blank region on the
1113  maps and image cutouts. There is a new flag parameter --
1114  drawBlankEdges -- that controls the use of this function (the
1115  default is true).
1116  Also, the edge of the field is plotted on the image cutouts as a
1117  yellow line. This distinguishes it from the purple blank pixel
1118  dividing line.
1119
1120* The size of the tick mark in the image cutout (indicating the
1121  spatial scale of the image) is now adaptable, so that it shows the
1122  scale that best fits the image (ranging from 1 arcsec to 15
1123  degrees). (This is a change from the previous standard length of 15
1124  arcmin.)
1125
1126* The zoomed spectrum had its flux scale determined by all points in
1127  the zoom box,including those flagged by the Milky Way (MW)
1128  range. This has been fixed so that any MW channels will not affect
1129  the flux scaling, potentially improving the appearance of objects
1130  detected close in velocity to strong MW emission or absorption.
1131
1132* Improved Karma Annotation file output, so that files without good
1133  WCS are dealt with appropriately.
1134
1135TEXT-BASED OUTPUT:
1136
1137* The position resolution in the IAU-format names has been increased,
1138  so that distinct objects close to one another are more readily
1139  distinguished.  Overlapping sources at different velocities may
1140  still have the same name, but this is a more general problem.
1141
1142* The presentation of the results in tabular format has been improved,
1143  so that the precision of faint sources is sufficient to show their
1144  flux, for instance, is not 0. This also applies to the information
1145  presented above the spectral plots.
1146
1147* Improved error and warning reporting, so that the formatting and
1148  presentation of information is clearer.
1149
1150FITS I/O RELATED:
1151
1152* The previous version was did not deal well with 4 (or greater)
1153  dimensions in the FITS file -- the memory allocation was badly done,
1154  and the axes needed to be in a particular order. This has been
1155  fixed, so that the FITS I/O routines can now handle any sized FITS
1156  file, with axes in any order (as long as there are two spatial and one
1157  spectral axes).
1158
1159* When the FITS file does not have a rest frequency defined, there is
1160  no way to transform the frequency axis into velocity. In this case,
1161  the spectral axis is left as frequency, and put into units of MHz.
1162
1163* If the beam size is not indicated by the FITS header (through the
1164  BMAJ and BMIN keywords), the user can manually specify the size
1165  through the parameter file (using parameter beamSize).
1166
1167* The FITS I/O routines make use of the command fits_file_exists. This
1168  was introduced into the CFITSIO package from version 2.5, so Duchamp
1169  will only work with this version or higher. Version 3+ is preferred.
1170
1171OTHER BUG-FIXING:
1172
1173* Several bugs related to the blank-pixels were dealt with.
1174  The correct values were not propagating through correctly, meaning
1175  that the default values were being used. Also, there was the
1176  potential for blank pixels to be detected when the FDR method was
1177  used. Finally, the image cutout displays occasionally did not
1178  obscure the blank pixels. All these problems should be fixed now.
1179
1180* The FDR setup routine was occasionally failing with a segmentation
1181  fault, due to it accessing unallocated memory. This has been fixed.
1182
1183* Fixed bugs that affected the way 2D images (rather than 3D cubes)
1184  were dealt with. The minChannels test failed when minChannels=0.
1185
1186* If there is no beam information in the FITS header, the user can now
1187  enter a beam size as a parameter.
1188
1189* Other minor fixes to the code, to improve its structure and
1190  robustness.
1191
1192
1193Duchamp version 1.0 (2006/06/30)
1194--------------------------------
1195
1196The first full release version.
1197
1198Several major changes compared to previous version:
1199
1200* Incorporated a FitsHeader class, to keep track of header information
1201  (including WCS) from the FITS file. All units-related calculations
1202  are done with this class, such as conversion to velocity, and
1203  pixel-to-WCS conversion.  The main advantage of this is improved
1204  speed of execution, more adaptable to different units, and improved
1205  code (easier for me!).
1206
1207* Able to select the dimensionality of the reconstruction, rather than
1208  just doing it in 3-D.
1209
1210* Robust text output, so that the columns are sized correctly given
1211  the data being written, and the relevant units (derived from the
1212  FITS header) are included in the output table.
1213
1214* Improved the way the reconstructed image is saved and read back in,
1215  with a better naming scheme and a better set of new FITS header
1216  keywords. The MW headers are now not written.
1217
1218Other, more minor changes include:
1219
1220* MW channels now not set to 0, but just ignored for searching
1221  purposes (and for the scaling of the spectral output -- they are
1222  plotted though and highlighted with a hashed box).
1223
1224* The scale bar in the moment map in the spectral output file has its
1225  length indicated.
1226
1227* Improved VOTable output, with more information about the way Duchamp
1228  was run.
1229
1230* Made sure all statistics calculations in the reconstruction are
1231  based on robust (ie. median) methods.
1232
1233Also moved to an autoconf based configure/make compilation, for ease of use.
1234
1235
1236Duchamp version 0.9.2 (2006/06/27)
1237--------------------------------
1238
1239* Added the use of warning flags for detections: for either edge
1240  location or negative enclosed flux.
1241
1242* A new command line option added so that the user can specify a FITS
1243  file to be searched with the default parameters, rather than giving
1244  a full parameter file.
1245
1246* The scale bar on the moment cutout now works in general case, using
1247  an iterative process to get to the correct length.
1248
1249* RA axes are now done correctly on the full-scale maps.
1250
1251* Various coding fixes, improving the readability of the code,
1252  particularly regarding the declaration of temporary variables.
1253
1254* Changed the way the time of execution is called : now using time.h
1255  functions, rather than a system call.
1256
1257* Changed output in Detection/outputDetection.cc for the pixel
1258  information of detected objects (the info printed above the spectral
1259  plots) -- now print out number of voxels in object, and have
1260  improved the formatting a little.
1261
1262* Improved the functionality of saving the FITS file containing the
1263  reconstructed array. Added ability to save and read back in the MW
1264  parameters when saving a reconstructed cube (if a reconstructed cube
1265  has different MW range to that being considered, it would cause
1266  problems). ReadRecon now just returns a FAILURE and will reconstruct
1267  the cube anew.
1268
1269* Changed default value of flagGrowth to 0.
1270
1271
1272
1273Duchamp version 0.9 (2006/05/05)
1274--------------------------------
1275
1276First public release version.
Note: See TracBrowser for help on using the repository browser.