possible memory issues with wavelet reconstruction

[MatthewWhiting]

Email from Tobias: Likely to be lack of memory, but perhaps Duchamp can be a bit smarter about reporting problems like this?

---

Hi Matthew,

last night I was trying to run Duchamp with wavelet reconstruction on the Westerbork model cube for WALLABY. However, Duchamp quit during the calculation of cube statistics without providing any useful error message other than "Abort".

Below please find a transcript of the complete Duchamp run. Without the wavelet reconstruction it works fine, but as soon as I turn the wavelet reconstruction on, Duchamp aborts prematureley.

I was wondering if this is a known issue or if anything is wrong with the parameters? Wavelet reconstruction does work on other data cubes.

Thank you! Tobias

parsec:/u/wes160/Documents/WALLABY/TWG4/WSRT_Test_Cube_1/> Duchamp -p duchamp.par
Opening image: wallaby_sky_beam.fits
Dimensions of FITS file: 360x360x1464x1
Reading data ... Done. Data array has dimensions: 360x360x1464
Opened successfully.

---- Parameters ----
Image to be analysed.........................[imageFile]  = wallaby_sky_beam.fits
Intermediate Logfile...........................[logFile]  = duchamp-logfile.txt
Final Results file.............................[outFile]  = duchamp-results.txt
Spectrum file..............................[spectraFile]  = duchamp-spectra.ps
Karma annotation file........................[karmaFile]  = duchamp-annotation.ann
0th Moment Map...............................[momentMap]  = duchamp-MomentMap.ps
Detection Map.............................[detectionMap]  = duchamp-DetectionMap.ps
Display a map in a pgplot xwindow?.........[flagXOutput]  = true
Saving reconstructed cube?.............[flagOutputRecon]  = false
Saving residuals from reconstruction?..[flagOutputResid]  = false
Saving mask cube?.......................[flagOutputMask]  =  true --> duchamp-mask.fits
Saving 0th moment to FITS file?.........[flagOutputMask]  =  true --> duchamp-mom0.fits
------
Type of searching performed.................[searchType]  =  spatial
Trimming Blank Pixels?........................[flagTrim]  =  false
Searching for Negative features?..........[flagNegative]  =  false
Removing Milky Way channels?....................[flagMW]  =  false
Area of Beam (pixels)...................................  =  10.1978
Removing baselines before search?.........[flagBaseline]  =  false
Smoothing data prior to searching?..........[flagSmooth]  =  false
Using A Trous reconstruction?...............[flagATrous]  =  true
Number of dimensions in reconstruction........[reconDim]  =  1
Minimum scale in reconstruction...............[scaleMin]  =  1
SNR Threshold within reconstruction...........[snrRecon]  =  4
Filter being used for reconstruction........[filterCode]  =  1 (B3 
spline function)
Using Robust statistics?...............[flagRobustStats]  =  true
Using FDR analysis?............................[flagFDR]  =  false
SNR Threshold (in sigma)........................[snrCut]  =  1
Minimum # Pixels in a detection.................[minPix]  =  5
Minimum # Channels in a detection..........[minChannels]  =  5
Growing objects after detection?............[flagGrowth]  =  false
Using Adjacent-pixel criterion?...........[flagAdjacent]  =  true
Max. velocity separation for merging....[threshVelocity]  =  7
Reject objects before merging?........[flagRejectBeforeMerge]  =  false
Merge objects in two stages?..........[flagTwoStageMerging]  =  true
Method of spectral plotting.............[spectralMethod]  =  peak
Type of object centre used in results......[pixelCentre]  =  centroid
--------------------

Commencing search in reconstructed cube...
  Reconstructing...  All Done.
  Calculating the cube statistics... Abort

[MatthewWhiting]

I have added [740] a calculation of the total memory that is to be read in, in the hope this will be of some use. It is hard to diagnose automatically that the program has run out of memory (which I think was this issue), but this might help prompt the user to see the issue.

I've also improved the way the memory is allocated - it was a bit convoluted the way I had it (although this clearly wasn't the problem in the example above).

[MatthewWhiting]

Improved the reporting in [758] so that all allocation is reported, and only when it is actually allocated.