Opened 14 years ago
Last modified 14 years ago
#90 assigned defect
possible memory issues with wavelet reconstruction
Reported by: | MatthewWhiting | Owned by: | MatthewWhiting |
---|---|---|---|
Priority: | normal | Milestone: | Release-2.0 |
Component: | Building/Installation | Version: | 1.1.9 |
Severity: | normal | Keywords: | |
Cc: | tobias.westmeier@… |
Description
Email from Tobias: Likely to be lack of memory, but perhaps Duchamp can be a bit smarter about reporting problems like this?
Hi Matthew,
last night I was trying to run Duchamp with wavelet reconstruction on the Westerbork model cube for WALLABY. However, Duchamp quit during the calculation of cube statistics without providing any useful error message other than "Abort".
Below please find a transcript of the complete Duchamp run. Without the wavelet reconstruction it works fine, but as soon as I turn the wavelet reconstruction on, Duchamp aborts prematureley.
I was wondering if this is a known issue or if anything is wrong with the parameters? Wavelet reconstruction does work on other data cubes.
Thank you! Tobias
parsec:/u/wes160/Documents/WALLABY/TWG4/WSRT_Test_Cube_1/> Duchamp -p duchamp.par Opening image: wallaby_sky_beam.fits Dimensions of FITS file: 360x360x1464x1 Reading data ... Done. Data array has dimensions: 360x360x1464 Opened successfully. ---- Parameters ---- Image to be analysed.........................[imageFile] = wallaby_sky_beam.fits Intermediate Logfile...........................[logFile] = duchamp-logfile.txt Final Results file.............................[outFile] = duchamp-results.txt Spectrum file..............................[spectraFile] = duchamp-spectra.ps Karma annotation file........................[karmaFile] = duchamp-annotation.ann 0th Moment Map...............................[momentMap] = duchamp-MomentMap.ps Detection Map.............................[detectionMap] = duchamp-DetectionMap.ps Display a map in a pgplot xwindow?.........[flagXOutput] = true Saving reconstructed cube?.............[flagOutputRecon] = false Saving residuals from reconstruction?..[flagOutputResid] = false Saving mask cube?.......................[flagOutputMask] = true --> duchamp-mask.fits Saving 0th moment to FITS file?.........[flagOutputMask] = true --> duchamp-mom0.fits ------ Type of searching performed.................[searchType] = spatial Trimming Blank Pixels?........................[flagTrim] = false Searching for Negative features?..........[flagNegative] = false Removing Milky Way channels?....................[flagMW] = false Area of Beam (pixels)................................... = 10.1978 Removing baselines before search?.........[flagBaseline] = false Smoothing data prior to searching?..........[flagSmooth] = false Using A Trous reconstruction?...............[flagATrous] = true Number of dimensions in reconstruction........[reconDim] = 1 Minimum scale in reconstruction...............[scaleMin] = 1 SNR Threshold within reconstruction...........[snrRecon] = 4 Filter being used for reconstruction........[filterCode] = 1 (B3 spline function) Using Robust statistics?...............[flagRobustStats] = true Using FDR analysis?............................[flagFDR] = false SNR Threshold (in sigma)........................[snrCut] = 1 Minimum # Pixels in a detection.................[minPix] = 5 Minimum # Channels in a detection..........[minChannels] = 5 Growing objects after detection?............[flagGrowth] = false Using Adjacent-pixel criterion?...........[flagAdjacent] = true Max. velocity separation for merging....[threshVelocity] = 7 Reject objects before merging?........[flagRejectBeforeMerge] = false Merge objects in two stages?..........[flagTwoStageMerging] = true Method of spectral plotting.............[spectralMethod] = peak Type of object centre used in results......[pixelCentre] = centroid -------------------- Commencing search in reconstructed cube... Reconstructing... All Done. Calculating the cube statistics... Abort
Change History (2)
comment:1 Changed 14 years ago by
Status: | new → assigned |
---|
comment:2 Changed 14 years ago by
Improved the reporting in [758] so that all allocation is reported, and only when it is actually allocated.
I have added [740] a calculation of the total memory that is to be read in, in the hope this will be of some use. It is hard to diagnose automatically that the program has run out of memory (which I think was this issue), but this might help prompt the user to see the issue.
I've also improved the way the memory is allocated - it was a bit convoluted the way I had it (although this clearly wasn't the problem in the example above).