1 | % ----------------------------------------------------------------------- |
---|
2 | % hints.tex: Section giving some tips & hints on how Duchamp is best |
---|
3 | % used. |
---|
4 | % ----------------------------------------------------------------------- |
---|
5 | % Copyright (C) 2006, Matthew Whiting, ATNF |
---|
6 | % |
---|
7 | % This program is free software; you can redistribute it and/or modify it |
---|
8 | % under the terms of the GNU General Public License as published by the |
---|
9 | % Free Software Foundation; either version 2 of the License, or (at your |
---|
10 | % option) any later version. |
---|
11 | % |
---|
12 | % Duchamp is distributed in the hope that it will be useful, but WITHOUT |
---|
13 | % ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or |
---|
14 | % FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License |
---|
15 | % for more details. |
---|
16 | % |
---|
17 | % You should have received a copy of the GNU General Public License |
---|
18 | % along with Duchamp; if not, write to the Free Software Foundation, |
---|
19 | % Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA |
---|
20 | % |
---|
21 | % Correspondence concerning Duchamp may be directed to: |
---|
22 | % Internet email: Matthew.Whiting [at] atnf.csiro.au |
---|
23 | % Postal address: Dr. Matthew Whiting |
---|
24 | % Australia Telescope National Facility, CSIRO |
---|
25 | % PO Box 76 |
---|
26 | % Epping NSW 1710 |
---|
27 | % AUSTRALIA |
---|
28 | % ----------------------------------------------------------------------- |
---|
29 | \secA{Notes and hints on the use of \duchamp} |
---|
30 | \label{sec-notes} |
---|
31 | |
---|
32 | In using \duchamp, the user has to make a number of decisions about |
---|
33 | the way the program runs. This section is designed to give the user |
---|
34 | some idea about what to choose. |
---|
35 | |
---|
36 | \secB{Memory usage} |
---|
37 | |
---|
38 | A lot of attention has been paid to the memory usage in \duchamp, |
---|
39 | recognising that data cubes are going to be increasing in size with |
---|
40 | new generation correlators and wider fields of view. However, users |
---|
41 | with large cubes should be aware of the likely usage for different |
---|
42 | modes of operation and plan their \duchamp execution carefully. |
---|
43 | |
---|
44 | At the start of the program, memory is allocated sufficient for: |
---|
45 | \begin{itemize} |
---|
46 | \item The entire pixel array (as requested, subject to any |
---|
47 | subsection). |
---|
48 | \item The spatial extent, which holds the map of detected pixels (for |
---|
49 | output into the detection map). |
---|
50 | \item If smoothing or reconstruction has been selected, another array |
---|
51 | of the same size as the pixel array. This will hold the |
---|
52 | smoothed/reconstructed array (the original needs to be kept to do the |
---|
53 | correct parameterisation of detected sources). |
---|
54 | \item If baseline-subtraction has been selected, a further array of |
---|
55 | the same size as the pixel array. This holds the baseline values, |
---|
56 | which need to be added back in prior to parameterisation. |
---|
57 | \end{itemize} |
---|
58 | All of these will be float type, except for the detection map, which |
---|
59 | is short. |
---|
60 | |
---|
61 | There will, of course, be additional allocation during the course of |
---|
62 | the program. The detection list will progressively grow, with each |
---|
63 | detection having a memory footprint as described in |
---|
64 | Section~\ref{sec-scan}. But perhaps more important and with a larger |
---|
65 | impact will be the temporary space allocated for various algorithms. |
---|
66 | |
---|
67 | The largest of these will be the wavelet reconstruction. This will |
---|
68 | require an additional allocation of twice the size of the array being |
---|
69 | reconstructed, one for the coefficients and one for the wavelets - |
---|
70 | each scale will overwrite the previous one. So, for the 1D case, this |
---|
71 | means an additional allocation of twice the spectral dimension (since |
---|
72 | we only reconstruct one spectrum at a time), but the 3D case will |
---|
73 | require an additional allocation of twice the cube size (this means |
---|
74 | there needs to be available at least four times the size of the input |
---|
75 | cube for 3D reconstruction, plus the additional overheads of |
---|
76 | detections and so forth). |
---|
77 | |
---|
78 | The smoothing has less of an impact, since it only operates on the |
---|
79 | lower dimensions, but it will make an additional allocation of twice |
---|
80 | the relevant size (spectral dimension for spectral smoothing, or |
---|
81 | spatial image size for the spatial Gaussian smoothing). |
---|
82 | |
---|
83 | The other large allocation of temporary space will be for calculating |
---|
84 | robust statistics. The median-based calculations require at least |
---|
85 | partial sorting of the data, and so cannot be done on the original |
---|
86 | image cube. This is done for the entire cube and so the temporary |
---|
87 | memory increase can be large. |
---|
88 | |
---|
89 | |
---|
90 | \secB{Timing considerations} |
---|
91 | |
---|
92 | Another intersting question from a user's perspective is how long you |
---|
93 | can expect \duchamp to take. This is a difficult question to answer in |
---|
94 | general, as different users will have different sized data sets, as |
---|
95 | well as machines with different capabilities (in terms of the CPU |
---|
96 | speed and I/O \& memory bandwidths). Additionally, the time required |
---|
97 | will depend slightly on the number of sources found and their size |
---|
98 | (very large sources can take a while to fully parameterise). |
---|
99 | |
---|
100 | Having said that, in \citet{whiting12} a brief analysis was done |
---|
101 | looking at different modes of execution applied to a single HIPASS |
---|
102 | cube (\#201) using a MacBook Pro (2.66GHz, 8MB RAM). Two sets of |
---|
103 | thresholds were used, either $10^8$~Jy~beam$^{-1}$ (no sources will be |
---|
104 | found, so that the time taken is dominated by preprocessing), or |
---|
105 | 35~mJy~beam$^{-1}$ (or $\sim2.58\sigma$, chosen so that the time taken |
---|
106 | will include that required to process sources). The basic searches, |
---|
107 | with no pre-processing done, took less than a second for the |
---|
108 | high-threshold search, but between 1 and 3~min for the low-threshold |
---|
109 | case -- the numbers of sources detected ranged from 3000 (rejecting |
---|
110 | sources with less than 3 channels and 2 spatial pixels) to 42000 |
---|
111 | (keeping all sources). |
---|
112 | |
---|
113 | When smoothing, the raw time for the spectral smoothing was only a few |
---|
114 | seconds, with a small dependence on the width of the smoothing |
---|
115 | filter. And because the number of spurious sources is markedly |
---|
116 | decreased (the final catalogues ranged from 17 to 174 sources, |
---|
117 | depending on the width of the smoothing), searching with the low |
---|
118 | threshold did not add much more than a second to the time. The spatial |
---|
119 | smoothing was more computationally intensive, taking about 4 minutes |
---|
120 | to complete the high-threshold search. |
---|
121 | |
---|
122 | The wavelet reconstruction time primarily depended on the |
---|
123 | dimensionality of the reconstruction, with the 1D taking 20~s, the 2D |
---|
124 | taking 30 - 40~s and the 3D taking 2 - 4~min. The spread in times for |
---|
125 | a given dimensionality was caused by the different reconstruction |
---|
126 | thresholds, with lower thresholds taking longer (since more pixels are |
---|
127 | above the threshold and so need to be added to the final spectrum). In |
---|
128 | all cases the reconstruction time dominated the total time for the |
---|
129 | low-threshold search, since the number of sources found was again |
---|
130 | smaller than the basic searches. |
---|
131 | |
---|
132 | |
---|
133 | \secB{Should there be preprocessing?} |
---|
134 | |
---|
135 | Why do preprocessing? Effect on completeness and reliability. Cite |
---|
136 | results from MNRAS paper and given basic summary of what each does. |
---|
137 | |
---|
138 | \secB{Reconstruction considerations} |
---|
139 | |
---|
140 | Several things: |
---|
141 | \begin{itemize} |
---|
142 | \item Beam effects and residual noise |
---|
143 | \item Memory and time recap |
---|
144 | \item Details on how it works. |
---|
145 | \item Effect of subsectioning |
---|
146 | \end{itemize} |
---|
147 | |
---|
148 | \secB{Smoothing considerations} |
---|
149 | |
---|
150 | Anything here? Edge effects? |
---|
151 | |
---|
152 | \secB{Threshold method} |
---|
153 | |
---|
154 | When it comes to searching, the FDR method produces more reliable |
---|
155 | results than simple sigma-clipping, particularly in the absence of |
---|
156 | reconstruction. However, it does not work in exactly the way one |
---|
157 | would expect for a given value of \texttt{alpha}. For instance, |
---|
158 | setting fairly liberal values of \texttt{alpha} (say, 0.1) will often |
---|
159 | lead to a much smaller fraction of false detections (\ie much less |
---|
160 | than 10\%). This is the effect of the merging algorithms, that combine |
---|
161 | the sources after the detection stage, and reject detections not |
---|
162 | meeting the minimum pixel or channel requirements. It is thus better |
---|
163 | to aim for larger \texttt{alpha} values than those derived from a |
---|
164 | straight conversion of the desired false detection rate. |
---|
165 | |
---|
166 | If the FDR method is not used, caution is required when choosing the |
---|
167 | S/N cutoff. Typical cubes have very large numbers of pixels, so even |
---|
168 | an apparently large cutoff will still result in a not-insignificant |
---|
169 | number of detections simply due to random fluctuations of the noise |
---|
170 | background. For instance, a $4\sigma$ threshold on a cube of Gaussian |
---|
171 | noise of size $100\times100\times1024$ will result in $\sim340$ |
---|
172 | single-pixel detections. This is where the minimum channel and pixel |
---|
173 | requirements are important in rejecting spurious detections. |
---|
174 | |
---|
175 | |
---|
176 | % \secB{Preprocessing} |
---|
177 | % |
---|
178 | % \secC{Should I do any preprocessing?} |
---|
179 | % |
---|
180 | % The main choice is whether to alter the cube to try and enhance the |
---|
181 | % detectability of objects, by either smoothing or reconstructing via |
---|
182 | % the \atrous method. The main benefits of both methods are the marked |
---|
183 | % reduction in the noise level, leading to regularly-shaped detections, |
---|
184 | % and good reliability for faint sources. |
---|
185 | % |
---|
186 | % The main drawback with the \atrous method is the long execution time: |
---|
187 | % to reconstruct a $170\times160\times1024$ (\hipass) cube often |
---|
188 | % requires three iterations and takes about 20-25 minutes to run |
---|
189 | % completely. Note that this is for the more complete three-dimensional |
---|
190 | % reconstruction: using \texttt{reconDim = 1} makes the reconstruction |
---|
191 | % quicker (the full program then takes less than 5 minutes), but it is |
---|
192 | % still the largest part of the time. |
---|
193 | % |
---|
194 | % The smoothing procedure is computationally simpler, and thus quicker, |
---|
195 | % than the reconstruction. The spectral Hanning method adds only a very |
---|
196 | % small overhead on the execution, and the spatial Gaussian method, |
---|
197 | % while taking longer, will be done (for the above example) in less than |
---|
198 | % 2 minutes. Note that these times will depend on the size of the |
---|
199 | % filter/kernel used: a larger filter means more calculations. |
---|
200 | % |
---|
201 | % The searching part of the procedure is much quicker: searching an |
---|
202 | % un-reconstructed cube leads to execution times of less than a |
---|
203 | % minute. Alternatively, using the ability to read in previously-saved |
---|
204 | % reconstructed arrays makes running the reconstruction more than once a |
---|
205 | % more feasible prospect. |
---|
206 | % |
---|
207 | % On the positive side, the shape of the detections in a cube that has |
---|
208 | % been reconstructed or smoothed will be much more regular and smooth -- |
---|
209 | % the ragged edges that objects in the raw cube possess are smoothed by |
---|
210 | % the removal of most of the noise. This enables better determination of |
---|
211 | % the shapes and characteristics of objects. |
---|
212 | % |
---|
213 | % \secC{Reconstruction vs Smoothing} |
---|
214 | % |
---|
215 | % While the time overhead is larger for the reconstruction case, it will |
---|
216 | % potentially provide a better recovery of real sources than the |
---|
217 | % smoothing case. This is because it probes the full range of scales |
---|
218 | % present in the cube (or spectral domain), rather than the specific |
---|
219 | % scale determined by the Hanning filter or Gaussian kernel used in the |
---|
220 | % smoothing. |
---|
221 | % |
---|
222 | % When considering the reconstruction method, note that the 2D |
---|
223 | % reconstruction (\texttt{reconDim = 2}) can be susceptible to edge |
---|
224 | % effects. If the valid area in the cube (\ie the part that is not |
---|
225 | % BLANK) has non-rectangular edges, the convolution can produce |
---|
226 | % artefacts in the reconstruction that mimic the edges and can lead |
---|
227 | % (depending on the selection threshold) to some spurious |
---|
228 | % sources. Caution is advised with such data -- the user is advised to |
---|
229 | % check carefully the reconstructed cube for the presence of such |
---|
230 | % artefacts. |
---|
231 | % |
---|
232 | % A more important effect that can be important for 2D reconstructions |
---|
233 | % is the fact that the pixels in the spatial domain typically exhibit |
---|
234 | % some correlation due to the beam. Since each channel is reconstructed |
---|
235 | % independently, beam-sized noise fluctuations can rise above the |
---|
236 | % reconstruction threshold more frequency than in the 1D case, providing |
---|
237 | % a greater number of spurious single-channel spikes in a given |
---|
238 | % reconstructed spectrum. This effect will also be present in 3D |
---|
239 | % reconstructions, although to a lesser degree since information in the |
---|
240 | % spectral direction is also taken into account. |
---|
241 | % |
---|
242 | % If one chooses the reconstruction method, a further decision is |
---|
243 | % required on the signal-to-noise cutoff used in determining acceptable |
---|
244 | % wavelet coefficients. A larger value will remove more noise from the |
---|
245 | % cube, at the expense of losing fainter sources, while a smaller value |
---|
246 | % will include more noise, which may produce spurious detections, but |
---|
247 | % will be more sensitive to faint sources. Values of less than about |
---|
248 | % $3\sigma$ tend to not reduce the noise a great deal and can lead to |
---|
249 | % many spurious sources (this depends, of course on the cube itself). |
---|
250 | % |
---|
251 | % The smoothing options have less parameters to consider: basically just |
---|
252 | % the size of the smoothing function or kernel. Spectrally smoothing |
---|
253 | % with a Hanning filter of width 3 (the smallest possible) is very |
---|
254 | % efficient at removing spurious one-channel objects that may result |
---|
255 | % just from statistical fluctuations of the noise. One may want to use |
---|
256 | % larger widths or kernels of larger size to look for features of a |
---|
257 | % particular scale in the cube. |
---|
258 | % |
---|
259 | % When it comes to searching, the FDR method produces more reliable |
---|
260 | % results than simple sigma-clipping, particularly in the absence of |
---|
261 | % reconstruction. However, it does not work in exactly the way one |
---|
262 | % would expect for a given value of \texttt{alpha}. For instance, |
---|
263 | % setting fairly liberal values of \texttt{alpha} (say, 0.1) will often |
---|
264 | % lead to a much smaller fraction of false detections (\ie much less |
---|
265 | % than 10\%). This is the effect of the merging algorithms, that combine |
---|
266 | % the sources after the detection stage, and reject detections not |
---|
267 | % meeting the minimum pixel or channel requirements. It is thus better |
---|
268 | % to aim for larger \texttt{alpha} values than those derived from a |
---|
269 | % straight conversion of the desired false detection rate. |
---|
270 | % |
---|
271 | % If the FDR method is not used, caution is required when choosing the |
---|
272 | % S/N cutoff. Typical cubes have very large numbers of pixels, so even |
---|
273 | % an apparently large cutoff will still result in a not-insignificant |
---|
274 | % number of detections simply due to random fluctuations of the noise |
---|
275 | % background. For instance, a $4\sigma$ threshold on a cube of Gaussian |
---|
276 | % noise of size $100\times100\times1024$ will result in $\sim340$ |
---|
277 | % single-pixel detections. This is where the minimum channel and pixel |
---|
278 | % requirements are important in rejecting spurious detections. |
---|
279 | % |
---|
280 | % %Finally, as \duchamp is still undergoing development, there are some |
---|
281 | % %elements that are not fully developed. In particular, it is not as |
---|
282 | % %clever as I would like at avoiding interference. The ability to place |
---|
283 | % %requirements on the minimum number of channels and pixels partially |
---|
284 | % %circumvents this problem, but work is being done to make \duchamp |
---|
285 | % %smarter at rejecting signals that are clearly (to a human eye at |
---|
286 | % %least) interference. See the following section for further |
---|
287 | % %improvements that are planned. |
---|