linefinder.find_lines significantly slower than in release 1.2

[george.kosugi@…]

The polynomial fit method in CASA/ASAP2.1a sometimes crashes by throwing exceptions of RuntimeError?. I'll attach several sdfits files and short python script which can re-generate errors in my environment. However, in ASAP1.2.1, which I used to use, no error occures. Could you please investigate it?

The other problem is the slowness of linefinder.find_lines, which is also used in my ALMA heuristic pipeline script. It became more than 4 times slower compared to ASAP1.2.1. Did anybody complained about it? I believe the robustness and accuracy got enhanced.... but no descriptions in the manual....

[george.kosugi@…]

sdfits for error case 1

[george.kosugi@…]

sdfits for error case 2

[george.kosugi@…]

sdfits for normal case

[george.kosugi@…]

short script

[george.kosugi@…]

Additional comments to scantable.poly_baseline: I also changed some of the data in the spectra, which caused crash on poly_baseline. If the change is small, i.e., 1/1000 or less, the result didn't change at all. I suspect that there may have some codes using insufficient precision for the calculation... However, I may be wrong....

[Malte Marquarding]

poly_baseline

I don't see a crash - it merely tells you

<snip>
RuntimeError: Function not yet fitted.

Fit didn't converge.

Secondly, you are fitting high order polynomials and when using the mask with a low number of channels. If you leave out the mask an order=11 fit gives quite a good result. (I only looked at the first sdfits file.)

I don't think this is a bug. A non-convergent fit is not something you can fix.

linefinder

Could you submit a sperate ticket please. It i s always good to have one per issue.

[Malte Marquarding]

Actually asap1.2 had a bug which didn't honour the mask (ticket:49). This is possibly the reason you didn't see this before.

[george.kosugi@…]

Thank you for your quick reply. It was my mistake to write it as 'crash', but actually it fails the fit. I think the number of channels (683) to be fit is enough for the order (9) of the polynomial for my sample case. I suppose the poly_baseline does something in addition to the simple polynomial fitting (i.e., rejection of lines or etc.). Am I right? If I need only a polynomial fit, should I use fitter class instead?

[Malte Marquarding]

No, the poly_baseline just wraps the fitter. auto_poly_baseline uses linefinder. I use the aips++ fitting class specifically NonLinearFitLM, which might not be necessary for polynomial fits (Linear fitting should be ok).

[george.kosugi@…]

OK, you are using Levenberg-Marquart method for the fit. If it's a one which included in the numerical recipe, it is not very robust. I understand that 'Fit didn't converge' means the iteration cycle get to the maximum number which you specified. Do you have any other methods in ASAP which provides linear LSQ fit for polynomial?

[Malte Marquarding]

I only just noticed that you might be using mask the wrong way. a mask in create_mask([100,200]) means that only the values [100,200] are used in the fit.

I have tried LinearFit?, but it doesn't actually make any difference. It will probably be faster though.

[george.kosugi@…]

The range of the mask in my attached script was right and it was what I intended. It'a a part of the baseline fit heuristics pipeline script, which divides spectrum into several to avoid oscillation of the fit at distant channels due to the limited precision of the power calculation.

Thank you for trying Linear Fit. What does "not actually make any difference" mean? Does it mean "not converged"? Do you have any method providing linear fit in ASAP? IF so, I'm pleased to use it in my script. A linear fit code I tentatively wrote in python was much slower than poly_baseline. Therefore, I want to replace.

[Malte Marquarding]

"not actually make any difference" means the resulting fit wasn't any better. By eye it didn't look any different.

I have tried conditional fitter slection depending on the function to be fitted. Unfortunately, ther seems to be a bug int the underlying code that prevents me form having this in the next ASAP stable release. I can send you a patch if you wan't to build it yourself.

[george.kosugi@…]

Thanks a lot. I'll wait for the release which includes linear fit. I hope it'll take not so long.

[Malte Marquarding]

Maxim is investigating this.

[Malte Marquarding]

We have tracke this down. linefinder.find_lines executes the fitter internally and the time this takes depends on the number of channels supplied. ASAP1.2 wasn't doing a proper job and only supplying a small number of channels to the fitter.

The next minor release 2.1.1 will include an option to tune the width of the lines to be searched for. If one only expects narrow lines the speed up should be significant.

[Malte Marquarding]

Actually, after speking to Maxim. linefinder already has this option set_options(avg_limit=8). If you set avg_limit=1 you get a significant speed up.

[Stephen]

hello gyus! ​http://hometown.aol.com/skylineblog/index.html sorry ;)

[Robert]

lol :) ​http://hometown.aol.com/hoodianp thx!

[Anthony]

hi! ​http://toyota-engine-info.blogspot.com/ ​http://kia-engine-info.blogspot.com/ ​http://nissan-engine.blogspot.com/

[James]

Hi! ​http://blogshot.nl/hoodiadietpill ​http://blogshot.nl/kiacars ​http://blogshot.nl/kiachicago