SDFITS export import doesn't keep flags.

[Malte Marquarding]

SDFITS does not preserve flag data

Hi there, I am writing out ASAP data in SDFITS format and have noticed that flagged scans or cycles become unflagged when read back into ASAP. ie. the process of writing to SDFITS file appears to delete flag information.

If I save in ASAP format, the flags are preserved. But I need to write into SDFITS format because the data then go through livedata/gridzilla.

Later. Andrew xxx

[Malte Marquarding]

Andrew, I can't reproduce this in a simple test.

#!/usr/bin/env python                                                         
from asap import *
s = scantable("data/mopra_dualif.rpf")
q = s.auto_quotient()
m = q.create_mask([0,100])
q.flag(m)
flags = q.get_mask(0)[95:105]
q.save('t.sdfits', 'SDFITS', True)
q2 = scantable('t.sdfits')
flags2 = q2.get_mask(0)[95:105]
assert flags == flags2

[Malte Marquarding]

Does the data contain crosspols?

[Andrew Walsh <andrew.walsh@…>]

Hi Malte, the crosspols are removed when reading the raw data in livedata, which is then exported to sdfits, to be read in by ASAP. So I think the answer is no, there's no crosspols.

As for reproducing, I have looked into this in more detail and am sorry to say I misled you in saying that no flags are saved when writing out to SDFITS. As far as I can tell, flags made through ASAP do appear to be saved.

HOWEVER, there is still a problem with ASAP not saving flags that are generated through livedata. The following includes a few commands that demonstrate this:

s=scantable('2012-05-13_1506-P817_G335.0-1.0_lon_dfb4.sdfits',average=False) sel=selector() sel.set_scans(1) sel.set_cycles(0) s.set_selection(sel) s.save('ttt.sdfits',format='SDFITS') q=scantable('ttt.sdfits',average=False) q.set_selection(sel)

The original sdfits file is created from livedata and you can find a copy at: kaputar:/DATA/KAPUTAR_3/wal645/sdfits/2012-05-13_1506-P817_G335.0-1.0_lon_dfb4.sdfits

When running livedata before creating this file, it flags out cycles that were taken when the telescope was not on source. The first cycle in the second scan is an example of this. If you go through the above commands, you'll see that the first attempt to set the selection (on 's') fails, but works on the second attempt (on 'q'), so the flag on this cycle was not preserved.

The reason I am using livedata at the start is because it:

1. removes cross polarisations
2. flags out all cycles that were taken not on source.

There must be some information in the rpfits file that says which cycles are bad data. So it might be possible to eliminate the livedata step all together, if the problem with ASAP reading crosspols is resolved, and if I knew how to identify cycles where the telescope was not on source and flag them out. I am just saying this in case you know of a way to flag out the not-on-source cycles. If so, then once we have a way to do away with the crosspol data, I can start from scratch with ASAP.

[Malte Marquarding]

Andrew,

this is not a problem of flagging. I don't know if your version of asap has scantable.getcyclenos() but if it does you can see that the the cycle numbers don't start at 0 (from the original). On export cycle numbers ( as well as scan numbers) get reset so will start at 0.

There was a decision made long ago that these numbers are transient, so would be reset. I can see that this is inconsistent to the saving to asap format but I don't think will change.

So in short, provided your version has getcyclenos ( and similar for beams/ifs/pols/scans) you should never iterate over the "index" but over the value itself, e.g.

s= scantable()
scanno = s.getscannos()
# here for examle this is [2,7,8,9]
s.set_selection(scans=scanno[0])

This makes your code more portable.

[Malte Marquarding]

For avoiding livedata:

1) I can implement this quite easily. In the meantime you can just ignore those or use

s.set_selection(pols=[0,1])

The data will sill report npol=4 but it should be ok.

2) Do you know how livedata identifies 'off source'? Again could be implemented if I know what it should do.

[Malte Marquarding]

I have implemented dropping of xpols as scantable.drop_xpols() in r2789

[Andrew Walsh <andrew.walsh@…>]

It appears that the rpfits data does have a mask saved with the spectra. So I have successfully managed to flag out the bad data based on this mask in ASAP. The remaining problem for me is just the xpols.

I have tried something like s.set_selection(pols=[0,1]) which does work when you look at a summary of the scantable. However, if I then try and process the scantable further, I get a segmentation fault or other nasty errors, suggesting to me that the xpols are still mucking things up somehow. I would dearly like to try your new fix of scantable.drop_xpols() but don't know how this gets installed/updated (I tried asap_update_data but saw no change).

I've tracked down the source of the segfault: At some stage in my code I am averaging polarisations. This appears to work fine for cycles where there is good data, but on cycles where the data have previously been flagged, I get the segmentation fault, or a whole bunch of python lib errors, starting something like:

* glibc detected * python: free(): invalid next size (fast): 0x000000000902c840 * ======= Backtrace: ========= /lib/libc.so.6(+0x71e16)[0x7f82fb2b8e16] /lib/libc.so.6(cfree+0x6c)[0x7f82fb2bdb8c]

etc etc.

[Malte Marquarding]

If you are using kaputar I can try and arrange an update of asap tomorrow. WIll need to talk to Nathan though.

Is the segmentation fault in average_pol" ?

[Malte Marquarding]

I have installed the new asap onto my desktop machine 'brage.atnf.csiro.au' If you log in you can create a directory in '/var/tmp/' and work in there.

[Andrew Walsh <andrew.walsh@…>]

Yes, the segmentation fault occurs on average_pol() I suspect it also occurs on average_time() as well. This happens when it tries to average data with flagged cycles.

Now I know where the problem is occurring, I have managed to get around it by not averaging polarisations. I was doing this to make a dummy scantable, so it was easy for me to make it another way. So for me at least, this does not need to be addressed straight away, but probably does need attention at some point since it looks like an important bug.

[Malte Marquarding]

I have updated kaputar. drop_xpol() should be available now. This hopefully solves the segfault,