Data Verification
The programs vsib_checker and fauto can be used to check data during
recording.
Vsib_checker does a number of simple checks on the data while fauto
computes the bandpass and plots it. The number of channels and
bandwidth are now written into the file headers. The common
options needed to describe the data format include:
-b/bits/
<nbits>
|
Number of bits of digitization.
Defaults to 2 (and higher precision is not supported yet)
|
-v/-vlba
|
Bit encoding is in vlba (offset
binary) rather than sign/magnitude. This is needed if the DAS "vsop"
mode was used.
|
vsib_checker
Vsib_checker has three main modes of operation. In PPS checking mode,
it checks the 1 PPS marker is set every second or returns an error.
This is a good way to check that recording is running smoothly. This is
the default mode of operation.
>
vsib_checker test_131_013622.lba # Check a
single file
> vsib_checker -fail vt01i1_131*
# Check multiple files at once. Stop running
if any problems are found
In sampler stats mode, the sampler statistics are calculated. By
default the data for the whole file is used, which is slow. The -n
<nbytes> option tells it to only check the first nbytes bytes.
However
this is rounds up to the internal buffer size (usually 2 MB). This
means running with -n 10 will check 2 MB of data, give an
accurate calculation and run fairly quickly. The sampler statistics
should be 17.3 % very high and very low and 32.7% for the rest.
>
vsib_checker -s test_131_013622.lba # Check
sample stats on a single file
> vsib_checker -s -vlba
test_131_013622.lba # Check sample stats on a single file,
assuming vlba encoding
> vsib_checker -n 100 -s test_131_013622.lba # Check single file, only
checking the first 2 MB of data
The third main mode for vsib_checker is to check data from the VSIC
countdown mode (mode 6). This is extremely useful way of testing the
VSIB and PC are working correctly (disks running fast enough etc).
Vsib_checker will check that each subsequent sample is numerically 1
less than the preceding sample. It reports an error if not. Generally
you will want to run with the -fail option as one error normally means
many errors will be reported (but not always for intermittent errors).
In countdown mode, vsib_checker uses a different definition of number
of bits compared to the other tests. When recording 256 Mbps
(vsib_record mode 3) the number of bits needs to be set to 8. For 512
Mbps rates (vsib_record mode 2), the number of bits must be set to 16.
>
vsib_checker -x -b 8 test_131_013622.lba #
Test
a single file
> vsib_checker -count
-b 8 test_131_013622.lba # The same test
> vsib_checker -x test_131_*
# Check multiple files
> vsib_checker -x -b 16 test_131_*
# Check data recorded in
vsib_record
mode 2 (16 bit/512 Mbps)
fauto
Fauto calculates the data bandpass and plots it using pgplot to either
postscript or the screen. It simply converts the digitized data to a
floating point number, FFTs and then averages the data stream. It can
do
a single 16 MHz channel in roughly realtime, but is computationally
very intensive. Fauto should not be run constantly during data
recording and probably not at all during 512 Mbps recording. The user
can choose which channels are processed, by default all are processed.
A zoom mode is available by setting the first and last channel to plot.
A mode is also available to calculate a series of integrations times
shorter than the entire length of the file, optionally pausing between
plots. Currently the number of FFTs to average needs to be specified
rather than the integration time in seconds (this will be changed when
time permits). Example usage (obviously options would normally be
combined):
>
fauto test_131_013622.lba
#
Calculate 4 bandpasses on a single file and plot to the screen
> fauto -v test_131_13622.lba
# Same but assume VLBA bit encoding
> fauto -n 1024 test_131_13622.lba
# Use a 1024 point FFT
(resulting in 512 spectral channels)
> fauto -C 1
test_131_13622.lba
# Only process the first
channel
> fauto -C 1 -C 3
test_131_13622.lba #
Process the first and third channel
> fauto test_131_13622.lba -dev plot.ps/cps # Save output as a
postscript file
> fauto
test_131_13622.lba -dev plot.png/png # Save output as a .png file
> fauto test_131_13622.lba -n
1024 -sp1 100 -sp2 300 # Plot a zoomed channel range
Pre-experiment testing
The following is a set of tests that are recommended before starting a
recording run (for example before the start of a VLBI
session).
VSIC
Countdown: Set the VSIC to mode 6 (0110). Run the recorder for
a few minutes (if time, it can be worthwhile running these tests for
many hours. However, you need to allow roughly the same amount of time
to test the data as record it). Check data is recorded, the PPS markers
are in place and run vsic_checker in countdown mode to check all the
data was recorded correctly. Generally you would want to run this at
512 Mbps (to stress the system the most), but some hardware setups may
not support 512 Mbps but are OK for 256 Mbps recording rates. To do a
512
Mbps test:
>
vsib_record -m 2 -t 100s # Record
data at 512 Mbps (16 bits per clock sample)
> vsib_checker test*
#
Check 1 PPS markers.
> vsib_checker -x test* # Check
each sample.
To do a 256 Mbps test:
> vsib_record -t 100s
# Record data at 256 Mbps (16 bits per clock sample)
> vsib_checker test*
# Check 1 PPS markers.
> vsib_checker -x test* # Check
each sample. Need to say 8 bit data.
Remember to set the VSIC back to mode E
(1110) after these tests
Data
quality: Set the VSIC back to mode E (1110), then record
another few minutes of data from the DAS (either directly from the DAS
or via the S2). Check the 1 PPS markers and sampler statistics are OK.
It is probably worth running this test with the VSIC directly connected
to the DAS and connected via the S2 and BG2 adapter.
When directly attached to the DAS:
> vsib_record -t 100s
# Record some data at 256 Mbps
> vsib_checker
test*
# Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test* # Check the sampler stats
are 17.3 and 32.7%
When directly attached via the S2,
enable data feed through in the S2 (
uic feedthru on).
> vsib_record -t 100s -c ooxx #
Record some data at 128 Mbps. The compress mode depends on the current
S2 mode
> vsib_checker
test*
# Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test* #
Check the sampler stats
are 17.3 and 32.7%
Bandpass:
Use fauto to check the bandpass. It is also useful to turn on a weak
test tone (much weaker than needed for coherence) a few MHz from the
band edge, to check the band inversion and sky frequency is as
expected. The same data as used for the data quality check can be used
>
fauto -n 512 test_131_135631.lba # For
direct recording modes
> fauto -n 512
test_131_135631.lba # For compressed
recording via the S2
> fauto -n 512 -vlba
test_131_135631.lba # If a DAS "vsop" mode was used
Testing during experiment
Data testing during an while recording is occurring is a good idea, but
should be kept to a minimum to avoid overloading the system. 1 PPS
checking should be fairly safe and checking sampler statistics on a
small fraction of the data also. fauto is computationally intensive so
generally should be avoided. vsib_record checks the 1PPS markers so
further checking is at most needed only occasionally. If these
checks are run, it is recommended to run the 1PPS checks on handful of
recent files and check only a small section of one file for sampler
statistics
> nice vsib_checker vt01i1_131_135*
#
Check 1PPS for 10 minutes of data
> nice vsib_checker -s -n 100 vt01i1_131_135620 # Check the sampler stats of a
single file