Version 2.5 ~~~~~~~~~~~ * Another 0.5ns jitter was found in Mark5B processing for data rates >= 2 Gbps. This is now fixed (2015 Feb 22 -- WFB) * Slightly more instructive messages when model files have disagreements with .calc files. * Check for time disagreements in polyco headers: the fractional MJD is supposed to be definitive, while the HHMMSS value can be rounded * Support .im files that contain some additional information (mpifxcorr now just ignores the info) * First bits of code for native Mark6 in place. This relies on some mark6 "gather" code in the vdifio library. * Add support for VDIF with multiple channels per thread and multiple threads. * Add support for complex VDIF with multiple threads * Add support for per-band data weights. Currently only implemented for Interlaced VDIF * Add support for fanned-out VDIF (one channel interleaved across more than one thread). This is relevant for some DDC3 modes. Untested as of 2015 Nov 17 * If arrarystride is zero, calculate a reasonable value for it - Allows arraystride to be set separately per datastream - This is of particular use when datastreams have discrepant bandwidths (e.g., ALMA's 62.5 MHz vs normal 2^n MHz ) * If xmacstride is zero, calculate a reasonable value for it based on output frequencies' channel counts * Introduce concept of rotatestridelen which is used for uvshifting. It is based purely on xmacstridelen, unlike previous implementation which also considered arraystridelen * If guardns is zero, set automatically. It likely will fail for any satellite observation. * checkmpifxcorr: option to set max message level * checkmpifxcorr: can operate on multiple input files * now require dirlist library for native Mark5 playback * Mark5 playback can now be based on new directory file format as documented in dirlist/doc/newdir.tex * Added support for unicast difxmessages. This (in current implementation) probably disables mpifxcorr control via difxmessages but outgoing messages work OK). The command thread used to receive DIFX messages is not started if DIFX_MESSAGE_GROUP is not set. Unicast will not work if multiple processes are run on the same node/computer. * Change to shutdown procedure which may reduce occurence of "DIFX hanging" bug * Promote some ints to long longs; fixes reporting of core memory use when lots of memory is needed. * Experimental support for VDIF data served over Shared Memory * For Mark5B and VDIF file-based correlation put file information into log stream * Adopted startdifx, genmachines.py, joblist, jobstatus, and jobdisks scripts from calcif2 as new competing delay model, difxcalc11, is coming in... * Improve accuracy of estimated memory usage (estimatedbytes) in VDIF datastreams * Peaceably ignore the new DELAY MODEL parameter of the .calc file * Calculate Mark5 module signature more consistently; the "NeoLegacy" formats used wrong number of bytes in the calculation * Change genmachines.py back to genmachines * Fix weights after a datastream dries up (developed after 2.5 branching) * Version for DiFX-2.5, May 16, 2017 * Fix extraction of VGOS and IVS 5/10 MHz pcal, but omit VDIFC fixes of 2.6.2 on purpose to retain 2.5.3 compatibility * Fix complex data PCal extractor to support other than 16 tones per band * Remove try/catch pair that fails with new openmpi (developed for 2.5.3) * Fix segfault on 'exiting gracefully' when a recording is not present * Fix genmachines lack of handling December doy's * Update genmachines to support Mark6 with multiple expansion chassis Version 2.4 ~~~~~~~~~~~ * Add INTERLACEDVDIF Fake mode * Change polId to recBandId in the PCAL files as per documentation * Add support for TCP, UDP and Raw Ethernet Interlaced VDIF. (TCP and UDP Not yet tested) * Test Mark5 XLRReadData() calls by tainting the end of the destination buffer and verifying it gets untainted. * Add more generalized support for the zoo of Mark5 module directory formats * When reading data from Mark5 units, verify that the data is in the same format class as requested by the .input file. * Improve Mark5B decoding * Fix Bonn Bug 1: on 64-bit systems sizeof(unsigned long) is 8 bytes, not 4 as it is on 32-bit systems. - some other variable types changed as well to minimize casting and prevent future similar problems. * Fix Bonn Bug 2: Lower sideband interacted badly with zoom bands * Emit Mark5 version information via cinfo on startup * PCAL: Pcal files generated are now consistent with http://cira.ivec.org/dokuwiki/lib/exe/fetch.php/difx/difxuserguide.pdf * PCAL: Write informational header in comments to top of PCAL files * Reduce sort window for VDIF to 32 frames to accomodate lower bandwidths. - Someday this should be tuned a bit better. * Adapt new call API of vdifio * VDIF: respect frame granularity when unpacking - Frame durations not integer nanoseconds in length were affected by a roundoff error. * Tagged for DiFX-2.4, 4 November 2014 Version 2.3 ~~~~~~~~~~~ * LO offset correction is now always done with fringe rotation (so in the time domain if fringe rotation is in the time domain). Allows larger LO offsets to be corrected without inducing decorrelation. * Working polarization dependent delay and phase offsets (from Chris) * Experimental linear2circular conversion (from Chris) * Complex Double sideband (RDBE/Xcube) sampling support (from Chris) * Explicitly set pthreads as joinable * Fix one aspect of non-power-of-two VDIF threads * Always close Mark5 device before freeing semaphore * Always set Mark5 into playback configuration * Move some Mark5 functions around to simplify things for the new VDIF code * Fully new Mark5B and VDIF datastream code for files and Mark5 units * Explicitly split up all Mark5 reads longer than 20MB * Never change Mark5 .dir files * Check Mark5 reads for lots of zero values and perform re-read if so * Support (maybe???) multi-threaded complex VDIF data * Bug fix: handle cases where the length in bytes of a first Mark5 scan is less than expected based on duration * (lightly tested?) support for single-thread Legacy VDIF data * fixed bug where send size could be computed incorrectly by 1-2 bytes for Mark4/VLBA/Mark5B/VDIF formats, potentially resulting in very small amounts of data loss * Tagged for DiFX-2.3, Dec 20, 2013 Version 2.2 ~~~~~~~~~~~ * Correct scaling of autocorrelations for LBA-format data * Optimized corner turners implemented for interlaced VDIF * Fixed a scaling bug for zoom band data * Allow building without IPP (needs fftw) [Thanks Richard Dodson!] * .pc files now include full compile-in info for ipp or fftw as appropriate * Check for a common .dir file anomoly caused by hand editing * Stop 0.5 ns delay wobble in 2 Gbps Mark5B format * c++11 compliance * Finally, kill off the MODE errors and Stale data errors for cases where subintegration time is between 1 and 2 seconds long. This enables 1 Hz resolution. * Use openmp to parallelize common corner turners for VDIF * Fix weights (and maybe more) for setups differing only by pcal setup (Adam) * Defeat data "echoes" that that would occur on each scan start after files ran out of data * Properly scale multicast weights when selecting bands or performing zooms * Avoid a rounding error which can lead to wrong scan choice when playing back nativemk5 VDIF data * Tagged as DiFX 2.2, May 31, 2013 Version 2.1 ~~~~~~~~~~~ * Version number change from 2.0.2 prior to tagging * Tagging Version 2.1, May 4, 2012 Version 2.0.2 ~~~~~~~~~~~~~ * Correctly scale cross-correlation amplitudes for pulsar binning when using Tsys>0 (accounts for varying # samples per bin c.f. nominal) * Correctly scale LBA format visibilities, reducing the size of ACCOR corrections * Fix bug when pulsar binning/gating and xmaclength > nfftchan * Add collection of drive statistics to native Mark5 * guardns was incorrectly (overzealously) calculated in mpifxcorr * "Mk5DataStream::calculateControlParams: bufferindex>=bufferbytes" bug fixed * Low weight reads could result in uninitialized memory * Streamstor XLRRead() bug work-around installed several places (read at position 0 before reading at position > 0) * mpifxcorr terminates correctly for all short jobs - previously it hung for jobs with a number of subints between nCores and 4 x nCores * multithread VDIF support enabled for nativemk5datastream (reading from a module) inclduing stripping of non-VDIF packets * Add "FAKE" as new datastream type. Results will be useless but benchmarking and bugfixing may benefit Version 2.0.1, 2011 June 12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Added new checkmpifxcorr utility * Updates to phase cal extraction * Added switched power detection (enable with TCAL FREQUENCY, optional line just above PHASE CAL INT (MHZ) ), currently only for mark5access based unpackers, and only works on 2-bit data. * Fix parsing of mark5 directory files: stepping through extra parameters had a bug. * Native Mark5: Swap order of fill pattern setting and bank mode setting to better work around certain module problems * Fix zoom-bands for all but special cases that just worked despite the logic in the code. (WFB paraphrasing for ATD) * Allow number of scans on a Mark5 module to exceed 1024 * Fix calculation of bytes from guardns Version 2.0.0, 2010 July 14 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Many bugfixes to both band matching and multiple phase centres * Visibilities now written out in binary format * New polyco parser and checker is more forgiving of TEMPO2 generated polycos * Can now produce kurtosis dumps every STA under the command of difxfilterbank * Many nativemk5 inmprovements migrated from 1.5 * Decorrelation correction is applied if uv shifting is done * Catch rare datastream error where file ends exactly at end of databuffer for a Mk5 files * Allow for new VEX FILE line in calc file to be optional * Bugfix for polycos with more than one entry * Many more minor bugfixes Version 2.0, 2009 September 24 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Many, many changes both visibly and under the hood * Allows support of multiple phase centres * Writes each pulsar bin/source to a different DiFX file * More efficient implementation of station-based processing, using complex multiplications to replace many sin/cos calculations (~20% speedup) * More cache-efficient implementation of baseline-based processing for correlations with many channels where whole output array can't fit in cache simultaneously (factor of ~2 speedup for >=2k channels) * Allows support of zoom bands * Replaces blockspersend with subintNS, guardBlocks with guardNS * Shifts numchannels/oversample/decimation etc from Configuration to Freq * Allows use of Rule table in input file * Adds support for LO offsets * mostly complete Mark5B support * Mark5B eVLBI support * Visibilities with zero weight are not written to disk * See http://cira.ivec.org/dokuwiki/doku.php/difx/documentation for more info on changes, particulaly for the file formats Version 1.5 Perth, 2008 March 7 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This version was born from the merge of the main trunk code and Walter's native-mark5 branch. In addition some new features were added: 1. Input file * HEADER O/RIDE is removed (never used) and replaced with VIS BUFFER LENGTH, which was a compile-time constant. Typical values for this number are upwards of 8. * In the configuration table, two additional parameters have been added after NUM CHANNELS: CHANNELS TO AVERAGE OVERSAMPLE FACTOR Currently both of these only accept value 1. * In the datastream table, the format and data source parameters have changed. The four following parameters are now used: DATA FORMAT QUANTIZATION BITS DATA FRAME SIZE DATA SOURCE 2. Difx format output * WEIGHTS WRITTEN replaced with DATA WEIGHT. The value is a float ranging from 0.0 to possibly a bit more than 1.0, but no more than 2.0; the average weight over many seconds of data should not exceed 1.0. * Exactly NUM CHANNELS of data are written out. For upper sideband data the first written channel is the DC channel and for lower sideband data the last written channel is the DC channel. The Nyquist channel is always dropped.