\documentclass[12pt]{article}

% Notes to potential editors:
% 1. Please don't change the line wrapping.  Exactly one sentence per line!
% 2. Update "date" and "version" below with each update
% 3. Notation : 
%       Program names and other text to be typed by user or returned by the computer in {\tt }
%       Variables or arguments in {\em } or in $<$ $>$


\usepackage[margin=0.75in,twoside]{geometry}
\usepackage{graphics}
\usepackage{float}
\usepackage{color}
\definecolor{darkblue}{rgb}{0,0.2,0.4}
\usepackage[colorlinks,linkcolor=darkblue,citecolor=blue,urlcolor=blue,pdftitle={How vex2difx works},pdfauthor={Walter Brisken}]{hyperref}

\floatstyle{ruled}
\newfloat{code}{thp}{lop}
\floatname{code}{Code}

\begin{document}

\newcommand{\vexdifx}{{\tt vex2difx} }
\newcommand{\vd}{{\tt .v2d} }
\newcommand{\vx}{{\tt .vex} }
\newcommand{\defname}{{\em def name} }
\newcommand{\fs}{{\tt /}}
\newcommand{\hy}{{\tt -}}
\newcommand{\us}{{\tt \_}}

\begin{center}
{\Large How does vex2difx work?}

\vspace{10pt}
Walter Brisken

\vspace{10pt}
{\em draft} 2016/05/15
\end{center}


\section{Introduction}

This document was produced to demystify the inner workings of \vexdifx.
The version of \vexdifx covered here is that which will become part of the DiFX 2.5 release series.
This version has had considerable changes since the most recent stable version (2.4.0), most of which are related to support for multiple datastreams per antenna and changes to the \vx file parser.

\section{Sources of information}

\vexdifx explicitly takes a single file as its input, the \vd file.
This file must contain a reference to a \vx file (using the {\tt vex} parameter.
Contents of the {\tt .vex} file will be used unless overridden in the {\tt .v2d} file.
Additional optional files referenced by the {\tt .v2d} file will be discussed as needed.
Some bits of information (e.g., names of files to correlate) can only be provided by the \vd file, and some information, such as frequency setups, can only be provided by the \vx file.
Little attention will be paid to these cases where the source of information is unambiguous.
The \vexdifx wiki page and vex documentation should be sufficient to understand these cases.
The more complicated options are those where settings in the \vd file are used to override those in the \vx file; that will be the primary focus of this document.

Note that as of now the only version of the vex format that is supported is vex 1.5\footnote{See \url{http://vlbi.org/vex/docs/vex\%20definition\%2015b1.pdf}.}.
This version of the standard itself is not fully capable of conveying many details of modern VLBI systems, including the VDIF format, some recording systems, and all eVLBI.
The lack of unambiguous support by vex 1.5 means that \vexdifx had to adopt some conventions and make some assumptions.
\vexdifx aims to warn the user when assumptions are being made.

Some of this information can be found on the DiFX wiki at \url{http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/vex2difx}.

\section{Output files}

\vexdifx generates three output files for each job created.
See Sec.~\ref{sec:break} for information on how \vexdifx breaks an observation into multiple jobs.
The {\tt .input} file is the master file for each DiFX job.
It is passed to {\tt mpifxcorr} and has reference to other files.
The {\tt .calc} file is primarily used as input to generate the delay model {\tt .im} file that {\tt mpifxcorr} requires.
Finally a {\tt .flag} file is written.
This file is used to mask data that may be inadvertently correlated but that is irrelevant to the particular job.
This is important espeically when the observing antenna array is broken into subarrays, each observing different sources.
It is also important in cases where an observation is interrupted but where data for the interrupting observation is accessible to the correlator.
See the DiFX reference manual \url{...} for more information.

\section{Details}

The sections below detail how various bits of information get conveyed to the DiFX input files.

\subsection{Frequency tunings}

At the moment the only way to specify frequency tunings is in the \vx file within a \$FREQ block.
Please refer to vex 1.5 documentation for details.

\subsection{Data format}

The data format is probably the most confusing and complicated parameter.
The format, in the context of DiFX, consists of a number of more elementary parts, some of which are not applicable to every format, that when used together completely determine the structure of the baseband data.
Things are further complicated by the fact that \vexdifx allows some of these parts to be determined by the \vx file and some to be set by the \vd file.
Various parts of the format include
\begin{itemize}
\item {\em class}: VDIF, Mark5B, VLBA, Mark4, and a few other variants and lesser used options.
\item {\em bits}: number of bits per sample, often equal to 1 or 2.
\item {\em chans}: the number of baseband channels has an impact on the layout of data so is considered part of the data format.  Note that not all of the channels need be correlated, but DiFX needs to know which channels are present in any case.  See more on this in Sec.~\ref{sec:channels}
\item {\em fanout}: used only in older formats (VLBA and Mark4).  From the DiFX perspective this describes the internal ordering of channel data in a data stream.
\item {\em threads}: currently used only for the VDIF format.  The thread is a numeric identifier for a channel set within a VDIF stream.  One or more channel can be contained in each thread.
\item {\em sampling type}: in most cases today this is real.  Two forms of complex sampling (single-sideband and double-sideband) are also supported, but only for VDIF format.
\item {\em size}: the size in bytes of one frame of data (currently required only for VDIF).  Unless otherwise noted, {\em size} includes the framing overhead.
\item {\em rate}: the data rate (excluding framing overhead), in Mbps.
\end{itemize}

\noindent
Each of these parts will be discussed in more detail below.

The \vx file does not provide all of this information in a single place, but instead it can (often) be constructed by combining information from various places in the \vx file, but mostly from the {\tt \$TRACKS} block.
Typically some parts of the format will be specified by the {\tt track\_frame\_format} parameter in the {\tt \$TRACKS} block.
\vexdifx is able to parse a wide variety of values for this parameter, most of which contain more information than just {\tt format class}.
The wide variety of syntaxes that are supported is in response to a wide variety of specifications and informal usage that has accumulated over the years.
Fortunately, there are no known cases where ambiguity arises.
The complete list of possible values is shown in Table~\ref{tab:format}.

\begin{table}
\begin{center}
\caption{
Allowed format specifiers.
The numbering is consistent with internal details of the format matching code in \vexdifx.
The {\em class} parameter is case insensitive.
Certain shorthands are allowed: {\tt MARK5B} format can be written as {\tt MK5B} and {\tt MARK4} can be written as {\tt MKIV}.
See Table~\ref{tab:vdifclass} for specifying variations of the {\tt VDIF} format class.
}
\label{tab:format}
\begin{tabular}{llll}
\# & Syntax & Example & Notes \\
\hline
0 & {\em class} & {\tt MARK5B} & \\
1a & {\em class}\fs{\em threads}\fs{\em size}\fs{\em bits} & {\tt VDIF/0:1:2:3/5032/2} & In {\tt .v2d}; for VDIF formats only. \\
1b & {\em class}\fs{\em threads}\fs{\em size}\fs{\em bits} & {\tt VDIF/0,1,2,3/5032/2} & In {\tt .vex}; for VDIF formats only. \\
2 & {\em class}\fs{\em size}\fs{\em bits} & {\tt VDIF/5032/2} & For VDIF formats only. \\
3 & {\em class} {\em size} & {\tt VDIF5032} & This syntax is discouraged. \\
4 & {\em class}{\tt 1\_}{\em fanout} & {\tt VLBA1\_4} & For VLBA and Mark4 formats only. \\
5 & {\em class}\us{\em size}\hy{\em rate}\hy{\em chans}\hy{\em bits} & {\tt VDIF\_5000-2048-16-2} & VDIF only. \\
  & & & {\em size} here excludes frame headers. \\
6 & {\em class}\hy{\em rate}\hy{\em chans}\hy{\em bits} & {\tt MARK5B-1024-8-2} & \\
7 & {\em class}{\tt 1\_}{\em fanout}\hy{\em rate}\hy{\em chans}\hy{\em bits} & {\tt VLBA1\_4-512-8-2} & For VLBA and Mark4 formats only. \\
8a & {\em class}\fs{\em bits} & {\tt MARK5B/2} & Assumes {\em bits} $\le 32$.  Use is discouraged. \\
8b & {\em class}\fs{\em size} & {\tt VDIF/5032} & Assumes {\em size} $> 32$.  Use is discouraged. \\
8c & {\em class}\hy{\em bits} & {\tt MARK5B-2} & Assumes {\em bits} $\le 32$.  Use is discouraged. \\
8d & {\em class}\hy{\em size} & {\tt VDIF-5032} & Assumes {\em size} $> 32$.  Use is discouraged. \\
\end{tabular}
\end{center}
\end{table}

\begin{table}
\begin{center}
\caption{
Supported classes for various VDIF formats.
}
\label{tab:vdifclass}
\begin{tabular}{p{3.5cm}p{12cm}}
Class & Description \\
\hline
{\tt VDIF} & Can describe any non-legacy VDIF.  Usually defaults to single-thread unless threads are explicitly defined.  See note on canonical threads (Sec.~\ref{sec:canonicalthreads}). \\
{\tt VDIFL} & Specify Legacy VDIF.  DiFX supports legacy VDIF only in single-thread cases. \\
{\tt VDIFC} & Same as {\tt VDIF}, but specifies single-sideband complex sampling. \\
{\tt VDIFD} & Same as {\tt VDIF}, but specifies double-sideband complex sampling. \\
{\tt INTERLACEDVDIF} & Same as {\tt VDIF}, but explicitly forces multiple threads.  The list of threads must be provided through some means. \\
{\tt INTERLACEDVDIFC} & Same as {\tt INTERLACEDVDIF}, but for single-sideband complex sampling. \\
{\tt INTERLACEDVDIFD} & Same as {\tt INTERLACEDVDIF}, but for double-sideband complex sampling. \\
\end{tabular}
\end{center}
\end{table}

Note that behavior is undefined if the \vx file contains conflicting information regarding the format of data.

The subsections below dictate how the various format parts are specified.
Everything in this section is relevant both for single and multiple datastreams per channel.
For details on configuring multiple datastreams, see Sec.~\ref{sec:mds}.

\subsubsection{Format class}

Format class can only be specified in two places: the {\tt track\_frame\_format} parameter in the \vx file's {\tt \$TRACKS} section and the {\tt format} parameter in the \vd file (which can live either in an {\tt ANTENNA} or {\tt DATASTREAM} section).
Collectively these parameters will be called ``format parameters''.
At least one file must provide this information.
If the information is provided in both files, the \vd file will override.
Both the \vx and \vd parameters accept the same set of possible values as enumerated in Table~\ref{tab:format}.
The sources of format class, listed in increasing priority, are:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}

Note: it is not always possible to change the format class via the {\tt .v2d} file.
Specifically, at the moment, it is not possible to change between VDIF and any other format.
This is because VDIF determines channel order strictly from the {\tt \$FREQ} section based on channel names and the other formats use the {\tt \$TRACKS} section and the parsing of the channel order is done while reading the {\tt .vex} file, before the contents of the {\tt .v2d} file are considered.

\subsubsection{Number of bits}

The number of bits per sample can be specified in the format parameters.
However, if the \vx file has tracks defined in the {\tt \$TRACKS} block, the number of bits is determined by the absense or presense of magnitude tracks: if a magnitude track is found, 2 bits is assumed, otherwise 1 bit is used.
Any format parameter in the \vd file containing number of bits will override that in the \vx file, however it is deduced.
These sources of bits per sample, listed in increasing priority, are:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item Existence of magnitude tracks listed in {\tt fanout\_def} statments in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}

\subsubsection{Number of channels}

The number of recorded baseband channels can be specified in the format parameters.
There are two additional sources of channel count within the \vx file: the {\tt format\_def} statements in {\tt \$TRACKS} section and the {\tt chan\_def} statements in the {\tt \$FREQ} block. 
These sources of channel count, listed in increasing priority, are:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item Number of {\tt chan\_def} entries in a {\tt \$FREQ} block of the \vx file
\item Number of sign tracks listed in {\tt fanout\_def} statments in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}

\subsubsection{Fanout}

Fanout is relevant only to VLBA and Mark4 formats, and some other very closely related formats\footnote{E.g., Mark3 and the so-called {\tt VLBN} format, which is equivalent to the VLBA format but without modulation.}

Fanout can be set three ways.
The format parameters cover two of these (by use of syntax numbers 4 or 7 in Table~\ref{tab:format}).
The repetition of the same bit of the same channel in the {\tt \$TRACKS} table is the third mechanism.
The VLBA and Mark4 formats as implemented within DiFX require that the fanout be the same for all channels and that fanout groups (sets of tracks used for one channel) use consecutive track numbers.
The sources of fanout, listed in increasing priority, are:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item Multiplicity of tracks for the same channel/bit in the {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}

\subsubsection{Threads}

Threads are relevant only to multi-threaded VDIF format.
With one exception (see Sec.~\ref{sec:canonicalthreads}) an explicit list of threads must be provided throug a format parameter for any multi-threaded VDIF format using syntax number 1 of Table~\ref{tab:format}.
The sources of thread information, listed in increasing priority, are:
\begin{enumerate}
\item Canonical thread values (see Sec.~\ref{sec:canonicalthreads})
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}
{\em Important note:} The separator used for the list of threads depends on which file the format is contained within.
For {\tt .vex} files the separator must be commas, and for the {\tt .v2d} files the separator must be colons.
Swapping the two interferes with the lexical rules for each of the two file types.

\subsubsection{Sample type}

Sample type describes the representation of data samples.
Three kinds are supported: real, single-sideband complex and double-sideband complex.
The choice of sampling type is independent of choice of upper- or lower-sideband, which is not considered here.
This can be overridden either in the \vx or \vd files.
Sampling type always defaults to real.
There are several ways to change to one of the complex types:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt sampling} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\item {\tt sampling} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}
Note that unlike for most parameters, a priority system does not strictly determine the final value of sampling.
Instead any indicator of non-real sampling will override a lower priority real value, but a real setting cannot override a complex value.

\subsubsection{Data frame size}

Data frame size is completely determined based on other data for all formats except for VDIF.
For VDIF the size must be explicitly provided.
The only means to provide frame size are through format parameters.
The sources, listed in increasing priority, are:
\begin{enumerate}
\item {\tt track\_frame\_format} statement in a {\tt \$TRACKS} block of the \vx file
\item {\tt format} statement in a {\tt ANTENNA} section of the \vd file
\item {\tt format} statement in a {\tt DATASTREAM} section of the \vd file
\end{enumerate}
\noindent
Note! For historical reasons some format syntaxes only consider the size of the data portion of a frame, while others specify the full frame size.
See comments in Table~\ref{tab:format} for details.

\subsubsection{Data rate}

Data rate is completely determined based on other parameters and the contents of the \vx {\tt \$FREQ} block.
The format parameters can include data rate, but in all cases it is ignored.
It is mentioned here only because some of the historical data format syntaxes include a data rate field.

\subsection{Special topics related to data format}

The following sections provide details on some special cases of format specification.

\subsubsection{Canonical VDIF threads} \label{sec:canonicalthreads}

The VLBA introduced the concept of ``canonical thread numbering'' as a thread numbering convention that allows thread numbers to be deduced purely by the contents of the {\$FREQ} block of a \vx file.
The numbering scheme works as follows:
\begin{enumerate}
\item Sort the baseband channels specified in {\tt chan\_def} statements in a {\tt \$FREQ} block in alphabetical order.
\item The channels are then given sequential integer thread numbers starting with 0 for the first channel alphabetically listed.
\end{enumerate}
\vexdifx only assigns canonical thread numbers if no other source of thread information is present {\em and} they are from one of the antennas known to produce channels in canonical order.
By default, these antennas are those of the VLBA, GBT (GB), and VLA (Y).
The set of antennas making use of canonical thread numbers can be changed through use of environment variable {\tt CANONICAL\_VDIF\_USERS}.

\subsubsection{Format overriding with multiple modes}

In general, format settings made within the \vd file apply to all modes.
It is not possible to explicitly set format parameters separately for each mode, but careful omission of parameters in the \vd file can allow the mode-dependent values from the \vx file to remain in control.
Essentially any format parameter that must change with mode must not be set in the \vd file.

\subsubsection{Specifying multi-channel multi-thread VDIF}

This mode is not yet supported, but will make use of the fact that the number of channels and number of threads is independently specified.
The number of channels per thread will be determined as the ratio of number of channels to number of specified threads, which must be an exact power of 2.
Details of channel ordering in this case are not yet established.

\subsection{Channel ordering} \label{sec:channels}

The \vx file is the only place where ordering of channels can be specified.
There are two mechanisms:
\begin{itemize}
\item If {\tt fanout\_def} statements are present in a {\tt \$TRACKS} block, the ordering, by track number, is used to specify the channel ordering.
\item If no {\tt fanout\_def} statements are present, the channels as specified by {\tt chan\_def} statements in the {\tt \$FREQ} table are ordered alphabetically by the name of the channels.
\end{itemize}


\subsection{Antenna properties}

Note that within the DiFX ecosystem three terms, ``station'', ``antenna'', and ``telescope'' are all used more or less interchangably.
Within the vex format itself there is a distinction between ``station'' and ``antenna'', and there is no concept of ``telescope''.

\subsubsection{Antenna name}

By default the antenna name assigned by \vexdifx is the \defname of the corresponding entry in the {\tt \$STATION} block of the \vx file, promoted to all capital letters.
Antenna names can be overridden in the \vd file within a corresponding {\tt ANTENNA} section through the use of the {\tt name} parameter.




\subsubsection{Antenna coordinates}

Antenna coordinates in the ITRF are usually provided in the {\tt vx} file.
They can also be provided in an {\tt ANTENNA} section with the {\tt X}, {\tt Y}, and {\tt Z} parameters.
Note that if one of these three parameters is specified the other two must be as well.
The values in the \vd file will override those in the \vx file if provided.


\subsubsection{Axis offsets}

At this time the antenna axis offsets can only be supplied by the \vx file.

\subsubsection{Clock models}

Specification of the clock model is somewhat complicated due to numerous options available.
The convention used by vex is that the clock offset is positive if the Data Acquisition System (DAS) time tick is early (i.e., the station clock is running fast) and accordingly the full name in the vex file is {\em clock\_early}.
In all other parts of the system the opposite sign convention is used, that is the clock offset is how late (slow) the DAS clock is.
Thus there is the added confusion that clock information provided by the \vd file has the opposite sign as equivalent data in the \vx file.
Internally within \vexdifx and the {\tt .input} file used by {\tt mpifxcorr} the clock model is represented by a 5 term polynomial and a refrence epoch.

The \vx file can provide multiple clock models per antenna through a {\tt clock\_early} statement, each with four parameters:
\begin{itemize}
\item {\em Start} : the time at which the clock model becomes valid (implicitly overriding all previous models for that antenna).
\item {\em Offset} : the amount the timestamp on data is early relative to actual time.
\item {\em Epoch} : the time at which the value of {\em Offset} was the actual clock offset.
\item {\em Rate} : the time derivative of {\em Offset}.
\end{itemize}
Note that the {\em Rate} value is a dimensionless number.
Some debate centered on the appropriate units to assume: sec/sec and microsec/sec were both considered.
DiFX assumes the units are sec/sec.
Vex2 will more clearly specify this and also provide the option to make this explicit by supporting dimensionless units sec/sec and microsec/sec.
If clock information is provided in both the \vx and \vd files the values in \vx will be considered first with the \vd parameters either adding to or replacing those in \vx.

The \vd file can provide partial, full, or incremental clock information through the following parameters which can be set separately for each antenna in the {\tt ANTENNA} sections:
\begin{itemize}
\item {\tt clockEpoch} : the time (expressd as an MJD, in vex time, or in any other format known to \vexdifx) when the zero-order term of the full clock polynomial was the actual clock offset.
\item {\tt clockOffset} (or {\tt clock0}) : the clock offset (in microseconds).  It is highly recommended that {\tt clockRate} and {\tt clockEpoch} also be provided if setting this.
\item {\tt clockRate} (or {\tt clock1}) : the clock rate (in microseconds per second); {\tt clockOffset} and {\tt clockEpoch} must also be set if this is used.
\item {\tt clock2} \ldots {\tt clock5} : the higher order clock polynomial coefficients.  Units for {\tt clock}$n$ are microseconds/sec$^n$.
\item {\tt deltaClock} : a simple mechanism to add a correction to clock value in the \vx file.  Remember that the sign has opposite sense as compared to the \vx file.
\item {\tt deltaClockRate} : a simple mechanism to add a correction to the clock rate value.  Note that if the clock epoch is very different than the experiment observation time this will also affect the clock offset at the start of the experiment, so it is not recommended to use this parameter unless the clock epoch occurs sometime during the experiment.
\end{itemize}
The \vx file can only provide a clock offset and a rate.
The \vd file can contain a polynomial with up to 5 terms.
The vex2 format will allow higher order clock models.


%\subsubsection{Ephemeris information}




\subsection{Specifying baseband data} \label{sec:data}

Specifying data is actually fairly complicated in the case of multiple data streams but is easily described.
The following description of which data is used and which data stream it gets assigned to is fully general.
Each data stream has a data source parameter which is equal to one of the following:
\begin{itemize}
\item {\tt None}: No data is to be associated with this datastream.  Such datastreams never make their way into the {\tt .input} files.
\item {\tt Fake}: Data will be generated internally.  The structure of the data should not be assumed to be of any particular form.
\item {\tt Module} (or {\tt Mark5}): Data is on a Mark5 module.  The associated ``filename'' must be a legal Mark5 Volume Serial Number (VSN).
\item {\tt Mark6}: Data is on a Mark6 module.  The name should either be a legal Mark6 Module Serial Number (MSN) or a filename corresponding to a recorded file.
\item {\tt File}: Data is stored in one or more file(s) in the Unix filesystem.
\item {\tt Network}: Enough network parameters must be defined to set up a socket to receive the relevant baseband data from an external source.
\end{itemize}
The data source can usually be left implicitly defined, but if it needs to be forced (such as in the case of fake data) it can be with the {\tt source} parameter in either the {\tt ANTENNA} section (if no explicit datastreams are defined) or in the {\tt DATASTREAM} section.

There are three possilbe places to specify baseband data: The {\tt TAPELOG\_OBS} section of a {\tt .vex} file, or either the {\tt ANTENNA} or {\tt DATASTREAM} section of a {\tt .v2d} file.
If one or more {\tt DATASTREAM} sections are defined for an antenna, data should not be specified in the corresponding {\tt ANTENNA} section.  
Data specified in the {\tt .v2d} file will take precedence over data in the {\tt .vex} file.


\subsection{Multiple datastreams} \label{sec:mds}

A datastream in the DiFX context is a logical stream of data containing data from one or more baseband channels.
A single file or a sequence of files that do not overlap in time could be a datastream.  A datasteam could also be recordings on a Mark5 module or a network socket from an eVLBI source.
Use of multiple datastreams becomes important in heterogeneous VLBI arrays where one or more antenna records data onto two records simultanously.
With the advent of very wideband backends (e.g., DBBC3, ALMA, other EHT telescopes, and VGOS) the use of multiple datastreams will become increasing important.

New with the DiFX 2.5 version of \vexdifx is support for multiple datastreams per antenna.
Support for this has always been present in {\tt mpifxcorr} but many of the support programs, such as \vexdifx let such capability lapse.
Support for this in \vexdifx required substantial restructuring of its code but has left a relatively simple interface for the user.

Currently datastreams can only be defined in the \vd file.
This is done via a {\tt DATASTREAM} structure.
If none of these are present then it is assumed that every antenna makes use of a single datastream.
For antennas with multiple datastreams the following \vd file additions are needed:
\begin{enumerate}
\item A {\tt datastreams} parameter needs to be set in the antenna's {\tt ANTENNA} section.
The argument is an ordered list of references to \vd {\tt DATASTREAM} sections.
For example: \\

{\tt ANTENNA PT \{ datastreams = PT1,PT2 \}} \\

\item One or more {\tt DATASTREAM} sections must be added.
An example:\\

{\tt DATASTREAM PT1 \{ format = VDIF/0:1:2:3/5032/2 vsn = NRAO+001 \} } \\

{\tt DATASTREAM PT2 \{ format = VDIF/4:5:6:7/5032/2 vsn = NRAO+002 \} } \\

\end{enumerate}

The contents of a {\tt DATASTREAM} section more or less mirror the data related parameters of the {\tt ANTENNA} section such as {\tt format}, {\tt file},  {\tt vsn}, {\tt filelist}, {\tt networkPort}.
A complete list is maintained on the DiFX wiki\footnote{See \url{http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/vex2difx}.}.

While it may not always be required it is strongly encouraged that the full data format for a datastream be fully described in the {\tt DATASTREAM} section, including explicit thread Ids for VDIF.

The \vx file explicitly states the number of baseband channels (called ``bands'' in DiFX parlance); call this number $n_{\rm band}$.
By default these are divided equally between the $n_{\rm datastream}$ datastreams listed in the {\tt ANTENNA} section with the first $n_{\rm band}/n_{\rm datastream}$ being assigned to the first listed datastream, and subsequent blocks of $n_{\rm band}/n_{\rm datastream}$ being divvied out to the other datastreams in the order listed.
It is possible to explicitly indicate how many bands are associated with a datastream with the {\tt nBand} parameter.
This would allow the assumption of equal distribution to be broken if need be.
It is, however, discouraged to make use of this parameter unless absolutely necessary as it causes problems in cases where the number of bands for a particular antenna changes through an experiment.
It may be necessary to use multiple \vd files in complicated cases.

Optionally each {\tt DATASTREAM} section can include a {\tt machine} parameter which can be used to assign a particular cluster node to host the {\tt mpifxcorr} datastream process.
Note that when this parameter is used (and presumably the global {\tt machines} parameter is used as well to specify hosts of manager and core processes), {\tt genmachines} should not be run when preparing to start the job. 
Instead, {\tt startdifx} can be used directly with the {\tt -n} option.
The case where some machines are determined through these {\tt machine}({\tt s}) parameters is currently not supported but could be in the future.


Multiple datastreams can be implicitly triggered through use of multiple recorders in use at a given station.
This would be specified in the {\tt TAPELOG\_OBS} section of a {\tt .vex} file.
In the event that both multiple recorders are specified here and there are multiple {\tt DATASTREAM}s for an antenna in the {\tt .v2d} file one can use the {\tt recorder} parameter of the {\tt DATASTREAM} section to explicitly associate a given recorder to a datastream.
If no such explicit assignment is provided, the ``reduced recorder numbers'' (see below) for a given antenna are sorted numerically, each becoming mapped to a datastream in the correct order.

The ``reduced recorder numbers'' is a function of the recorder number specified as the first parameter in a {\tt TAPELOG\_OBS} entry.
For most antennas the ``reduced recorder numbers'' is just equal to that number.
Such numbers for the VLBA are different for each bank of a Mark5C recorder, with odd number corresponding to Bank A and even numbers to Bank B.
Since these are not unique recorders, the ``reduced recorder numbers'' is computed by subtracting 1 and dividing by 2.
This is essentially a short-cut to prevent the need to explicitly use the {\tt recorder} parameter for every DATASTREAM of every antenna.

\subsection{Earth orientation parameters}

Earth orientation parameters (EOPs) quantify the actual orientation of the earth relative to a uniformly rotation model.
There are three quantities, the two coordinates defining the instantaneous spin axis and the spin phase, UT1-UTC.
These are referenced to time in the TAI time system, which differs from UTC by number of leap seconds.
Thus for each EOP entry 5 numbers are needed.

The \vx file can provide these in the {\tt \$EOP} section.
Due to some subtleties this is handled with a separate {\tt def} statement for each EOP entry even through the vex format nominally allows a single {\tt def} statement to accommodate multiple EOPs.
Due to concerns about the ability for post-processing software to properly handle more than 5 EOP values (which will hopefully soon be reevaluated), it is strongly suggested that no matter how EOPs are provided that exactly 5 EOPS on consecutive UTC midnights are provided.

\vd sets EOP values in order as follows:
\begin{enumerate}
\item EOP values in \vx file are stored in memory.
\item EOP values from \vd are loaded.
\begin{enumerate}
\item EOPs that have the same time as those from \vx file replace those from the \vx file.
\item EOPs that have unique times will be appended to the list from the \vx file.
\end{enumerate}
\item The EOPs are sorted in ascending time order.
\end{enumerate}

Note that one of the 5 numbers within a \vd EOP section, the UTC time for that entry, forms the name of that block.
For example:

{\tt EOP 55005 \{ tai\_utc=34 ut1\_utc=0.236958 xPole=0.10597 yPole=0.53906 \} }

\noindent
would deliver values valid at MJD 55055.

\subsection{Correlator parameters}

The \vx file currently has no provision for supplying correlator-specific parameters such as integration time, spectral resolution.
These and other parameters can be set in the \vd file in the {\tt SETUP} sections.
The following parameters are in this category:
\begin{itemize}
\item {\tt tInt} : the integration time (also called correlator dump interval or acquisition interval).  Default is 2 seconds.
\item {\tt specRes} : the spectral resolution produced by the correlator.  Default is 0.5 MHz.
\item {\tt fftSpecRes} : the spectral resolution produced at the FFT stage of the FX algorithm.  This value must be an integer multiple of {\tt specRes}.  Larger values will lead to increasingly confined spectral response but possibly at the expense of performance.  Default is 0.125 MHz.
\item {\tt doPolar} : if true, correlate cross hands; otherwise only correlate parallel hands.  Default is true.  Savings in disk space and some execution time can be had by setting to false.

\end{itemize}

\subsubsection{Optimization parameters}

\begin{itemize}
\item {\tt subintNS} : the length (in seconds) of a work package to be assigned to each core process.
This number must be a multiple of the FFT size and it must evenly divide into the integration time.
If excessively long the increased latency will cause slower performance on short jobs and it may require excessive memory.  The maximum allowed value is 2 seconds.  If too short loop overhead and increased interprocess bandwidth between core processes and the master process may cause performance problems, especially on long jobs.
\item {\tt guardNS} : starting with DiFX 2.5, for all jobs this should be left unset for normal astrometry using Earth-based radio antennas.
\item {\tt postFFringe} : if set to true, the fringe rotation will be performed after the FFT stage of the correlation.
This leads to improved performance but decreased coherence of results. 
For all long-baseline observations this should be left at its default value of ``false''.
\item {\tt fringRotOrder} : the polynomial order of the fringe rotation function as applied on a per-FFT basis.
Three options are currently supported: 0 means a constant function, effectively the same as setting {\tt postFFringe} to true; 1 means fringe rotation phase is a linear function of time (the default, and for VLBI usually is the correct option); and 2 means quadratic, which might be wanted for long FFT sizes on especially long baselines or in cases (spacecraft related) where the delay function is accelerating much faster than usual.
\item {\tt strideLength} : in calculating the fringe rotation function this parameter is used to subdivide the problem.
In general a number that divides the FFT size {\em for every antenna} and that is roughly the square root of this number is ideal.
In cases where incommensurate sample rates (e.g., for EHT with 62.5~MHz vs.\ 64~MHz) it is difficult to set this optimally.
With DiFX 2.5 and later, leaving it unset, or setting it to 0, will allow {\tt mpifxcorr} to determine this automatically {\em per antenna}, and that is recommended in just about every situation.
\item {\tt xmacLength} : in performing the cross multiplications the problem is divided into ``strides'' of this size to keep recently loaded values in cache memory to improve performance.
For typical $\sim 10$ antenna arrays a good value is around 128.
For much larger arrays a smaller value may be better.
Starting with DiFX 2.5 leaving this unset, or setting to 0, will cause the legal value nearest 128 to be used.
Note that the value of this parameter must evenly divide the output spectrum size {\em for all bands}.
When using zoom bands this is most constraining, so it is best to keep the number of channels in all zoom bands the same and to keep them a product of small numbers.

\end{itemize}



\subsection{Zoom bands}



\subsection{Pulse cal extraction}

Pulse cal tones are extracted using an efficient time-domain algorithm in {\tt mpifxcorr}.
The algorithm is equally efficient to extract a single tone or all tones in a baseband channel.
Given this {\tt mpifxcorr} essentially has two modes of operation: either extract no tones or extract all tones.
The selection of which tones to retain in downstream processing can be deferred to post-correlation time.

The version 1.5 \vx file contains insufficient information to fully specify pulse cal tone detection, however sensible defaults suffice in most cases (today), but this will change as baseband channels get wider and tone spacings other than 1~MHz become common. 

In modes where a {\tt \$PHASE\_CAL\_DETECT} block is specified \vexdifx will by default set up pulse cal extraction assuming 1~MHz tone spacing and will recommend to downstream software to retain two tones near (but not too near) the band edges.
See the \vexdifx wiki for details on changing the tones to be extracted and selected through use of the {\tt phaseCalInt}, {\tt toneSelection} and {\tt toneGuard} parameters.



\section{Job separation} \label{sec:break}

This section describes what triggers splitting of a project into more than one job.
There are two fundamentally different reasons for jobs to be split into smaller pieces.
\vexdifx will first split the scans into seperate groups depending on their observing mode.
Observations made with different frequencies, bandwidths, sidebands, time resolution, or frequency resolution will be divided into different scan groups.
Each of these scan groups may then be broken down futher according to the following criteria:
\begin{itemize}
\item {\em Single scan flag} : If the parameter {\tt singleScan} is set to true, there will be a job boundary between every scan.
\item {\em Scan gap} : If the time between two scans in a scan group is larger than a preset duration they will be split (default is 3 minutes but this can be overridden with the {\tt maxGap} global parameter).
\item {\em Max length} : If the total duration of a scans is to exceed a preset duration, then a job break will be forced between scans (default is 2 hours, but this can be overridden with the {\tt maxLength} global parameter).
\item {\em Manual job break} : The \vd file can contain {\tt break} parameters; a job break will happen at the time provided to each of these.  These breaks can happen in the middle of a scan.
\item {\em Clock break} : If an antenna has multiple clock models a job break will occur at the time of transition from one clock model to the next.
\item {\em Media change} : If correlating off Mark5 or Mark6 media, a break will occur whenever media changes.  This can occur in the middle of a scan.
\end{itemize}

\end{document}
