
% FIXME: make sure the following get documented:
% 2dirlist
% bbsum
% calcifMixed
% checkCalcServer
% difxcalc11
% difxfilelist
% plotpcal2
% mvdifxhere
% computetotals (not yet tested)
% polconvert
% fuseMk5
% filterVDIF
% generateVDIF
% fakemultiVDIF
% fileto5c
% mk6vmux
% mk6summary
% mk6mon
% mk6copy
% m6sg_mount
% m6sg_gather
% m6sg_blocknrs
% calcsig
% printdirlist
% splitVDIFbygap


% document whole mark6sg packages in here

\section{Reference guide to programs and utilities} \label{sec:programs}

This section has usage information for the numerous programs and scripts used in the DiFX system.
Basic help information for most or all of these programs can be gotten by typing the program name with either no command line arguments or with a {\tt -h} option, depending on the program.
In the usage descriptions below, arguments in square brackets $[$\ $]$ are optional and can often include multiple different parameters.
Cases where 1 or more arguments of a certain type (such as files) can be passed to the program, the usage instructions will look like {\em arg}1 $[\ \cdots$ {\em arg}N$]$, with the implication that N arguments of this type were passed.
In cases where 0 arguments of that type is also allowed, that first argument will also be in square brackets.
If it is not obvious from the program name, the software package containing the program follows the section header.
The package that includes each program is included in its section heading.

Note that several VLBA specific programs are discussed in this manual that are not documented here, such as {\tt tsm}.
These are preexisting programs that may be documented elsewhere and are less likely to be useful outside VLBA operations.
Also note that programs from the {\tt nrao\_difx\_db} package are internal to NRAO and in general are not applicable outside VLBA operations.
The code for these programs can be made available upon request.




% TODO
% add more hyperlinks (e.g., to vdifd, etc)
% add mark5access python programs: m5selfcorr.py  m5spec.py  m5stat.py  m5subband.py  m5time.py  m5tone.py
% add showcal
% add difxcalc (11)



% apd2antenna -----------------------------------------------------------------

\subsection{apd2antenna {\small $\mathrm{(package: difx2fits)}$}} \label{sec:apd2antenna}

Python program {\tt apd2antenna} will read a {\tt .apd file} (see Sec.~\ref{sec:apd}), which stores baseline-based fringe-fit solutions, and writes to {\tt stdout} an antenna-based set of fringe-fit values.
These are determined through a least-squares fit.
A reference antenna must be specfied.

\begin{itemize}
\item[] Usage: {\tt apd2antenna} {\em apdFile refAnt}
\item[] {\em apdFile} is a {\tt .apd} file (\S\ref{sec:apd}), created by {\tt difx2fits}
\item[] {\em refAnt} is a 1-based number or text string indicating the reference antenna to be used
\item[] Example: {\tt apd2antenna DQ1206.apd 2} 
\end{itemize}

A reference antenna is required as the least-squares solution is ill-determined otherwise.
The reference antenna has its delay, rate, and phase set to zero in the process.

The output of this program, which is sent to {\tt stdout} is documented in Sec.~\ref{sec:aapd}.
It is conventional to redirect the output of this program to a file ending in {\tt .aapd} .





% avgDiFX ---------------------------------------------------------------------

\subsection{avgDiFX {\small $\mathrm{(package: difxio)}$}} \label{sec:avgDiFX}

{\tt avgDiFX} is a utility packaged with difxio.
It reads two complete DiFX filesets that need to cover the same timerange and have essentially the same array structure and produces a new complete fileset.
The {\tt .input}, {\tt .calc}, and {\tt .im} files, along with any extracted pulse cal data, are copied from the first of the two datasets.
The visibility data from the two file sets is averaged.
Certain pulsar processing may benefit from this capability.

\begin{itemize}
\item[] Usage: {\tt avgDiFX} {\em configFile1 configFile2 outputConfigFile}
\item[] Example: {\tt avgDiFX pass1\_01.input pass2\_01.input output\_01.input} 
\end{itemize}



% bp2antenna ------------------------------------------------------------------

\subsection{bp2antenna {\small $\mathrm{(package: difx2fits)}$}} \label{sec:bp2antenna}

Python program {\tt bp2antenna} will read a {\tt .bandpass file} (see Sec.~\ref{sec:filebandpass}), which stores baseline-based bandpass solutions determined by {\tt difx2fits}, and writes to {\tt stdout} an antenna-based set of bandpasses.
These are determined through a least-squares fit separately to phase and amplitude.
This program requests a reference antenna be provided and a minimum of three antennas are required.
This antenna's phases are fixed to be zero and all other antennas' phases are determined relative to that antenna.
If a negative number is provided for the reference antenna, the antenna phases will be adjusted after antenna-based solutions are found such that the average bandpass phase at each frequency is zero.
The bandpass can be smoothed using the low-pass filter {\tt LPF} option.
The value to be provided is measured in MHz; bandpass features smaller than this are smoothed out.
Optionally a pulse cal data file (determined by non-DiFX program pcalanal) can be applied to create ``absolute phase'' antenna-based bandpasses.
This feature is experimental.
If autocorrelation bandpasses are included in the {\tt .bandpass} file, they will be ignored.

\begin{itemize}
\item[] Usage: {\tt bp2antenna} {\em bandpassFile refAnt} $[$ {\em LPF} $[$ {\em pcalFile} $]$ $]$
\item[] {\em bandpassFile} is a {\tt .bandpass} file (\S\ref{sec:filebandpass}), created by {\tt difx2fits}
\item[] {\em refAnt} is a 1-based number or text string indicating the reference antenna to be used
\item[] {\em LPF} is the low pass filter parameter, measured in MHz
\item[] {\em pcalFile} contains pulse cal phases after removing dominant delays (FIXME: to be documented...)
\item[] Example: {\tt bp2antenna DQ1206.bandpass 2} 
\end{itemize}

The output of this program, which is sent to {\tt stdout} is documented in Sec.~\ref{sec:abp}.
It is conventional to redirect the output of this program to a file ending in {\tt .abp} .





% calcif2 ---------------------------------------------------------------------

\subsection{calcif2} \label{sec:calcif2}

Program {\tt calcif2} evaluates the delay model, producing a delay model file (ending with {\tt .im}) from a file containing the source, antenna and scan timing information (ending with {\tt .calc}).
The detailed calculations are performed by the Goddard CALC program.
Prior to this writing (May 4, 2013), the only option for the calculation back-end was CALC version 9 with NRAO additions which add ocean loading and near-field corrections (accurate as close as a few $\times 10^5$~km). 
Now new options are being introduced, including CALC 9 with the Sekido-Fukushima near-field model (using the {\tt --sekido} option).
Note that this option requires an installation of a special version of CALC that is not covered in this document.

Instead of calling CALC for every tabulated model row, {\tt calcif2} computes a 5th degree polynomial every 120 seconds (typically),
very closely resembling the delay model generation used at the VLBA hardware correlator.
These polynomials are then evaluated at each model point.
This results in a tremendous speedup at negligible loss of accuracy.
By default {\tt calcif2} will call CALC three times for each model point and calculates more accurate $u, v, w$ coordinates from delay measurements made over a small patch of the sky:
\begin{equation}
(u, v, w) = \left(-c \frac{d \tau}{d l}, c \frac{d \tau}{d m}, c \tau \right)
\end{equation}
where $l, m$ are angular coordinates (in radians) relative to the delay center on the sky, $\tau$ is the delay at the delay center and $c$ is the speed of light.
%Note that the sign convention for $\tau$ in this equation is that used by {\tt CALC}

Normally {\tt calcif2} will be called by {\tt difxqueue}, {\tt startdifx}, or another higher-level program if needed.

{\tt calcif2} connects via Remote Procedure Call (RPC) to an instance of {\tt CalcServer} which must be running on a computer identified by environment variable {\tt \$CALC\_SERVER}, or by the specified computer if the {\tt -s} option is used.
If the output files (specified in the {\tt .calc} file) exist and are current (have newer modification times than the {\tt .calc} file, then the files will not be recreated unless the force option is used.

In addition to calculating the delay model, this program computes the baseline vectors, $u,v,w$ (relative to Earth center on a per-antenna basis) and source elevation vs.\ time.

\begin{itemize}
\item[] Usage: {\tt calcif2} $[$ {\em options} $]$ $\{$ {\tt -a} $\mid$ {\em calcFile1} $[$ {\em calcFile2} $[\cdots]$ $]$ $\}$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -a} or {\tt --all} : run on all {\tt .calc} files found in the current directory
\item[] {\tt -v} or {\tt --verbose} : print more verbose logging/debug info
\item[] {\tt -q} or {\tt --quiet} : print less verbose logging/debug info
\item[] {\tt -f} or {\tt --force} : rerun even if output files exist and are current
\item[] {\tt -n} or {\tt --noaber} : don't perform aberration $u,v,w$ corrections
\item[] {\tt -F} or {\tt --fit} : Instead of producing an $n$ term polynomial from $n$ samples, calculate more samples and perform a fit.  This is not of general use as tests have shown that the improvement is negligible.
\item[] {\tt -z} or {\tt --allow-neg-delay} : don't zero delays that are negative (i.e., shadowed by Earth)
\item[] {\tt -A} or {\tt --noatmos} : don't include atmosphere in calculation of $u,v,w$
\item[] {\tt -s} {\em server} or {\tt --server} {\em server} : connect to {\em server}, not {\tt \$CALC\_SERVER}
\item[] {\tt -o} {\em order} or {\tt --order} {\em order} : make polynomials with {\em order}+1 terms (default 5)  
\item[] {\tt -i} {\em int} or {\tt --interval} {\em int} : make a polynomial every {\em int} seconds (default 120)
\item[] {\tt --override-version} ignore difx version clashes
\end{itemize}
\item[] {\em calcFile} is a {\tt .calc} file (\S\ref{sec:input}), such as one generated by {\tt vex2difx} (\S\ref{sec:vex2difx})
\item[] Example 1: {\tt calcif2 job1420.000.calc job1421.000.calc} 
\item[] Example 2: {\tt calcif2 -s kepler job1420.000.calc}
\item[] Example 3: {\tt calcif2 -a -i 60} 
\end{itemize}







% CalcServer ------------------------------------------------------------------

\subsection{CalcServer} \label{sec:CalcServer}

Program {\tt CalcServer} contains the Goddard Space Flight Center CALC package version 9.1, used to compute geometric delay models for VLBI applications.
It is a repackaged version of the same source code that is used to compute models on the VLBA correlator.
It is configured to run as a server.
All of its interactions are via RPC calls from other programs, such as {\tt calcif2}, which could be running on the same or different computer.
\newcommand{\oa}[1]{\hspace{-12pt}\makebox[12pt]{$\star$}#1}
This program only needs to be started once on a given machine using the {\tt startCalcServer} script.
It should probably be set to start automatically upon boot of the machine on which {\tt CalcServer} runs.
Environment variable {\tt \$CALC\_SERVER} should be set to the name of the computer on which {\tt CalcServer} is running.

\begin{itemize}
\item[] Start: {\tt startCalcServer}
\item[] Test: {\tt checkCalcServer \$CALC\_SERVER}
\item[] Stop: {\tt killall CalcServer}
\end{itemize}

\noindent
Note that {\tt CalcServer} must be installed (with {\tt make install}) to be usable as the paths for various files are permanently set in the executables at compile time.
At this time it seems {\tt CalcServer} cannot be compiled for 64-bit machines.







% checkdir --------------------------------------------------------------------

\subsection{checkdir {\small $\mathrm{(package: mk5daemon)}$}}

Program {\tt checkdir} can be used to check the integrity of one or more {\tt .dir} files that are stored at a location pointed by environment variable {\tt MARK5\_DIR\_PATH} .
Even after many years of use, the Mark5 units tend to be a weak point in the reliability of correlation.
Since reading the module directory and examining a bit of data from each scan are the first actions done to a module, many of the possible problems show up at this time.
This utility looks for a number of possible problems, including scans that could not be decoded, overlapping or out-of-order scans, scans with illegal format parameters and others.
This program makes no attempt to fix problems.
It is up to the operator to determine if a problem is real or not and if further action should be taken.
In cases where many scans are not properly decoded it is worthwhile to rename (or remove) the {\tt .dir} file in question and regenerate the directory.  
A second directory read often succeeds when a first one does not.

\begin{itemize}
\item[] Usage: {\tt checkdir} $[$ {\em options} $]$ $[$ {\em module list} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution ({\tt -v -v} for more)
\item[] {\tt -q} or {\tt --quite} : be less verbose in execution
\item[] {\tt -a} or {\tt --all} : run on all files in \${\tt MARK5\_DIR\_PATH} 
\item[] {\tt -s} or {\tt --show} : print the entire directory file to screen
\item[] {\tt -H} or {\tt --histogram} : print a histogram of record rates
\end{itemize}
\item[] Example 1: {\tt checkdir -a}
\item[] Example 2: {\tt checkdir NRAO-123 NRAO+266}
\item[] Example 3: {\tt checkdir -s NRAO+233}
\end{itemize}

Either {\tt -a} or a list of module names can be provided (but not both simultaneously).
If the former, a less verbose output will be generated by default.
Except in the lowest verbosity mode (the default for {\tt -a}), module directories without any detected problems will show a one line summary consisting of the number of scans and the time range of the module.







% checkmpifxcorr --------------------------------------------------------------

\subsection{checkmpifxcorr {\small $\mathrm{(package: mpifxcorr)}$}}

Program {\tt checkmpifxcorr} reads the {\tt .input} and other associated files for a DiFX job and parses them with the same logic used by {\tt mpifxcorr} in order to determine their validity.

\begin{itemize}
\item[] Usage: {\tt checkmpifxcorr}  $[$ {\em options} $]$ {\em configFile} \ldots
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} : print usage information and exit
\item[] {\tt -f -s -e -w -i -v -d} : select the verbosity level of output (options refer to ``fatal'', ``severe'', ``error'', ``warning'', ``info'', ``verbose'', and ``debug'' levels).
\end{itemize}
\end{itemize}

All of the files referenced from the provided configuration ({\tt .input}) files are read as well (excepting any baseband files or the {\tt .vex} file).
This check has proven especially useful for pulsar processing.
The default verbosity level will lead to printing of any problems at the ``warning'' level or worse.
See Sec.~\ref{sec:difxalertmessage} for details on the severity levels.





% cleanVDIF -------------------------------------------------------------------

\subsection{cleanVDIF {\small $\mathrm{(package: vdifio)}$} \label{sec:printVDIF}}

Program {\tt cleanVDIF} loops through a VDIF file writing valid content to a new output file.

\begin{itemize}
\item[] Usage: {\tt cleanVDIF} {\em inputvdiffile outputvdiffile Mbps} $[$ {\em options} $]$ 

\item[] {\em inputvdiffile} is the recorded VDIF file to clean
\item[] {\em outputvdiffile} is the corrected VDIF file to write
\item[] {\em Mbps} is the data rate in megabits/second
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -v} or {\tt --verbose} : be verbose in execution
\end{itemize}

\item[] Example: {\tt cleanVDIF bad.vdif good.vdif 256}
\end{itemize}




% condition -------------------------------------------------------------------

\subsection{condition {\small $\mathrm{(package: nrao\_difx\_db)}$}}

This is an NRAO-only program owing to its ties to the VLBA database.

Program {\tt condition} is mainly used to extract Mark5 module conditioning reports from the database but also has the means to manually import data into the database.
When querying (with the {\tt find} action), one or more ``identifiers'' can be supplied which can be either the names of the Mark5 modules or serial number of individual disks (or a mix of the two!).
Environment variable {\tt VLBA\_DB} must be set to point to the correct database.

\begin{itemize}
\item[] Usage: {\tt condition} $[$ {\em options} $]$ {\em action + args}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be verbose in execution
\end{itemize}
\item[] {\em action} can be one of:
\begin{itemize}
\item[] add {\em report1} $[${\em report2} $\cdots$ $]$
\item[] find {\em identifier1} $[${\em identifier2} $\cdots$ $]$
\end{itemize}
\item[] {\em report} is the name of a file containing one or more condition reports from {\tt SSErase}
\item[] {\em identifier} is either a Mark5 module VSN or a hard disk serial number
\item[] Example 1: {\tt condition add NRAO-040}
\item[] Example 2: {\tt condition find NRAO-042}
\item[] Example 3: {\tt condition find NRAO+342 NRAO+270}
\item[] Example 4: {\tt condition find Y66M3BQE}
\end{itemize}







% condition_watch -------------------------------------------------------------

\subsection{condition\_watch {\small $\mathrm{(package: nrao\_difx\_db)}$}}

This is an NRAO-only program owing to its ties to the VLBA database.

Program {\tt condition\_watch} is meant to run as a background process on the correlator head node.
Its function is to receive {\tt Mark5ConditionMessage}s emitted by a special version of {\tt SSErase} (the module conditioning program) and stuff this data into the database.
This program is automatically started by {\tt mk5daemon} when it is supplied with the {\tt -w} or {\tt --condition-watch} arguments.
When restarting {\tt mk5daemon} by hand, make sure that a duplicate copy of {\tt condition\_watch} is not left running.
Environment variable {\tt VLBA\_DB} must be set to point to the correct database.

\begin{itemize}
\item[] Usage: {\tt condition\_watch} $[$ {\em options} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] Example: {\tt condition\_watch}
\end{itemize}







% countVDIFpackets ------------------------------------------------------------

\subsection{countVDIFpackets {\small $\mathrm{(package: vdifio)}$} \label{sec:countVDIFpackets}}

Program {\tt countVDIFpacket} loops through a VDIF file and counts number of valid and skipped frames.
Packet counts are performed only on the thread ID requested.

\begin{itemize}
\item[] Usage: {\tt countVDIFpackets} {\em vdiffile Mbps threadId}

\item[] {\em vdiffile} is the recorded VDIF file
\item[] {\em Mbps} is the data rate in megabits/second
\item[] {\em threadId} is the threadId to report on

\item[] Example: {\tt countVDIFpackets example.vdif 256 3}
\end{itemize}








% cpumon ----------------------------------------------------------------------

\subsection{cpumon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:cpumon} 

Program {\tt cpumon} is a program that listens for {\tt difxLoad} messages multicast from the Mark5 units and displays the information; updating the display as new messages are received.

\begin{itemize}
\item[] Usage: {\tt cpumon}
\end{itemize}

\noindent
Make sure the terminal is at least 60 characters wide and is at least as tall as there are computers that may transmit information.
To quit, use ctrl-C.
The columns displayed are:
\begin{enumerate}
\item Computer name
\item CPU load averaged over 10 seconds
\item Memory usage / Total memory
\item Network receive rate (Mbps)
\item Network transmit rate (Mbps)
\item Number of CPU cores
\end{enumerate}









% db2vex --- OBSOLETE ---------------------------------------------------------

%\subsection{db2vex {\small $\mathrm{(package: nrao\_difx\_db)}$}}
%
%{\bf This program has been superceded by {\tt vex2obs}}
%
%This is an NRAO-only program owing to its ties to the VLBA database.
%
%Python program {\tt db2vex} adds three tables to a vex file based on data found within the VLBA database and VLA databases.
%For legacy VLBA/HSA projects, the Oracle database is accessed using the {\tt cx\_Oracle} library to retrieve information from the {\tt TAPELOG\_OBS}, {\tt CLOCK}, and {\tt EOP} tables.
%For RDBE VLBA/GBT projects, a new Postgres VLBA database (vlbampts) using the EVLA monitor data format is accessed.
%For projects containing the (modern) phased-VLA, the EVLA database (evlampts) is accessed for similar information.
%
%This program should be run after the Earth Orientation Parameters (EOPs) for a project have reached a suitable maturity and after the clocks for the antennas have been inserted into the (legacy) database.
%If a preliminary run of this program is made for the purposes of doing a clock search correlation pass it is recommended that the output file (a vex format file with the new tables attached) get renamed to indicate that this is not to be used to derive jobs for production correlator passes.
%
%\begin{itemize}
%\item[] Usage: {\tt db2vex} $[$ {\em options} $]$ {\em vexFile} $[$ {\em fsLog} $\cdots ]$
%\item[] {\em options} can be:
%\begin{itemize}
%\item[] {\tt -h} or {\tt --help} : print usage information and exit
%\item[] {\tt -v} or {\tt --verbose} : be verbose in execution
%\item[] {\tt -f} or {\tt --force} : force program to overwrite existing db2vex output files
%\item[] {\tt -p} or {\tt --preobs} : don't look for VSNs, but get clock and EOP values (used for USNO observations by goUSNO)
%\item[] {\tt -a} or {\tt --mark5a} : force to run in Mark5A mode
%\item[] {\tt -c} or {\tt --mark5c} : force to run in Mark5C mode
%\item[] {\tt -u} or {\tt --usno} : force to run in USNO Mark5C mode
%\item[] {\tt --vsnProjectCode} {\em projectCode} : use {\em projectCode} when getting VSNs from the database
%\end{itemize}
%\item[] {\em vexFile} is the name of the vex file produced by {\tt sched}
%\item[] {\em fsLog} is a field system log file; multiple such files can be provided
%\item[] Example 1: {\tt db2vex bx123.skd}
%\item[] Example 2: {\tt db2vex -v gb070.vex gb070nt.log gb070mc.log}
%\end{itemize}
%
%\noindent
%The output of {\tt db2vex} will be another vex format file called {\em vexFile}{\tt .obs} .
%This output file is to be used by {\tt vex2difx} for conversion into jobs to run by {\tt mpifxcorr} .
%The {\tt --vsnProjectCode} was intended for a pathological situation where a project code had to be renamed.
%
%\noindent
%Notes: 
%\begin{enumerate}
%\item Environment variable {\tt VLBA\_DB} must be set to point to the Oracle database in question.
%\item Mark5C observations require environment variable {\tt VLBAMPTS\_DB} to point to the new EVLA-style VLBA database
%\item Observations using the phased VLA require environment variable {\tt EVLAMPTS\_DB} to point to the EVLA monitor database
%\end{enumerate}




% dedisperse_difx ------------------------------------------- INCOMPLETE -------





% diffDiFX.py ------------------------------------------------------------------

\subsection{diffDiFX.py {\small $\mathrm{(package: vis2screen)}$}} \label{sec:diffDiFX} 

Program {\tt diffDiFX.py} generates a context-sensitive difference of two DiFX output files for detailed version testing.  Corresponding visibility records are differenced and statistics on the differences are accumulated and printed at the end of the processing.

\begin{itemize}
\item[] Usage: {\tt diffDiFX.py} $[$ {\em options} $]$ $\{$ {\em difxfile1} {\em difxfile2}$]$ $\}$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -f FREQ} or {\tt --freq=FREQ} : Only look at visibilities from this FREQ index
\item[] {\tt -b BASELINE} or {\tt --baseline=BASELINE} : Only look at visibilities from this BASELINE num
\item[] {\tt -t THRESHOLD} or {\tt --threshold=THRESHOLD} : Display any difference that exceeds THRESHOLD
\item[] {\tt -e EPSILON} or {\tt --epsilon=EPSILON} : Display any differences that exceeds allowed numerical error EPSILON
\item[] {\tt -s SKIPRECORDS} or {\tt --skiprecords=SKIPRECORDS} : Skip SKIPRECORDS records before starting comparison
\item[] {\tt -m MAXRECORDS} or {\tt --maxrecords=MAXRECORDS} : Stop after comparing MAXRECORDS (if >0) records
\item[] {\tt -p PRINTINTERVAL} or {\tt --printinterval=PRINTINTERVAL} : Print a summary every PRINTINTERVAL records
\item[] {\tt -c MAXCHANNELS} or {\tt --maxchannels=MAXCHANNELS} : The length of the array that will be allocated to hold vis results
\item[] {\tt -v} or {\tt --verbose} : Turn verbose printing on
\item[] {\tt -i INPUTFILE} or {\tt --inputfile=INPUTFILE} : Parse INPUTFILE for the correlation setup
\item[] {\tt --matchheaders} : On seeing a header mismatch, skip through file 2 looking for next match
\end{itemize}
\item[] {\em difxfile1} is the first difx file to compare
\item[] {\em difxfile2} is the second difx file to compare 
\item[] Example: {\tt difxDiFX.py -i example\_1.input example\_1.difx/DIFX\_55523\_025239.s0000.b0000} \\ {\tt comparison\_1.difx/DIFX\_55523\_025239.s0000.b0000}
\end{itemize}


\noindent 
If the error for any record exceeds the specified threshold a verbose error message is printed.
Summary statistics are printed at the end of the file.
Warnings are printed if the headers do not match between the two files.




% difx2fits -------------------------------------------------------------------

\subsection{difx2fits} \label{sec:difx2fits}

Program {\tt difx2fits} creates a FITS output file from the native output format created by {\tt mpifxcorr} and several other files carrying information about the observation.
Multiple input file sets can be specified.
A separate output FITS file is created for each unique frequency setup encountered.
When run, {\tt difx2fits} requires the following files to be present for each DiFX file set being converted:
\begin{enumerate}
\item {\em baseFilename}{\tt .difx/}
\item {\em baseFilename}{\tt .input}
\item {\em baseFilename}{\tt .calc}
\item {\em baseFilename}{\tt .im}
\end{enumerate}
Several other files are optional and are typically used to populate calibration and ancillary tables:
\begin{enumerate}
\item {\em baseFilename}{\tt .flag} 
\item {\tt flags}
\item {\tt pcal}
\item {\tt tsys}
\item {\tt weather}
\item {\tt \$GAIN\_CURVE\_PATH/}
\item {\tt .difx/*.history}
\end{enumerate}
With the exception of the gain curve files, all the input files to {\tt difx2fits} are expected to be in the current working directory or in the place indicated by the {\tt .input} file.
As the visibility file ({\tt .difx}) is read, any records that are all zero are omitted.

\begin{itemize}
\item[] Usage: {\tt difx2fits} $[$ {\em options} $]$ $\{$ {\tt -d} $\mid$ {\em baseFilename1} $[\cdots${\em baseFilenameN}$]$ $[${\em outFile}$]$ $\}$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -n} or {\tt --no-model} : don't write model (ML) table
\item[] {\tt -s} {\em scale} or {\tt --scale} {\em scale} : scale visibility data by {\em scale}
\item[] {\tt -t} {\em interval} or {\tt --deltat} {\em interval} : generate {\tt .jobmatrix} file with time intervals of length {\em interval} seconds
\item[] {\tt --difx-tsys-interval} {\em interval} : the Difx-derived tsys interval (sec) (default 30.0s averaging)
\item[] {\tt --difx-pcal-interval} {\em interval} : the Difx-derived pcal interval (sec) (default 30.0s averaging)
\item[] {\tt -S} or {\tt --sniff-all} : sniff all bins and phase centers, not just the first
\item[] {\tt -T} {\em interval} or {\tt --sniff-time} {\em interval} : use {\em interval} as the sniffer time resolution
\item[] {\tt -v} or {\tt --verbose} : increase verbosity of output; use twice or thrice to get even more
\item[] {\tt -d} or {\tt --difx} : run on all {\tt .difx} files found in the directory
\item[] {\tt -k} or {\tt --keep-order} : don't sort the antennas by name
\item[] {\tt -1} or {\tt --dont-combine} : make a separate FITS file for each input job
\item[] {\tt -x} or {\tt --dont-sniff} : don't generate sniffer output files
\item[] {\tt -0} or {\tt --zero} : don't put visibility data in FITS file
\item[] {\tt --bin} {\em b} : Select on this pulsar bin number
\item[] {\tt --phasecentre} {\em p} : (U.S.\ spelling okay too) Create a FITS file for all the {\em p}$^{\rm th}$ phase centers (default 0)
\item[] {\tt --override-version} : ignore difx version clashes
\item[] {\tt --bandpass} : write the {\tt .bandpass} file (see Sec.~\ref{sec:filebandpass})
\item[] {\tt -m} {\em nJob} or {\tt --max-jobs} {\em nJob} : split into more FITS files after reaching {\em nJob} input files.
\item[] {\tt --eop-merge-mode} {\em mode} : sets conditions for allowing jobs with different EOPs to be merged or not; options are {\tt strict} (default), {\tt drop}, {\tt relaxed}
\item[] {\tt --clock-merge-mode} {\em mode} : sets conditions for allowing jobs with different clock models to be merged or not; options are {\tt strict} (default) or {\tt drop}
\item[] {\tt --antpol} : use antenna-based polarization labels as in VEX. Note: fits-idi file will violate original specifications and abide extended specifications.
\item[] {\tt --polxy2hv} : re-labels all polarizations XY to HV. Requires --antpol option. 
\end{itemize}
\item[] {\em baseFilename} is the prefix of the jobfile to convert; it is okay to use the {\tt .difx} filename instead
\item[] {\em outFile} is the name of the {\tt FITS} file to produce; if not provided one will be made based on the project code
\item[] Example 1: {\tt difx2fits dq109\_1 DQ109.FITS}
\item[] Example 2: {\tt difx2fits -v -v -d}
\end{itemize}

\noindent Environment variables respected:
\begin{itemize}
\item {\tt DIFX\_GROUP\_ID} : if set, run difx2fits with {\tt umask(2)}.
\item {\tt DIFX\_LABEL} : the local name of the difx install.  Used to verify matching versions and put inside FITS file; if not set, {\tt DIFX\_VERSION} will be used instead.
\item {\tt DIFX\_MAX\_SNIFFER\_MEMORY} : maximum amount mf memory (bytes) to allow sniffer to use.
\item {\tt DIFX\_VERSION} : the difxbuild version name.
\item {\tt GAIN\_CURVE\_PATH} : directory containing gain files.
\item {\tt TCAL\_PATH} : a directory containing Tcal value files.
\item {\tt TCAL\_FILE} : if {\tt TCAL\_PATH} is not set, use the file pointed to by this env. var.
\end{itemize}

Unless adjusted with the {\tt --difx-pcal-interval} and {\tt --difx-tsys-interval} parameters, the respective PC and TY data will be time averaged to the default of 30 seconds. For geodetic
VLBI or other observations with very short scans you may want to shorten the averaging time.

Unless disabled with the {\tt --dont-sniff} or {\tt -x} flag, four ``sniffer'' output files ({\tt .acb}, {\tt .apd}, {\tt .wts} and {\tt .xcb}) will be written for each {\tt .FITS} file produced.  
These files are used by {\tt difxsniff} and its associated programs to produce data plots that are used to assess data quality.

Unless disabled by setting {\em interval} to a non-positive number with the {\tt -t} or {\tt --deltat} option, an output file with suffix {\tt .jobmatrix} will be produced. 
This file contains an ASCII art diagram of which jobs contributed to each {\tt .FITS} file produced as a function of both time and antenna.

Unless augmented with option {\tt --antpol} the produced FITS files are fully compliant with the original FITS-IDI specifications. In case of mixed mode polarization (XY against RL)
or certain feed types (HV in particular) the option {\tt --antpol} allows to force output of noncompliant but polarization correct FITS files; FITS-IDI numerical parameter STK\_1 is set to
the new value of -9, indicating to post-processing software that it should refer to the existing FITS-IDI character parameters Antenna1Feed1 and Antenna2Feed1 (populated from VEX)
for the polarization details.

If submitting a bug report for {\tt difx2fits}, please include in it the full output of {\tt difx2fits -v -v} and the {\tt .input} and {\tt .calc} files.

{\tt difx2fits} displays several diagnostics during the conversion process, separately for each output FITS file.
The size of each FITS table is printed; a zero size indicates that table is not produced.
For the visibility table, input files contributing to the output are printed.
Scan information is printed at increased verbosity levels.
Not all DiFX output visibilities are written to the FITS file.
Accounting of the disposition of the visibilities is provided.
Invalid records are those containing infinite or NaN values and indicate a likely bug in the software.
Flagged records are those identified by vex2script (or other DiFX file set generation programs) in the {\tt .flag} file as being illogical, such as cases where a particular baseline during a job belongs to a different subarray.
When using integration times longer than 1 second it is possible for one visibility to span two scans.  
Such records are dropped.
Finally any visibilities, produced outside a normal scan start/stop time are dropped; this should not occur unless the {\tt .calc} file is modified between correlation and FITS creation.

If there are any files matching {\tt .difx/*.history} for {\tt .difx/} output being converted to {\tt .FITS}, the contents of these files will be inserted into the FITS HISTORY table.






% difx2mark4 ------------------------------------------------------------------

\subsection{difx2mark4} \label{sec:difx2mark4}

Program {\tt difx2mark4} creates a Mark4 output file set from {\tt mpifxcorr} input and output files.
When run, {\tt difx2mark4} requires the following files to be present for each file set being converted:
\begin{enumerate}
\item {\em baseFilename}{\tt .difx/}
\item {\em baseFilename}{\tt .input}
\item {\em baseFilename}{\tt .im}
\end{enumerate}
as well as the {\tt .vex} file referenced in the {\tt .input} file, which may be common to many DiFX file sets.

\begin{itemize}
\item[] Usage: {\tt difx2mark4} $[$ {\em options} $]$ {\em baseFilename1} $[\cdots${\em baseFilenameN}$]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity of output; use twice or thrice to get even more
\item[] {\tt -d} or {\tt --difx} : run on all {\tt .difx} files found in the directory
\item[] {\tt -k} or {\tt --keep-order} : don't sort the antennas by name
\item[] {\tt --override-version} : ignore difx version clashes
\item[] {\tt -r} or {\tt --raw} : suppresses normalization of amplitudes
\item[] {\tt -p} or {\tt --pretend} : do a dry run
\item[] {\tt -e} or {\tt --experiment-number} {\em n} : set the experiment number to {\em n} which must be a 4 digit number (default is 1234)
\item[] {\tt -b} {\em code} {\em flo} {\em fhi} : Override freq band codes.  Frequencies are in MHz.  Multiple parameters of this kind can be specified.
\end{itemize}
\item[] {\em baseFilename} is the prefix of the jobfile to convert {\em without} the underscore and job number
\item[] Example {\tt difx2mark4 dq109}
\end{itemize}






% difxarch --------------------------------------------------------------------

\subsection{difxarch {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:difxarch}

Program {\tt difxarch} is a simple script that moves {\tt FITS} files produced by {\tt makefits} from the correlation queue staging area (defined by the {\tt DIFX\_QUEUE\_BASE} environment variable) to the archive staging area (defined by environment variable {\tt DIFX\_ARCHIVE\_ROOT}).
A process running on the archive computer will periodically monitor new files in this staging area and will then copy them to the actual archive.
In order to prevent premature pick-up of these files, they are first moved into a directory with a name beginning with a period ({\tt .}).
This directory is renamed without the period once all files to be archived are copied.

\begin{itemize}
\item[] Usage: {\tt difxarch} $[$ {\em options} $]$ {\em passName1} $[\cdots${\em passNameN}$]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity of output
\item[] {\tt -p} or {\tt --pretend} : generate SQL and bash commands but don't execute them
\item[] {\tt --override-version} ignore DiFX version clashes 
\end{itemize}
\item[] {\em passName} is the name of a correlator pass; a file called {\em passName}{\tt .fitslist} is expected to be present
\item[] Example: {\tt difxarch -v clock}
\end{itemize}







% difxbuild -------------------------------------------------------------------

\subsection{difxbuild} \label{sec:difxbuild}

Program {\tt difxbuild} aids in the installation of DiFX onto a cluster.
Full documentation on the install process can be found in \S\ref{sec:installdifxbuild}, so details will not be shown here.
Command syntax is as follows:

\begin{itemize}
\item[] Usage: {\tt difxbuild} $[$ {\em options} $]$ {\em command} $[$ {\em command arguments} $]$ 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -d} or {\tt --documentation} : print full in-line documentation to screen
\item[] {\tt -t} ot {\tt --todo} : print developer's to-do list
\item[] {\tt -v} or {\tt --verbose} : increase verbosity of output
\item[] {\tt -q} or {\tt --quiet} : decrease output verbosity
\item[] {\tt -p} or {\tt --pretend} : generate SQL and bash commands but don't execute them
\item[] {\tt -V} or {\tt --version} : print version and quit
\end{itemize}
\item[] {\em command} is one of the {\tt difxbuild} commands, such as {\tt build} or {\tt svn}; the program help information will list all options
\item[] {\em command arguments} are options for some commands
\end{itemize}





% difxcalc11 ------------------------------------------------------------------

\subsection{difxcalc11}




% difxcalculator --------------------------------------------------------------

\subsection{difxcalculator {\small $\mathrm{(package: difxio)}$}} \label{sec:difxcalculator}

Program difxcalculator looks at a set of DiFX input files ({\tt .input}, {\tt .calc}, etc.) and reports/calculates key operating parameters.
This program is inspired by the {\tt difx\_calculator.xls} spread sheet available at \url{http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/calculator}.

\begin{itemize}
\item[] Usage: {\tt difxcalculator} $[$ {\em options} $]$ {\em baseName} $[${\em speedUp}$]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em baseName} is the base name of a correlator job
\item[] {\em speedUp} is the expected processing rate relative to real-time
\item[] Example: {\tt difxcalculator mt933\_01}
\end{itemize}

%FIXME
\noindent
Known bugs:
\begin{enumerate}
\item Does not take into consideration multiple phase centers or zoom bands.
\end{enumerate}






% difxclean -------------------------------------------------------------------

\subsection{difxclean {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:difxclean}

Program {\tt difxclean} simply deletes all data from {\tt \$DIFX\_QUEUE/} for a particular project.
It also removes all jobs with status not equal to {\tt COMPLETE} for this project from the {\tt DIFXQUEUE} table of the database.
It is intended that is be run at the same time the project is released, meaning data has been correlated and archived.
The user does not have to be in any particular directory when running this program.

\begin{itemize}
\item[] Usage: {\tt difxclean} $[$ {\em options} $]$ {\em project}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -p} or {\tt --pretend} : don't actually do the erasure
\end{itemize}
\item[] {\em project} is the name of the project to be ``cleaned'' out
\item[] Example: {\tt difxclean mt917}
\end{itemize}








% difxcopy --------------------------------------------------------------------

\subsection{difxcopy {\small $\mathrm{(package: misc\_utils)}$}} \label{sec:difxcopy}

Python program {\tt difxcopy} is used to copy DiFX input (and other) files to a different directory.
In the process, explicit references to other files that are being copied are changed to reflect their
new file system path.
For a given file prefix, {\em prefix}, the following files are copied if they exist: {\em prefix}{\tt .input}, {\em prefix}{\tt .calc}, {\em prefix}{\tt .flag}, {\em prefix}{\tt .delay}, {\em prefix}{\tt .uvw}, {\em prefix}{\tt .rate}, and {\em prefix}{\tt .im}.
The {\tt .vex} file referenced within the {\tt .calc} file is also copied.

\begin{itemize}
\item[] Usage: {\tt difxcopy} $[$ {\em options} $]$ {\em jobPrefix1} $[\cdots${\em jobPrefixN}$]$ {\em destDir}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : possibly increase verbosity of output
\end{itemize}
\item[] {\em jobPrefixN} is the file prefix of a job, e.g., {\tt mt911\_04} would be the prefix corresponding to input file {\tt mt911\_04.input}
\item[] {\em destDir} is the directory in which the copied and modified files will be placed
\item[] Example: {\tt difxcopy mt911\_02 mt911\_03 mt911\_04 /home/difx/queue/MT911}
\end{itemize}










% difxdiagnosticmon -----------------------------------------------------------

\subsection{difxdiagnosticmon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:difxdiagnosticmon} 

Program {\tt difxdiagnosticmon} listens for multicast messages of the {\tt difxStatus} variety and simply prints their contents to the terminal.
This is mainly useful for debugging {\tt mpifxcorr}.
Diagnostic information that is produced includes status of internal buffers, memory usage, execution time, data throughput and lost subintegrations.

\begin{itemize}
\item[] Usage: {\tt difxdiagnosticmon} $[$ {\em options} $]$ 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\end{itemize}





% difxlog ---------------------------------------------------------------------

\subsection{difxlog {\small $\mathrm{(package: difxmessage)}$}} \label{sec:difxlogprogram} 

Program {\tt difxlog} can be used to collect DiFX multicast messages for a particular correlator job and write them to a file.
A new instance of {\tt difxlog} must be started for each job being run.
Normally this will be done automatically if implemented in the particular deployment of DiFX.
Both {\tt startdifx} and {\tt mk5daemon} can start DiFX and will instantiate a {\tt difxlog} process as needed.
Only messages of type {\tt DifxAlertMessage} and {\tt DifxStatusMessage} are collected and written to the output file.

\begin{itemize}
\item[] Usage: {\tt difxlog} {\em jobIdentity} {\em outFile} $[$ {\em logLevel} {\em pidWatch} $]$ 
\item[] {\em jobIdentity} : the name of the job being run (specifically, it should match the {\tt identifier} field in the DifxMessage being sent).
\item[] {\em outFile} : the name of the output file containing log information.
\item[] {\em logLevel} : the minimim message severity to retain (see \S\ref{sec:difxalertmessage}).
\item[] {\em pidWatch} : the program id (in the Unix sense) of the mpifxcorr process running.
\item[] Example: {\tt difxlog mt911\_04 mt911\_04.difxlog 4 1243}
\end{itemize}

Unless a {\em pidWatch} value is specified, {\tt difxlog} will run until killed.
If a {\em pidWatch} value is provided, {\tt difxlog} will quit as soon as that process stops running.
The {\em loglevel} parameter can be used to select the maximum severity level to write to the log.
The possible values and their meanings are:
\begin{center}
\begin{tabular}{lll}
0 & Fatal & {\tt mpifxcorr} cannot continue because of the noted problem \\
1 & Severe & an internal error that should never happen happened (likely bug) \\
2 & Error & a problem was encountered in the data processing \\
3 & Warning & something suboptimal was noted \\
4 & Informative & a note containing progress information \\
5 & Verbose & more detailed progress information \\
6 & Debug & values probably of use only to software developers \\
\end{tabular}
\end{center}







% difxqueue -------------------------------------------------------------------

\subsection{difxqueue {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:difxqueue}


This is an NRAO-only program owing to its ties to the VLBA database.

Python program {\tt difxqueue} is a program used to maintain the DiFX correlator queue.
There are two main responsibilities in doing so: copying or deleting files in the correlator queue directory (which is project specific: {\tt \$DIFX\_QUEUE\_BASE/}{\em projectName}) and maintaining the database entries for each queued job.
In the VLBA context, this program is the main interface between the analysts and the correlator operators.
This program is mainly intended to work on one {\em job pass} at a time rather than single jobs or whole projects.
In some cases one job pass could be one job, or it could be a whole project (or both), but in many cases there will be multiple passes per project with possibly multiple jobs per pass.
It is possible for {\tt difxqueue} to operate on individual jobs when a list of job numbers is provided.
The first command line argument describes the action to perform.
Each subsequent argument is then context dependent; see the examples or run with the {\tt -h} command line parameter to get a feel for the variety of options allowed.
Once a job has been correlated successfully, its status will be COMPLETE.
There is no need to {\tt del}ete a job from the queue once it is complete. 
Doing so will require recorrelation if the results of that job are still needed.
Each job in the queue has a priority.
The smaller the priority, the lower the number.
By default a queued job will have priority 2.

\begin{itemize}
\item[] Usage: {\tt difxqueue} $[$ {\em options} $]$ {\em action} $[$ {\em args} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -p} {\em priority} or {\tt --priority} {\em priority} : set the priority of jobs to {\em priority}
\item[] {\tt -q} {\em queuedir} or {\tt --queuedir} {\em queuedir} : manually set the staging directory
\item[] {\tt -v} or {\tt --verbose} : increase verbosity of output
\item[] {\tt -d} or {\tt --db-only} : do not copy/delete/move files; operate only on database
\item[] {\tt --override-version} : ignore DiFX version clashes 
\end{itemize}
\item[] {\em action} : the action to perform
\begin{itemize}
\item[] {\tt add} : add job(s) to the queue, usually a whole pass at a time.  Default priority is 2; use the {\tt -p} option to set the priority if a different priority is required.
\item[] Example 1: {\tt difxqueue add clock}
\item[] Example 2: {\tt difxqueue add mt911 1 2 3 4}
\item[] Example 3: {\tt difxqueue -p 3 add geodesy}
\vspace{5pt}
\item[] {\tt bump} : increase the priority of queued job(s)
\item[] Example: {\tt difxqueue bump clock}
\vspace{5pt}
\item[] {\tt del} : remove job(s) from the queue
\item[] Example 1: {\tt difxqueue del clock}
\item[] Example 2: {\tt difxqueue del mt911 2 3}
\vspace{5pt}
\item[] {\tt list} : list all jobs within a pass
\item[] Example: {\tt difxqueue list mt911}
\vspace{5pt}
\item[] {\tt listall} : list all incomplete jobs in the queue; note that this is not restricted even to any particular project.  If one or more projects is specified, all jobs, complete or not, for those projects will be listed.  If no segment code is appended to a project name, then all matching proposal codes will be listed.
\item[] Example 1: {\tt difxqueue listall}
\item[] Example 2: {\tt difxqueue listall BX123 BY321}
\item[] Example 3: {\tt difxqueue listall BR138A}
\vspace{5pt}
\item[] {\tt log} : list all correlations that have happened for a given project.  This simply searches the DIFXLOG database table and dumps it to the screen in a readable fashion.
\item[] Example 1: {\tt difxqueue log BX123}
\vspace{5pt}
\item[] {\tt prod} : print production queue list, possibly sending to a file
\item[] Example 1: {\tt difxqueue prod}
\item[] Example 2: {\tt difxqueue prod queue.txt}
\vspace{5pt}
\item[] {\tt set} : set the status of queued job(s)
\item[] Example 1: {\tt difxqueue set tc015d COMPLETE}
\item[] Example 2: {\tt difxqueue set tc015d QUEUED 3 4}
\vspace{5pt}
\item[] {\tt slide} : decrease the priority of queued job(s)
\item[] Example: {\tt difxqueue slide mt911 6}
\end{itemize}
\item[] {\em args} : {\em action} dependent arguments, usually a pass name and possible list of job numbers
\end{itemize}

Note that exept for the {\tt listall} and {\tt prod} actions, the current working directory must contain the {\tt .joblist} file for a project.






% difxsniff -------------------------------------------------------------------

\subsection{difxsniff {\small $\mathrm{(package: SniffPlots)}$}} \label{sec:difxsniff}

Program {\tt difxsniff} is a reimplementation of the VLBA analysts' program {\tt sniff.pd} to be more appropriate for software correlation where the sniffer data is generated at the same time as the FITS files.
It uses the same underlying set of plotting programs ({\tt plotwt}, {\tt plotbp}, and {\tt plotapd}) as {\tt sniff.pd} did.
It should be run in a project directory as it will create a subdirectory (if not existing already) which by default is called {\tt sniffer/}{\em refant} within the current directory.
All files created by {\tt difxsniff} will be placed in this directory, overwriting existing files with the same filenames.
Unlike {\tt sniff.pd}, {\tt difxsniff} is a purely non-interactive command line program.
Note that although {\tt .FITS} files are provided to {\tt difxsniff}, it is the associated files ending in {\tt .apd}, {\tt .wts}, {\tt .acb} and {\tt .xcb} that are actually read.

\begin{itemize}
\item[] Usage: {\tt difxsniff} $[$ {\em options} $]$ {\em refants} {\em FITS}1 $[\ \cdots$ {\em FITS}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em refants} is a list reference antennas, separated by spaces
\item[] {\em FITS} is a FITS file created by {\tt difx2fits}; multiple FITS files can be specified together
\item[] Example 1: {\tt difxsniff LA *.FITS}
\item[] Example 2: {\tt difxsniff NL FD *.FITS}
\end{itemize}






% difxspeed -------------------------------------------------------------------

\subsection{difxspeed {\small $\mathrm{(package: vex2difx)}$}} \label{sec:difxspeed}

Program {\tt difxspeed} does processing benchmarking, possibly over a range of parameters, of DiFX.
To ensure that data playback (reading from files, Mark5 modules or network) are not limiting performance, the FAKE mode of DiFX (see \S\ref{sec:fake}) is used; thus the output data are meaningless.
{\tt difxspeed} takes a {\tt .speed } (\S\ref{sec:speed}) file as input.
This file contains various parameters, many of which are identical to those in the {\tt .v2d} (\S\ref{sec:v2d}) files.
An important difference with the parameters specified in {\tt .speed} files is that multiple values can be provided for many of the parameters.
In the benchmarking process, a separate run of DiFX for each combination of the supplied parameters is performed.
The first combination is run twice, with the first being labeled a {\em dummy} run.
This is because the timing of the first execution can vary depending on recent usage of the correlator.

\begin{itemize}
\item[] Usage: {\tt difxspeed} $[$ {\em options} $]$ {\em inputFile} $[$ {\em numIterations} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em inputFile} is the {\tt .speed} file describing the series of benchmarks to run
\item[] {\em numIterations} is the number of times to execute all test combinations
\end{itemize}

Each run of {\tt difxspeed} will append a new column of data to a file called {\em inputFile}{\tt .out} ; if the file does not exist, a new file will be created.
Documentation of this output file format can be found in \S\ref{sec:speed.out}.







% difxusage -------------------------------------------------------------------

\subsection{difxusage {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:difxusage}

This is an NRAO-only program owing to its ties to the VLBA database.

Program {\tt difxusage} mines the VLBA database for correlator usage statistics.
Usage is as follows:

\begin{itemize}
\item[] Usage: {\tt difxusage} $[$ {\em options} $]$ {\em mjdStart} {\em mjdStop}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution
\item[] {\tt -l} or {\tt --list} : print all matching jobs
\item[] {\tt -a} or {\tt --all} : select jobs in all states
\item[] {\tt -c} or {\tt --complete} : select only complete jobs (default)
\item[] {\tt -k} or {\tt --killed} : select only killed jobs
\item[] {\tt -u} or {\tt --unknown} : select only unknown jobs
\end{itemize}
\item[] {\em mjdStart} is the start time (in Modified Julian Days) to look for jobs
\item[] {\em mjdStop} is the stop time (in Modified Julian Days) to look for jobs
\end{itemize}

\noindent
Note that environment variable {\tt VLBA\_DB} must be set to point to the postgres database in question.






% difxvmf ---------------------------------------------------------------------

\subsection{difxvmf {\small $\mathrm{(package: calcif2)}$}} \label{sec:difxvmf}

{\em Note: this is coming in DiFX 2.7 series\ldots}

{\tt difxvmf} takes a DiFX fileset (including the {\tt .im} file) and modifies the wet and dry troposphere values based on the Vienna Mapping Functions.
This program retrieves the needed external data from \url{http://vmf.geo.tuwien.ac.at}. 
The {\tt .im} file will be replaced with an updated version.

\begin{itemize}
\item[] Usage: {\tt difxvmf} $[$ {\em options} $]$ {\em filebase1} $[$ {\em filebase2} \ldots $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution
\item[] {\tt -w} or {\tt --usewx} : use metrology data from each site rather than defaults
\end{itemize}
\item[] {\em filebase}n is the {\tt .input} file or prefix to be processed; multiple can be provided.
\end{itemize}

If {\tt --usewx} is specified, files of the form {\em project}{\tt .}{\em station}{\tt .weather} will be looked for in the local directory and used to supply metrology data, overriding defaults.

\noindent Environment variables:
\begin{itemize}
\item[] {\tt DIFX\_VMF\_DATA} : a writable directory for caching downloaded VMF data
\item[] {\tt DIFX\_VERSION} : to enforce DiFX version compatibility
\end{itemize}



% difxwatch -------------------------------------------------------------------

\subsection{difxwatch {\small $\mathrm{(package: difxmessage)}$}} \label{sec:difxwatch} 

Program {\tt difxwatch} can be used to monitor progress of ongoing DiFX jobs and kill jobs that appear to be hung.

\begin{itemize}
\item[] Usage: {\tt difxwatch} $[$ {\em options} $]$ 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt --version} : show program's version number and exit
\item[] {\tt -i} {\em idletime} or {\tt --idle-time} {\em idletime} : maximum number of seconds a job is allowed to be idle before it is killed.
\end{itemize}
\end{itemize}





% dir2filelist.pl ---------------------------------------- INCOMPLETE ---------

%----------------------------------------------------------------------------------
%Script to convert a .dir file created by mk5dir into a filelist that can be 
%evaluated by vex2difx (parameter filelist) 
%---------------------------------------------- -----------------------------------

%Usage: /export/home/shiitake/difx/WFBDIFX-TRUNK/bin/dir2filelist.pl dirfile [prefix]

%dirfile: the .dir file created by mk5dir
%prefix:  [optional] if given the prefix will be prepended to the disk filenames

%output will be written to dirfile.filelist

%---------------------------------------------- -----------------------------------



% doi -------------------------------------------------------------------------

\subsection{DiFX Operator Interface} \label{sec:doi}

The DiFX Operator Interface (DOI) is a java-based application to monitor and control the correlation of DiFX jobs.
Jobs to be correlated can be selected with a file browser or retrieved with a database request.
A separate manual \cite{doi} will be made available with instructions for its use.
The only specific detail that will be mentioned here is the contorted path the starting of a job takes:
\begin{enumerate}
\item The job to run is selected.
\item The DOI determines which resources (Mark5 units and processor nodes) are required.
\item If the intended output file already exists, a dialog will ask the operator whether to overwrite this file or not.
\item The DOI allocates resources.
\item The DOI assembles a {\tt DifxStartMessage} XML document and multicasts it with the correlator head node as the recipient.
\item The {\tt mk5daemon} process running on the head node captures this message.
\item {\tt mk5daemon} fork()s; the child process changes its userId to {\tt difx} and spawns an {\tt mpirun} process via {\tt ssh} to ensure the proper environment variables are set.
\item The {\tt mpirun} process starts a copy of {\tt mpifxcorr} on each of the Mark5 units and processing nodes that is requested.
\item {\tt mk5daemon} fork()s again; the child process changes its userId to {\tt difx} and spawns a {\tt difxlog} process via {\tt ssh} to ensure the proper environment variables are set.
\item All processes continue until job end is reached or the job is killed.
\item When the first fork()ed {\tt mk5daemon} process ends, the {\tt difxlog} process stops automatically, causing the second fork()ed process also to stop.
\item The DOI receives messages suggesting the job has ended and frees the allocated resources.
\end{enumerate}







% e2ecopy ---------------------------------------------------------------------

\subsection{e2ecopy {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:e2ecopy}

Program {\tt e2ecopy} copies files one directory to another, changing the ownership to 
{\tt \$DIFX\_ARCH\_USERNAME} in the process.
This program must be setuid root; the person installing the program must run {\tt chmod +s e2ecopy} after installation if {\tt make install} is not run by root.
Normally this program is run by {\tt difxarch} (see \S\ref{sec:difxarch}).
This is a VLBA-centric program, but could be used by others.

\begin{itemize}
\item[] Usage: {\tt e2ecopy} $[$ {\em options} $]$ {\em fromDir} {\em toDir} {\em file1} $[\ \cdots$ {\em file}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose
\end{itemize}
\item[] {\em fromDir} : the source directory of the file(s) to copy
\item[] {\em toDir} : the destination directory
\item[] {\em file} : a file to copy (multiple files may be provided)
\end{itemize}

\noindent
Note: ``e2e'' is NRAO terminology for ``End to End'', a philosophy of providing user software covering the full project lifecycle from proposal handling to archive access.
In this particular case the name arose due to the location of the archive staging area at NRAO.





% errormon --------------------------------------------------------------------

\subsection{errormon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:errormon} 

Program {\tt errormon} listens for multicast messages of the {\tt difxError} variety and simply prints their contents to the terminal.
It is effectively the same as {\tt difxlog} except that log data is sent to {\em stdout} rather than a systematically named file.

\begin{itemize}
\item[] Usage: {\tt errormon} $[$ options $]$ $[${\em maxSeverity}$]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em maxSeverity} : maximum severity level to display (default = 8)
\end{itemize}

See \S~\ref{sec:difxalertmessage} for a list of severity codes.
If no {\em maxSeverity} is provided, the default level of 8 will cause no selection to occur; all messages will be printed.

A similar program, {\tt errormon2}, does nearly the same thing, but defaults to a less verbose output, and sends output to {\em stderr} rather than {\em stdout} (so use with {\tt grep} or other *nix tools is more cumbersome.
It also writes its output to a log file.

See documentation for {\tt difxlog} to see the list of alert levels.









% extractSingleVDIFThread -----------------------------------------------------

\subsection{extractSingleVDIFThread {\small $\mathrm{(package: vdifio)}$}}

This program has been superceded by {\tt vmux} (see Sec.~\ref{sec:vmux}).





% extractVDIFThreads ----------------------------------------------------------

\subsection{extractVDIFThreads {\small $\mathrm{(package: vdifio)}$}} 

This program has been superceded by {\tt vmux} (see Sec.~\ref{sec:vmux}).






% fakemultiVDIF ----------------------------------------- INCOMPLETE ---------

\subsection{fakemultiVDIF {\small $\mathrm{(package: vdifio)}$}} \label{sec:fakemultivdif}

Note: the input file for {\tt fakemultiVDIF} must have a single thread or unpredictable results will occur.




% fileto5c ----------------------------------------------- INCOMPLETE ---------

\subsection{fileto5c {\small $\mathrm{(package: mark5daemon)}$}} \label{sec:fileto5c}





% filterVDIF --------------------------------------------- INCOMPLETE ---------

\subsection{filterVDIF {\small $\mathrm{(package: vdifio)}$}} \label{sec:filtervdif}



% generateVDIF ------------------------------------------- INCOMPLETE ---------

\subsection{generateVDIF {\small $\mathrm{(package: vdifio)}$}} \label{sec:generatevdif}




% genmachines -----------------------------------------------------------------

\subsection{genmachines {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:genmachines}

Program {\tt genmachines} uses the information in a {\tt .input} file and a file containing information about the members of the compute cluster (such as the file pointed to by {\tt \$DIFX\_MACHINES}) to produce a {\tt .machines} file (\S\ref{sec:machines}) needed by {\tt mpifxcorr}.
Note that {\tt genmachines} is not intended to be run by hand anymore as {\tt startdifx} does this,
if necessary.
If playback directly off Mark5 units is to be done, {\tt genmachines} will send a multicast request to all Mark5 units on the correlator requesting an inventory of loaded Mark5 modules.
The {\tt mk5daemon} process on each unit will respond with another multicast message containing the loaded modules and the status of the unit, i.e., whether busy or available to be used.
This information is collected by {\tt genmachines} which will look for availability of all the modules and detect conflicts (i.e., two needed modules loaded in the same unit).
If all needed modules are found and enough resources remain for the computations, a {\tt .machines} file and a {\tt .threads} file are written.
Note that the {\tt .machines} file contains a certain number of comment lines so that the use of Unix command {\tt wc -l} can be used to determine exactly how many processes will be started.
It is suggested to run this program immediately before starting the software correlator to minimize the chance that the Mark5 units change their status or that information about the modules whereabouts becomes stale; it is thus discouraged to run with {\tt *.input}.

\begin{itemize}
\item[] Usage: {\tt genmachines} $[$ {\em options} $]$ {\em input}1 $[\ \cdots$ {\em input}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose
\item[] {\tt -o} or {\tt --overheadcores} {\em ohc} : leave at least {\em ohc} on each compute node unscheduled
\item[] {\tt -m} {\em file} or {\tt --machinesfile} {\em file} : use {\em file} instead of {\tt \$DIFX\_MACHINES}
\item[] {\tt -n} or {\tt --no-threads} : don't write a {\tt .threads} file.
\item[] {\tt -d} or {\tt --difxdb} : lookup module locations in a database.
\end{itemize}
\item[] {\em input} is a {\tt .input} file; multiple files can be specified, each producing its own {\tt .machinesfile}
\end{itemize}


% FIXME: info on which database and associated env. vars.






% getshelf --------------------------------------------------------------------

\subsection{getshelf {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:getshelf}

This is an NRAO-only program owing to its ties to the VLBA database.

Program {\tt getshelf} retrieves the shelf location of specified modules from the legacy VLBA database and prints them to the screen.
While possibly useful, this program is not required for the software correlation process.

\begin{itemize}
\item[] Usage 1: {\tt getshelf} $[$ {\em options} $]$ {\em module1} $[$ {\em module2} $[ \cdots ] ]$
\item[] Usage 2: {\tt getshelf} $[${\em options}$]$ {\em shelfFile}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : print the database query string as well
\end{itemize}
\item[] {\em moduleN} is the volume serial number (VSN) of a module; multiple modules can be specified
\item[] {\em shelfFile} is a {\tt .shelf} file (\S\ref{sec:shelf}), as may be written by {\tt db2vex}
Default is the current working directory if none is provided.
\item[] Example 1: {\tt getshelf NRAO+267}
\item[] Example 2: {\tt getshelf bx123a.skd.shelf}
\end{itemize}








% getvex -------------------------------------------------- OBSOLETE ----------
%
%\subsection{getvex {\small $\mathrm{(package: nrao\_difx\_db)}$}} \label{sec:getvex}
%
%This is an NRAO-only program owing to its assumption of file locations.
%
%Python program {\tt getvex} can be used to create a new software correlation project directory and copy {\tt .skd} files (vex files), {\tt .oms} files (from output of sched) and a {\tt cal.vlba} file from {\tt /home/vlbiobs} into the new project directory.
%If the {\tt cal.vlba} file is found, it is {\tt gunzip}ed if needed and run through {\tt vlog} (\S\ref{sec:vlog}).
%The sched {\tt .oms} file is parsed to create a base {\tt .v2d} file with program {oms2v2d} that simplifies difx job creation.
%
%\begin{itemize}
%\item[] Usage: {\tt getjobs} $[$ {\em options} $]$ {\em project} $[$ {\em rootDir} $]$
%\item[] {\em options} can be:
%\begin{itemize}
%\item[] {\tt -h} or {\tt --help} : print usage information and exit
%\item[] {\tt -f} or {\tt --force} : allow overwrite of files
%\end{itemize}
%\item[] {\em project} is the full name of a project, in lower case, including segment
%\item[] {\em rootDir} is an optional parameter specifying the directory into which the new project directory will be created.
%Default is the current working directory if none is provided.
%\item[] Example 1: {\tt getjobs bw088r}
%\item[] Example 2: {\tt getjobs bg160 \$JOB\_ROOT}
%\end{itemize}
%
%
%\noindent
%Note: This program currently will not work with Mark5C-based projects.
%
%% FIXME: option in code for Mark5C












% jobdisks --------------------------------------------------------------------

\subsection{jobdisks {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:jobdisks}

Program {\tt jobdisks} looks through job files to see which modules (disks) are needed for correlation.
It reads through {\tt .input} files, as used by {\tt mpifxcorr}, to get the needed information.
There are two modes of operation.
By default, a matrix of all modules for all stations is displayed, with a {\tt --} symbol indicating that a particular station is not used in a particular job.
An asterisk ({\tt *}) indicates a module change.
The second mode, instigated with command line argument {\tt -c}, summarizes only module changes.
Running without any arguments will cause {\tt jobdisks} to look at job files within the current directory, prioritizing on {\tt .input} files if any exist and falling back on {\tt .fx} files otherwise.
Listings for a subset of jobs can be made by specifying particular files.

\begin{itemize}
\item[] Usage: {\tt jobdisks} $[$ {\em options} $]$ $[${\em file}1$] \cdots [$ {\em file}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -c} or {\tt --changes} : print module changes only
\end{itemize}
\item[] {\em file} is a {\tt .fx} or {\tt .input} file; mixed types are not supported. 
Multiple input files may be supplied.
\item[] Example 1: {\tt jobdisks}
\item[] Example 2: {\tt jobdisks job1420*.input}
\item[] Example 3: {\tt jobdisks *.fx}
\item[] Example 4: {\tt jobdisks -c}
\end{itemize}


% FIXME
\noindent
Known bugs:
\begin{enumerate}
\item Program should make sure datastream type is MODULE.
\end{enumerate}






% joblist ---------------------------------------------------------------------

\subsection{joblist {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:joblist}

Program {\tt joblist} prints useful information about DiFX correlator jobs to {\em stdout}.
Six columns of output are produced:
\begin{enumerate}
\item Job file base filename
\item File indicator, showing a particular character for each one of the files associated with that job that is found within a pair of square brackets, $[\ ]$:
\begin{itemize}
\item[{\tt c}] {\tt .calc} file (\S\ref{sec:calc})
\item[{\tt m}] {\tt .machines} file (\S\ref{sec:machines})
\item[{\tt t}] {\tt .threads} file (\S\ref{sec:threads})
\item[{\tt i}] {\tt .im} file (\S\ref{sec:im})
\item[{\tt v}] {\tt .difx} file (\S\ref{sec:difx})
\end{itemize}
\item Band code of first scan in file
\item Observation duration of correlation (in minutes)
\item Recording mode triplet; three integers(data rate(Mbps), number of baseband channels \& quantization bits) separated by dashes
\item Comma separated list of antennas
\end{enumerate}
One line is printed for each {\tt .input} file found in the list of directories provided (or current directory if not listed).

\begin{itemize}
\item[] Usage: {\tt joblist} $[$ {\em options} $]$ $[$ {\em dir}1 $] \cdots [$ {\em dir}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em dir} is a directory for which to print job information (default is current shell directory).
Multiple directories can be specified.
\item[] Example 1: {\tt joblist}
\item[] Example 2: {\tt joblist \$JOB\_ROOT/*}
\end{itemize}







% jobstatus -------------------------------------------------------------------

\subsection{jobstatus {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:jobstatus}

\noindent
{\bf Warning: As of DiFX 2.0.1, this utility has not yet been updated to work with DiFX 2 output format.}

\noindent
Program jobstatus lists the current correlation progress for each DiFX job in one or more directories.
This program is normally run without any command line arguments from within the project directory.
For each job, the base filename is listed with 5 or 6 additional columns of data.
These columns are
\begin{enumerate}
\item Observation duration (minutes)
\item Record mode triplet ({\em Mpbs-nChan-nBit})
\item Number of stations in job
\item Speed up factor (ratio of correlation time to observe time), or zero if correlation has not yet begun.
\item Percentage complete
\item Number of minutes remaining (only if Percentage complete isn't 0\% or 100\%)
\end{enumerate}
Below these lines, five more lines containing information about the group of jobs as a whole is are presented.
The contents of these lines are:
\begin{enumerate}
\item Total job time : Minutes of observe time in listed jobs
\item Fraction complete : Percentage in time through the entire project
\item Job time remaining : Minutes of observation left to be correlated
\item Wall time remaining : Minutes of real time needed to complete jobs
\item Average speedup : Ratio of total correlation time to run time, up to current point
\end{enumerate}
Note that the speedup and time remaining values are estimates and don't include model calculation, conversion to FITS, and job startup time.

\begin{itemize}
\item[] Usage: {\tt jobstatus} $[$ {\em options} $]$ $[$ {\em dir}1 $] \cdots [$ {\em dir}N $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\item[] {\em dir} is a directory for which to print job information (default is current shell directory).
Multiple directories can be specified.
\item[] Example 1: {\tt jobstatus}
\item[] Example 2: {\tt jobstatus \$JOB\_ROOT/*}
\end{itemize}


% FIXME: code needs updating
\noindent
Known bugs:
\begin{enumerate}
\item This program has not been updated to work with DiFX 2.0 output
\end{enumerate}






% listcpus --------------------------------------------------------------------

\subsection{listcpus {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:listcpus}

Python program {\tt listcpus} uses ssh to connect to each machine listed in a file (usually {\tt \$DIFX\_MACHINES}) and peaks at the list of CPUs on that machine and prints to stdout.
Only the first column of this file is used and any content after a {\tt \#} is ignored.
For each CPU on the machine, the model name, which usually also contains the CPU speed, is listed.
For multi-core CPUs, each core will appear as its own CPU.

\begin{itemize}
\item[] Usage: {\tt listcpus} $[$ {\em options} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
\item[] {\tt -m} {\em file} or {\tt --machines} {\em file} : use {\em file} instead of {\tt \$DIFX\_MACHINES} for list of machines to probe
\end{itemize}
Multiple directories can be specified.
\item[] Example 1: {\tt listcpus}
\item[] Example 2: {\tt listcpus -m myCPUs.list}
\end{itemize}








% makefits --------------------------------------------------------------------

\subsection{makefits {\small $\mathrm{(package: difx2fits)}$}} \label{sec:makefits}

This program is tuned for NRAO use; some modifications may be required for use at other sites.
In particular, this program requires that jobs were queued to be run in {\tt \$DIFX\_QUEUE\_BASE/}/{\em experiment}{\tt /}.

Program {\tt makefits} is basically a wrapper for {\tt difx2fits} (\S\ref{sec:difx2fits}) that does some sanity checking and ensures that files end up in the proper places with the proper names.
This program is intrinsically {\em pass-based} and it bases its functionality on the {\tt .joblist} (\S\ref{sec:joblistfile}) file that is written by {\tt vex2difx} (\S\ref{sec:vex2difx}).
One must run this program on the software correlator head node ({\tt swc000} in the current VLBA DiFX implementation).
Upon successful completion, FITS-IDI files are created in the same directory in the correlator job staging area ({\tt \$DIFX\_QUEUE\_BASE/}{\em projectName}) and the sniffer output files are left in a subdirectory of the current working directory.
An additional output file is left in the current working directory called {\em passName}{\tt .fitslist} .
This file has a list of the FITS files that are to be archived once the data for this pass are deemed valid.

The checks that {\tt makefits} performs will by default not allow an incomplete set of FITS files to be produced.
This can be overridden with a special command line argument (below).
This is part of an accountability chain that aims to ensure that nothing gets omitted.

\begin{itemize}
\item[] Usage: {\tt makefits} $[$ {\em options} $]$ {\em passName}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
\item[] {\tt --override-version} : ignore potential difx version conflicts
\item[] {\tt --allow-partial} : bypass check for complete set of correlated output and proceed
\end{itemize}
Multiple directories can be specified.
\item[] Example: {\tt makefits clock}
\end{itemize}







% makemark4 -------------------------------------------------------------------

\subsection{makemark4 {\small $\mathrm{(package: difxdb)}$}} \label{sec:makemark4}

This program is tuned for NRAO use; some modifications may be required for use at other sites.

Program {\tt makemark4} is essentially a wrapper for {\tt difx2mark4} (\S\ref{sec:difx2mark4}) that does some sanity checking and ensures that files end up in the proper places with the proper names.
This program is intrinsically {\em pass-based} and it bases its functionality on the {\tt .joblist} (\S\ref{sec:joblistfile}) file that is written by {\tt vex2difx} (\S\ref{sec:vex2difx}).
One must run this program on the software correlator head node ({\tt swc000} in the current VLBA DiFX implementation).
Upon successful completion, Mark4 file sets are created in the same directory in the correlator job staging area ({\tt \$DIFX\_QUEUE\_BASE/}{\em projectName}).
An additional output file is left in the current working directory called {\em passName}{\tt .mark4list} .
This file has a list of the Mark4 file sets that are to be archived once the data for this pass are deemed valid.

\begin{itemize}
\item[] Usage: {\tt makemark4} $[$ {\em options} $]$ {\em passName}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
%\item[] {\tt --override-version} : ignore potential difx version conflicts
\end{itemize}
\item[] Example: {\tt makemark4 rdv95}
\end{itemize}







% m5bstate --------------------------------------------------------------------

\subsection{m5bstate {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5bstate}

Program {\tt m5bstate} will perform state counts on a baseband data file.

\begin{itemize}
\item[] Usage: {\tt m5bstate} {\em file} {\em format} {\em nFrames} $[$ {\em offset} $]$ 
\item[] {\em file} is the file to decode
\item[] {\em format} is the format of the data
\item[] {\em nFrames} is the number of data frames (typically a few kB in size) to decode
\item[] {\em offset} is the number of bytes into the file to start decoding
\item[] Example: {\tt m5bstate sample.vlba VLBA1\_2-256-8-2 100}
\end{itemize}

\noindent
Notes:
\begin{enumerate}
\item See documentation for {\tt m5b} for details on specifying the data format.
\item Only real-sampled data with 1 or 2 bits per sample is supported at this time.
\item In the case of VDIF data, only single thread data with $2^n$ channels is supported.
For equivalent functionality in the multi-thread VDIF case see {\tt vdifd}.
\end{enumerate}






% m5d -------------------------------------------------------------------------

\subsection{m5d {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5d}

Program {\tt m5d} is a very simple example program using the mark5access decoding library.
It turns out to be useful enough as a stand-alone program to be separately documented.
This program takes as command line input the name of a file containing (or thought to be containing) VLBI baseband data, the expected format of the data, and the number of samples per baseband to decode.
Optionally a starting file offset can be supplied.
If the data can be decoded correctly, information about the data will be printed to the screen along with a table of decoded data.
The output values, -3, -1, 1, or 3 for valid data, are printed in {\em nchan} columns.
Data that cannot be decoded (either due to data replacement headers, data fill pattern replacing the actual data after unloading from a Mark5 module, or identified via the VDIF invalid bit) will show as 0.
It should be invoked with the following parameters:

\begin{itemize}
\item[] Usage: {\tt m5d} {\em file} {\em format} {\em n} $[$ {\em offset} $]$ 
\item[] {\em file} is the file to decode
\item[] {\em format} is the format of the data
\item[] {\em n} is the number of samples to decode
\item[] {\em offset} is the number of bytes into the file to start decoding
\item[] Example 1: {\tt m5d sample.vlba VLBA1\_2-256-8-2 24}
\item[] Example 2: {\tt m5d sample.mk4 MKIV1\_4-128-2-1 600 200} 
\item[] Example 3: {\tt m5d sample.5b Mark5B-512-16-2 1200}
\end{itemize}

The format parameter is constructed from four parts as {\em type}-{\em rate}-{\em nchan}-{\em nbit} where:
\begin{itemize}
\item[] {\em type} is the type of format and should be one of {\tt VLBA1\_1}, {\tt VLBA1\_2}, {\tt VLBA1\_4}, {\tt MKIV1\_1}, {\tt MKIV1\_2}, {\tt MKIV1\_4}, {\tt Mark5B}, or {\tt VDIF}
\item[] {\em rate} is the data rate in Mbps
\item[] {\em nchan} is the number of baseband channels
\item[] {\em nbit} is the number of bits per recorded sample
\end{itemize}
See the usage examples above for some explicit values.
Note for the {\tt VLBA} and {\tt MKIV} format types the fanout is appended as this affects the decodability of the files.

\noindent
Notes:
\begin{enumerate}
\item In the case of VDIF data, only single thread data with $2^n$ channels is supported.
For equivalent functionality in the multi-thread VDIF case see {\tt vdifd}.
\end{enumerate}




% m5findformats ---------------------------------------------------------------

\subsection{m5findformats {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5findformat}

Program {\tt m5findformats} attempts to determine which format a baseband data file may be.
Currrently it searches over 16 to 2048 Mbps data rates in factors of 2 and checks only for MKIV, VLBA and Mark5B types.

\begin{itemize}
\item[] Usage: {\tt m5findformats} {\em filename}
\item {\em filename} is the name of the baseband data file.
\end{itemize}

Run with no command line arguments to get help information.






% m5fold ----------------------------------------------------------------------

\subsection{m5fold {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5fold}

Program {\tt m5fold} takes a baseband data stream and integrates the power (formed by squaring the voltage) in a number of time bins that equally divide a given period.
This is a simplifed version of ``folding'' such as is used in pulsar processing.
A typical use of such functionality would be to investigate the waveform of the switched power injected into the receiver for calibration.
This program has found considerable utility in determining time offsets between the sample clock and formatter time (modulo the period of the calibration cycle).
In the case of 2-bit sampling a non-linear correction is applied before results are written to a file.
This correction takes the form
\begin{equation}
P = \frac{1}{\left({\rm erf}^{-1}\left(\frac{\hat{P} - v_{\rm high}^2}{1 - v_{\rm high}^2} \right)\right)^2},
\end{equation}
where $P$ is a value proportional to true power and $\hat{P}$ is the value obtained by calculating $\left<\hat{v}^2\right>$ when the bitstream is reproduced with values $\hat{v} \in ( -v_{\rm high}, -1, 1, v_{\rm high} )$. 
This non-linear correction can be turned off by setting {\em nbin} to a negative value.
Note that this program is not useful for 1-bit quantized data.
The program should be used as follows:

\begin{itemize}
\item[] Usage: {\tt m5fold} {\em infile} {\em format} {\em nbin} {\em nchunk} {\em freq} {\em outfile} $[$ {\em offset} $]$ 
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data`
\item[] {\em nbin} is the number of bins to calculate per period; if negative, power correction is not performed and the absolute value of {\em nbin} is used
\item[] {\em nchunk} is the number of 10000 sample chunks to operate on
\item[] {\em freq} is the reciprocal of the period to be observed (Hz)
\item[] {\em outfile} is the name of the output file
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt m5fold sample.vlba VLBA1\_2-256-8-2 128 10000 80 sample.fold}
\end{itemize}

See the documentation for {\tt m5d} for information on specifying the data format.

The output file will contain {\em nchan}+1 columns where {\em nchan} is the number of baseband channels in the data stream.
The first column contains the time (seconds) within the period.
Each remaining column is folded power for one baseband channel.
If {\em nbin} is positive and the data is 2-bit quantized, the scaling is such that $\left<v^2\right> = \sigma^2$ yields a power reading of 1.0, for sampler threshold $\sigma$.
Optimal signal to noise ratio occurs for a value of about 1.03.
For non 2-bit quantization, the power will be in units of reconstituted ${\rm counts}^2$.

In the case of VDIF data, only single thread data with $2^n$ channels is supported.
For equivalent functionality in the multi-thread VDIF case see {\tt vdiffold}.








% m5pcal ----------------------------------------------------------------------

\subsection{m5pcal {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5pcal}

Program {\tt m5pcal} can be used to extract pulse cal tones from baseband data in Mark4, VLBA, Mark5B and single-thread VDIF formats.

\begin{itemize}
\item[] Usage: {\tt m5pcal} $[$ {\em options} $]$ {\em infile} {\em format} {\em freq1} $[$ {\em freq2} $[ \ldots ] ]$ {\em outfile}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -c} {\em n} or {\tt --chunksize} {\em n} : use a fixed rather than automatic chunk size
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
\item[] {\tt -q} or {\tt --quiet} : decrease output verbosity
\item[] {\tt -n} {\em n} : loop over {\em n} chunks of data (default is 1000)
\item[] {\tt -N} {\em N} : perform {\em N} outer loops, each yielding a result set
\item[] {\tt -o} {\em o} or {\tt --offset} {\em o} : jump {\em o} bytes into file
\item[] {\tt -i} {\em i} or {\tt --interval} {\em i} : use pulse cal comb interval of {\em i} MHz (default is 1)
\item[] {\tt -e} {\em e} or {\tt --edge} {\em e} : don't use channels closer than {\em e} MHz from band edges when computing delay (default is 1/8 of bandwidth)
\end{itemize}
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data`
\item[] {\em freq1} \ldots is/are the frequencies (MHz) relative to baseband of the first tone to detect; there should be one {\em freq} specified per baseband channel
\item[] {\em outfile} is the name of the output file
\end{itemize}

Note: The position of the first tone in a baseband channel ({\em freq1} for baseband 1, and so on) must not be larger than the tone interval (set with {\tt -i} {\em i}).
All tones are extracted from each baseband channel.
The tone interval is allowed to exceed the bandwidth of a baseband channel in which case {\em freqN} will effectively select just a single tone from the baseband.

See the documentation for {\tt m5d} for information on specifying the data format.







% m5slice ------------------------------------------------- INCOMPLETE --------

\subsection{m5slice {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5slice}





% m5spec ----------------------------------------------------------------------

\subsection{m5spec {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5spec}

Program {\tt m5spec} is an example program using the mark5access decoding library that is a bit more advanced than the {\tt m5d} program is.
It forms total power spectra for each baseband channel in the data, including cross spectra for polarization pairs, assuming data is in alternating polarization pairs (if not, the cross spectra should make no sense, but they are formed anyway).
The results are written to a text file with the following columns: Column 1 is the frequency offset from baseband for each channel; Columns 2 to {\em nchan}+1 are the total power spectra for each baseband channel; Columns {\em nchan}+2 to $4 \times${\em nchan}+1 contain, in pairs, the amplitude and phase of the cross spectra for each pair of channels.
It should be invoked with the following parameters:

\begin{itemize}
\item[] Usage: {\tt m5spec} {\em infile} {\em format} {\em npoint} {\em n} {\em outfile} $[$ {\em offset} $]$ 
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data
\item[] {\em npoint} is the number of points to calculate for each spectrum
\item[] {\em n} is the number of FFT frames to include in the calculation
\item[] {\em outfile} is the name of the output file
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt m5spec sample.vlba VLBA1\_2-256-8-2 256 1000 vlba.spec}
\end{itemize}

See the documentation for {\tt m5d} for information on specifying the data format.

In the case of VDIF data, only single thread data with $2^n$ channels is supported.
For equivalent functionality in the multi-thread VDIF case see {\tt vdifspec}.





% m5test ----------------------------------------------------------------------

\subsection{m5test {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5test}

Program {\tt m5test} is an example program using the mark5access decoding library that works its way through a VLBI baseband data stream attempting to decode data and header information to look for problems.
Every million samples (per baseband channel) a summary line containing frame number, decoded date and time, and counts of valid and invalid frames are shown.
After 20 invalid frames are encountered the program will stop.  
Otherwise the program will run until end of file or until interrupted by the user.
Usage is as follows:

\begin{itemize}
\item[] Usage: {\tt m5test} {\em infile} {\em format} $[$ {\em offset} $]$ 
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt m5test sample.vlba VLBA1\_2-256-8-2}
\end{itemize}

See the documentation for {\tt m5d} for information on specifying the data format.





% m5time ----------------------------------------------------------------------

\subsection{m5time{\small $\mathrm{(package: mark5access)}$}} \label{sec:m5time}

Program {\tt m5time} decodes the time of the beginning of a Mark4, VLBA, or Mark5B datastream and prints the result in integer MJD and UT hours, minutes, seconds to the screen.

\begin{itemize}
\item[] Usage: {\tt m5time} {\em infile} {\em format}
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data`
\end{itemize}




% m5timeseries ----------------------------------------------------------------

\subsection{m5timeseries {\small $\mathrm{(package: mark5access)}$}} \label{sec:m5timeseries}

Program {\tt m5timeseries} produces a power measurments for each of channel of a baseband data file, averaging over a specified time interval.

\begin{itemize}
\item[] Usage: {\tt m5timeseries} {\em infile} {\em format} {\em tint} {\em ntime} {\em outfile} $[$ {\em offset} $]$
\item[] {\em infile} is the file to decode
\item[] {\em format} is the format of the data`
\item[] {\em tint} is the integration time per sample in milliseconds
\item[] {\em ntime} is the number of samples to generate
\item[] {\em outfile} is the name of the output file
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt m5timeseries sample.vlba VLBA1\_2-256-8-2 6.25 8000 sample.series}
\end{itemize}

The output file contains {\em nchan}+2 columns of data where {\em nchan} is the number of channels in the data file.
The first column is sample number.
The second column is time since beginning of series, in seconds.
The remaining columns are power measurements for the channels.




% m5tsys -------------------------------------------------- INCOMPLETE --------

\subsection{m5tsys {\small $\mathrm{(package: mark5access)}$}} \label{sec:tsysal}







% mk5cat ----------------------------------------------------------------------

\subsection{mk5cat {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:mk5cat}

This program sends data on a module to standard out.
See additional documentation under {\tt mk5cp} which operates on similar principles (mk5cat is mk5cp writing to {\em stdout}
Note that the executable for mk5cat is identical to that for mk5cp and only the name of the program actually differs.

\begin{itemize}
\item[] Usage: {\tt mk5cat} $[$ {\em options} $]$ \{ {\em bank} $\mid$ {\em VSN} \} {\em scans}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity, e.g., print directory to screen
\end{itemize}
\item[] {\em bank} is either {\tt A} or {\tt B}
\item[] {\em VSN} is a valid 8-character VSN of a loaded module
\item[] {\em scans} is one or more scan numbers (starting at 1) with scan numbers separated by commas.
\end{itemize}

Many of the other baseband data utilities documented here such as {\tt m5d}, {\tt m5spec} and {\tt vmux} can take input from {\em stdin} and thus can be mated with {\tt mk5cat}.
Usually a single hyphen ({\tt -}) as the name of the input file indicates this to these programs.

\begin{itemize}
\item[] Example: {\tt mk5cat B PT\_BB241\_No0111 | m5spec - Mark5B-2048-16-2 128 10000 methanol.spec}
\end{itemize}





% mk5control ------------------------------------------------------------------

\subsection{mk5control {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:mk5control} 

{\tt mk5control} is a program that sends XML messages of type {\tt DifxCommand} to the {\tt mk5daemon} programs that run on the software correlator cluster members.
This program is a superset of {\tt mk5take} and {\tt mk5return}, allowing any allowed command to be sent.

\begin{itemize}
\item[] Usage: {\tt mk5control} $[$ {\em options} $]$ {\em command} {\em unit}1 $\cdots$ {\em unit}N
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit.
\end{itemize}
\item[] {\em command} is the (non-case-sensitive) command to be executed; see list below.
\item[] {\em unit} is the number of a correlator Mark5 unit, a range, {\tt all} for all software correlator cluster members, {\tt mark5} for all Mark5 units, or {\tt swc} for all software correlator compute nodes.
\item[] Example 1: {\tt mk5control stopmark5a 07 08 09 11 14}
\item[] Example 2: {\tt mk5control resetmark5 14-24} 
\item[] Example 3: {\tt mk5control startmark5a mark5}
\end{itemize}

The list of supported {\em command} types is below.
All commands are not case sensitive.
\begin{itemize}
\item {\tt GetVSN} Request a {\tt Mark5Status} XML document to be multicast from the {\em unit}
\item {\tt ResetMark5} Execute {\tt SSReset} and {\tt ssopen}; this cures many/most mark5 hangs
%\item {\tt StartMark5A} Start the {\tt Mark5A} program (see \S\ref{sec:mk5returnprogram})
%\item {\tt StopMark5A} Stop the {\tt Mark5A} program (see \S\ref{sec:mk5takeprogram})
\item {\tt Clear} Clear the stat of the Mark5 unit and get the VSNs, can be dangerous if other programs are currently using the Streamstor card
\item {\tt Reboot} Reboot the machine
\item {\tt Poweroff} Shut down the machine
\item {\tt StopMk5Daemon} Stop the {\tt mk5daemon} program; you probably never need to do this
\item {\tt GetDir} Extract the directory from the modules in both banks and save to files in \$MARK5\_DIR\_PATH
\item {\tt GetDirA} Same as above, but look only at bank A
\item {\tt GetDirB} Same as above, but look only at bank B
\item {\tt StopDir} Stop a directory read that is in progress
\item {\tt KillMpifxcorr} Send sigkill (like {\tt kill -9}) to all processes on machine called mpifxcorr
\item {\tt Copy} Copy data from a module to files in a provided directory.
At least three parameters must be provided that match the parameters of {\tt mk5cp}.
Because of the way mk5control parses the command line, the word {\tt copy} and the parameters must all be enclosed in quotes.
\item {\tt StopCopy} Stop a data copy process
\item {\tt GetVer} Request send of a {\tt DifxMessageMk5Version} XML message
\item {\tt mount}{\em XX} Cause Linux device {\tt /dev/sd}{\em XX} to be mounted on {\tt /mnt/usb}
\item {\tt umount} Cause {\tt /mnt/usb to be unmounted}
\item {\tt Test} Used in debugging --- for developers only
\end{itemize}







% mk5cp -----------------------------------------------------------------------

\subsection{mk5cp {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:mk5cp}

Program {\tt mk5cp} copies baseband VLBI data from a module to a file somewhere on the operating system filesystem, perhaps an external USB disk.
This program is often started using {\tt mk5control} to tell the instance of {\tt mk5daemon} running on the desired Mark5 unit to start the copy.
Status information is multicast via a {\tt Mark5StatusMessage} document.

\begin{itemize}
\item[] Usage: {\tt mk5cp} $[$ {\em options} $]$ \{ {\em bank} $\mid$ {\em VSN} \} {\em scans} {\em outputDirectory}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity: print directory to screen
\end{itemize}
\item[] {\em bank} is either {\tt A} or {\tt B}
\item[] {\em VSN} is a valid 8-character VSN of a loaded module
\item[] {\em scans} is one or more scan numbers (starting at 1) with scan numbers separated by commas.
No spaces are allowed in the list.
A range can be specified with a hyphen (see examples).
Alternatively, a scan name, or portion thereof, can be specified.
When a partial scan is provided, any scan name matching that partial scan name will be copied.
The scans parameter can also take either a time range (two floating point modified Julian Days connected with an underscore, or a byte range (must be a multiple of 4) via two integers separated by an underscore, or a byte start and a length can be specified with two integers separated by a plus sign.
\item[] {\em outputDirectory} is the directory to which files will be copied.
Make sure the destination directory exists before running this program and make sure sufficient free space remains on that filesystem.
If the {\em outputDirectory} parameter is set to {\tt -} (the hyphen), data will go to stdout.
Another utility called {\tt mk5cat} is derived from this behavior.
\item[] Example 1: {\tt mk5cp NRAO-123 4 /mnt/usb/WaltersProject}
\item[] Example 2: {\tt mk5cp A 1,2,3 /tmp}
\item[] Example 3: {\tt mk5cp B BC120A /mnt/usb/BC120A/PT}
\item[] Example 4: {\tt mk5cp NRAO+255 6-12 /tmp}
\item[] Example 5: {\tt mk5cp NRAO+255 123123100\_124123100 /tmp}
\item[] Example 6: {\tt mk5cp NRAO+255 54327.13124\_54327.15124 /tmp}
\item[] Example 7: {\tt mk5cp NRAO+322 123123100+1000000 /tmp}
\end{itemize}








% mk5daemon -------------------------------------------------------------------

\subsection{mk5daemon {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:mk5daemon} 

{\tt mk5daemon} is a program that started automatically at boot time on all of the software correlator cluster nodes (not only the Mark5 units!) that performs a number of operations in support of the software correlator.

The functions that mk5daemon performs are:
\begin{itemize}
\item {\bf Logging}

All received multicast messages, significant internal functions, and interactions of the {\tt Mark5A} program are logged to human readable log files.
These log files are restarted at the beginning of each day.  
By default these log files are saved in {\tt /tmp}.

\item {\bf High level control of Mark5 units}

The {\tt Mark5A} program (written by Haystack) is the principle program used to access the Mark5 systems at the VLBA stations and the hardware correlator.
DiFX directly accesses the Streamstor card via a library level programming interface.
Since only one program is allowed to do this (or face a crash of varying degree of seriousness), access to the Streamstor
card must be carefully managed.
One function of {\tt mk5daemon} is to maintain knowledge of who ``owns'' the Streamstor card at a given time to prevent conflicts.
The starting and stopping of the {\tt Mark5A} program can be requested by two messages of type {\tt DifxCommand} : {\tt startmark5a} and {\tt stopmark5a}.
When these commands are received by {\tt mk5daemon}, the requested action is taken unless Streamstor conflict is likely.
This type of command and others can be sent to {\tt mk5daemon} with the {\tt mk5control} program (\S\ref{sec:mk5control}).

\item {\bf Low level control of Mark5C units}

This program exposes a VLBI Standard Interface (VSI) over TCP port 2620 that very closely emulates equivalent functionality of the Mark5C Data Recording System (DRS) program provided by Haystack observatory.
The implementation of the DRS command set is not complete but is sufficient for monitor and control at
record time.
At the time of writing this program is used in lieu of DRS at the two Mark5C units provided by USNO.

\item {\bf CPU, memory, and network monitoring}

Every 10 seconds, {\tt mk5daemon} extracts data from the {\tt /proc} directory to get information about the CPU load, memory usage, and network traffic.
These numbers are multicast in a {\tt DifxLoad} message and logged.

\item {\bf Module VSN and state determination}

Receipt of a multicast {\tt getvsn} command will result in {\tt mk5daemon} multicasting out a {\tt Mark5Status} message containing information on the VSNs of the inserted modules as well as the state of the Mark5 unit.
When {\tt Mark5A} is running, a socket is opened to this program and the {\tt bank\_set?} query is issued, which returns the VSNs, regardless of the activity.
When {\tt Mark5A} is not running, {\tt mk5daemon} either directly determines the VSNs through a Streamstor API library call if the Mark5 unit is idle, or doesn't respond if the Mark5 unit is busy.
With each {\tt Mark5Status} message that is multicast from {\tt mk5daemon} the state of the Mark5 unit is included.
See \S\ref{sec:xml} for details on these XML messages.

\item {\bf Starting of mpifxcorr}

If {\tt mk5daemon} is started with the {\tt -H} or {\tt --head-node} option, it will be allowed to start new correlations.
A correlation will be started when a {\tt difxStart} message is received if it passes some minor sanity checks.
Since {\tt mk5daemon} runs as root, it has the capability of changing file ownerships.
By default, the output files from {\tt mpifxcorr} and {\tt difxlog} will have their ownership and permissions changed to match those of the {\tt .input} file.

%\item {\bf Miscellaneous other functionality}

\end{itemize}

Normally {\tt mk5daemon} is started automatically, either by {\tt /etc/rc.local} or by a script in {\tt /etc/init.d} .
The command line options supported are:
\begin{itemize}
\item[] Usage: {\tt mk5daemon} $[$ {\em options} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -q} or {\tt --quiet} : be less verbose and don't mulitcast state
\item[] {\tt -H} or {\tt --head-node} : give head-node permissions
\item[] {\tt -m} or {\tt --isMk5} : force mk5daemon on this host to act as Mark5 regardless of hostname (default is mark5fx??)
\item[] {\tt -u} {\em userID} or {\tt --user} {\em userID} :  use {\em userID} when executing remote commands (default is 'difx')
\item[] {\tt -l} {\em logPath} or {\tt --log-path} {\em logPath} : put logs in directory {\em logPath}, not {\tt /tmp}
\end{itemize}
\end{itemize}

Please be sure not to have multiple instances of {\tt mk5daemon} running at any one time on any individual Mark5 or correlator unit!







% mk5dir ----------------------------------------------------------------------

\subsection{mk5dir {\small $\mathrm{(package: mark5daemon)}$}} \label{sec:mk5dir}

Program {\tt mk5dir} extracts the directory from a module.
Normally one would not call this program directly but would use the {\tt getdir} option of {\tt mk5control}.
By default this program will change the disk module state to {\tt played}.
There is a danger that if this is done with an SDK~9 unit and the disk is later needed in an SDK~8 unit that it will no longer be readable in the later without a full reset of its VSN.
As of April 2014 this program supports decoding of all directory types described by Mark5 Memos 81 (\url{http://www.haystack.mit.edu/tech/vlbi/mark5/mark5_memos/081.pdf}) and 100 (\url{http://www.haystack.mit.edu/tech/vlbi/mark5/mark5_memos/100.pdf}).

\begin{itemize}
\item[] Usage: {\tt mk5dir} $[$ {\em options} $]$ \{ {\em bank} $\mid$ {\em VSN} \}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity: print directory to screen
\item[] {\tt -f} or {\tt --force} : force directory read even if current
\item[] {\tt -F} or {\tt --fast} : get format details from new-style directory (Mark5B format only)
\item[] {\tt -n} or {\tt --nodms} : get directory but don't change disk module state
\item[] {\tt -s} or {\tt --safedms} : only change disk module state if SDK~8 unit or new dir type
\item[] {\tt -d} or {\tt --dmsforce} : proceed with change of module state even if this makes module unreadable in SDK~8 units
\item[] {\tt -b} {\em b} or {\tt --begin} {\em b} : begin with scan number {\em b}
\item[] {\tt -e} {\em e} or {\tt --end} {\em e} : end with scan number {\em b}
\item[] {\tt -w} {\em file} or {\tt --write} {\em file} : write directory file {\em file} to the module
\end{itemize}
\item[] {\em bank} is one of {\tt A}, {\tt B} or {\tt AB}
\item[] {\em VSN} is a valid 8-character VSN of a loaded module
\end{itemize}

\noindent
The resultant directory file will be saved in a file called {\em VSN}{\tt .dir} in the directory pointed to by environment variable {\tt MARK5\_DIR\_PATH} .

This program responds to the value of environment variable {\tt DEFAULT\_DMS\_MASK}.
This variable should be an integer representing the state of three bits.
{\tt mk5dir} only responds to the setting of bit 1 (value 2); if this bit is set, the disk module state will not be updated on directory reading.
It is recommended to set this environment variable at recording stations so auto-erasure of modules does not occur.

In the mode where a specified {\tt .dir} file is to be written to a Mark5 module directory the VSN must be provided explicitly (i.e., selecting by bank is not allowed)






% mk5erase --------------------------------------------------------------------

\subsection{mk5erase {\small $\mathrm{(package: mark5daemon)}$}} \label{sec:mk5erase}

Program {\tt mk5erase} replaces the functionality of {\tt SSErase}.
It is used to either erase or condition a Mark5 module.  It supports SDK9 and
earlier revisions of the Conduant API and either legacy or new (see Mark5 memop 81) module directories.
Conditioning results are multicast upon conclusion of conditioning, to be received bt {\tt condition\_watch}.
Conditioning (which is started with the {\tt -c} option) causes an entire read/write cycle of the entire module to be performed.
This can require a good fraction of 24 hours to complete.
Status updates and progress are sent every 10 seconds as well.
By default the original directory version will be restored on the module, with zero scans.
The version of directory to use can be forced with either the {\tt -l} or {\tt -n} options.

\begin{itemize}
\item[] Usage: {\tt mk5erase} $[$ {\em options} $]$ {\em VSN}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase verbosity: print directory to screen
\item[] {\tt -f} or {\tt --force} : force directory read even if current
\item[] {\tt -c} or {\tt --condition} : Do full conditioning, not just erasing
\item[] {\tt -r} or {\tt --readonly} : Perform read-only conditioning
\item[] {\tt -w} or {\tt --writeonly} : Perform write-only conditioning
\item[] {\tt -d} or {\tt --getdata} : Save the performance data to a file called {\em VSN}{\tt .timedata}
\item[] {\tt -l} or {\tt --legacydir} : Put an empty legacy directory on the module when complete
\item[] {\tt -n} or {\tt --newdir} : Put an empty new style directory on the module when complete
\item[] {\tt -0} or {\tt --nodir} : Put a zero-size directory on the module when complete
\end{itemize}
\item[] {\em VSN} is a valid 8-character VSN of a loaded module
\end{itemize}

\noindent
Note that this program will not run without specifying a legal mounted module VSN.
If you wish to erase a module that has no VSN set, use the {\tt vsn} program first.

\noindent
Control-C can be used to safely abort conditioning early.
The directory will be left in an indeterminate state, so use caution when doing this; if conditioning is stopped before completion use the {\tt vsn} program to assess and possibly modify the current module state.







% mk5mon ----------------------------------------------------------------------

\subsection{mk5mon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:mk5mon} 

Program {\tt mk5mon} is a program that listens for {\tt mark5Status} messages multicast from the Mark5 units and displays the information; updating the display as new messages are received.

\begin{itemize}
\item[] Usage: {\tt mk5mon}
\end{itemize}

Make sure the terminal is at least 110 characters wide and is at least as tall in characters as there are Mark5 units that may transmit information.
To quit, use ctrl-C.
The columns being displayed are:
\begin{enumerate}
\item Mark5 unit name
\item VSN of module in Bank A
\item VSN of module in Bank B
\item State of the Mark5 unit
\item Playback rate, if playing, in Mbps
\item Playback position, in bytes from beginning of module
\item Scan number of data being played, if playing
\item Scan name of data being played, if playing
\end{enumerate}


% mk6copy ---------------------------------------------------------------------


% mk6cp -----------------------------------------------------------------------

\subsection{mk6cp {\small $\mathrm{(package: mark6sg)}$} \label{sec:mk6cp}}

{\tt mk6cp} is a wrapper around {\tt mk6gather} which makes the operation of copying multiple files from a Mark6 module easier.

\begin{itemize}
\item[] Usage: {\tt mk6cp} $[$ {\em options} $]$ {\em filematch1} $[$ {\em filematch2} \ldots $]$ {\em destination}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
\item[] {\tt -r} or {\tt --resume} : don't copy files that exist in the destination path
\end{itemize}
\item[] {\em filematch1} is a shell-style pattern for matching scan names
\item[] {\em destination} is the output path to place files; needs to start with {\tt .} or {\tt /} or end with {\tt /}  
\end{itemize}




% mk6gather -------------------------------------------------------------------

\subsection{mk6gather {\small $\mathrm{(package: mark6sg)}$} \label{sec:mk6gather}}

{\tt mk6gather} extracts data from a Mark6 module, assembling as necessary the data scattered across the disks in the module.

\begin{itemize}
\item[] Usage: {\tt mk6gather} $[$ {\em options} $]$ {\em template}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : increase output verbosity
\item[] {\tt -a} or {\tt --append} : append to existing file; this continues a previous gather
\item[] {\tt -o} {\em outfile} or {\tt --output} {\em outfile} : send output to {\em outfile} (default is {\tt gather.out})
\item[] {\tt -b} {\em bytes} or {\tt --bytes} {\em bytes} : stop writing after {\em bytes} are written
\item[] {\tt -s} {\em bytes} or {\tt --skip} {\em bytes} : skip first {\em bytes} of file
\end{itemize}
\item[] {\em template} specifies a file list to match, in a shell-style wildcard pattern, such as the name of a scan (e.g., {\tt BB407A\_LA\_No0001})
\end{itemize}

If more than one file is to be copied, it is best to use the {\tt mk6cp} (\ref{sec:mk6cp}) program instead. 




% mk6ls -----------------------------------------------------------------------

\subsection{mk6ls {\small $\mathrm{(package: vdifio)}$} \label{sec:mk6ls}}

{\tt mk6ls} looks in the standard Mark6 mountpoint locations (either in {\tt /mnt/disks/*/*/data} or in the path pattern set by environment variable {\tt \$MARK6\_ROOT} for VDIF formatted files.
Found files will be probed and summarized.

Three printing levels are supported:
\begin{itemize}
\item {\em short} : Simply prints the file names.  The portion of the filename corresponding to the mountpoint location is excised.
\item {\em long} : Prints the file names (same as for {\em short}), the number of actual files making up the virtual Mark6 file (e.g., scattered across multiple mountpoints), the full size of the virtual file, and an indication of the completeness of the virtual file set.
\item {\em full} : Prints the same information as for {\em long} followed by details of the files, such as details of the Mark6 file version, block sizes, packet numbers.  This mode is mainly useful for developers with access to the vdifio source code.
\end{itemize}

\begin{itemize}
\item[] Usage: {\tt mk6ls} $[$ {\em options} $]$
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -s} or {\tt --short} : print long form output (default)
\item[] {\tt -l} or {\tt --long} : print long form output
\item[] {\tt -f} or {\tt --full} : print full information for each file
\end{itemize}
\end{itemize}



% mk6mon --------------------------------------------------- INCOMPLETE -------

\subsection{mk6mon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:mk6mon} 



% mk6summary ----------------------------------------------- INCOMPLETE -------

\subsection{mk6summary {\small $\mathrm{(package: mark6sg)}$}} \label{sec:mk6summary} 


% mk6vmux -------------------------------------------------- INCOMPLETE -------

\subsection{mk6vmux {\small $\mathrm{(package: vdifio)}$}} \label{sec:mk6vmux} 




% mpifxcorr -------------------------------------------------------------------

\subsection{mpifxcorr} \label{sec:mpifxcorr}

The core of the DiFX software correlator is the program called {\tt mpifxcorr}.
This program uses Message Passing Interface (MPI) to exploit parallel computing to make correlation practical on a cluster of ordinary computers.
This program runs on all the machines listed in the {\tt .machines} file that is passed to {\tt mpirun}, the program that starts {\tt mpifxcorr}.
It should be initiated from the cluster head node from within the project directory.
The usage line below is appropriate for use with OpenMPI (\S\ref{sec:mpi}) and within the DiFX context; other incantations may provide better results depending on the setup.
See the OpenMPI documentation for more details.

\begin{itemize}
\item[] Usage: {\tt mpirun -np} {\em nProcess} {\tt --bynode --hostfile} $[$ {\em otherOptions} $]$ {\em machinesFile} {\tt mpfixcorr} {\em inputFile} $[$ {\em options} $]$
\item[] {\em nProcess} is the number of processes to start; found with {\tt wc -l} {\em machineFile}
\item[] {\em machinesFile} the {\tt .machineFile}
\item[] {\em inputFile} the {\tt .input} to run; the full path to this file needs to be given, so prepending the file with {\tt `pwd`/} is typical
\item[] {\em otherOptions} can be any additional option to {\tt mpirun}; {\tt startdifx} uses the {\tt --mca btl $\hat{\ }$udapl,openib --mca mpi\_yield\_when\_idle 1} to suppress some warning messages and be less aggressive on networking
\item[] {\em options} are additional options that {\tt mpifxcorr} can take which include:
\begin{itemize}
\item[] {\tt -M}{\em monHostname}{\tt :}$[${\em monSkip}$]$ : hostname of a machine serving as a monitor data server, with optional value indicating how many records to skip between sends.
\item[] {\tt -r}{\em startSec} : start {\em startSec} seconds into the job, writing a new set of files into the visibility directory ({\tt .difx/})
\item[] {\tt --vgoscomplex} : flips sideband of all Complex VDIF stations by complex conjugating the unpacked Complex VDIF data
\end{itemize}
\end{itemize}

Within the DiFX framework, the user should never have to directly start {\tt mpifxcorr} as this is done more simply with {\tt startdifx} or via the DiFX Operator Interface in conjunction with {\tt mk5daemon}.




% multi2singlethreadVDIF ----------------------------------- INCOMPLETE -------





% oms2v2d ---------------------------------------------------------------------

\subsection{oms2v2d {\small $\mathrm{(package: vex2difx)}$} \label{sec:oms2v2d}}

The VLBI scheduling program {\tt sched} generates a file with extension {\tt .oms} which is used to populate some fields in the VLBA database.
These fields are usually used to feed the dynamic scheduler but can also be used to reduce the tedium of transferring information from the {\tt sched} input file ({\tt .key}) to the {\tt vex2difx} input file {\tt .v2d}.
For simple experiments this resulting {\tt .v2d} file can be used unedited, but for more typical experiments editing will be required.
The resulting file will have the same file prefix as the input file and will end with {\tt .v2d}.

\begin{itemize}
\item[] Usage: {\tt oms2v2d} $[$ {\em options} $]$ {\em file}{\tt .oms}

\item[] {\em file}{\tt .oms} is an {\tt .oms} file written by {\tt sched}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -f} or {\tt --force} : allow overwrite of output file
\end{itemize}
\item[] Example: {\tt oms2v2d bx123.oms}
\end{itemize}

\noindent
Note: {\tt sched} now produces a {\tt .tv2d} file (template vex2difx) that can contain useful information for some projects (especially multi-phase-center projects), however, this file is not tied to any particular version of DiFX (or more importantly, version of {\tt vex2difx}) and thus cannot be guaranteed to be legal.
It is thus suggested to use {\tt oms2v2d} and transfer over needed information by hand after the fact.







% padVDIF ---------------------------------------------------------------------

\subsection{padVDIF {\small $\mathrm{(package: vdifio)}$} \label{sec:padVDIF}}

Program {\tt padVDIF} takes an input VDIF file and inserts additional packets as needed to create a contiguous without gaps.
Newly inserted frames will have the invalid bit set.

\begin{itemize}
\item[] Usage: {\tt padVDIF} {\em infile} {\em outfile} {\em Mbps} $[$ {\em newStartMJD} $]$

\item[] {\em infile} is the input VDIF file
\item[] {\em outfile} is the output VDIF file
\item[] {\em Mbps} is the data rate in megabits/second
\item[] {\em newStartMJD} is the MJD (with fractional component) to overwrite the times with

\item[] Example: {\tt padVDIF raw.vdif smooth.vdif 2048}
\end{itemize}





% peekVDIF --------------------------------------------- INCOMPLETE -----------




% plotapd ---------------------------------------------------------------------

\subsection{plotapd {\small $\mathrm{(package: SniffPlots)}$} \label{sec:plotapd}}

Program {\tt plotapd} takes a text file containing {\em sniffer} fringe fit results and makes plot files.
Separate plots for Amplitude, Phase and Delay (hence the name suffix ``apd''are made for each baseline in the resultant file.
A plot of delay rate is also produced.

This is an interactive command line program; running {\tt plotapd} will prompt the user for inputs.
The program {\tt difxsniff} will run {\tt plotapd} and its sister programs automatically, so usually it won't be necessary to run by hand.

The {\tt PGPLOT\_FONT} environment variable must be set, otherwise all plot text will be missing.





% plotbp ----------------------------------------------------------------------

\subsection{plotbp {\small $\mathrm{(package: SniffPlots)}$} \label{sec:plotbp}}

Program {\tt plotbp} takes a text file containing {\em sniffer} bandpass output files and creates plots.
This program can produce both auto- and cross-correlation data plots.

This is an interactive command line program; running {\tt plotbp} will prompt the user for inputs.
The program {\tt difxsniff} will run {\tt plotbp} and its sister programs automatically, so usually it won't be necessary to run by hand.

The {\tt PGPLOT\_FONT} environment variable must be set, otherwise all plot text will be missing.





% plotwt ----------------------------------------------------------------------

\subsection{plotwt {\small $\mathrm{(package: SniffPlots)}$} \label{sec:plotwt}}

Program {\tt plotwt} takes a text file containing {\em sniffer} data weights.
In this context, a data weight of 0 indicates complete loss of data and a weight of 1 indicates complete data.
Each plotted data point will usually span many correlator integration periods.
A solid dot will be plotted for the mean of these points, and ``error bars'' will indicate the range of weights over the averaging period.
Note that in some cases where awkward integration times are used it may be possible for the weight to occasionally exceed 1, as long as the long running average never does.

This is an interactive command line program; running {\tt plotwt} will prompt the user for inputs.
The program {\tt difxsniff} will run {\tt plotwt} and its sister programs automatically, so usually it won't be necessary to run by hand.

The {\tt PGPLOT\_FONT} environment variable must be set, otherwise all plot text will be missing.







% printDiFX -------------------------------------------------------------------

\subsection{printDiFX.py {\small $\mathrm{(package: vis2screen)}$} \label{sec:printDiFX}}

Program {\tt printDiFX} prints a summary of the visibility information in a DiFX output file.
It loops through all the records printing some basic info about frequencies, baselines, polarizations, times etc., plus a couple of selected visibility values from the start and middle of the record, to the screen.

\begin{itemize}
\item[] Usage: {\tt printDiFX} {\em difxfile} {\em inputfile}

\item[] {\em difxfile} is the full name of the visibility file in the {\tt .difx} directory
\item[] {\em inputfile} is the path to the input file used to generate this difx output
\end{itemize}
\begin{itemize}
\item[] Example: {\tt printDiFX example\_1.difx/DIFX\_55523\_025239.s0000.b0000 example\_1.input}
\end{itemize}




% printVDIF -------------------------------------------------------------------

\subsection{printVDIF {\small $\mathrm{(package: vdifio)}$} \label{sec:printVDIF}}

Program {\tt printVDIF} loops through a VDIF file inspecting each packet header and printing some basic summary info (time etc.) to the screen.

\begin{itemize}
\item[] Usage: {\tt printVDIF} {\em vdiffile} {\em Mbps}

\item[] {\em vdiffile} is the recorded VDIF file to inspect
\item[] {\em Mbps} is the data rate in megabits/second

\item[] Example: {\tt printVDIF example.vdif 256}
\end{itemize}



% printVDIFgaps ---------------------------------------------- INCOMPLETE -----

\subsection{printVDIFgaps {\small $\mathrm{(package: vdifio)}$}} \label{sec:printVDIFgaps}



% printVDIFheader -------------------------------------------------------------

\subsection{printVDIFheader {\small $\mathrm{(package: vdifio)}$}} \label{sec:printVDIFheader}

Program {\tt printVDIFheader} is a powerful diagnostic tool that prints details of each VDIF header found in a VDIF file.
All fields of each header, including information in known Extended VDIF Headers (see Sec.~\ref{sec:vdifedv}) are printed.
Three different print formats are allowed: compact {\tt short} version, detailed {\tt long}, and hexidecimal {\tt hex}.

\begin{itemize}
\item[] Usage: {\tt printVDIFheader} {\em inputFile} $ [ $ {\em inputFrameSize} $ [ $ {\em printLevel} $ ] ] $
\item[] {\em inputFile} is the input multi-thread VDIF file, or {\tt -} for {\em stdin}
\item[] {\em inputFrameSize} is the size of one thread's data frame, including header (for RDBE VDIF data this is 5032)
\item[] {\em printLevel} describes what is to be printed; one of {\tt short}, {\tt long}, or {\tt hex}
\end{itemize}

Run this program with no arguments to get some additiona explanation.




% psrflag ---------------------------------------------------------------------

\subsection{psrflag {\small $\mathrm{(package: difxio)}$}} \label{sec:psrflag}

{\tt psrflag} is a program that reads one or more DiFX filesets and produces a flag table readable by AIPS UVFLG that contains flags for times when the fringe rate, on a per-baseline basis, resonates with the pulse period, allowing DC bias to correlate.
The pulsar {\tt .binconfig} file is used to determine the harmomic content of the pulsar profile.

\begin{itemize}
\item[] Usage: {\tt psrflag} $ [ $ {\em options} $ ] $ {\em inputFile} \ldots
\item[] {\em inputFile} is the input multi-thread VDIF file; multiple may be provided
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print help information and quit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution
\end{itemize}
\end{itemize}


% record5c ------------------------------------------------- INCOMPLETE -------

\subsection{record5c {\small $\mathrm{(package: mark5daemon)}$}} \label{sec:record5c}




% recover ---------------------------------------------------------------------

\subsection{recover {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:recover}

{\tt recover} is a program that wraps the XLRRecover call for convenient use.
This replaces the functionality of the {\tt recover=} command of the {\tt Mark5A} program.

\begin{itemize}
\item[] Usage: {\tt recover} $[$ {\em options} $]$ {\em type} {\em bank}

\item[] {\em type} is the type of recovery to attempt.  See below.
\item[] {\em bank} should be either {\tt A} or {\tt B} and is the bank containing the module to address.
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -f} or {\tt --force} : allow overwrite of output file
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution
\end{itemize}
\item[] Example: {\tt recover -v 2 A}
\end{itemize}

\noindent
There are three possible modes of operation that are selected with the {\em type} argument:
\begin{itemize}
\item[0] Fix directory if power failed during recording
\item[1] Allow access to data that might have been overwritten
\item[2] Unerase the module
\end{itemize}
These recovery attempts will not always be successful.





% reducepoly ------------------------------------------------------------------

\subsection{reducepoly {\small $\mathrm{(package: difxio)}$}} \label{sec:reducepoly}

Program {\tt reducepoly} takes one or more DiFX file sets as input and will modify each fileset's delay model ({\tt .im} file) to have polynomial representations with fewer terms.
All polynomials in the {\tt ,im} file will be reduced, including the baseline vectors, atmospheric components, azimuth, and elevation.
The original {\tt .im} files will be overwritten.
The main purpose of this program is to evaluate the impact of using different polynomial orders,

\begin{itemize}
\item[] Usage: {\tt reducepoly} $ [ $ {\em options} $ ] $ {\em inputFile} \ldots
\item[] {\em inputFile} is the input multi-thread VDIF file, or {\tt -} for {\em stdin}; multiple may be provided
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print help information and quit
\item[] {\tt -2} : reduce polynomials to two terms
\item[] {\tt -3} : reduce polynomials to three terms
\item[] {\tt -4} : reduce polynomials to four terms
\item[] {\tt -5} : reduce polynomials to five terms
\end{itemize}
\end{itemize}


% restartdifx ---------------------------------------------- INCOMPLETE -------




% searchVDIF ----------------------------------------------- INCOMPLETE -------

\subsection{searchVDIF {\small $\mathrm{(package: vdifio)}$}} \label{sec:searchVDIF}




% splitVDIFbygap ------------------------------------------- INCOMPLETE -------

\subsection{splitVDIFbygap {\small $\mathrm{(package: vdifio)}$}} \label{sec:splitVDIFbygap}



% startdifx -------------------------------------------------------------------

\subsection{startdifx {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:startdifx} 

Starting {\tt mpifxcorr} generally requires a lengthy command.
This inspired the creation of {\tt startdifx} which vastly simplifies use of the DiFX correlator.
In addition to spawning the {\tt mpifxcorr} processes, {\tt startdifx} can orchestrate some of the preparatory work (for example running {\tt calcif2} and {\tt genmachines}) and optionally run {\tt difx2fits} to create a {\tt .FITS} file for each job.
This program is meant to work within the DiFX environment and would probably require modification to be useful in other situations.

\begin{itemize}
\item[] Usage: {\tt startdifx} $[$ {\em options} $] [$ {\em startDelay} $]$ {\em input1} $[$ {\em input2} $\cdots$ $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose
\item[] {\tt -q} or {\tt --quiet} : be quieter
\item[] {\tt -f} or {\tt --force} : proceed on files even if correlator output already exists and is up to date
\item[] {\tt -a} or {\tt --automachines} : run {\tt genmachines} only if no {\tt .machines} file exits
\item[] {\tt -g} or {\tt --genmachines} : run {\tt genmachines} unconditionally (default)
\item[] {\tt -n} or {\tt --nomachines} : don't run {\tt genmachines}
\item[] {\tt -d} or {\tt --dont-calc} : don't run {\tt calcif} even if needed -- will skip file
\item[] {\tt -m} or {\tt --message} : start {\tt mpifxcorr} by sending {difxStart} message to {\tt mk5daemon}
\item[] {\tt -F} or {\tt --fits} : run {\tt difx2fits} on output of each job separately
\item[] {\tt -l} or {\tt --localhead} : use the current host as the headnode, not {\tt \$DIFX\_JEAD\_NODE}
\item[] {\tt --override-version} : ignore potential difx version conflicts
\item[] {\tt -A} {\em agent} or {\tt --agent=}{\em agent} : call mpirun through {\em agent} with filebase as only argument
\item[] {\tt -M} {\em machinesFile} or {\tt --machines-file=}{\em machinesFile} : use {\em machinesFile} instead of the one expected based on the job name
\end{itemize} 
\item[] {\em startDelay} is an optional number of seconds to jump into the job upon start
\item[] {\em input}N is a {\tt .input} file, or its prefix
\item[] Example 1: {\tt startdifx job1420.000.input}
\item[] Example 2: {\tt startdifx -f -n job1420.000 job1421.000}
\item[] Example 3: {\tt startdifx -F *.input}
\end{itemize}

\noindent Environment variables respected:
\begin{itemize}
\item {\tt DIFX\_MESSAGE\_GROUP} : multicast group to use (when using -m option), overriding default 224.2.2.1
\item {\tt DIFX\_MESSAGE\_PORT} : multicast port to use (when using -m option), overriding default 50200
\item {\tt DIFX\_HEAD\_NODE} : when using -m option, this must be set, which specifies which machine will serve as the head node
\item {\tt DIFX\_MPIRUNOPTIONS} : used to pass options to the mpirun command
\item {\tt DIFX\_CALC\_PROGRAM} : can be used to change the delay model program (default is {\tt calcif2}); only needed if model calculations have not been done
\item {\tt DIFX\_CALC\_OPTIONS} : used to override options to the delay model program
\end{itemize}







% statemon --------------------------------------------------------------------

\subsection{statemon {\small $\mathrm{(package: difxmessage)}$}} \label{sec:statemon} 

Program {\tt statemon} listens for multicast messages of the {\tt difxStatus} variety and simply prints their contents to the terminal.
This is mainly useful for debugging {\tt mpifxcorr} and any programs responsible for launching it.

\begin{itemize}
\item[] Usage: {\tt statemon} $[$ {\em options} $]$ 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\end{itemize}
\end{itemize}

\noindent Environment variables respected:
\begin{itemize}
\item {\tt DIFX\_MESSAGE\_GROUP} : multicast group to use, overriding default 224.2.2.1
\item {\tt DIFX\_MESSAGE\_PORT} : multicast port to use, overriding default 50200
\end{itemize}









% stopmpifxcorr ---------------------------------------------------------------

\subsection{stopmpifxcorr {\small $\mathrm{(package: mpifxcorr)}$}} \label{sec:stopmpifxcorr}

If software correlation is in progress and it is desired to stop it, it is best to gently stop it rather than killing it abruptly.
In most circumstances this can be accomplished with {\tt stopmpifxcorr}.
This program must be run on the machine running the {\em manager} process of the software correlator (usually this is {\tt swc000} for the VLBA).
If multiple {\tt mpifxcorr} processes are found running on a machine, {\tt stopmpifxcorr} will not proceed unless the
{\tt -f} option is used.

\begin{itemize}
\item[] Usage: {\tt stopmpifxcorr} $[$ {\em options} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -f} or {\tt --first-pid} : send stop message to the numerically first process ID found
\item[] {\tt -q} or {\tt --quiet} : don't produce much output
\end{itemize}
\end{itemize}









% stripVDIF -------------------------------------------------------------------

\subsection{stripVDIF {\small $\mathrm{(package: vdifio)}$} \label{sec:stripVDIF}}

Program {\tt stripVDIF} strips network headers from a VDIF format basebad data file (e.g., captured from wireshark) and dumps a pure VDIF stream.

\begin{itemize}
\item[] Usage: {\tt stripVDIF} {\em infile} {\em outfile} $[$ {\em skipbytesfront} $[$ {\em skipbytesback} $[$ {\em skipbytesinitial} $] ] ]$

\item[] {\em infile} is the input VDIF file
\item[] {\em outfile} is the output VDIF file
\item[] {\em skipbytesfront} is the number of bytes to skip over before each frame (default is 54)
\item[] {\em skipbytesback} is the number of bytes to skip over after each frame (default is 4)
\item[] {\em skipbytesinitial} is the number of bytes to skip over only once after opening the file (default is 28)

\item[] Example: {\tt stripVDIF vdif.wireshark vdif.pure 54 4 28}
\end{itemize}






% tabulatedelays --------------------------------------------------------------

\subsection{tabulatedelays {\small $\mathrm{(package: difxio)}$}} \label{sec:tabulatedelays}

Program {\tt tabulatedelays} takes one or more DiFX filesets and produces a text file containing, for each scan, a table of interferometer delays ($\mu$s) and rate ($\mu$s/s) based on the values in the {\tt .im} file at a cadence of one entry every 8 seconds.
Based on command line options values other than the delay can be extracted and tabulated.
The details of the output files are documented in comments at the top of the output file.

\begin{itemize}
\item[] Usage: {\tt tabulatedelays} $ [ $ {\em options} $ ] $ {\em inputFile} \ldots
\item[] {\em inputFile} is the input multi-thread VDIF file, or {\tt -} for {\em stdin}; multiple may be provided
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print help information and quit
\item[] {\tt --az} : print azimuth (deg) and azimuth rate (deg/s) instead of delay, rate
\item[] {\tt --el} : print elevation (deg) and azimuth rate (deg/s) instead of delay, rate
\item[] {\tt --dry} : print the dry component of atmospheric delay ($\mu$s) instead of delay, rate
\item[] {\tt --wet} : print the wet component of atmospheric delay ($\mu$s) instead of delay, rate
\item[] {\tt --uvw} : print the antenna-based baseline coordinates $(x, y, z)$ (meters) instead of delay, rate
\item[] {\tt --clock} : print the clock offset ($\mu$s) and rate ($\mu$s/s) instead of delay, rate
\item[] {\tt --perint} : print values at the center of every integration rather than every 8s
\item[] {\tt --addclock} : include clock offset/rate in printed delay/rate values
\end{itemize}
\end{itemize}

This program reads through one or more difx datasets and evaluates delay polynomials in the {\tt .im} files on a regular time grid (every 8 seconds).
Delays and rates are both calculated.
Output should be self explanatory.
Plotting utilities such as gnuplot can be used directly on the output.

When operating without {\tt --perint}, the entirety of the delay polynomials are plotted, even exceeding the time range of the scans to which they belong.
Comments in the output separate scans cleanly.
When {\tt --perint} is used, only the time covered by the scans is output.

Sign conventions:
\begin{itemize}
  \item Delay: a positive delay indicates wavefront arrival at the station before wavefront arrival at earth center.
	The delay includes contribution from wet and dry atmosphere components.
  \item Rate: simply the time derivative of Delay.
  \item Clock Offset: sign convention is opposite that of {\tt .vex "clock\_early"} parameter; a positive clock offset indicates slow station clock.
	The sum of Clock Offset and Delay is the total correlator delay.
  \item Clock Rate: simply the time derivative of Clock Offset.
\end{itemize}


% testdifxinput ---------------------------------------------------------------

\subsection{testdifxinput {\small $\mathrm{(package: difxio)}$}} \label{sec:testdifxinput}

This program was intended mainly for helping debug parsing of {\tt .input} files and associated other files.
It turned out to be useful as a general tool to investigate the contents of such files.
When multiple input files are provided on the command line merging of the resultant data structures is attempted.
Two output files are created when run: {\tt input.out} and {\tt calc.out}.
These files should closely resemble the input files if the parsing was done properly.

\begin{itemize}
\item[] Usage: {\tt testdifxinput} $[$ {\em options} $]$ {\em inputFilePrefix1} $[$ {\em inputFilePrefix2} $ \cdots ]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -v} or {\tt --verbose} : be a bit more verbose
\item[] {\tt -h} or {\tt --help} : print help information and quit
\end{itemize}
\item[] {\em inputFilePrefix}$n$ is the base name of an input file
\end{itemize}






% testdifxmessagereceive ------------------------------------------------------

\subsection{testdifxmessagereceive{\small $\mathrm{(package: difxmessage)}$}} \label{sec:testdifxmessagereceive}

Test program {\tt testdifxmessagereceive} captures multicast DiFX messages and prints them to the screen.
Both the raw XML is shown and the decoded values.
It is mostly useful as a tool for examining the correctness of the multicast messages that are broadcast and is not intended to be part of an operational system.
There is a special binary mode which instead listens for the multicast high time resolution autocorrelations.
In this mode, only a terse summary of what is received is printed (see the source code for more information).

\begin{itemize}
\item[] Usage: {\tt testdifxmessagereceive}  $[$ {\em options} $]$  $[$ {\em type} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print help information
\item[] {\tt -b} or {\tt --binary} : generate output based on binary records
\end{itemize}
\item[] {\em type} is the kind of message to capture (not for use with binary records):
\begin{enumerate}
\item DifxLoadMessage
\item DifxAlertMessage
\item Mark5StatusMessage
\item DifxStatusMessage
\item DifxInfoMessage
\item DifxDatastreamMessage
\item DifxCommand
\item DifxParameter
\item DifxStart
\item DifxStop
\item Mark5VersionMessage
\item Mark5ConditionMessage
\item DifxTransientMessage
\end{enumerate}
\end{itemize}

If {\em type} is not provided, all message types will be captured.





% testmod ---------------------------------------------------------------------

\subsection{testmod {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:testmod}

{\tt testmod} is a program that is used to perform read and write tests on Mark5 modules.  
It is meant as a replacement of the {\tt ResetModule} program that relies on the {\tt Mark5A} program which is being phased out of VLBA operations.
Read-only tests can be performed without risk of erasing astronomical data recorded on the disks.
The more invasive write-read tests (which are the default) will erase all preexisting data!
A matrix of numbers similar to what is produced by {\tt ResetModule} or {\tt SSerase} in condition mode, but with statistics from a much smaller volume of reading/writing is produced.
Usually badly performing disks will occur in pairs with both bad disks belonging to the same bus (e.g., disks 0 and 1, 2 and 3, 4 and 5, or 6 and 7).  
Badly performing drives should have their directory files {\tt .dir} (see \S~\ref{sec:dir}) updated by hand to include {\tt RT} at the end of the top line.

\begin{itemize}
\item[] Usage: {\tt testmod} $[$ {\em options} $]$ {\em bank}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : produce more informative/diagnostic output; {\tt -v -v} for even more
\item[] {\tt -f} or {\tt --force} : continue to produce files despite warnings
\item[] {\tt -r} or {\tt --readonly} : Perform read-only test
\item[] {\tt -R} or {\tt --realtime} : Switches to real-time mode (see below)
\item[] {\tt -d} or {\tt --skipdircheck} : Disable directory checking (see below)
\item[] {\tt -S} or {\tt --speed} : Disable correctness testing to better test throughput
\item[] {\tt -n} {\em n} or {\tt --nrep} {\em n} : Repeat the test {\em n} times (default is 2)
\item[] {\tt -s} {\em s} or {\tt --blocksize} {\em s} : Read and write {\em bytes} at a time (default is 10~MB)
\item[] {\tt -b} {\em b} or {\tt --nblock} {\em b} : Perform {\em b} reads per test (default is 50)
\item[] {\tt -p} {\em p} or {\tt --pointer} {\em p} : Start read-only tests at byte position {\tt p}
\item[] {\tt -o} {\em file} or {\tt --dirfile} {\em file} : Write the module directory to file {\em file}
\end{itemize}
\item[] {\em bank} is the Mark5 bank containing the disk to be studied ({\tt A} or {\tt B})
\end{itemize}

Many modules being tested are perhaps damaged in some way and may require the {\tt -R} and/or {\tt -d} options above.
Is is generally safe to use these options, but the diagnostic power of this program may be reduced in cases where some drives are intrinsically slow, but still produce valid data.




% testseqnumbers --------------------------------------------------------------

\subsection{testseqnumbers {\small $\mathrm{(package: difxmessage)}$}} \label{sec:testsequnumbers} 

Program {\tt testseqnumbers} is a utility to listen for DiFX multicast messages and identify any that come with a sequence number that is not sequential.
This is a good way to identify possible packet loss or duplication on a DiFX cluster network.

\begin{itemize}
\item[] Usage: {\tt testseqnumbers} $[$ {\em options} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : produce more output; {\tt -v -v} for even more
\end{itemize}
\end{itemize}

If run without the {\tt -v} option, only unexpected packets will be noted.
If run with one {\tt -v} flag, each received packet will be identified with a period being written to the screen.
If run with 2 {\tt -v} flags, each packet received will have its source and sequence number printed.






% vdif2to8 --------------------------------------------------------------------

\subsection{vdif2to8 {\small $\mathrm{(package: vdifio)}$}} \label{sec:vdif2to8}

Program {\tt vdifb2to8} takes a 2-but sampled VDIF stream and reencodes it as 8-bit samples.
It should work on any form of VDIF with 2 bits per sample (2+2 bits complex).
It is anticipated that any 8-bit decoder will be performed with linear level spacings, unlike the case for 2-bit samples which use a high to low ratio of 3.3359 to minimized quantization noise.
Given this anticipation the levels chosen in the output 8-bit stream correspond to levels of 35.5 and 118.5 counts for the low and high states respectively.
These levels lead to a ratio as close to 3.3359 as possible.
The 0.5 offset comes about from assuming that the 256 output states are centered on zero and thus range from -127.5 \ldots -0.5, 0.5 \ldots 127.5.
Extra bytes in the input stream (not corresponding to valid VDIF frames) will be excised but invalid frames (as marked with the invalid bit) will be retained and encoded to 8 bits.

\begin{itemize}
\item[] Usage: {\tt vdif2to8} {\em inputFile} {\em frameSize} {\em outputFile}
\item[] {\em inputFile} is the file to read (2 bits per sample)
\item[] {\em frameSize} is the size of the VDIF frames including frame headers (5032 for VLBA or VLA VDIF data)
\item[] {\em outputFile} is the output file (8 bits sper sample)
\item[] Example: {\tt vdif2to8 input.vdif 5032 output.vdif}
\end{itemize}







% vdifbstate ------------------------------------------------------------------

\subsection{vdifbstate {\small $\mathrm{(package: vdifio)}$}} \label{sec:vbstate}

Program {\tt vdifbstate} will perform state counts on a multi-thread VDIF baseband data file.

\begin{itemize}
\item[] Usage: {\tt vdifbstate} {\em file} {\em frameSize} {\em dataRate} {\em threadlist} {\em nFrames} $[$ {\em offset} $]$ 
\item[] {\em file} is the file to decode
\item[] {\em frameSize} is the size of the VDIF frames including frame headers (5032 for VLBA or VLA VDIF data)
\item[] {\em dataRate} is the file data rate, measured in Mbps, not including frame headers
\item[] {\em threadList} is a comma-separated list of thread ids to decode
\item[] {\em nFrames} is the number of data frames (typically a few kB in size) to decode
\item[] {\em offset} is the number of bytes into the file to start decoding
\item[] Example: {\tt vdifbstate sample.vdif 5032 1024 1,2,3,4 100}
\end{itemize}

\noindent
Notes:
\begin{enumerate}
\item See documentation for {\tt m5b} for details on specifying the data format.
\item Only real-sampled data with 1 or 2 bits per sample is supported at this time.
\item If a non-power-of-two number of threads is requested, extra channels will be invented to pad out the next power of two.
Data in these extra channels has undefined qualities.
\end{enumerate}



% vdifChanSelect ----------------------------------------- INCOMPLETE ---------

\subsection{vdifChanSelect {\small $\mathrm{(package: vdifio)}$}} \label{sec:vdifChanSelect}




% vdifd -----------------------------------------------------------------------

\subsection{vdifd {\small $\mathrm{(package: vdifio)}$}} \label{sec:vdifd}

Program {\tt vdifd} takes a multi-thread VDIF file and decodes some samples.
It is implemented as a python script that makes use of {\tt vmux} and {\tt m5d} to do most of the work.
It is thus a good program to study to understand how vmux can be used.
This program takes as command line input the name of a file containing (or thought to be containing) multi-thread VDIF baseband data, information about the VDIF stream, and the number of samples per baseband to decode.
Optionally a starting file offset can be supplied.
If the data can be decoded correctly, information about the data will be printed to the screen along with a table of decoded data.
The output values, -3, -1, 1, or 3 for valid data, are printed in {\em nchan} columns.
Data that cannot be decoded (either due to data replacement headers, data fill pattern replacing the actual data after unloading from a Mark5 module, or identified via the VDIF invalid bit) will show as 0.
It should be invoked with the following parameters:

\begin{itemize}
\item[] Usage: {\tt vdifd} {\em file} {\em frameSize} {\em dataRate} {\em threadlist} {\em n} $[$ {\em offset} $]$ 
\item[] {\em file} is the file to decode
\item[] {\em frameSize} is the size of the VDIF frames including frame headers (5032 for VLBA or VLA VDIF data)
\item[] {\em dataRate} is the file data rate, measured in Mbps, not including frame headers
\item[] {\em threadList} is a comma-separated list of thread ids to decode
\item[] {\em n} is the number of samples to decode
\item[] {\em offset} is the number of bytes into the file to start decoding (default is 0)
\item[] Example: {\tt vdifd sample.vdif 5032 1024 1,2,3,4 24}
\end{itemize}

In the above example, a stream consisting of 4 channels, each with 64~MHz bandwidth and 2 bits per sample and with thread ids 1, 2, 3 and 4 are to be decoded.
If the thread ids were to be permuted, the decoded output data would be permuted in the same manner.

\noindent
Notes:
\begin{enumerate}
\item At this time only 2-bit real-valued data can be properly decoded.
\item If a non-power-of-two number of threads is requested, extra channels will be invented to pad out the next power of two.
Data in these extra channels has undefined qualities.
\end{enumerate}




% vdiffold --------------------------------------------------------------------

\subsection{vdiffold {\small $\mathrm{(package: vdifio)}$}} \label{sec:vdiffold}

Program {\tt vdiffold} takes a baseband data stream and integrates the power (formed by squaring the voltage) in a number of time bins that equally divide a given period.
This is a simplifed version of ``folding'' such as is used in pulsar processing.
A typical use of such functionality would be to investigate the waveform of the switched power injected into the receiver for calibration.
This program has found considerable utility in determining time offsets between the sample clock and formatter time (modulo the period of the calibration cycle).

In the case of 2-bit sampling a non-linear correction is applied before results are written to a file.
This correction takes the form
\begin{equation}
P = \frac{1}{\left({\rm erf}^{-1}\left(\frac{\hat{P} - v_{\rm high}^2}{1 - v_{\rm high}^2} \right)\right)^2},
\end{equation}
where $P$ is a value proportional to true power and $\hat{P}$ is the value obtained by calculating $\left<\hat{v}^2\right>$ when the bitstream is reproduced with values $\hat{v} \in ( -v_{\rm high}, -1, 1, v_{\rm high} )$. 
This non-linear correction can be turned off by setting {\em nbin} to a negative value.
Note that this program is not useful for 1-bit quantized data.

The program should be used as follows:

\begin{itemize}
\item[] Usage: {\tt vdiffold} {\em infile} {\em frameSize} {\em dataRate} {\em threadlist} {\em nbin} {\em nchunk} {\em freq} {\em outfile} $[$ {\em offset} $]$ 
\item[] {\em infile} is the file to decode
\item[] {\em frameSize} is the size of the VDIF frames including frame headers (5032 for VLBA or VLA VDIF data)
\item[] {\em dataRate} is the file data rate, measured in Mbps, not including frame headers
\item[] {\em threadList} is a comma-separated list of thread ids to decode
\item[] {\em nbin} is the number of bins to calculate per period; if negative, power correction is not performed and the absolute value of {\em nbin} is used
\item[] {\em nchunk} is the number of 10000 sample chunks to operate on
\item[] {\em freq} is the reciprocal of the period to be observed (Hz)
\item[] {\em outfile} is the name of the output file
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt vdiffold sample.vdif 5032 1024 1,2,3,4 128 10000 80 sample.fold}
\end{itemize}

The output file will contain {\em nchan}+1 columns where {\em nchan} is the number of baseband channels in the data stream.
The first column contains the time (seconds) within the period.
Each remaining column is folded power for one baseband channel.
If {\em nbin} is positive and the data is 2-bit quantized, the scaling is such that $\left<v^2\right> = \sigma^2$ yields a power reading of 1.0, for sampler threshold $\sigma$.
Optimal signal to noise ratio occurs for a value of about 1.03.
For non 2-bit quantization, the power will be in units of reconstituted ${\rm counts}^2$.

\noindent
Notes:
\begin{enumerate}
\item At this time only 2-bit real-valued data can be properly decoded.
\item If a non-power-of-two number of threads is requested, extra channels will be invented to pad out the next power of two.
Data in these extra channels has undefined qualities.
\item The output columns are in the same order as the thread id list.
Thus, you can rearrange the output order by changing the order of the thread list.
This enables reordering of data so that polarization pairs occur consecutively, allowing more sensible cross-correlation columns.
\end{enumerate}





% vdifspec --------------------------------------------------------------------

\subsection{vdifspec {\small $\mathrm{(package: vdifio)}$}} \label{sec:vdifspec}

Program {\tt vdifspec} forms total power spectra for each baseband channel in the data, including cross spectra for polarization pairs, assuming data is in alternating polarization pairs (if not, the cross spectra should make no sense, but they are formed anyway).
The results are written to a text file with the following columns: Column 1 is the frequency offset from baseband for each channel; Columns 2 to {\em nchan}+1 are the total power spectra for each baseband channel; Columns {\em nchan}+2 to $4 \times${\em nchan}+1 contain, in pairs, the amplitude and phase of the cross spectra for each pair of channels.
It should be invoked with the following parameters:

\begin{itemize}
\item[] Usage: {\tt vdifspec} {\em infile} {\em frameSize} {\em dataRate} {\em threadlist} {\em npoint} {\em n} {\em outfile} $[$ {\em offset} $]$ 
\item[] {\em infile} is the file to decode
\item[] {\em frameSize} is the size of the VDIF frames including frame headers (5032 for VLBA or VLA VDIF data)
\item[] {\em dataRate} is the file data rate, measured in Mbps, not including frame headers
\item[] {\em threadList} is a comma-separated list of thread ids to decode
\item[] {\em npoint} is the number of points to calculate for each spectrum
\item[] {\em n} is the number of FFT frames to include in the calculation
\item[] {\em outfile} is the name of the output file
\item[] {\em offset} (optional) is the number of bytes into the file to start decoding
\item[] Example: {\tt vdifspec sample.vdif 5032 1024 1,2,3,4 256 1000 vlba.spec}
\end{itemize}

\noindent
Notes:
\begin{enumerate}
\item At this time only 2-bit real-valued data can be properly decoded.
\item If a non-power-of-two number of threads is requested, extra channels will be invented to pad out the next power of two.
Data in these extra channels has undefined qualities.
\item The output columns are in the same order as the thread id list.
Thus, you can rearrange the output order by changing the order of the thread list.
This enables reordering of data so that polarization pairs occur consecutively, allowing more sensible cross-correlation columns.
\end{enumerate}




% vex2difx --------------------------------------------------------------------

\subsection{vex2difx} \label{sec:vex2difx}

{\tt vex2difx} is a program that takes a {\tt .vex} files (such as one produced by {\tt sched} with various tables based on observe-time data appended, probably by {\tt db2vex} in the case of VLBA operations) and a {\tt .v2d} configuration file (see \S\ref{sec:v2d}) and generates one or more {\tt .input} and {\tt .calc} file pairs for use with the DiFX correlator. 
Note specifically that {\tt .ovex} files, as used at many/most Mark4 correlators, are not supported.
{\tt vex2difx}, along with {\tt calcif2}, supercedes the functionality of {\tt vex2config} and {\tt vex2model}, two programs that were widely used but never fully integrated into the VLBA's software chain. 
Don't forget that {\tt oms2v2d} can be used to create a valid baseline {\tt .v2d} file from the {\tt .oms} file made by {\tt sched}, perhaps saving some time.

The following guiding principles drove the design of {\tt vex2difx}:
\begin{enumerate}
  \item The output files should never need to be hand edited
  \item Simple experiments should not require complicated configuration
  \item All features implemented by mpifxcorr should be accessible
  \item All experiments expressible by vex should be supported
  \item The configuration file should be human and machine friendly
  \item Command line arguments should not influence the processing of the vex file
\end{enumerate}
Note that not all of these ideals have been completely reached as of now. It is not the intention of the developer to guess all possible future needs of this program. 
Most new features will be easy to implement so send a message to the difx-users mailing list or file a JIRA \cite{jira} bug tracking ticket for requests.

\begin{itemize}
\item[] Usage: {\tt vex2difx} $[$ {\em options} $]$ {\em v2dFile}
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -o} or {\tt --output} : write a configuration file called {\em v2dFile}{\tt .params} (see \S\ref{sec:params}) as output
\item[] {\tt -v} or {\tt --verbose} : produce more informative/diagnostic output; {\tt -v -v} for even more
\item[] {\tt -d} or {\tt --delete-old} : delete old output from same .v2d file
\item[] {\tt -f} or {\tt --force} : continue to produce files despite warnings
\item[] {\tt -s} or {\tt --strict} : treat some warnings as errors and quit (default)
\end{itemize}
\item[] {\em v2dFile} is a {\tt .v2d} file (see \S\ref{sec:v2d}) that controls the operation of this program; the filename cannot contain underscore characters
\item[] Example: {\tt vex2difx bx123.v2d}
\end{itemize}

\subsubsection{VDIF issues}

Unlike for the other formats, VDIF does not make use of the {\tt \$TRACKS} section for numeric assignment of channel.
Instead the channels as listed in the {\tt \$FREQ} section are sorted alphabetically by their link name (usually something like {\tt Ch01}.
The alphabetical list is matched against the thread-channel order where the threads are listed in numeric order; the ``thread index'' takes precedence over the ``channel index''.
The {\tt track\_frame\_format} parameter of the {\tt \$TRACKS} section is still required.

\subsubsection{Mark5B issues}

The Mark5B format, including its 2048 Mbps extension, is now supported by {\tt vex2difx}. 
The {\tt .vex} file track assignments for Mark5B format has never been formally documented.
{\tt vex2difx} has adopted the track assignment convention used by Haystack. 
Formally speaking, Mark5B has no tracks. 
Instead it stores up to 32 bitstreams in 32 bit words. 
The concept of fanout is no longer used with Mark5B. 
Instead, the equivalent operation of spreading one bitstream among 1 or more bits in each 32 bit word is performed automatically. 
Thus to specify a Mark5B mode, only three numbers are needed: Total data bit rate (excluding frame headers), number of channels, and number of bits per sample (1 or 2). 
The number of bitstreams is the product of channels and bits.

The {\tt \$TRACKS} section of the vex file is used to convey the bitstream assignments. 
Individually, the sign and magnitude bits for each channel are specified with {\tt fanout\_def} statements. 
In unfortunate correspondence with existing practice, 2 is the first numbered bitstream and 33 is the highest. 
In 2-bit mode, all sign bits must be assigned to even numbered bitstreams and the corresponding magnitude bit must be assigned to the next highest bitstream. 
To indicate that the data is in Mark5B format, one must either ensure that a statement of the form 

\indent
{\tt track\_frame\_format = MARK5B;}

\noindent
must be present in the appropriate {\tt \$TRACKS} section or

\indent
{\tt format = MARK5B}

\noindent
must be present in each appropriate {\tt ANTENNA} section of the {\tt .v2d} file. 
As a concrete example, a {\tt \$TRACKS} section may resemble:

\begin{verbatim}
$TRACKS;
def Mk34112-XX01_full;
  fanout_def = A : &Ch01 : sign : 1 : 02;
  fanout_def = A : &Ch01 : mag  : 1 : 03;
  fanout_def = A : &Ch02 : sign : 1 : 04;
  fanout_def = A : &Ch02 : mag  : 1 : 05;
  fanout_def = A : &Ch03 : sign : 1 : 06;
  .
  .
  .
  fanout_def = A : &Ch15 : mag  : 1 : 31;
  fanout_def = A : &Ch16 : sign : 1 : 32;
  fanout_def = A : &Ch16 : mag  : 1 : 33;
  track_frame_format = MARK5B;
enddef;

\end{verbatim}

\subsubsection{Media specification}

{\tt vex2difx} allows {\tt .input} file generation for two types of media.
A single {\tt .input} file can have different media types for different stations. 
Ensuring that media has been specified is important as antennas with no media will be dropped from correlation.
The default media choice is Mark5 modules.
The {\tt TAPELOG\_OBS} table in the input vex file should list the time ranges valid for each module.
Jobs will be split at Mark5 module boundaries; that is, a single job can only support a single Mark5 unit per station.
All stations using Mark5 modules will have {\tt DATA SOURCE} set to {\tt MODULE} in {\tt .input} files. 
If file-based correlation is to be performed, the {\tt TAPELOG\_OBS} table is not needed and the burden of specifying media is moved to the {\tt .v2d} file. 
The files to correlate are specified separately for each antenna in an {\tt ANTENNA} block.
Note when specifying filenames, it is up to the user to ensure that full and proper paths to each file are provided and that the computer running the datastream for each antenna can see that file.
Two keywords are used to specify data files.
They are not mutually exclusive but it is not recommended to use both for the same antenna.
The first is {\tt file}.
The value assigned to {\tt file} is one or more (comma separated) file names.
It is okay to have multiple file keywords per antenna; all files supplied will be stored in the same order internally.
The second keyword is {\tt filelist} which takes a single argument, which is a file containing the list of files to read.
The file pointed to by {\tt filelist} only needs to be visible to {\tt vex2difx}, not the software correlator nodes. 
This file contains a list of file names and optionally start and stop MJD times.
Comments can be started with a \# and are ended by the end-of-line character.
Like for the file keyword, the file names listed must be in time order, even if start and stop MJD values are supplied.
An example file as supplied to {\tt filelist} is below:

\begin{verbatim}
# This is a comment.  File list for MK for project BX123
/data/mk/bx123.001.m5a  54322.452112 54322.511304
/data/mk/bx123.002.m5a  54322.512012 54322.514121 # a short scan
/data/mk/bx123.003.m5a  54322.766323 54322.812311 
\end{verbatim}

If times for a file are supplied, the file will be included in the {\tt .input} file DATA TABLE only if the file time range overlaps with the {\tt .input} file time range.
If not supplied, the file will be included regardless of the {\tt .input} file time range, which could incur a large performance problem.

A few sample ANTENNA blocks are shown below:

\begin{verbatim}
ANTENNA MK 
{
  filelist=bx123.filelist.mk
}
\end{verbatim}

\begin{verbatim}
ANTENNA OV { file=/data/ov/bx123.001.m5a, 
                  /data/ov/bx123.002.m5a,
                  /data/ov/bx123.003.m5a }
\end{verbatim}

\begin{verbatim}
ANTENNA PT { file=/data/pt/bx123.003.m5a } # recording started late here
\end{verbatim}


\subsubsection{Pulsars}

Some information, including example {\tt .v2d} sections, on setting up pulsar correlation can be found in \S\ref{sec:pulsars}.


You may find additional information at \url{http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/vex2difx} .






% vexpeek ---------------------------------------------------------------------

\subsection{vexpeek {\small $\mathrm{(package: vex2difx)}$}} \label{sec:vexpeek}

Program {\tt vexpeek} takes a vex file as input and sends to {\em stdout} the experiment name, segment, and a list of antennas and the MJD times that they were included in the observation.
This program is mainly intended to be called from python program {\tt db2vex} which needs to know a little about the file before appending the CLOCK and TAPELOG\_OBS tables.  
The VLBA operations system relies on such functionality but there is no reason other operations couldn't use this.
This program uses the same parsing infrastructure as {\tt vex2difx} so the warnings that may be produced and sent to {\tt stderr} in running {\tt vex2difx} will also do so with {\tt vexpeek}.
Thus, when {\tt db2vex} is run some of these error messages may be seen.

\begin{itemize}
\item[] Usage: {\tt vexpeek} $[${\em options}$]$ {\em vexFile} 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : print decoded version of {\em vexFile} to screen
\item[] {\tt -f} or {\tt --format} : add per-antenna format description to output
\item[] {\tt -b} or {\tt --bands} : print list of bands used by {\em vexFile}
\item[] {\tt -u} or {\tt --diskusage} : add per-antenna disk usage to output
\item[] {\tt -s} or {\tt --scans} : print list of scans and antennas used by each
\end{itemize}
\item[] {\em vexFile} is the vex format file to be inspected.
\end{itemize}





% vlog ------------------------------------------------------------------------

\subsection{vlog {\small $\mathrm{(package: vex2difx)}$}} \label{sec:vlog}

Program {\tt vlog} takes as input a calibration file ({\tt cal.vlba}, \S\ref{sec:cal}).
This file is parsed to produce four files containing formatted arrays that are convenient for use in the construction of FITS tables:
{\tt flag}, {\tt pcal}, {\tt tsys}, and {\tt weather} (\S\ref{sec:flag}, \S\ref{sec:pcal}, \S\ref{sec:tsys} and \S\ref{sec:weather}).
This program is named after AIPS task {\tt vlog} that does nearly the same thing.

\begin{itemize}
\item[] Usage: {\tt vlog} {\em calFile} $[${\em antennaList}$]$
\item[] {\em calFile} is the {\tt cal.vlba} file produced by {\tt tsm} to be processed.
\item[] {\em antennaList} is an optional comma-separated list of antennas to process.
If omitted, all antennas with calibration data will be processed.
\end{itemize}
Running with no command line arguments will print usage information to the terminal and exit.








% vmux ------------------------------------------------------------------------

\subsection{vmux {\small $\mathrm{(package: vdifio)}$}} \label{sec:vmux}

Program {\tt vmux} takes a VDIF file with multiple threads of one channel each and multiplexes the data into a new VDIF file consisting of a single thread containing all the channels.
In the case that a non-power-of-two number of channels are contained in the input file (or equivalently, a non-power-of-two number of threads are specified on the command line), the next power of two will be selected as the number of channels in the output thread and any unused channel slots will contain random data.

\begin{itemize}
\item[] Usage: {\tt vmux} $[${\em options}$]$ {\em inputFile} {\em inputFrameSize} {\em framesPerSecond} {\em threadList} {\em outputFile} $[${\em offset} $[${\em chunkSize}$] ]$
\item[] {\em inputFile} is the input multi-thread VDIF file, or {\tt -} for {\em stdin}
\item[] {\em inputFrameSize} is the size of one thread's data frame, including header (for RDBE VDIF data this is 5032)
\item[] {\em framesPerSecond} is the number of frames per second in the input file for each thread (and is thus the number of output frames per second as well)
\item[] {\em threadList} is a comma-separated list of integers in range 0 to 1023; the order of the numbers is significant and dictates the order of channels in the output data
\item[] {\em outputFile} is the name of the output, single-thread VDIF file, or {\tt -} for {\em stdout}
\item[] {\em offset} is an optional offset into the input file (in bytes)
\item[] {\em chunkSize} is (roughly) how many bytes to operate on at a time (default is 2000000)
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in execution
\item[] {\tt -q} or {\tt --quiet} : be less verbose in execution
\item[] {\tt -f} {\em f} or {\tt --fanout} {\em f} : set fanout factor to {\em f}
\item[] {\tt -e} or {\tt --EDV4} : convert VDIF extended data version to EDV4 (default)
\item[] {\tt -n} or {\tt --noEDV4} : don't convert VDIF extended data version to EDV4
\end{itemize}
\end{itemize}

The concept of fanout applies to certain variants of VDIF data where one logical sampled channel is interleaved multiple threads.
Certain modes of the DBBC3 makes use of this mode.
See Sec.~\ref{sec:vdiffanout} for details.

VDIF Extended Data Version (EDV) 4 is used to contain per-channel validity flags within a multi-channel VDIF file.
See Sec.~\ref{sec:vdifedv4} for details.





% vsn -------------------------------------------------------------------------

\subsection{vsn {\small $\mathrm{(package: mk5daemon)}$}} \label{sec:vsntool}

Program {\tt vsn} is used to check or set the Volume Serial Number (VSN), the write protect state, and the Disk Module State (DMS) of a module.
This program can not be used when the Mark5 unit is being used for something else.
You must be logged into the Mark5 unit that contains the module to inspect/change.
In addition to displaying the VSN of the module, this utility will list information about each disk in the module.
The columns displayed are:

\begin{enumerate}
\item {\em Disk number:} in the range 0 to 7.
\item {\em Drive model:} the model number of the disk.
\item {\em Serial number:} the serial number of the disk (in parentheses).
\item {\em Drive model revision number:} addition model information.
\item {\em SMART capable:} 1 indicates SMART information is available; 0 otherwise.
\item {\em SMART state:} (only valid if SMART capable) 1 indicates good health.
\end{enumerate}

Note that the drive model, serial number and revision number can have spaces making it hard to tell when one field start and the next begins.
Thus the serial number is contained within parentheses to make this clear.

\begin{itemize}
\item[] Usage: {\tt vsn} $[$ {\em options} $]$ {\em bank} $[$ {\em newVSN} $]$
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -f} or {\tt --force} : proceed without asking
\item[] {\tt -v} or {\tt --verbose} : be more verbose in operation
\item[] {\tt -p} or {\tt --played} : set DMS to played
\item[] {\tt -r} or {\tt --recorded} : set DMS to recorded
\item[] {\tt -e} or {\tt --erased} : set DMS to erased
\item[] {\tt -w} or {\tt --writeprotect} : set write protection
\item[] {\tt -u} or {\tt --unwriteprotect} : clear write protection
\end{itemize}
\item[] {\tt -s} or {\tt --smart} : Get S.M.A.R.T.\ data from disks and write to file {\em VSN}{\tt .smart})
\item[] {\em bank} is the Mark5 unit bank to look at (must be {\tt A} or {\tt B})
\item[] {\em newVSN} is the new name to assign to the module and must be a legal VSN
\item[] Example 1: show VSN: {\tt vsn A}
\item[] Example 2: set VSN: {\tt vsn A NRAO+456}
\end{itemize}

If you get a message such as ``Watchdog caught a hang-up executing \ldots''\ that means access to the module failed.
This could indicate a bad module.
The module should be reinserted (perhaps in a different bank or unit) and the unit rebooted before coming to a firm conclusion.







% vsum ------------------------------------------------------------------------

\subsection{vsum {\small $\mathrm{(package: vdifio)}$}} \label{sec:vsum}

Program {\tt vsum} prints a summary of one or more VDIF files, printing such information as list of threads found, collectively, in the first and last few MB of the file, the VDIF epoch and other time information, and packet size.
If the data is not recognized as VDIF, an error code will be printed; see below for the list of codes and their meanings.
Legacy format VDIF data is not supported.

\begin{itemize}
\item[] Usage: {\tt vsum} $[$ {\em options} $]$ {\em file1} $[$ {\em file2} $[ \ldots ] ]$
\item[] {\em file1} \ldots is/are the VDIF file(s) to summarize
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -s} or {\tt --shortsum} : print one-line summary per file
\item[] {\tt -6} or {\tt --mark6} : interpret provided file names as Mark6 scans
\item[] {\tt --allmark6} : summarize all files found on mounted Mark6 modules
\end{itemize}
\end{itemize}

If the {\tt -6} or {\tt --mark6} or {\tt --allmark6} option is used, it is assumed that the files are to be found in their expected location, which could be altered by an environment variable.
See sec~\ref{sec:mark6path} for more details.

If the summary operation failed for a file, one of the following error codes will be returned:
\begin{center}
\begin{tabular}{ll}
-1 & The file size could not be determined \\
-2 & The file could not be opened \\
-3 & Memory allocation failure \\
-4 & File read failed \\
-5 & Frame size could not be determined \\
-6 & First frame could not be found \\
-7 & Seek to near-end-of-file failed \\
-8 & Read at end of file failed \\
-9 & A valid frame at the end of the file could not be found \\
\end{tabular}
\end{center}

The {\tt -s} or {\tt --shortsum} option produces output that can be used directly by {\tt vex2difx} as a file list.



% zerocorr --------------------------------------------------------------------

\subsection{zerocorr {\small $\mathrm{(package: mark5access)}$}} \label{sec:zerocorr}

Program {\tt zerocorr} is intended to cross correlate data with zero time delay (and a window determined by the spectral resolution) between two recordings made at the same station.
It is possible to correlate data observed in different formats and even mis-matched bandpasses, through the construction of a file describing the details of the single sub-band that is to be correlated.

\begin{itemize}
\item[] Usage: {\tt zerocorr} $[$ {\em options} $]$ {\em confFile} 
\item[] {\em options} can be:
\begin{itemize}
\item[] {\tt -h} or {\tt --help} : print usage information and exit
\item[] {\tt -v} or {\tt --verbose} : be more verbose in operation
\end{itemize}
\item[] {\em confFile} is a file describing the correlation parameters
\item[] Example: {\tt zeroconf td006.zc}
\end{itemize}

See documentation on {\em confFile} (or {\tt .zc} file) in section \ref{sec:zc}.
Output data documentation can be found for {\tt .vis} files in section \ref{sec:vis} and for {\tt .lag} files in section \ref{sec:lag}.
