Experiment Editor

The Experiment Editor allows you to create a new DiFX experiment (or "session"), optionally containing a pass and/or a list of jobs.  It also allows an existing experiment to be edited and new passes and job sets to be created.  All directory structures and files associated with the experiment are created on the DiFX host, as well as in the DiFX database (if it is being used), and the experiment, pass(es), and job(s) are listed on the Queue Browser on the GUI from where DiFX can be run.

As a user you control the specifics of a DiFX job using two files - the ".vex" file, which describes your observations, and the ".v2d" file, which (loosely) describes how you wish DiFX to process those observations (these definitions with apologies to the DiFX creators who defined these files and can certainly better - and perhaps differently - describe what they do).  The Experiment Editor needs to be provided with a .vex file - one was presumably made available with your observations.  Almost all of what the Experiment Editor does involves creating a detailed .v2d file.  The .v2d file is then used by the vex2difx application to produce a ".input" file - which in turn can be used to run DiFX.  There are associated files and directory structures that are created as well, and some other details are seen to, but the point to take away is that until you hit the "Apply" button on the Experiment Editor you are using it to define a .v2d file.

Alternatively, specified as an outline, the steps to creating a new DiFX experiment with the Experiment Editor involve:
  1. Specify where your experiment will go and what it will be called
  2. Get a .vex file
  3. Detail all of the stuff that goes into the .v2d file
  4. Hit "Apply" to generate the specified directory and dump the .vex and .v2d file in it

The Experiment Editor is launched either to create a new experiment (by picking "New" from the Queue Browser) in which case its title is "Create New Experiment", or to edit an existing experiment (by right-clicking on the experiment name in the Queue Browser and choosing "Edit Properties") in which case its title is "Edit Experiment Properties".  While the controls are identical, applying changes to an existing experiment has slightly different effects than those applied to a new experiment, as described below.

Some General Information

The Experiment Editor controls are organized in hierarchical indexed panels, with a small number of "top level" panels that can be opened or closed based on whether you are interested in them or not.  The "Identification Data" panel at the top of the Editor and the "Apply/Cancel" panel at the bottom are "permanent" - they cannot be closed.

Many of the settings in the Experiment Editor are "remembered" if you run it a subsequent time (and even between sessions of running the GUI).  This is done to ease repetitive tasks, making the assumption that any time you run the Experiment Editor, a very good guess for the way you wish to run it is somewhat similar to the way you last ran it.  For the most part this should save you time, however it bears watching here and there.  If you change a setting that you never touch and don't even fully understand the consequences of just for fun, it may very well stay changed until you actively undo it.

Nothing happens to the DiFX host, the database, or your experiments until you hit the "Apply" button, so don't worry about playing with the controls (bearing in mind the caveat in the previous paragraph).  You can always "Cancel" and start again if you get in a position you can't figure out how to back out of.

Identification Data

The experiment name, ID, directory, and some other items are used to uniquely identify an experiment and to access it in the database.  Some of these items are not changed after the experiment is created, others can be but generally aren't, and some are meant to be changed.  They are contained in the top section of Experiment Editor.

Name
This is the name of the experiment which appears in the Queue Browser.  When an experiment is initially created, a default name will be generated - a combination of the word "Experiment" and the Database ID number.  It can be changed to reflect something that better describes the experiment to the user.  Due to restrictions imposed by the database, a 20 character limit is placed on the name length.  Changes to the name are propagated to the "Directory" and ".vex File" parameters, unless those items have been previously changed (you have to hit return to make changes do this).

Number
The number is contained in the database, and I'm not sure what it is.  I'll get back to you on this one.  This is not the database ID.

Status
The status describes the current state of the experiment.  No doubt this will be used by the GUI in the future, but for the moment it is simply something the user can set for their own tracking purposes.  If the database is being used the options permitted by the database are given for status.  If no database is being used the status currently doesn't work and can't be changed, but ultimately a way of allowing the user to specify their own status strings will be implemented.

In Database
This check box will determine whether the experiment is put in the database (for experiments that are already created, it will indicate whether they exist in the database).  If the database is not being used, this item should remain unchecked.  An experiment that was not, for whatever reason, originally created with this item checked can later be put in the database by checking it.

Database ID
This is the unique ID generated by the database to track the experiment.  When an experiment is initially created, a "guess" for this number is generated (usually it is a pretty good guess).  Later if the experiment is edited the number will accurately reflect the ID used by the database.  If the database is not being used, the original guess is retained.  This number cannot be changed by the user.

Created
This field contains the creation time and date for the experiment.  When a new experiment is created, the field will show the local time of the GUI host at the moment the Experiment Editor was opened.  Once the experiment is put in the database, this time will change (hopefully not much) to reflect the actual creation time on the database host.  This is not a user-settable item.

Working Directory
This is the "working directory" on the DiFX host under which all files and sub-directories associated with an experiment reside.  It should be an absolute (not relative) path name - starting with a "/".  When an experiment is created, this directory is created on the DiFX host (if necessary).  If this parameter is later changed, it will cause the directory to be moved on the DiFX host.  For either of these operations to occur, the DiFX user must have write permission to do them on the DiFX host.

.vex File
This is the name of the .vex file that will be stored (if creating a new experiment) or is already stored (if editing an experiment) under the working directory.  A session of the Experiment Editor may work with only one .vex file at a time, however any number of .vex files may be associated with an experiment.  The data stored in the .vex file is obtained using the "Get .vex File Content" as described in the next section.

If you already have a .vex file associated with an experiment and wish to create a new one, enter a new name in this field and obtain new content using "Get .vex File Content".  If you have several .vex files associated with an experiment and wish to use a previous one, the Previous Files button can be used to load it.

Starting .v2d File

Get .vex File Content

An experiment is associated with at least one .vex file - in most cases only one (although more can be added).  The .vex file contains a complete description of an observing session, including times, frequencies, antennas used, sources observed, and scans collected (as well as other things).  The GUI uses the .vex file to provide the user with the options that may be applied to tailor DiFX processing of the observations.

If you are creating a new experiment, you need a .vex file before you can go anywhere.  Any set of observations should have an associated .vex file supplied with them (or provide some way of producing a file).  The "Get .vex File Content" of the Experiment Editor provides you with a number of ways of obtaining .vex file data, including:

  1. Grabbing a file from a location on the DiFX Host.
  2. Obtaining a file via HTTP.
  3. Obtaining a file via FTP.
  4. Using a file on the GUI Host.
  5. Copying a file used in another experiment.
Note that when you pick a .vex file in this interface (when you hit "Go"), you are getting the content of the source you specify - the source itself is not used for your experiment.  The content is stored in the file location specified in the .vex File field in the Identification Data section (see above) when the "Apply" button is hit.

.vex File Editor

The .vex File Editor panel provides a view of the .vex file content - the text that is written to the .vex file on the DiFX host (and that ultimately governs how DiFX will be run).  The panel was created to support GUI development, and may be removed in the future.

The "Alternate Editor" button doesn't do anything.

The content can be edited (the panel is a rudimentary editor with cut, copy and paste capabilities), bearing in mind that no effort is made to check edits for sanity.  Any changes are propagated verbatim to the .vex file stored on the DiFX host, which is then used to produce .input files that are read by DiFX when it runs.  Typographic or logic errors in the .vex content can produce any number of obscure and opaque errors.  Don't mess around with the .vex file content unless you are confident you know what you are doing!

Correlation Parameters

The Correlation Parameters section is something of a catch-all for items that don't fit neatly in any other sections.  Most of these parameters end up in the "Setup" section of the .v2d file.

Spectral Resolution/Channels

This is the spectral resolution of the visibilities produced by the correlation (in MHz).  This should be the bandwidth divided by the number of channels.  The "Channels" setting can be used to change the spectral resolution, although the reverse is not true.  The reason for this is that the channels setting is limited to factors of 2 in the DiFX documentation, while the spectral resolution has no such limitations.  Whether or not this makes any sense is another matter.

Stations

Using a File List

Sources

Scan Selection

EOP Data


Earth Orientation Parameter (EOP) data are necessary for a correlation process to run (they are required by calcif2).  The Experiment Editor provides EOP data, when available, from two different sources:
  1. The .vex file may contain EOP data.  Often these data are a "projection" into the future, as they are produced when observations occur (and before actual EOP measurements are made).  They are usually reasonably accurate.
  2. EOP data is obtained from a URL, as specified in the Settings menu.  If the URL is set up correctly, and the connection to the internet is good, these data should always be available.  They will be at least as accurate as the "projected" data that may already exist in the .vex file, and in fact may represent actual measurements and thus be better (they most likely will never be worse).  The GUI updates these data when it is started and roughly every hour thereafter.
Data from either source can be selected using the check boxes next to the title bar for each source.  Of the two sources, the updated EOP data are probably preferred (the Experiment Editor will pick them automatically when available), however there are imaginable circumstances where data from the .vex file may be preferred.  For instance, it should be noted that because they are constantly updated the "Updated From Source" EOP measurements will not produce identical DiFX results for multiple runs on the same data.  As stated before, changes will be small, and probably for the better, but they will not exactly match.  If for some reason you need to eliminate changing EOP data as a source of changes in calculated results, use the .vex file EOP.

Names, Etc.

The Names section can be used to change the way vex2difx will name the input files it creates (an .input file is created for each "job").  By default, the vex2difx naming scheme is to use the "base" name of the vex file (e.g. "foo" for the file "foo.vex") with an appended integer followed by ".input".  For the first input file created the appended integer will be 1; the integer will increment for subsequent input files.  For instance, if the "foo.vex" file is used to create three jobs, their input files will be called "foo_1.input", "foo_2.input", and "foo_3.input".

If for some reason you don't like the default scheme, the Names section can give you some control over the names of the files that will be created.

.v2d File

The .v2d file is the "configuration" file used by vex2difx to implement user adjustments when using the .vex file content to create one or more .input files.  Most of the controls in the Experiment Editor are used to make changes to this file.

The .v2d File section shows the content the .v2d file that will be created in your working directory (when you hit "Apply").  This content is continually updated as you make changes using other controls.  If you feel confident you know what you are doing you can edit the content by hand, however any changes you make to other controls (any of those above the .v2d file section in the GUI that is) will cause the .v2d content to be recreated and will erase your changes.  If you need to change something by hand, do it last.

Applying Your Changes

Creating a New Pass (or not!)
Combine Scans in One Job

The GUI can create jobs out of the specified scans in one of two ways, either as a series of individual scans with one job per scan, or a single job containing all scans.  Those who know vex2difx well will recognize this as a slight change from what it does.  Vex2difx can be set to create single-scan jobs, but otherwise it arranges scans in one or more jobs based on the "maxGap" parameter - breaking into new jobs only when the gap between two correlations exceeds this value.  In fact, the GUI essentially cheats by setting the "maxGap" parameter to something huge so scans are never broken into more than one job.

Status Bar

What is the GUI Doing?

When you hit the "Apply" button, the GUI "creates" jobs on the DiFX Host that can then be run.  But what exactly is going on?  Following is a step-by-step description of the operations performed.

EOP Data

Your choice of source (either .vex file or external URL) and the availability of EOP data will cause a number of things to happen.  The following is a list of actions taken by the GUI based on what it can do, and what it is told to do (note that the GUI will not allow you to choose a source for which data are unavailable):