set up a tree of test case data files (for regression tests)

[VincentMcIntyre]

We need to know when we do something to the code that breaks existing functionality, i.e. we introduce a regression. The usual path for this is ​unit testing. There are lots of ​testing frameworks available

This is difficult to do with scientific software because the answers each subroutine are not always possible to predict. Also, the only free unit ​test framework for FORTRAN is aimed at FORTRAN-90, it doesn't handle FORTRAN-77. Some other frameworks are mentioned in ​this report.

So the best we can do right now is "smoke tests" - read in some data files and process them with a shell (or perl) script that calls MIRIAD subroutines, and look for errors (or possibly for large differences). To do this we need to start collecting data files that exercise the various corners of the MIRIAD code. We should also collect any data files that give errors. Once the bugs that gave rise to the problem are fixed, we should be able to rerun those files at any time and see no errors.

The task for this ticket is to:

• set up a suitable directory structure
• plunk a few files in it (uvdata and images)
• write some scripts that at least run the data through ATLOD or FITS

We can build up more complex tests as we go along.

[VincentMcIntyre]

see also ​tutorial for testing in perl

[StacyMader]

"Smoke testing" received its name from the hardware industry. When a unit was first powered on and no smoke was produced, it pass the smoke test. These type of tests are likened to window shopping in that only the superficial aspects of the item are tested. A good example are the quality assurance smoke tests done for the MOZILLA project:

​http://www.mozilla.org/quality/smoketests/

By performing these smoke tests, one can identify bugs when the reported behaviour differs from expected behaviour. A bug report is filled in and acted on before that change can be released into the on-line version.

MORE LATER...

[StacyMader]

To start with, I have written a script (smoke.pl; attatched) which will catch any differences (using DIFF) between the output given from the stable and development versions. The stable version log is generated if not present. The script simply executes the development routine and compares the output between that and the on-disk stable log. If there are differences, you are alerted (either via the terminal you start from or by using tkdiff if you add the appororiate switch "-t 1" to the script.)

To start with, a proposed directory structure is as follows (in the users personal MIRIAD area):

1. miriad/smoke -> top-level
   1. data/ -> Top-level directory where our good/bad data files live
      1. temp/ -> Where temporary data output files (produced by running the test script) go
   2. stable/ -> Output logs from from routines in the "stable" version of the code
   3. test/ -> Output logs from development routines
   4. switches/ -> Files containing switches for routine. There can be more than one file per routine to test various aspects of that routine.

So to test atlod with various switch files (and be alerted via tkdiff if there are differences), simply type "smoke.pl -s atlod -t 1".

TODO?

• Add a flag to allow user to define where temporary data files go (won't work with all routines as some dump output files in the current directory, ie uvsplit).

[StacyMader]

Replying to StacyMader:

To start with, I have written a script (smoke.pl; attatched) which will catch any differences (using DIFF) between the output given from the stable and development versions. The stable version log is generated if not present. The script simply executes the development routine and compares the output between that and the on-disk stable log. If there are differences, you are alerted (either via the terminal you start from or by using tkdiff if you add the appororiate switch "-t 1" to the script.)

To start with, a proposed directory structure is as follows (in the users personal MIRIAD area):

1. miriad/smoke -> top-level
   1. data/ -> Top-level directory where our good/bad data files live
      1. temp/ -> Where temporary data output files (produced by running the test script) go
   2. stable/ -> Output logs from from routines in the "stable" version of the code
   3. test/ -> Output logs from development routines
   4. switches/ -> Files containing switches for routine. There can be more than one file per routine to test various aspects of that routine.

So to test atlod with various switch files (and be alerted via tkdiff if there are differences), simply type "smoke.pl -s atlod -t 1".

TODO?

• Add a flag to allow user to define where temporary data files go (won't work with all routines as some dump output files in the current directory, ie uvsplit).

[StacyMader]

Replying to StacyMader:

Replying to StacyMader:

To start with, I have written a script (smoke.pl; attatched) which will catch any differences (using DIFF) between the output given from the stable and development versions. The stable version log is generated if not present. The script simply executes the development routine and compares the output between that and the on-disk stable log. If there are differences, you are alerted (either via the terminal you start from or by using tkdiff if you add the appororiate switch "-t 1" to the script.)

To start with, a proposed directory structure is as follows (in the users personal MIRIAD area):

1. miriad/smoke -> top-level
   1. data/ -> Top-level directory where our good/bad data files live
      1. temp/ -> Where temporary data output files (produced by running the test script) go
   2. stable/ -> Output logs from from routines in the "stable" version of the code
   3. test/ -> Output logs from development routines
   4. switches/ -> Files containing switches for routine. There can be more than one file per routine to test various aspects of that routine.

So to test atlod with various switch files (and be alerted via tkdiff if there are differences), simply type "smoke.pl -s atlod -t 1".

TODO?

• Add a flag to allow user to define where temporary data files go (won't work with all routines as some dump output files in the current directory, ie uvsplit).