Index: /trunk/docs/ProjectBook/conversionSoftware.tex
===================================================================
--- /trunk/docs/ProjectBook/conversionSoftware.tex	(revision 259)
+++ /trunk/docs/ProjectBook/conversionSoftware.tex	(revision 260)
@@ -37,4 +37,5 @@
 
 \section{Pre-existing data format}
+\label{sec-dataFormat}
 
 The current formats of the data files for all three surveys are very
@@ -148,22 +149,9 @@
 information about (\eg time and position information from the
 header). The dimensions of each entry will be (\# samples per
-block)$\times$(\# channels per sample). Since there is a fixed time
-per pointing for each survey, there will be a fixed number of blocks
-for each pointing position, and hence a fixed number of rows in the
-\subint table. Table~\ref{tab-datasize} summarises the size of the
-table and the dimensions of each data entry (note that there is no
-polarisation information in any of this data):
-
-\begin{table}[h]
-\begin{tabular}{lll}
-Survey  &Number of \subint rows &Dimensions of data table\\ 
-\hline
-S70          &330                            &$1536\times256$\\
-PM           &2051                           &$4096\times96$\\
-PH           &517                            &$4096\times96$\\
-\end{tabular}
-\caption{The size of the \subint data table for each survey.}
-\label{tab-datasize}
-\end{table}
+block)$\times$(\# channels per sample). The number of rows in the
+\subint table will be governed by the survey length and the required
+size. See \S\ref{sec-fitsLenth} for details on how the length of the
+FITS file is determined.
+
 
 \section{Algorithm}
@@ -193,4 +181,5 @@
   \begin{itemize}
   \item \texttt{block.readBlock(fin);}
+  \item Each byte in the data array is reversed.
   \item \texttt{block.telescopeOK();}
   \item This information will be used to determine future changes in
@@ -237,5 +226,5 @@
   changed, close the file.
   \begin{itemize}
-  \item See \S\ref{sec-moving} for details on what ``full'' means.
+  \item See \S\ref{sec-fitsLenth} for details on what ``full'' means.
   \item This is done with \texttt{PSRFits::closeFile()}.
   \end{itemize}
@@ -329,5 +318,5 @@
 
 \subsection{How we know when a FITS file is finished}
-\label{sec-moving}
+\label{sec-fitsLenth}
 
 The S70 survey includes many pointings in a given (existing) data
@@ -350,126 +339,4 @@
 the number of samples reaches 512K, so that the last few rows in the
 \subint table will have different positions to the rest.
-
-
-\section{Location of code and usage}
-
-\subsection{Location}
-
-The software is part of the pulsarVO Subversion repository described
-in \S\ref{sec-svn}. The working code, however, has been installed in
-the pulsar filesystem, in the directory \texttt{/pulsar/psr/pvo},
-where a Makefile uses the standard make system to install the
-executables in the directory \texttt{/psr/linux/bin} (in the case of
-installation on a linux machine). The Makefile used in this case is
-\texttt{Makefile.psr} in the trunk/ directory of the repository.
-
-\subsection{How to run \pvoconvert}
-
-The program is run very simply. If you have an existing tape file, \eg
-S31201, then you can convert it by typing
-\begin{quote}
-\texttt{> pvoconvert S31201}
-\end{quote}
-
-More than one file can be done at once, and wildcards can be accepted
-according to the usual shell rules \eg
-\begin{quote}
-\texttt{> pvoconvert S31201 S31204 S3131*}
-\end{quote}
-
-If an output file already exists, the default behaviour is to stop
-rather than write over the file. If you want to overwrite files in
-this situation, use the \texttt{-c} or \texttt{--clobber} option.
-
-If the datafile is from the PH Survey, you need to say so explicitly
-so that \pvoconvert knows to apply the PH settings (as the PH files
-have the same basic structure as the PM ones). This is done by using
-the \texttt{-H} flag:
-\begin{quote}
-\texttt{> pvoconvert -H <file(s)>}\hspace{1cm}\\
-\texttt{> pvoconvert --highlat <file(s)>}
-\end{quote}
-If a list of files is provided, and some are S70 files, they will be
-processed according to the S70 setup. However, you cannot provide a
-mix of PH and PM files, as all will get processed in the same way. 
-
-Other possible flags are \texttt{-h} or \texttt{--help} to provide a
-help message, and \texttt{-v} or \texttt{--verbose} to set verbose
-mode for maximum information for any error or warning messages.
-
-\section{Other PulsarVO programs}
-
-We briefly detail here two utility programs that have been developed
-for use with the newly created FITS files. They are not sufficiently
-complex to warrant their own document.
-
-\subsection{\pvosummary}
-
-This is a utility program to print out key metadata from PSRFITS
-files. The standard functionality of this program is to print out
-basic information about a FITS file: filename, RA, Dec, Beam size,
-Frequency, Date of Observation. It can also produce a csv file that
-lists many more parameters -- the full list, obtainable from
-\texttt{pvosummary -p}, is:
-{\small
-\begin{verbatim}
-Basic parameters written to screen:
- FILENAME       The FITS file.
- SRC_NAME       The SRC_NAME FITS keyword.
- RAJ            Right Ascension, hh:mm:ss.ss
- DECJ           Declination, dd:mm:ss.sss 
- BEAM FWHM      The average full-width-half-maximum of the beam
-                 (averaged between BMAJ and BMIN)
- OBSFREQ        The central observing frequency
- DATE-OBS       The UT time and date of the observation, in FITS
-                 format YYYY-MM-DDThh:mm:ss.ssss
-
-Additional parameters included only in csv file:
- File Location  The full path of the FITS file.
- Header Version The version of the PSRFITS definition used.
- Survey         Which survey the data comes from:
-                  left blank if not known or not a survey file (e.g. .rf files).
- Source Name    The SRC_NAME FITS keyword (printed as SRC_NAME to screen).
- Telescope      Where the observations were made.
- Data_type      The type of data -- usually the backend instrument used.
- RAJd           Right Ascension, in decimal degrees.
- DecJd          Declination, in decimal degrees.
- GL             Galactic Longitude, in degrees.
- GB             Galactic Latitude, in degrees.
- BMAJ           The FWHM of the major axis of the beam.
- BMIN           The FWHM of the minor axis of the beam.
- BPA            The position angle of the beam.
- MJD            The Modified Julian Date of the start of the observation.
- ChanBW         The channel bandwidth (from the HISTORY table).
- Nchan          Number of channels (from the HISTORY table).
- Scan length    The SCANLEN keyword.
- Tsamp          Time per sample [ms] (from the SUBINT table).
- NBits          Number of bits per datum (from the SUBINT table).
-\end{verbatim}
-}
-
-\subsection{\pvoconesearch}
-
-\pvoconesearch is a simple command-line cone-search tool that searches
-around a given (RA,DEC) location, within a given radius, and returns
-the same metadata as pvosummary for all FITS files whose beams overlap
-the search area. The returned files are listed in order of increasing
-separation from the requested (RA,DEC) position. RA and DEC can be in
-hh:mm:ss.s format or decimal degrees, while the search radius is in
-degrees.
-
-\subsection{\pvotapesummary}
-
-\pvotapesummary is a tool to summarise some basic metadata of the
-first block of PKS survey tape files. This is designed to give you an
-overview of the contents and type of data of the tape files. It
-produces a list in a similar format to \pvosummary, but with slightly
-different columns: FILENAME, RAJ, DECJ, OBSFREQ, SCANLEN, MOVE,
-DATE-OBS, where MOVE is the moving flag, indicating whether the
-telescope is scanning or doing a pointed (timing) observation. The
-main purpose is to allow one to discriminate between 70cm Survey files
-and timing observations that are not part of the survey. It works
-either a list of one or more files, or on all files in a given
-directory.
 
 
@@ -508,5 +375,4 @@
 \end{itemize}
 
-
 \subsection{Writing the data to the FITS file's binary table}
 \label{sec-1bit1st}
@@ -526,5 +392,5 @@
 \end{enumerate}
 
-The latter method might be the best option, as it is much less memory
+The latter method is the better option, as it is much less memory
 intensive. For instance, if one read a character from the existing
 data, that is 1 byte or 8 bits, and so 8 separate measurements. This
@@ -533,6 +399,68 @@
 sure there is no loss of information.
 
-
-\section{Implementation of Header analysis}
+\subsection{Preserving the data}
+
+The observational data (as opposed to the metadata in the headers) is
+in the form of 1-bit data, stored as 8 bits to a byte -- see
+\S\ref{sec-dataFormat} for details. When this data was written, the
+bits were in frequency order, with the least-significant-bit (LSB)
+having the lowest frequency channel in that byte, and the
+most-significant-bit (MSB) having the highest. 
+
+However, the FITS writing functions detailed above actually store the
+bits the other way around, from MSB to LSB. Thus, to have the bit
+order match the order of the frequency channels (as given by the
+\textsc{dat\_freq} array in the \subint table), we need to reverse the
+order of the bits in each byte in the data array.
+
+This is done by using a simple 256-element look-up table, where we
+reverse a char \verb|c| in the following manner:
+
+\begin{quote}
+\footnotesize{
+  \begin{verbatim}
+    static const unsigned char BitReverseTable256[] = 
+      {
+        0x00, 0x80, 0x40, 0xC0, 0x20, 0xA0, 0x60, 0xE0, 
+        0x10, 0x90, 0x50, 0xD0, 0x30, 0xB0, 0x70, 0xF0, 
+        0x08, 0x88, 0x48, 0xC8, 0x28, 0xA8, 0x68, 0xE8, 
+        0x18, 0x98, 0x58, 0xD8, 0x38, 0xB8, 0x78, 0xF8, 
+        0x04, 0x84, 0x44, 0xC4, 0x24, 0xA4, 0x64, 0xE4, 
+        0x14, 0x94, 0x54, 0xD4, 0x34, 0xB4, 0x74, 0xF4, 
+        0x0C, 0x8C, 0x4C, 0xCC, 0x2C, 0xAC, 0x6C, 0xEC, 
+        0x1C, 0x9C, 0x5C, 0xDC, 0x3C, 0xBC, 0x7C, 0xFC, 
+        0x02, 0x82, 0x42, 0xC2, 0x22, 0xA2, 0x62, 0xE2, 
+        0x12, 0x92, 0x52, 0xD2, 0x32, 0xB2, 0x72, 0xF2, 
+        0x0A, 0x8A, 0x4A, 0xCA, 0x2A, 0xAA, 0x6A, 0xEA, 
+        0x1A, 0x9A, 0x5A, 0xDA, 0x3A, 0xBA, 0x7A, 0xFA,
+        0x06, 0x86, 0x46, 0xC6, 0x26, 0xA6, 0x66, 0xE6, 
+        0x16, 0x96, 0x56, 0xD6, 0x36, 0xB6, 0x76, 0xF6, 
+        0x0E, 0x8E, 0x4E, 0xCE, 0x2E, 0xAE, 0x6E, 0xEE, 
+        0x1E, 0x9E, 0x5E, 0xDE, 0x3E, 0xBE, 0x7E, 0xFE,
+        0x01, 0x81, 0x41, 0xC1, 0x21, 0xA1, 0x61, 0xE1, 
+        0x11, 0x91, 0x51, 0xD1, 0x31, 0xB1, 0x71, 0xF1,
+        0x09, 0x89, 0x49, 0xC9, 0x29, 0xA9, 0x69, 0xE9, 
+        0x19, 0x99, 0x59, 0xD9, 0x39, 0xB9, 0x79, 0xF9, 
+        0x05, 0x85, 0x45, 0xC5, 0x25, 0xA5, 0x65, 0xE5, 
+        0x15, 0x95, 0x55, 0xD5, 0x35, 0xB5, 0x75, 0xF5,
+        0x0D, 0x8D, 0x4D, 0xCD, 0x2D, 0xAD, 0x6D, 0xED, 
+        0x1D, 0x9D, 0x5D, 0xDD, 0x3D, 0xBD, 0x7D, 0xFD,
+        0x03, 0x83, 0x43, 0xC3, 0x23, 0xA3, 0x63, 0xE3, 
+        0x13, 0x93, 0x53, 0xD3, 0x33, 0xB3, 0x73, 0xF3, 
+        0x0B, 0x8B, 0x4B, 0xCB, 0x2B, 0xAB, 0x6B, 0xEB, 
+        0x1B, 0x9B, 0x5B, 0xDB, 0x3B, 0xBB, 0x7B, 0xFB,
+        0x07, 0x87, 0x47, 0xC7, 0x27, 0xA7, 0x67, 0xE7, 
+        0x17, 0x97, 0x57, 0xD7, 0x37, 0xB7, 0x77, 0xF7, 
+        0x0F, 0x8F, 0x4F, 0xCF, 0x2F, 0xAF, 0x6F, 0xEF, 
+        0x1F, 0x9F, 0x5F, 0xDF, 0x3F, 0xBF, 0x7F, 0xFF
+      };
+    
+    unsigned char r = BitReverseTable256[c & 0xff];
+\end{verbatim}
+}
+\end{quote}
+
+
+\section{Implementation of Header information}
 
 We list here some important details about how the header information
@@ -620,2 +548,125 @@
   function in the header class above.
 \end{itemize}
+
+
+\section{Location of code and usage}
+
+\subsection{Location}
+
+The software is part of the pulsarVO Subversion repository described
+in \S\ref{sec-svn}. The working code, however, has been installed in
+the pulsar filesystem, in the directory \texttt{/pulsar/psr/pvo},
+where a Makefile uses the standard make system to install the
+executables in the directory \texttt{/psr/linux/bin} (in the case of
+installation on a linux machine). The Makefile used in this case is
+\texttt{Makefile.psr} in the trunk/ directory of the repository.
+
+\subsection{How to run \pvoconvert}
+
+The program is run very simply. If you have an existing tape file, \eg
+S31201, then you can convert it by typing
+\begin{quote}
+\texttt{> pvoconvert S31201}
+\end{quote}
+
+More than one file can be done at once, and wildcards can be accepted
+according to the usual shell rules \eg
+\begin{quote}
+\texttt{> pvoconvert S31201 S31204 S3131*}
+\end{quote}
+
+If an output file already exists, the default behaviour is to stop
+rather than write over the file. If you want to overwrite files in
+this situation, use the \texttt{-c} or \texttt{--clobber} option.
+
+If the datafile is from the PH Survey, you need to say so explicitly
+so that \pvoconvert knows to apply the PH settings (as the PH files
+have the same basic structure as the PM ones). This is done by using
+the \texttt{-H} flag:
+\begin{quote}
+\texttt{> pvoconvert -H <file(s)>}\hspace{1cm}\\
+\texttt{> pvoconvert --highlat <file(s)>}
+\end{quote}
+If a list of files is provided, and some are S70 files, they will be
+processed according to the S70 setup. However, you cannot provide a
+mix of PH and PM files, as all will get processed in the same way. 
+
+Other possible flags are \texttt{-h} or \texttt{--help} to provide a
+help message, and \texttt{-v} or \texttt{--verbose} to set verbose
+mode for maximum information for any error or warning messages.
+
+\section{Other PulsarVO programs}
+
+We briefly detail here two utility programs that have been developed
+for use with the newly created FITS files. They are not sufficiently
+complex to warrant their own document.
+
+\subsection{\pvosummary}
+
+This is a utility program to print out key metadata from PSRFITS
+files. The standard functionality of this program is to print out
+basic information about a FITS file: filename, RA, Dec, Beam size,
+Frequency, Date of Observation. It can also produce a csv file that
+lists many more parameters -- the full list, obtainable from
+\texttt{pvosummary -p}, is:
+{\small
+\begin{verbatim}
+Basic parameters written to screen:
+ FILENAME       The FITS file.
+ SRC_NAME       The SRC_NAME FITS keyword.
+ RAJ            Right Ascension, hh:mm:ss.ss
+ DECJ           Declination, dd:mm:ss.sss 
+ BEAM FWHM      The average full-width-half-maximum of the beam
+                 (averaged between BMAJ and BMIN)
+ OBSFREQ        The central observing frequency
+ DATE-OBS       The UT time and date of the observation, in FITS
+                 format YYYY-MM-DDThh:mm:ss.ssss
+
+Additional parameters included only in csv file:
+ File Location  The full path of the FITS file.
+ Header Version The version of the PSRFITS definition used.
+ Survey         Which survey the data comes from:
+                  left blank if not known or not a survey file (e.g. .rf files).
+ Source Name    The SRC_NAME FITS keyword (printed as SRC_NAME to screen).
+ Telescope      Where the observations were made.
+ Data_type      The type of data -- usually the backend instrument used.
+ RAJd           Right Ascension, in decimal degrees.
+ DecJd          Declination, in decimal degrees.
+ GL             Galactic Longitude, in degrees.
+ GB             Galactic Latitude, in degrees.
+ BMAJ           The FWHM of the major axis of the beam.
+ BMIN           The FWHM of the minor axis of the beam.
+ BPA            The position angle of the beam.
+ MJD            The Modified Julian Date of the start of the observation.
+ ChanBW         The channel bandwidth (from the HISTORY table).
+ Nchan          Number of channels (from the HISTORY table).
+ Scan length    The SCANLEN keyword.
+ Tsamp          Time per sample [ms] (from the SUBINT table).
+ NBits          Number of bits per datum (from the SUBINT table).
+\end{verbatim}
+}
+
+\subsection{\pvoconesearch}
+
+\pvoconesearch is a simple command-line cone-search tool that searches
+around a given (RA,DEC) location, within a given radius, and returns
+the same metadata as pvosummary for all FITS files whose beams overlap
+the search area. The returned files are listed in order of increasing
+separation from the requested (RA,DEC) position. RA and DEC can be in
+hh:mm:ss.s format or decimal degrees, while the search radius is in
+degrees.
+
+\subsection{\pvotapesummary}
+
+\pvotapesummary is a tool to summarise some basic metadata of the
+first block of PKS survey tape files. This is designed to give you an
+overview of the contents and type of data of the tape files. It
+produces a list in a similar format to \pvosummary, but with slightly
+different columns: FILENAME, RAJ, DECJ, OBSFREQ, SCANLEN, MOVE,
+DATE-OBS, where MOVE is the moving flag, indicating whether the
+telescope is scanning or doing a pointed (timing) observation. The
+main purpose is to allow one to discriminate between 70cm Survey files
+and timing observations that are not part of the survey. It works
+either a list of one or more files, or on all files in a given
+directory.
+
Index: /trunk/docs/ProjectBook/formulae.tex
===================================================================
--- /trunk/docs/ProjectBook/formulae.tex	(revision 259)
+++ /trunk/docs/ProjectBook/formulae.tex	(revision 260)
@@ -1,2 +1,28 @@
+
+\section{Angular Separation}
+\label{sec-angsep}
+
+To calculate the angular separation of two points on the celestial
+sphere, one needs to use vector geometry. Let the two points, given by
+right ascensions and declinations as $(\alpha_1,\delta_1)$ and
+$(\alpha_2,\delta_2)$ be represented by unit vectors 
+\begin{eqnarray*}
+\vec{u} &= &(\cos\alpha_1\cos\delta_1,
+\sin\alpha_1\cos\delta_1,\sin\delta_1)\\
+\vec{v} &= &(\cos\alpha_2\cos\delta_2,
+\sin\alpha_2\cos\delta_2,\sin\delta_2).
+\end{eqnarray*}
+
+Then, the angle between them, $\theta$, is given by
+\begin{eqnarray*}
+  \cos\theta &= &\vec{u}\cdot\vec{v}\\
+   &= &\cos\alpha_1\cos\delta_1\cos\alpha_2\cos\delta_2 +
+   \sin\alpha_1\cos\delta_1\sin\alpha_2\cos\delta_2 +
+   \sin\delta_1\sin\delta_2 \\
+   &= &\cos(\alpha_1-\alpha_2)*\cos(\delta_1)*\cos(\delta_2) +
+   \sin\delta_1\sin\delta_2 \\ 
+\end{eqnarray*}
+
+
 
 \section{Galactic coordinates}
Index: /trunk/docs/ProjectBook/web.tex
===================================================================
--- /trunk/docs/ProjectBook/web.tex	(revision 259)
+++ /trunk/docs/ProjectBook/web.tex	(revision 260)
@@ -52,5 +52,5 @@
 \item[xmlFunctions.php] Functions for use in constructing a
   VOTable. See Chapter~\ref{ch-vo}.
-\item[security.php] 
+\item[security.php] Controls authentication and authorisation.
 \item[pulsar\_archive\_guide.html] A Users' Guide on how the archive works.
 \item[header.html] The top-of-page matter for the web pages.
@@ -59,13 +59,92 @@
 \end{entry}
 
+\section{How the web interface works}
 
-\section{Possible queries}
+The primary use for the web interface is to enable users to see what
+observations exist according to certain criteria, and to download the
+data when desired. 
 
+\subsection{Possible queries}
 
-\section{How the queries are done}
+The primary query made with this archive is on the position. The user
+provides either a pulsar name (whose position is then extracted from
+the Pulsar
+Catalogue\footnote{\href{http://www.atnf.csiro.au/research/pulsar/psrcat/}%
+{http://www.atnf.csiro.au/research/pulsar/psrcat/}}) or a J2000
+Equatorial position. A search radius is also given.
 
+The query can be refined by giving limitations on the frequency band
+of the observation, the observation type and instrument type, and the
+date range to search in.
 
-\section{Presenting the results}
+The user can also select which table parameters should be returned. A
+default list is provided, but more can be selected via checkboxes.
 
+\subsection{How the queries are done}
 
-\section{Obtaining data}
+The queries are performed by constructing an SQL query using the given
+parameters from the previous section. The angular separation for the
+query is given by the formula in \S\ref{sec-angsep}. Note that we make
+allowance for the size of the primary beam of each observation -- if
+the primary beam of an observation covers part of our search region,
+then we include that observation, even if the central position is
+further than the search radius. Effectively, this means we add the
+primary beam to the search radius (or, rather,  the query becomes ``if
+angular separation $-$ primary beam is less than search radius'').
+
+\subsection{Presenting the results}
+
+The list of results returned by the query are displayed in a
+table. Only 20 results are displayed at any given time -- links to the
+rest are provided (\ie to page 1, 2, 3, ... of results). The columns
+of the table are dictated by the chosen parameters from the query
+page. 
+
+Some observations (timing observations that have been scrunched with
+the pulsar period) are able to have their profile displayed. The
+observations are selected via their check boxes, then the ``Display
+profiles'' option is selected from the drop-down menu. This uses the
+\textsc{pav} command to produce a profile image that is displayed. 
+ 
+\subsection{Obtaining data}
+
+The user can then select data from the table and store it in the
+``cart''. The cart is used so that the user can make a selection of
+particular observations from the list, and only download them. Since
+the results table is spread over many pages, using a cart is an
+efficient way of selecting a number of observations from different
+parts of the results list. 
+
+To download data, the user selects the desired file and selects the
+``Add to Cart'' option to store it. They then go to the cart screen
+(by clicking the link in the top right), and select one of the
+download options. The selected observations are archived in a tar
+file, and downloaded by the user. A link to the tar file is also
+provided. There is a limit of 10GB on the size of downloads.
+
+\section{Implementation Notes}
+
+The implementation of the web archive has been developed by a number
+of authors over a couple of years. It is therefore not as smoothly
+engineered as it could be. A potential future project could be to
+refactor the code so that it is cleaner and less prone to bugs (and
+easier to fix when bugs arise), but this is out of the scope of the
+current project. We describe here some of the details of the
+implementation. 
+
+\subsection{Parameters}
+
+The parameters entered by the user on the index page are stored as
+\verb|_SESSION| parameters, to be passed on to other pages.
+
+\subsection{Results}
+
+A single result is stored as a simple PHP array of values, referenced by
+their column name. The cart items are referenced simply by the number
+in the full list of results \textbf{Note: this may be changed}.
+
+\subsection{SQL queries}
+
+\subsection{Error reporting}
+
+\subsection{Logging of user interactions}
