Index: /trunk/docs/ProjectBook/ProjectBook.tex
===================================================================
--- /trunk/docs/ProjectBook/ProjectBook.tex	(revision 233)
+++ /trunk/docs/ProjectBook/ProjectBook.tex	(revision 234)
@@ -38,4 +38,5 @@
 \newcommand{\pvosummary}{\textsc{pvosummary}\xspace}
 \newcommand{\pvoconesearch}{\textsc{pvoconeseach}\xspace}
+\newcommand{\pvotapesummary}{\textsc{pvotapesummary}\xspace}
 \newcommand{\hi}{H\textsc{i}\xspace}
 \newcommand{\entrylabel}[1]{\mbox{\texttt{#1:}}\hfil}
@@ -113,4 +114,7 @@
 \input{conversionAppendix}
 
+\chapter{Database Table Specification}
+\label{app-DBtable}
+
 \bibliographystyle{plainnat}
 \bibliography{pulsars}
Index: /trunk/docs/ProjectBook/conversionSoftware.tex
===================================================================
--- /trunk/docs/ProjectBook/conversionSoftware.tex	(revision 233)
+++ /trunk/docs/ProjectBook/conversionSoftware.tex	(revision 234)
@@ -447,6 +447,29 @@
  NBits          Number of bits per datum (from the SUBINT table).
 \end{verbatim}
+}
 
 \subsection{\pvoconesearch}
+
+\pvoconesearch is a simple command-line cone-search tool that searches
+around a given (RA,DEC) location, within a given radius, and returns
+the same metadata as pvosummary for all FITS files whose beams overlap
+the search area. The returned files are listed in order of increasing
+separation from the requested (RA,DEC) position. RA and DEC can be in
+hh:mm:ss.s format or decimal degrees, while the search radius is in
+degrees.
+
+\subsection{\pvotapesummary}
+
+\pvotapesummary is a tool to summarise some basic metadata of the
+first block of PKS survey tape files. This is designed to give you an
+overview of the contents and type of data of the tape files. It
+produces a list in a similar format to \pvosummary, but with slightly
+different columns: FILENAME, RAJ, DECJ, OBSFREQ, SCANLEN, MOVE,
+DATE-OBS, where MOVE is the moving flag, indicating whether the
+telescope is scanning or doing a pointed (timing) observation. The
+main purpose is to allow one to discriminate between 70cm Survey files
+and timing observations that are not part of the survey. It works
+either a list of one or more files, or on all files in a given
+directory.
 
 
@@ -511,5 +534,5 @@
 
 
-\section{Details of Implementation of Header analysis}
+\section{Implementation of Header analysis}
 
 We list here some important details about how the header information
@@ -597,57 +620,2 @@
   function in the header class above.
 \end{itemize}
-
-\section{Implementation of FITS file access}
-
-The FITS file will be accessed through a special C++ Class that has
-functions that will control the writing of data to the PHU and to the
-SUBINT binary table. These functions will be front-ends to the
-updateKeyword and writeColumnValue functions that access the
-individual keywords and table columns.
-
-\subsection{Positional information}
-
-WARNING! Contrary to first thought, the existing header parameters
-\textsc{ra\_strt} and \textsc{dec\_strt} are not the appropriate ones
-for the \textsc{stt\_crd1,2} FITS keywords. They are the starting
-positions \emph{as far as the file is concerned}, and so are the same
-even when the telescope has moved to a different position. They are
-not the position of the start of the subints for a given sky location! 
-
-\subsection{Storing 1-bit binary data}
-
-As discussed earlier (Section~\ref{sec-1bit1st}), there are a couple of
-ways to write 1-bit data to a binary table using \cfitsio. However, we
-have run up against several problems. 
-
-First, the \texttt{TFORMn} keyword that goes with the data column
-needs to be `X', but the PSRFITS definition file currently has it as
-`I'. This is not a problem -- in fact, one probably does not want the
-general definition file to be 1-bit by default. It can easily be
-changed to `X' by the \verb|fits_update_key| function. 
-
-However, a word of caution. The implicit size of the row needs to be
-preserved, and so since the `I' datatype is 2-bytes, we need to
-replace `I' by `16X' (since there are 16 bits in 2 bytes). This is
-incorporated into a \texttt{fixPSRFITS} function in the PSRFits class.
-
-Next, however, there is a bug in \cfitsio that incorrectly calculates
-the number of bytes for a given number of bits when the `X' datatype
-is being used. This bug is in \texttt{editcol.c} in the \cfitsio
-distribution and is easily fixed:
-\begin{quote}
-delbyte = ((newveclen + 1) / 8) - ((repeat + 1) / 8);
-\end{quote}
-should be replaced by
-\begin{quote}
-delbyte = ((newveclen + 7) / 8) - ((repeat + 7) / 8);
-\end{quote}
-
-Another bug/feature to be wary of: when writing the first block to a
-FITS file, it seems to want to write a second row with nothing in
-it. This doesn't happen for any subsequent rows/blocks, so I'm not
-really sure what is happening. It is easy to delete that extra row on
-the first time round (for the first block, we also need to redefine
-the data cell's dimensions and size, so this can be done in the same
-way). 
-
Index: /trunk/docs/ProjectBook/database.tex
===================================================================
--- /trunk/docs/ProjectBook/database.tex	(revision 233)
+++ /trunk/docs/ProjectBook/database.tex	(revision 234)
@@ -34,5 +34,5 @@
 in the \texttt{constants.php} file in the \texttt{htdocs}
 directory\footnote{I won't quote it here as this document may be
-  publicly visible at some point.}.The last command lists the
+  publicly visible at some point.}. The last command lists the
 available tables. The first two commands can be combined in one by:
 \begin{quote}
@@ -59,5 +59,8 @@
 \end{entry}
 
-All three tables have been designed to have the same format 
+All three tables have been designed to have the same format. This
+means some columns for some data will not be relevant -- these values
+should be set to NULL. The table format is shown in
+Appendix~\ref{app-DBtable}. 
 
 \subsection{Deprecated tables}
@@ -82,4 +85,32 @@
 
 \section{Updating the data}
+
+\subsection{Updating the surveys tables}
+
+The data in the table ``s70'' needs only to be written once, as there
+will not be any new data of this format created. The way this table is
+populated is as follows
+\begin{enumerate}
+\item Use \texttt{pvoconvert} to convert all ``tape files'' in the
+  \texttt{\$S70TAPES}\footnote{\texttt{/pulsar/archive00/S70\_TAPES/}}
+  directory, putting them in the directory
+  \texttt{\$S70V}\footnote{\texttt{/pulsar/archive09/S70V/}}. Lucyna
+  was responsible for this.
+\item Run \texttt{pvosummary} on the whole directory, writing the
+  output to a csv file. This was done with the file
+  \texttt{make\_s70v\_summary\_table.csv} (in the trunk directory of
+  the subversion repository), which creates the output file
+  \texttt{/pulsar/archive13/S70V/summary\_S70.csv}
+\item Use \texttt{sql/create\_new\_permanent\_tables.sql} to import
+  the data into the database. This simply reads the csv file and
+  writes the data into the correct columns, as defined by the table
+  schema. 
+\end{enumerate}
+
+Should the conversion process need to be changed slightly, and the
+files re-converted, this procedure should be used to re-import the
+data (after first having DROPped the existing table).
+
+\subsection{Updating the observations table}
 
 The data in ``observations'' and ``cals'' are extracted from data
@@ -90,6 +121,4 @@
 metadata into the database; and delete database entries that no longer
 correspond to disk files. Some details of the script are given here.
-
-\subsection{Structure of perl script}
 
 The script uses the \texttt{Find} function to crawl through the given
@@ -156,4 +185,9 @@
 constitute many files, but is included for the sake of completeness.
 
+Note also that some files fail to be read by \texttt{vap}. These files
+will not be included in the integration time calculation, and if all
+files in a directory fail, that directory will not be written to the
+database. This is a problem with \texttt{vap} and \textsc{psrchive}.
+
 \subsubsection{Writing to the database}
 
Index: /trunk/docs/ProjectBook/introduction.tex
===================================================================
--- /trunk/docs/ProjectBook/introduction.tex	(revision 233)
+++ /trunk/docs/ProjectBook/introduction.tex	(revision 234)
@@ -33,12 +33,13 @@
 the project:
 \begin{entry}
-\item[src/] C++ source code for the conversion process, as well as
-  other related functions.
-\item[htdocs/] The web-page source code (php and html), as well as the
-  perl script for ingesting data into the database.
-\item[sql/] SQL functions that create tables in the database, and that
-  were used to test the cone-searching routines.
-\item[docs/] Documentation I developed during the course of the
-  project. Most of this is essentially deprecated by this document.
+\item[\texttt{src/}] C++ source code for the conversion process, as
+  well as other related functions.
+\item[\texttt{htdocs/}] The web-page source code (php and html), as
+  well as the perl script for ingesting data into the database.
+\item[\texttt{sql/}] SQL functions that create tables in the database,
+  and that were used to test the cone-searching routines.
+\item[\texttt{docs/}] Documentation I developed during the course of
+  the project. Most of this is essentially deprecated by this
+  document.
 \end{entry}
 There are also a range of files in the top trunk directory that were
Index: /trunk/docs/ProjectBook/vo.tex
===================================================================
--- /trunk/docs/ProjectBook/vo.tex	(revision 233)
+++ /trunk/docs/ProjectBook/vo.tex	(revision 234)
@@ -6,5 +6,137 @@
 
 
+
+
 \section{Registering the Cone Search}
 
+For the conesearch service to be widely accessible, it should be
+registered with a VO registry. There are a number of such registries,
+which typically have forms that allow registration of new
+services. Two sites that provide links to registries are:
+\begin{enumerate}
+\item NVO at \href{http://www.us-vo.org/pubs/files/PublishHowTo.html}%
+  {http://www.us-vo.org/pubs/files/PublishHowTo.html}, linking to
+  three different NVO registries.
+\item Euro-VO, at
+  \href{http://www.euro-vo.org/pub/tc/registries.html}%
+  {http://www.euro-vo.org/pub/tc/registries.html}, which also links to
+  three different registries.
+\end{enumerate}
 
+It is unclear whether all these registries are linked together, in
+such a way that registering at one provides access through all. This
+is the aim of VO registries, but it is not apparent whether this is
+reality yet. This needs to be sorted out.
+
+The process of registering a service involves providing the registry
+with metadata about the service. The full list of Resource Metadata
+can be found in the IVOA document ``Resource Metadata for the Virtual
+Observatory'', at \href{http://www.ivoa.net/Documents/latest/RM.html}%
+{http://www.ivoa.net/Documents/latest/RM.html}. We summarise the
+relevant metadata used to describe our service here.
+
+\subsubsection{Identity metadata}
+
+\begin{entry}
+\item[Title] ATNF Pulsar Data Archive
+\item[ShortName] APDA
+\item[Identifier] As a URI: ivo://atnf.csiro/apda\\
+As an XML form:
+\begin{verbatim}
+<ResourceID>
+  <AuthorityID>atnf.csiro</AuthorityID> 
+   <ResourceKey>apda</ResourceKey> 
+</ResourceID>
+\end{verbatim}
+\end{entry}
+
+\subsubsection{Curation metadata}
+
+\begin{entry}
+\item[Publisher] Australia Telescope National Facility
+\item[PublisherID] ivo://atnf.csiro
+\item[Creator] Various.
+\item[Contributor] Various.
+\item [Date] 2008-02-22
+\item [Version] Not Applicable
+\item [Contact] 
+\item [Contact.Name]
+\item [Contact.Address] ``PO. Box 76, Epping, NSW, 1710, AUSTRALIA''
+\item [Contact.Email]
+\item [Contact.Telephone]
+\end{entry}
+
+\subsubsection{General content metadata}
+
+\begin{entry}
+\item [Subject] Pulsar observations search timing
+\item [Description] [To be filled in]
+\item [Source] Various.
+\item [ReferenceURL] \texttt{http://pulsararchive-int.atnf.csiro.au/pulsar\_archive\_guide.html}
+\item [Type] Archive, Catalog
+\item [ContentLevel] Research
+\item [Relationship] 
+\item [RelationshipID]
+\end{entry}
+
+\subsubsection{Collection and service content metadata}
+
+\begin{entry}
+\item [Facility] Parkes
+\item [Instrument] FB\_1BIT, DFB1, DFB2, WBC, CPSR2
+\item [Coverage]
+  \begin{entry}
+  \item [Coverage.Spatial] AllSky
+  \item [Coverage.RegionOfRegard] 0.75
+  \item [Coverage.Spectral] Radio 
+  \item [Coverage.Spectral.Bandpass] 
+  \item [Coverage.Spectral.CentralWavelength]
+  \item [Coverage.Spectral.MinimumWavelength] 
+  \item [Coverage.Spectral.MaximumWavelength] 0.7
+  \item [Coverage.Spectral.StartTime] 1991-05-04
+  \item [Coverage.Spectral.StopTime] [empty]
+  \item [Coverage.Depth] [empty]
+  \item [Coverage.ObjectDensity]
+  \item [Coverage.ObjectCount]
+  \item [Coverage.SkyFraction]
+  \end{entry}
+\item [Resolution]
+  \begin{entry}
+  \item [Resolution.Spatial]
+  \item [Resolution.Spectral]
+  \item [Resolution.Temporal]
+  \end{entry}
+\item [UCD]
+\item [Format]
+\end{entry}
+
+\subsubsection{Correspondence of collection and service content metadata with the Space-Time Coordinate schema}
+
+\begin{entry}
+\item 
+\end{entry}
+
+\subsubsection{Data and metadata quality assessment}
+
+\begin{entry}
+\item [DataQuality] B
+\item [ResourceValidationLevel] 
+\item [ResourceValidatedBy]
+\item [Uncertainty.Photometric]
+\item [Uncertainty.Spatial]
+\item [Uncertainty.Spectral]
+\item [Uncertainty.Temporal]
+\end{entry}
+
+\subsubsection{Service metadata}
+
+\begin{entry}
+\item [Service.AccessURL]
+\item [Service.DefinitionURL]
+\item [Service.BaseURL]
+\item [Service.HTTPResultsMIMEType]
+\item [Service.StandardID]
+\item [Service.MaxSearchRadius]
+\item [Service.MaxReturnRecords]
+\item [Service.MaxReturnSize]
+\end{entry}
Index: /trunk/docs/ProjectBook/web.tex
===================================================================
--- /trunk/docs/ProjectBook/web.tex	(revision 233)
+++ /trunk/docs/ProjectBook/web.tex	(revision 234)
@@ -52,4 +52,5 @@
 \item[xmlFunctions.php] Functions for use in constructing a
   VOTable. See Chapter~\ref{ch-vo}.
+\item[security.php] 
 \item[pulsar\_archive\_guide.html] A Users' Guide on how the archive works.
 \item[header.html] The top-of-page matter for the web pages.
