= Details on the work needed to finish the project =

This page goes into more detail about the specific tasks remaining to be done. Links to the relevant tickets are also provided. [[BR]]
Names of people responsible for each task to be filled in:[[BR]]
MW=Matthew Whiting, DS=David Smith, VM=Vince McIntyre, AK=Arkadi Kosmynin, DM=Dick Manchester, GH=George Hobbs, LK=Lucyna Kedziora-Chudczer

== Working Query pages ==

The main query page is at http://pulsararchive-int.atnf.csiro.au.

The conesearch web service can be found at http://pulsararchive-int.atnf.csiro.au/conesearch.php and can be called by using the RA, DEC and SR parameters, for instance: http://pulsararchive-int.atnf.csiro.au/conesearch.php?RA=0.&DEC=-49.&SR=1. should provide an XML file in your browser. Alternatively, use something like `wget` to download it quickly:
{{{
wget -O output_file http://pulsararchive-int.atnf.csiro.au/conesearch.php\?RA=0.\&DEC=-49.\&SR=1.
}}}
and view it with something like TOPCAT: http://www.starlink.ac.uk/topcat/.


== Outstanding tasks (as at 20080204) ==

'''Data'''
 * [MW,DM,DS] Re-do the [source:trunk/htdocs/perl/find_archives.pl find_archives.pl] script. It will need to be fixed so that it matches the new schema of table "obs", and then re-run. Will have to delete the "obs" table that is in the database currently, so that it puts everything in there the first time round. 
   * ~~ This requires `vap` to be able to produce the correct output -- not the case with latest versions.~~ `vap` has been fixed.
   * ~~ Note that we need to include the project_id.~~ project ID part of `vap` output.
   * '''20080204''': script is working. Waiting for removal of "bad" data files from the storage disks, as well as new format for CPSR2 data [GH, DM] before running again. #31

'''Web Archive'''
 * [MW, AK] Need to enable OPAL-style authentication. #40, #42
   * The code is there, ~~but need to have an Oracle driver installed~~. Have not been able to check this yet.[[BR]]
     '''Update 20080703:''' we have the Oracle driver set up now. But we need access to the appropriate tables in OPAL. [ticket:63]
   * Also enable anonymous access for non-proprietary data.
 * [MW, AK] Also enable checking of project IDs to enable/deny access accordingly. #38
 * [MW] ~~Fix minor issues with conesearch,~~ and register the conesearch with a VO registry (see notes on WebQuery). #25, #39, #41
 * [DS] Enable logging of queries and downloads, so we know what is happening with the archive. #37
 * [DS] Enable query parameters to be modified without having to restart the query page from scratch. #36
 * [VM, MW, ?] Make webpage live. #43
 * [MW] Write up a "Project Book": a guide to the archive & web pages, for use by future developers. #44

 * There was a suggestion from the GW meeting (Dec 13) that the option of downloading scruched data be offered, perhaps with user-selectable parameters. This may be future work beyond the scope of the current project.
----

=== Old tasks, mostly complete but kept for reference ===

'''Data'''
 * ~~ [MW, LK] The tables in the database have duplicate entries for some CPSR2 data, showing the data files in the directory with pulsar name (eg. /pulsar/archive10/CPSR2/J2330-2005) and the symbolic links to those same files in the scrunched directory (eg. /pulsar/archive10/CPSR2/J2330-2005S). It would be good to get rid of these -- this will need to be done either by removing the symbolic links (can this be done in the scrunching scripts?) or tinkering with the find_archives.pl script to ignore the *S directories. ~~
   * ~~ Update 20071213 -- The files with *S directories have been removed from the updated tables in the database, so they won't appear in the query results. This is a temporary fix that won't apply when the find_archives.pl script is running again, so it needs to be able to avoid them. ~~
   * '''20080204''' New version of script will avoid them.
 * ~~ [MW, VM] Enable the www-data account to use `psrcat`. This will enable cone-searching based on a pulsar name, by extracting the pulsar position from the catalogue. ~~

'''Web archive'''
 * ~~ [MW] Make the names of the selectable parameters a bit more easy to understand (e.g. "Bandwidth" instead of "bw") ~~
 * ~~ [MW, DM, GH] Fill in README page, as well as how-to notes on archive and any tool tips. ~~


'''mySQL database'''
 * [MW, LK] Finalise the data conversion, complete with correct filenames:
   * ~~ Verify that all data has been converted. ~~ ~~ 20071011: Re-doing the conversion.~~ 20071015: Done
   * ~~ Use {{{pvofixfilenames}}} on $S70V. ~~ 20071011: Will be unnecessary.
 * [MW] ~~ Finalise the schema for the 70cm data - what data fields do we want in the table? ~~ 20071011: Done.
 * [MW] ~~ Use this to finalise the output from {{{pvosummary}}}. ~~ 20071011: Done
 * [MW] ~~ Use {{{pvosummary}}} to produce a .csv file(s).~~ 20071015: Done
 * [MW] ~~ Ingest .csv file(s) into the database. ~~
   * ~~ 20071015: Can do this in temporary fashion. Will do so finally once confirmed that everything necessary is present and working. ~~
   * 20071018: All 70cm data stored in database.
 * [MW] Fix up the existing tables, particularly the positions. 
   * ~~ Need to create new fields that hold the RA & DEC in decimal degrees. ~~ Done
   * ~~ Add correct values of GL and GB. ~~ Done
   * ~~ 20071015: This has been done for a temporary table. Currently testing full search capability to see if everything works. ~~
   * 20071018 : Tables made permanent.
 * [MW] ~~ Fix up bugs with downloading data. Names of files not well stored. ~~ 20071031: Done
 * [MW, AK] Look at what needs to be done to manage proprietary access for timing data - survey data is presumably public.
   * 20071011: Authentication could be done by connecting to OPAL database.
   * 20071011: Similarly, can use OPAL to check proprietary access to files. Will need to make sure project IDs are present for new files.

'''Querying functionality'''
 * [MW] ~~ Implement the cone-search queries to work on all data, not just the 70cm data.  20071018: Done, but some values come out zero -- need checking. ~~ 20071031: Fixed.
 * [MW] Allow other searches to be made on survey data: 
   * Date-based search : 20071018 - Available.
   * Refine search according to data_type and survey mode: 20071031: Done
 * [MW] Register the conesearch web service with a VO registry (see notes on WebQuery).

'''Web page'''
 * [DS] Improve the look and feel of the main query page: use consistent style sheet with other ATNF archives, eg. ATOA
 * [DS] Allow searching to include one or all of "surveys", "timing", specific instruments, ...
   * Pull down menu: Timing, Search,... --> If Timing, a "Pulsar name" box appears.
 * [DS, MW] Allow ability to select which table fields to be returned - exists for timing data. Incorporate for S70.
   * 20071018: Field specifications have changed, and "hidden" ones such as data location now do not appear.
 * [MW, DS, consult with DM, GH] Have link to a "README" or information page, explaining what the surveys etc are, and even what the different fields that get returned are.
 * [DS, MW] Introduction / "How-to" page on how to use the archive.
 * [MW, AK] Sort out security and proprietary access.
 * Make webpage live!