Details on the work needed to finish the project

This page goes into more detail about the specific tasks remaining to be done. Links to the relevant tickets are also provided.
Names of people responsible for each task to be filled in:
MW=Matthew Whiting, DS=David Smith, VM=Vince McIntyre?, AK=Arkadi Kosmynin, DM=Dick Manchester, GH=George Hobbs, LK=Lucyna Kedziora-Chudczer

Working Query pages

The main query page is at ​http://pulsararchive-int.atnf.csiro.au.

The conesearch web service can be found at ​http://pulsararchive-int.atnf.csiro.au/conesearch.php and can be called by using the RA, DEC and SR parameters, for instance: ​http://pulsararchive-int.atnf.csiro.au/conesearch.php?RA=0.&DEC=-49.&SR=1. should provide an XML file in your browser. Alternatively, use something like wget to download it quickly:

wget -O output_file http://pulsararchive-int.atnf.csiro.au/conesearch.php\?RA=0.\&DEC=-49.\&SR=1.

and view it with something like TOPCAT: ​http://www.starlink.ac.uk/topcat/.

Outstanding tasks (as at 20080204)

Data

• [MW,DM,DS] Re-do the find_archives.pl​ script. It will need to be fixed so that it matches the new schema of table "obs", and then re-run. Will have to delete the "obs" table that is in the database currently, so that it puts everything in there the first time round.
  • This requires vap to be able to produce the correct output -- not the case with latest versions. vap has been fixed.
  • Note that we need to include the project_id. project ID part of vap output.
  • 20080204: script is working. Waiting for removal of "bad" data files from the storage disks, as well as new format for CPSR2 data [GH, DM] before running again. #31

Web Archive

• [MW, AK] Need to enable OPAL-style authentication. #40, #42
  • The code is there, but need to have an Oracle driver installed. Have not been able to check this yet.
    Update 20080703: we have the Oracle driver set up now. But we need access to the appropriate tables in OPAL. 63
  • Also enable anonymous access for non-proprietary data.
• [MW, AK] Also enable checking of project IDs to enable/deny access accordingly. #38
• [MW] Fix minor issues with conesearch, and register the conesearch with a VO registry (see notes on WebQuery). #25, #39, #41
• [DS] Enable logging of queries and downloads, so we know what is happening with the archive. #37
• [DS] Enable query parameters to be modified without having to restart the query page from scratch. #36
• [VM, MW, ?] Make webpage live. #43
• [MW] Write up a "Project Book": a guide to the archive & web pages, for use by future developers. #44

• There was a suggestion from the GW meeting (Dec 13) that the option of downloading scruched data be offered, perhaps with user-selectable parameters. This may be future work beyond the scope of the current project.

---

Old tasks, mostly complete but kept for reference

Data

• [MW, LK] The tables in the database have duplicate entries for some CPSR2 data, showing the data files in the directory with pulsar name (eg. /pulsar/archive10/CPSR2/J2330-2005) and the symbolic links to those same files in the scrunched directory (eg. /pulsar/archive10/CPSR2/J2330-2005S). It would be good to get rid of these -- this will need to be done either by removing the symbolic links (can this be done in the scrunching scripts?) or tinkering with the find_archives.pl script to ignore the *S directories.
  • Update 20071213 -- The files with *S directories have been removed from the updated tables in the database, so they won't appear in the query results. This is a temporary fix that won't apply when the find_archives.pl script is running again, so it needs to be able to avoid them.
  • 20080204 New version of script will avoid them.
• [MW, VM] Enable the www-data account to use psrcat. This will enable cone-searching based on a pulsar name, by extracting the pulsar position from the catalogue.

Web archive

• [MW] Make the names of the selectable parameters a bit more easy to understand (e.g. "Bandwidth" instead of "bw")
• [MW, DM, GH] Fill in README page, as well as how-to notes on archive and any tool tips.

mySQL database

• [MW, LK] Finalise the data conversion, complete with correct filenames:
  • Verify that all data has been converted. 20071011: Re-doing the conversion. 20071015: Done
  • Use pvofixfilenames on $S70V. 20071011: Will be unnecessary.
• [MW] Finalise the schema for the 70cm data - what data fields do we want in the table? 20071011: Done.
• [MW] Use this to finalise the output from pvosummary. 20071011: Done
• [MW] Use pvosummary to produce a .csv file(s). 20071015: Done
• [MW] Ingest .csv file(s) into the database.
  • 20071015: Can do this in temporary fashion. Will do so finally once confirmed that everything necessary is present and working.
  • 20071018: All 70cm data stored in database.
• [MW] Fix up the existing tables, particularly the positions.
  • Need to create new fields that hold the RA & DEC in decimal degrees. Done
  • Add correct values of GL and GB. Done
  • 20071015: This has been done for a temporary table. Currently testing full search capability to see if everything works.
  • 20071018 : Tables made permanent.
• [MW] Fix up bugs with downloading data. Names of files not well stored. 20071031: Done
• [MW, AK] Look at what needs to be done to manage proprietary access for timing data - survey data is presumably public.
  • 20071011: Authentication could be done by connecting to OPAL database.
  • 20071011: Similarly, can use OPAL to check proprietary access to files. Will need to make sure project IDs are present for new files.

Querying functionality

• [MW] Implement the cone-search queries to work on all data, not just the 70cm data. 20071018: Done, but some values come out zero -- need checking. 20071031: Fixed.
• [MW] Allow other searches to be made on survey data:
  • Date-based search : 20071018 - Available.
  • Refine search according to data_type and survey mode: 20071031: Done
• [MW] Register the conesearch web service with a VO registry (see notes on WebQuery).

Web page

• [DS] Improve the look and feel of the main query page: use consistent style sheet with other ATNF archives, eg. ATOA
• [DS] Allow searching to include one or all of "surveys", "timing", specific instruments, ...
  • Pull down menu: Timing, Search,... --> If Timing, a "Pulsar name" box appears.
• [DS, MW] Allow ability to select which table fields to be returned - exists for timing data. Incorporate for S70.
  • 20071018: Field specifications have changed, and "hidden" ones such as data location now do not appear.
• [MW, DS, consult with DM, GH] Have link to a "README" or information page, explaining what the surveys etc are, and even what the different fields that get returned are.
• [DS, MW] Introduction / "How-to" page on how to use the archive.
• [MW, AK] Sort out security and proprietary access.
• Make webpage live!