Details on the work needed to finish the project

Using the summary list of jobs on ToDo, this page goes into more detail about the specific tasks that will need to be done.
Details about work that has been done so far are on WebQuery.
Names of people responsible for each task to be filled in:
MW=Matthew Whiting, DS=David Smith, AK=Arkadi Kosmynin, DM=Dick Manchester, GH=George Hobbs, LK=Lucyna Kedziora-Chudczer

Outstanding tasks (as at 20071031)

Data

• [MW, LK] The tables in the database have duplicate entries for some CPSR2 data, showing the data files in the directory with pulsar name (eg. /pulsar/archive10/CPSR2/J2330-2005) and the symbolic links to those same files in the scrunched directory (eg. /pulsar/archive10/CPSR2/J2330-2005S). It would be good to get rid of these -- this will need to be done either by removing the symbolic links (can this be done in the scrunching scripts?) or tinkering with the find_archives.pl script to ignore the *S directories.
• [MW] Re-do the find_archives.pl script. It will need to be fixed so that it matches the new schema of table "obs", and then re-run. Will have to delete the "obs" table that is in the database currently, so that it puts everything in there the first time round.
  • Note that we need to include the project_id.

Web Archive

• [MW, AK] Need to enable OPAL-style authentication.
• [MW, AK] Also enable checking of project IDs to enable/deny access accordingly.
• [MW] Make the names of the selectable parameters a bit more easy to understand (e.g. "Bandwidth" instead of "bw")
• [MW] Register the conesearch with a VO registry (see notes on WebQuery).
• [MW, DM, GH] Fill in README page, as well as how-to notes on archive and any tool tips.
• Make webpage live.

---

Old tasks, mostly complete but kept for reference

mySQL database

• [MW, LK] Finalise the data conversion, complete with correct filenames:
  • Verify that all data has been converted. 20071011: Re-doing the conversion. 20071015: Done
  • Use pvofixfilenames on $S70V. 20071011: Will be unnecessary.
• [MW] Finalise the schema for the 70cm data - what data fields do we want in the table? 20071011: Done.
• [MW] Use this to finalise the output from pvosummary. 20071011: Done
• [MW] Use pvosummary to produce a .csv file(s). 20071015: Done
• [MW] Ingest .csv file(s) into the database.
  • 20071015: Can do this in temporary fashion. Will do so finally once confirmed that everything necessary is present and working.
  • 20071018: All 70cm data stored in database.
• [MW] Fix up the existing tables, particularly the positions.
  • Need to create new fields that hold the RA & DEC in decimal degrees. Done
  • Add correct values of GL and GB. Done
  • 20071015: This has been done for a temporary table. Currently testing full search capability to see if everything works.
  • 20071018 : Tables made permanent.
• [MW] Fix up bugs with downloading data. Names of files not well stored. 20071031: Done
• [MW, AK] Look at what needs to be done to manage proprietary access for timing data - survey data is presumably public.
  • 20071011: Authentication could be done by connecting to OPAL database.
  • 20071011: Similarly, can use OPAL to check proprietary access to files. Will need to make sure project IDs are present for new files.

Querying functionality

• [MW] Implement the cone-search queries to work on all data, not just the 70cm data. 20071018: Done, but some values come out zero -- need checking. 20071031: Fixed.
• [MW] Allow other searches to be made on survey data:
  • Date-based search : 20071018 - Available.
  • Refine search according to data_type and survey mode: 20071031: Done
• [MW] Register the conesearch web service with a VO registry (see notes on WebQuery).

Web page

• [DS] Improve the look and feel of the main query page: use consistent style sheet with other ATNF archives, eg. ATOA
• [DS] Allow searching to include one or all of "surveys", "timing", specific instruments, ...
  • Pull down menu: Timing, Search,... --> If Timing, a "Pulsar name" box appears.
• [DS, MW] Allow ability to select which table fields to be returned - exists for timing data. Incorporate for S70.
  • 20071018: Field specifications have changed, and "hidden" ones such as data location now do not appear.
• [MW, DS, consult with DM, GH] Have link to a "README" or information page, explaining what the surveys etc are, and even what the different fields that get returned are.
• [DS, MW] Introduction / "How-to" page on how to use the archive.
• [MW, AK] Sort out security and proprietary access.
• Make webpage live!