Observing Operations | Reviews | Survey Management

The Charge

For the Review of SDSS
Data Processing and Distribution Operations 

Fermilab 
July 20-21, 2000


          The construction phase of the Sloan Digital Sky Survey is complete and the commissioning phase is essentially complete. Commissioning operations were resumed in February after completing the repair of the secondary mirror. Science quality data has been obtained from the commissioning operations, analyzed, and published. The Survey began the transition from commissioning to routine operations in late April. While this transition is not complete, we have achieved a reasonable approximation of routine operations at the Apache Point Observatory since March. Nevertheless, there is much to be done as the project staff shifts its focus from building and testing equipment and software to making the equipment and software reliable and efficient.

          Our primary goal is to improve the efficiency of our operations at the mountain so that we can approach our original goals in five years of observing. These original goals were to image about 10,000 square degrees of the Northern Galactic Cap; to select from those images a million objects, primarily galaxies and quasars and to obtain spectra of those objects; and to create a catalog of those images and spectra, first for the astronomers from the participating institutions and then for the astronomy community-at-large. These goals have been scaled back by about 25% to make them realistic for a survey of five years duration beginning April 1, 2000.

          Two reviews of the Survey were planned in order to help the SDSS management set its course for the next five years. The first review, which was held at Apache Point from April 25 to 27, assessed the state of the Observing Systems and the Observatory Support. The committee was asked to determine whether these systems were ready to begin the five year of observation period and whether they were capable of producing the quality of data that is needed to achieve our scientific objectives. The committee's report for the first review will be posted on the SDSS web site prior to the second review. The second review will be held at Fermilab on July 20-21, the site of the SDSS data processing and distribution.

          I want this committee to examine whether the pipelines have achieved sufficient functionality to support the start of survey operations, whether the data processing operation can process the data in a timely manner, and whether the distribution operation can effectively distribute the data to the SDSS collaboration. Finally, I want the committee to examine the plan to distribute the data to the astronomy community at large in accordance with the distribution plan developed in cooperation with the National Science Foundation.

          The specific charge for the review of the Data Processing and Distribution review is contained in five sets of questions, which we would like this Committee to answer:

  • Do the pipelines have sufficient functionality to support the start of the five-year survey? Are the pipelines, as you find them on July 20 and 21, sufficiently mature to achieve our scientific goals? Is the level of effort assigned to each pipeline for that pipeline appropriate considering its current maturity?
  • Has the data processing staff developed the software tools to support a five-year survey? Is the size of the data processing staff sufficient to support the data, which will be obtained by a five-year survey?
  • Is the hardware (computers, disk, and tape storage) for the data processing and storage operations sufficient to support the analysis and distribution of the data volume that will be generated by a five-year survey? Are there adequate plans to deal with hardware and system software obsolescence? Is medium for data storage adequate for the present (next ten years)?
  • The plan for distributing the data to the collaboration will be presented at the review. Will the plan and its proposed implementation meet the needs of the collaboration? The SDSS management welcomes suggestions for changes to the plan or the method of implementation with in the framework of the existing resources.
  • The plan for distributing the data to the astronomy community at-large is defined by a plan drawn up by the SDSS, subsequently reviewed and approved by the NSF. Is the proposed implementation plan adequate and can it be done with the existing resources.
  • We hope to distribute a fraction of the data obtained prior to April 11, to the astronomy community early in 2001. We do not plan to do this until the data is fully calibrated. Will this processed data be sufficiently valuable to the astronomy community to justify the effort that the SDSS project will have to expend in distributing the data, considering the current status of the project?

 


Review of SDSS Data Processing & Distribution Operations 
Fermilab
July 20-21, 2000