Sloan Digital Sky Survey

Sloan Digital Sky Survey

Operations Readiness Review

conducted

April 25 to 27, 2000

Reviewer's Report

Prepared by the Review Team

Jim Crocker, Chair

Version 2, May 31, 2000

1. Introduction and overall impressions

This document is the report of impressions and recommendations written by the Review Team for the Operations Readiness Review of the Sloan Digital Sky Survey (SDSS) conducted from April 25 to 27, 2000 at Apache Point Observatory (APO).

The overall impression is very positive. All project personnel have every reason to be very proud of their accomplishments. The project has confirmed the basic concept of the Survey and, even though the actual Survey has not officially started, already has some impressive science achievements to its credit.

The Review Team was extremely impressed with the progress the SDSS Project has made to date. The imaging camera and spectrographs appear to be performing at or near survey levels and are producing impressive scientific results. The facilities for instrument handling and configuration seem professionally engineered and well suited to safely perform the required task. The ability to acquire and process the staggering amount of astronomical data is striking. The high level of professionalism and dedication of the project personnel is widely evident.

The Project already has some impressive achievements to its credit, including not only the highest redshift quasar found to date but a large enough sample of quasars to make an estimate of the quasar luminosity function at redshifts higher than 3.5; the discovery of several brown dwarfs; and the identification of enough halo stars to identify tidal remnants. These early results are widely known within the astronomical community and already demonstrate the types of discoveries that will be enabled by such large, uniformly selected samples of objects. The dataset that will be provided by the complete Sloan survey will have a profound and lasting impact on many fields of astrophysics.

The overall impression from the review is that the engineering effort is well under control with a fairly well defined set of issues to be completed in order to begin full-scale survey operations. There were no glaring mechanical issues that had not been addressed already, in some fashion, by the engineering team. With the exception of some issues to do with image quality, the telescope performance appears to be quite good and instrument performance meets specification. The telescope interlock system is well thought out with only a few remaining issues to do with instrument interlocks. Several efficiency related hardware enhancements were presented and the scope and implementation of these improvements seems reasonable.

2. The Review

2.1 Review Overview

Overview-level documentation for all aspects of the project was provided to the Review Team via www posting previous to the review. More in-depth information was provided in presentations at the review for all aspects of the project except the data processing that is performed at Fermilab and the Project Management structure and process. A separate review of the data processing performed at Fermilab will be conducted at Fermilab before the end of July. Paper copies of the www posted information and the presentations were provided to the Review Team at the review.

The initial review planning, schedule, and logistics for housing, meals, and meetings were performed by the SDSS project. The actual review details and preparation of this report were performed by the Review Team.

In addition to attendance and discussion at the presentations, the Review Team organized themselves into three breakout sessions, roamed the observatory watching operations and interacting with Observers, met several times in executive session, and presented a close-out session at the conclusion of the review.

1) Systems Requirements & Science Commissioning (Comments in section 5, Q&As in section 8.1)

2) Observers and Observatory Operations (Comments in section 6, Q&As in section 8.2)

3) Hardware Systems and Their Maintenance (Comments in section 7, Q&As in section 8.3)

2.2 A note about duplication and negativity

The Review Team has taken the position that this report should serve as a management tool to aid the Project in the very difficult transition from construction/commissioning to production operations. "Production" is emphasized here because this project has set very stringent quality, quantity, and uniformity goals to achieve in a relatively short period of time. The "production" pressure in this project will be a constant source of conflict adding to the already arduous task of managing such an ambitious effort. To that end, we have assumed the fundamental question asked of us is not, What is right? although a list of what is right would be very long, but, What is wrong? and recommendations on approaches to address these issues. To the credit of the project, there follow more than 20 pages of What is wrong? We say "to the credit of the project" because essentially all of the items listed were raised by project personnel, most with proposed solutions.

The lists of items appear longer than they really are because there is duplication of items from different Team Members but from different perspectives. The duplications remain to preserve the different perspectives in the hope they will be useful to produce a more effective solution. The size of the observing staff is a good example. It is clear the observing staff is not large enough to accomplish all the work required. One perspective is the number of people required, another is how to most effectively use the valuable resource represented by the high-level of personnel now on staff, and another is the skill mix that may be suitable after some years of operations.

The questions and answers are included as recorded to preserve the original context. This has resulted in more duplication but is retained to preserve the additional insight provided by the different perspectives of the sources.

While there are many issues to address, as would be expected in such an ambitious endeavor, one must keep in mind we are addressing the last few % of data quality and processes related to observing efficiency, not any fundamental flaw in either concept or execution.

Again, the Review Team is very positively impressed with the accomplishments to date and predicts a high probability of success.

3. About this report

This report is largely a compilation of writings by the Review Team members, questions asked by the Review Team members, and answers to these questions written by Project staff within a few hours. Section and subsection numbering and headings have been added to facilitate organization and referencing, however, the original text is retained to preserve the tone and context of the original writings.

Sections 5, 6, and 7 of this report correspond to the breakout session numbers 1, 2, and 3 above and contain the Review Team member's writings organized by breakout session. Section 8 is a list of the Review Team members.

The original hard copy report contained a section of the questions and answers organized by breakout session and a section about an Email discussion conducted by two reviewers a few days after the review about the performance versus efficiency controversy. It also contained a section with comments by one reviewer, who was also a reviewer for the secondary mirror damage investigation, about the justification for replacing the mount with the new design and a section of one reviewer's list of work items. These sections are not included in the on-line version of the report.

4. Charge to the Review Team

The construction phase of the Sloan Digital Sky Survey is complete, the testing phase is nearly complete, and commissioning operations have been underway for some time. Science quality data has been obtained from the commissioning operations, analyzed, and published. The Survey is now making the transition from commissioning to routine operations. This transition is not complete and there is much to be done as we shift our focus from building and testing equipment to making the equipment reliable and efficient. At the same time, we are building a scientific staff, which will bring to fruition the goals that we established a decade ago. These goals were to image about 10,000 square degrees of the Northern Galactic Cap; to select from those images a million objects, primarily galaxies and quasars; to obtain spectra of those objects; and to create a catalog of those images and spectra, first for the astronomers from the participating institutions and then for the astronomy community-at-large.

We are planning two reviews of the Survey in order to help us set our course for the next five years. The first review, which will be held at Apache Point, from April 25 to 27, will examine whether the Observing Systems and the Observatory Support will be able to sustain the five years of observing and produce the quality of data that is needed to achieve our scientific objectives. The second review will be held at Fermilab, the site of the SDSS data processing and distribution, prior to the end of July. In that review we will ask the reviewers to examine whether the data processing operations can process the data in a timely manner and effectively distribute it to the collaboration and then the astronomy community at large in accordance with our distribution plan developed in cooperation with the National Science Foundation.

The specific charge for the review of the Observing Systems and Observatory support is contained in three sets of questions, which we would like this Review Team to answer:

� "Will the Observing Systems, with the proposed improvements, be ready to support the five-year survey? Will they, as you find them on April 25, be sufficient to achieve our scientific goals?"

A: As of April 25, the data do meet, apart from possible questions of image quality, the quality requirements. The project is far short of being able to produce data in the required quantities. Solving that problem will require a substantial investment in real-time software, and it is not clear whether adequate resources have been identified.

� "Has the observing staff been prepared to support a five-year survey? In particular has the SDSS Management given them the tools and training to carry out the survey? Is the size of the observing staff sufficient to support a five-year survey?"

A: The observing staff appears to have the commitment, competence, and training required, apart possibly from issues of quality assessment of the data. It is not clear they have the tools required to achieve adequate efficiency of operation. The observing staff needs to be augmented by one, but in addition, closer attention needs to be paid to optimizing the range of skill levels, the way shifts are scheduled, and the effective use of the observers.

� "Does the project have sufficient technical personnel at APO and the participating institutions to support the maintenance and continuous improvement of efficiency and reliability? Given the geographic dispersion of people will our management plan work?"

A: We were not given enough management information to assess the answer to this question. Better project management is a prerequisite for evaluating these questions.

5. Systems Requirements and Science Commissioning

5.1 Project Status and Production Tracking

5.1.1 Quality versus quantity conflict management

In any production activity, the desire to improve the product will be in conflict with the desire to produce a larger quantity of data. It will be extremely important to define "good enough" and to resist requirements creep. The expansion of the science utility of the survey beyond its current state is evident and future expansion, while noble in ideal, may place at the risk the ability to achieve the necessary volume within the available resources.

5.1.2 Mechanism for tracking production progress, conic progress charts

Transitioning into the production mode of the survey will be challenging. As we reviewed the presentations, the challenges being experienced in the transition from commissioning to operation were evident.

More detailed and thoughtful analysis will be necessary to identify the areas of efficiency or performance improvements required to achieve the necessary production rates. It must focus attention on the trades that will be required between performance improvements and efficiency improvements. These items are at different ends of the project time horizon and need to be viewed in harmony.

The Review Team concluded that a top-level metric would be helpful in both focusing the goals of the survey as well as tracking the progress. Such a tool would assist in identifying where actions are required to ensure the end result meets the scientific requirements within the allocated budget. The example charts shown in Figures 1 and 2 could be used to track the progress of the survey over the funded five years. The survey objectives are expressed as minimum accomplishments for survey success. The lower level bounds the science projects enabled with the currently achieved system performance over square degrees of sky in Figure 1 and number of redshifts in Figure 2. The upper level is what additional science projects would be enabled with slightly improved performance and a more optimistic assessment of sky coverage or number of spectra. By plotting the progress in these areas, everyone associated with the project can tell at a glance the progress to date.

At this juncture, the team suggests that more attention should be applied to improving the operational efficiency of the survey even if it is at the risk of delaying or eliminating additional performance improvements.

5.2 Project Management

5.2.1 Project Management Overview

Some data of the caliber required for the survey have already been obtained, even at this early stage of operation. However, we also heard about many improvements that could be made in both hardware and software that would improve performance, reliability, efficiency, and safety. What was not presented to us was a systems level overview of relative importance of these various improvement projects when measured against the scientific requirements of the survey, a method for establishing the priorities among the different projects, an integrated schedule for completing them, or a mapping of projects onto available budget and manpower resources. The tools for managing projects were not in evidence during our visit, and so we are not able to determine the extent to which they are being used. However, in a project of this complexity, given the distributed nature of the staff and resources, and the competing and probably conflicting claims for regular observations and continued enhancements to the facility, the use of project management tools is essential.

5.2.2 Real-time software

Management of software development is notoriously more difficult than the management of hardware projects. Not surprisingly, it is our impression that the real-time software for supporting operations is converging more slowly than the non-real-time software and hardware components of the system. Since a very high level of efficiency as measured in terms of time on the sky is required in order to achieve the project goals in a period of five years, resources will have to be identified for continued software development. Quantitative goals for efficiency should be established, and priorities for new software should be determined in order to achieve those goals as early in the project as resources permit. Improved automation in instrument set up, more efficient instrument changes, and real-time quality assessment of data would all enhance throughput.

5.2.3 Personnel safety

Safety must be a major concern for any astronomical observatory. Small groups of people are working in remote locations around massive moving machinery. Equipment worth up to a million dollars or even more must be operated, handled, and repaired, often late at night when people are tired. Severe environmental conditions, including lightning storms or blizzards that prevent access for days at a time, are common. The staff of the SDSS is very aware of these issues and has taken some steps to address them. We particularly commend the staff for scheduling an external review of safety by experts from other organizations. However, we recommend that priority be given to establishing clear and enforced procedures for tagging and lockout of systems that may present dangers to personnel; to providing systems for lightning protection and specifying procedures for disconnecting key equipment and taking other appropriate steps during storms; and to creating an environment that encourages safety consciousness on the part of staff and visitors and facilitates the reporting of safety problems.

5.2.4 Unauthorized network access

We are concerned about external access to the SDSS computer network. Hacking is an increasingly serious problem for all networked systems, and observatories are frequently targets. Appropriate security measures should be put in place that allow reasonably convenient access by external staff but that provide a substantial challenge to unauthorized users. This is an area that will have to be continuously reviewed and updated.

5.2.5 Equipment safety, Instrument-change risk

The current plan of operations calls for changing between imaging and spectroscopy during the night as image quality and cloud cover change. The overhead of this change is substantial and each instrument change involves some degree of risk. Steps should be taken to: 1) speed up the process of changing of instruments; 2) make the process as risk free as possible including possibly limiting the number of instrument changes per night to one and/or restricting how late at night they can occur; and 3) accumulating data that enable simulations to determine whether such instrument changes actually enhance throughput. For example, if seeing is typically strongly variable over hourly intervals during the night, it may be better to wait out an hour of bad seeing rather than switch to spectroscopy on photometric nights. Some primitive forecasting tools may be helpful; if the jet stream is over the site or a front is moving through, for example, good seeing is very unlikely, and the night should probably be dedicated to spectroscopy.

5.3 Management of Science Requirements for the SDSS

5.3.1 Requirements Overview

A detailed evaluation of the science requirements for the SDSS is available in Scientific Requirements and Scientific Commissioning for the SDSS, written and/or edited by M. Strauss. This document outlines the scope of the project, its scientific goals, and provides quality specifications for the data to be cataloged. The Review Team is truly impressed at the degree to which most of these goals have been met or exceeded. For example, the 100mas astrometric accuracy greatly exceeds the 180mas "Drop-Dead" requirement and already meets the goal required for proper motion determinations. Throughput goals for the imager are reached for all but one CCD. The spectrographs essentially meet or exceed all criteria tested thus far, including throughput, resolution, wavelength coverage, and stability. These and other technical feats attest to exceptional talent, effort, and dedication on the part of all involved.

5.3.2 Observing efficiency overview

The one survey criterion that is still far from adequate is overall efficiency. If the project is to achieve its stated imaging and spectroscopic goals, it must achieve a collective change in perspective from one of "building" to one of "producing". This change in perspective also necessitates a change in attitude and management philosophy. When evaluating a proposed modification in hardware or procedure, consideration must be given not only to cost (in dollars) and benefit (in data quality), but also to the project downtime required to implement the modification. Once the 5-year clock begins to run, every hour that the telescope is not on the sky is another portion of a stripe that is not imaged, or a field that will not produce spectra. This reality poses a real dilemma for most astronomers, who are not accustomed to thinking in terms of production. As research scientists, it is our nature to push an idea or technology to its physical limits. Yielding to that tendency at this point in the SDSS must be very critically weighed against the impact of the decision on the eventual science that is obtainable from the data. A case in point is the current desire to improve the secondary mirror support to potentially improve an astrometric accuracy that already meets or exceeds all projected goals. However, the quandary is evident across the project: from image quality to CCD performance to calibration requirements.

5.3.3 Method for evaluating observing efficiency versus performance

In delving into this question, it is the opinion of the Review Team that the SDSS scientific/ management staff has not yet implemented a firm mechanism for evaluating the scientific impact of technical and operational decisions that must be made. For example, the Strauss document fails to quantify the scientific compromise that would result from failing to reach ANY of the technical specifications. It is our recommendation that a small group of scientists from the member institutions with deep interests in the project should be assembled to carefully consider the scientific goals of the project and justify the technical requirements and real coverage which are motivated by that science. This input would be an essential ingredient to the setting of the performance benchmarks by which progress will be judged. The group should also evaluate the scientific impact of proposed upgrades in hardware or changes in operational policy. At a minimum, this process would weigh possible improvements in data quality against the likely reductions in sky coverage. In some cases, tradeoffs of performance vs. instrument safety might also need to be considered. Final decisions on such proposals would, of course, be made by the Project Director on the basis of the recommendation of that group as well as the advice of the Project Scientist and Project Manager. A starting point for this scientific advisory committee might be the existing SOC.

The Observing efficiency versus performance section 5.3.3 highlighted the need for a committee of survey scientists that can trade performance and coverage against scientific goals. That point is relevant to a question that Jim Gunn asked us about near-term tactics. Jim wanted to know if they should shut down for a couple of months and fix everything or continue to have a mix of science verification data gathering and improvements. Our point should be to emphasize that the project needs a process whereby they can answer such questions for themselves. That process requires adequate management information and the benchmarks to assess impact on science. If such questions are handed to the science committee for evaluation, their advice then informs the director's decision, mediated by the Project Scientist. The existence of such an apparatus would engage the scientific partners and allow the kind of trades we've been advocating.

5.3.4 Observing efficiency versus observing staff

The required increase in survey efficiency also drives a need for better organization of effort within the mountain staff. Because the Review Team was not presented with an overall organization chart of personnel, it was not possible to accurately evaluate lines of communication or the adequacy of the staff to address the outstanding technical and operational problems. Clearly, this must improve. Timetables must be generated for the completion of key outstanding software and hardware projects. Manpower must be identified and their time budgeted. Safety procedures must be solidified, followed, and monitored. The existing talent mix must be evaluated in the context of a production operation, and adjustments made where necessary. In short, a "top-down" approach must be adopted, where the primary benchmark is progress toward completion of the survey.

5.3.5 Observers as resource, Lead Observer

An important resource which appears to be largely overlooked is the team of Observers. Perhaps more than any other group at this point, these individuals will be essential in realizing the project�s potential. As Ph.D.-level scientists, they are an expensive team with a great deal of talent to offer toward reaching the overall efficiency goals. Indeed, it is clear to this Review Team that the Observers also take a personal stake in the success of the project. The current organization, however, does not provide an effective means for them to exercise their talents and education or a good path for their ideas to be communicated to the higher-level scientific and managerial staff where they might be implemented. The result is reduced survey productivity and a disillusioned observing staff. It is the very strong opinion of the Review Team that a Head Observer should be hired, among whose duties would be the management and organization of the Observers, the setting of their schedules, their hiring and training, and the setting of their duties. He/she would also monitor the overall performance of the team, fill in for sickness/vacation, and coordinate observing procedures with relevant management personnel. The Head Observer would act not only as a conduit for raising important suggestions and concerns of the Observers, but also as a filter for the many competing influences that they otherwise receive.

6. Observers and Observatory Operations

6.1 Observers 1

The SDSS survey is in a transition mode between commissioning and operations. The Review Team feels that it is important that this transition occur and SDSS moves into regular operations with high efficiency. Two specific questions related to this are:

2) Who makes the list of things to be done and who decides priorities and schedules?

6.1.1 Migration of responsibility to Observers

During the commissioning phase, these issues were best addressed by engineering and project people. However, moving now into an operations mode, it is felt that the people charged with delivering the "product", i.e. the Observers, should now play a more prominent role. This is particularly true if one considers a chart showing % complete survey versus time, which is reflected in the strategy that SDSS needs to adopt over the coming years. The Observers group has direct responsibility flowing from this, to play a leading role.

6.1.2 Impediments to observing efficiency

The Observers group was asked one by one to identify the major impediments to efficient completion of the survey from their experience with nightly operations. The following major issues were mentioned:

6.1.2.1 Spectroscopic Overhead

It is felt that with the current problems and inefficiencies, there was no way the spectroscopic portion of the survey could be completed in any reasonable time scale. The particular problems mentioned were: guide fiber acquisition speed, poor pointing model, stable focus, and extra (possibly unnecessary) calibrations.

6.1.2.2 On-Line Quality Assessment Tools

The lack of these prevents a real-time assessment of the data quality and completeness. This means the exposure times are probably being extended unnecessarily longer to guarantee good quality data.

6.1.2.3 Documentation and troubleshooting experience

A lack of proper documentation and troubleshooting experience has meant that problems often take longer to fix than they should.

6.1.2.4 Building-motion conservatism

The complicated operation of the building roll-off and roll-on requires they be more conservative with observing conditions since the SDSS telescope is more exposed and vulnerable in case of a sudden weather change.

6.1.3 Informed rather than involved

Since the Observers are now playing a leading role in the completion of the SDSS survey, their involvement in the process of problem solving and follow-up needs to change. The issues/problems they have raised which effect the efficiency of their observations are of critical importance with the SDSS moving into an Operations mode. A greater involvement in the Prioritization, follow-up, and deadlines for improvements and problem fixes should be given to the Observers.

This process requires closer consultation and collaboration with Engineering. One comment we heard repeated several times was that the Observers are not internal to this process. They are "informed" rather than "involved". One way of improving this situation might be to provide greater overlap with daytime engineering operations. Both for the evening hand-over and start up, but also in the morning when engineering work is being planned. Some type of regular formal meetings between Observers and Engineering to plan and prioritize should be organized.

6.2 Observers 2

6.2.1 Observers Overview

In the data production phase of the survey, the observers represent a key asset for efficient and successful completion. They were selected to have research experience and interest in the survey data. That level of expertise is highly valuable during the current commissioning stage, during which they can identify, verify, and occasionally implement improvements to operational efficiency, particularly at the software and user interface level. Several personnel issues related to this group should be addressed in order to assure their productivity and longevity with the project.

6.2.2 Observing Staffing Schedule cannot be maintained

In the documents provided for the review and in the presentations and discussions, it was clear that the current staffing schedule is not maintainable over the long run. In order to avoid burnout, two objectives must be realized in the near term, without compromising safety and with minimal impact on operational efficiency. One is to significantly reduce or eliminate the frequency of "shift slipping", in which schedules are changed among full night, early evening, late evening, and days. The observers are currently showing the signs of fatigue that come to any shift workers that have to alternate shifts frequently. The other is to allow more substantial overlap with day workers, with benefits to on-site operations as discussed above. More extensive overlap with consortium scientists during working hours is another significant advantage. Several creative alternatives to the current paired shift assignments are possible, and should be explored.

6.2.3 Lead Observer, off-shift time recommended

The observers themselves suggested two additional observers should be added to the staff. We recommend that the current complement be augmented by one, in order to create the position of Lead Observer. The primary purpose is effective management of this key scientific resource. A major goal of the position is to give the observers a more significant voice in the scientific planning and execution of the survey. This individual will interact with (and preferably be part of) the project management team, assuring that the priorities as articulated by the observers for investments to enhance survey efficiency are considered carefully in the course of project prioritization. This lead person will communicate the priorities and status of the project to the observers, coordinate their scheduling, plan their non-shift observatory projects, guide their skill enhancement and career development, and personally serve on shift, to be both an effective group leader and to create more off-shift time for the observers.

6.2.4 Migration of responsibility to Observers

With a strong site-based cadre of observers, responsibility for tactical decision-making on execution of the survey should devolve to the site. Following the models of queue observing at ESO and WIYN, the imaging and spectroscopic goals for each full lunation would be developed by the science planning team, then transmitted to APO, where the observers would make the nightly choices about the details of program execution. The observers still desire frequent communication with the off-site survey team members. This level of delegation of responsibility is appropriate for the skill level of the observers and the nature of the "production" phase of the Survey.

6.2.5 Future Observer Skill-mix

As the survey progresses and data acquisition becomes a more nearly routine activity, it will be worthwhile to consider a change of skill mix if the necessity arises to fill a vacant observer position. Ph.D.-level research experience is a worthwhile prerequisite for data quality assurance and assessment of short- and long-term impact on science of instrument and telescope anomalies. Such a background will be less critical for telescope, instrument, and facility operations when these functions are mature. At that time, Survey management should consider replacement with a more traditional "telescope operator" profile. For the remaining, relatively short duration of the survey, that person could be assigned nearly exclusively to operating shifts, further freeing the observers to concentrate more on data quality, pipeline applications, and the option of longer blocks of non-work hours to make progress on science projects.

6.3 Observers 3, Observatory Operations, Engineering - Software

6.3.1 Observatory Operations Overview

The operational software appears to be in good shape overall to support initial observing. There is clearly additional functionality needed and many rough edges to be addressed to provide software to support efficient operations that are maintainable through five years of production data taking.

6.3.2 Improved Observer Interfaces, dedicated status displays

Many different software sub-systems have been integrated to support operations. We agree with the assessment that a layer of software integration, including user friendly interfaces, and easy to organize and use status displays, is needed to support 5 years of operations. We encourage the survey to find the resources to implement a first version of this over the next few months. Since these are the panels that are in daily use by the Observing staff, the requirements should be identified through watching their use of the software during observations and discussion of their needs and ideas towards more efficient operation.

We recommend the addition or allocation of dedicated status screens in the control room to show existing and enhanced status displays.

6.3.3 Increased functionality of Observer's Programs (IOP, SOP, MOP), companion observer

There is a need for increased functionality and stability of the xOP programs and for real-time quality analysis of the data. For increased operational efficiency, both in the short and long term, the Observers must have continued interaction and the opportunity to work with the developers responsible for this software. The quality analysis programs should be configured to ensure they do not perturb the main observing process. If the speed of the networks allow, perhaps there is an opportunity for a "companion observer" at Fermilab to help with QA during observing.

6.3.4 Plan to transition the focus from development to production operations

We were not presented with a prioritized list of software tasks, assignments, and schedule. We recommend that one be posted. We were not presented with any plan for the transition from development of the software to an operations focus and any accompanying transfer of knowledge and responsibility. We recommend that such a plan be developed to address operational use of the software for the next five years.

6.3.5 Configuration management and documentation

We endorse a long-term commitment to configuration management and to provide operational and reference documentation. We encourage including documentation in configuration management. We recommend including regular inspections (code and documentation) in the configuration management plan.

6.3.6 Support for MCP and real-time programming

We agree with the concern and recommendation to identify someone with long-term responsibility for the maintenance and support of the Motion Control Processor. We recommend the "intermittent axis-motion problems" be fixed. Attention should continue to be paid to this software and if problems persist effort identified to do the necessary rewrite of the PVT code. Concern that the MCP code is not interpolating all TCC commands correctly should be understood. We identified a lack of real-time programming expertise at APO and recommend that some be acquired to provide backup support for the experts at Fermilab.

6.3.7 One-person software risk

We encourage the software project management to maintain awareness of those parts of the software that have only one person with any knowledge of the internals or one person who has contributed to the code.

6.3.8 Plan and schedule for upgrades and maintenance

A schedule for deployment of the semi-automated instrument-change system should be posted and the necessary resources and priorities assigned.

Plans for incremental upgrade of the data acquisition system to address those parts becoming unsupportable seem appropriate. Sufficient time should be planned for the integration and testing of changes.

We recommend construction and posting of a calendar of all the maintenance tasks with their time and resource budget.

7. Hardware Systems and Their Maintenance

7.1 Hardware Systems

The overall goal should be to, as expeditiously as possible, bring the instrumentation to a level of performance consistent with a finalized set of scientific objectives. An aggressive, yet plausible, timeline should be established for this objective so as to guarantee the timely completion of the survey.

Listed below is a summary of key improvements and enhancements that, in some way, lead to improved safety, performance, or efficiency. This list is not intended to be all-inclusive but rather to emphasize the issues the reviewers felt were most important.

7.2 Equipment-Safety Related Mechanical Issues

Catastrophic risk to the instrumentation due to lightning should be mitigated further via implementation of proper telescope grounding and the installation of some sort of formal early warning system. As it stands now, the instrumentation is particularly susceptible when the enclosure is rolled off.

Safe handling of the spectroscopic corrector should be ensured through completion of the dedicated handling cart and storage area.

The completion of the interlock system for safe engagement of the imager optical benches needs to be ironed out. The understanding is that engagement must be performed with the telescope near the zenith to prevent misalignment, and possible damage, of the kinematic mounts.

There is an issue with potential outgassing of the camera getter in the event of a camera warm-up. The engineering team has planned to remove the getter during the summer shutdown. We agree with this line of thinking.

A wind screen shock absorber is needed to prevent damage to the secondary support system in the event of a collision.

The frequency of recoating the primary and secondary should be minimized to mitigate the risk to the optics.

7.3 Image-Quality Related Mechanical Issues

There is uncertainty regarding the source and severity of the image degradation that is observed in the telescope. Before the relatively risky operation is undertaken to ship the secondary mirror and corrector to the Lick Observatory for characterization, the project should gather enough image data to confidently characterize the problem and investigate other possible sources of the degradation. In particular, thermometry should be implemented as soon as possible to investigate in particular any effect of the following on local seeing:

4) temperature of the wind baffle relative to the air, and the possibility of a temperature gradient across the baffle.

This should be done before the removal of the secondary mirror and common corrector for characterization. The Max Plank Institute has agreed to lend the SDSS a thermal imaging camera for thermal studies.

7.4 Observing-Efficiency Related Mechanical Issues

It is not clear whether a manual or automated instrument-change system is most beneficial. Whichever method is implemented, the method must:

3) be implemented as a routine, standard procedure with the users thoroughly trained in that procedure.

The throughput of the Photometric Telescope must be improved to keep pace with the rest of the observing system.

The mechanical spare parts list should be established and the critical spares kept on the site.

Spectroscopic corrector safety latches should be implemented to minimize the inconvenience of using the special bolts.