MAINTENANCE AT THE WORLD’S LARGEST CYCLOTRON
    
     Fred W. Bach, Operations Coordinator,
      TRIUMF, 4004 Wesbrook Mall, Vancouver, British Columbia, Canada
     Chad Hogan, University of Victoria
     Joseph Su, University of British Columbia

 Extended Abstract for WAO'98 Poster

   The TRIUMF 500-MeV cyclotron located in Vancouver, Canada is the
   world's largest cyclotron. As far as the two maintenance paradigms
   go (i.e. scheduled maintenance and 'run it until it quits'), TRIUMF 
   has the best of both worlds.  Although regular maintenance periods
   are scheduled, they may be canceled or severely shortened if the
   amount of required maintenance is small.  Conversely, an 
   emergency-maintenance period can be scheduled if a serious problem
   develops. Regularly scheduled one-day maintenance periods occur
   weekly, (usually Wednesdays), two-day maintenance periods 
   approximately every three weeks (usually Tuesday-Wednesday), and
   two-month shutdowns about twice annually (usually January-February
   and September-October).  Maintenance items are tracked using a
   number of systematic forms, all developed in-house. Most of the
   data is also stored electronically; the Controls ORACLE database
   and ASCII text files managed by VMS DCL command procedures are
   used to keep maintenance records.  Extensive use of e-mail has been
   effective in reducing the use of paper, as well as in improving
   response time and facilitating record searches.  The Operations
   Coordinator monitors, manages, and maintains these systems, with
   occasional assistance from the Head of Operations, the shift
   supervisors and co-op students.  Operations provides downtime 
   records to group leaders for the production of reports and to plan
   equipment upgrades.

   The general maintenance process begins when a problem is discovered
   during normal operation of the cyclotron.  Usually an operator, or
   sometimes an experimenter or a support-group member will file a 
   Fault Report on paper.  This report must identify the device(s)
   that caused the fault, and include a detailed description of the
   fault and its effect on beam performance and any immediate actions
   taken to resolve the problem.  Each day, all groups send a
   representative to the main control room to review their active
   faults. In conjunction with Operations, the group leaders decide
   what repairs must be carried out.  When the problem is solved
   (usually after the next maintenance period) the group leader or 
   his designate returns the fault report with comments about the
   repair and other operating instructions.  If the Operations
   Coordinator finds that the solution is satisfactory, then the
   report is placed in a book to be read by all operators.  However,
   if the fault requires further (likely extended) maintenance, it is 
   filed as “deferred”.  When fault reports are no longer active (or
   deferred) and have been read by all operators, they are filed in
   the Operations manuals or yearbooks for future reference.  A brief
   summary of each fault report is recorded in ascii text form, and
   in an ORACLE database.

   Occasionally, urgent beam requirements make it necessary to
   continue operation of the cyclotron even if device interlocks have
   shut the beam off.  The decision to override a device interlock
   must be made jointly by the shift supervisor on duty and the
   safety officer, who must ensure that the defeat will create no
   danger to personnel.  A defeat form is filled out to record the
   incident and to ensure that proper lockout procedures are
   followed.  While a device is defeated, red tags are attached to it
   and are also posted at all locations from which the device can be
   controlled.  This notifies all personnel that a defeat has been
   implemented.  As well, a red-ink message is placed in the control
   room logbook, and a fault report must be filed about each defeat.
   After repairs, and before removing a defeat and resuming normal
   operation, the Operations shift supervisor must inspect all related
   safety devices and make sure that they are functioning normally. 
   The removal of each defeat is also recorded in the control room 
   logbook in red ink, and the defeat form is filed as “inactive”.

   In order to schedule both immediate and future maintenance periods,
   once a week every active fault report is summarized and attached
   to a maintenance form.  All group leaders receive this summary, 
   usually on Sunday evenings.  Group leaders fill out the work they
   intend to do that week and return the maintenance forms to
   Operations by Monday afternoon.  From these, the Operations
   Coordinator makes the Maintenance Activities List, listing the
   maintenance items and maintenance times for the week.  This list
   is discussed among Operations personnel, and distributed to all
   group leaders and posted on the Cycinfo system and on the Control
   Room door.  Weekly Cyclotron Performance and Operations meetings
   are held to analyze any unexpected problems with beam performance
   and to plan maintenance to solve them, while maximizing beam for
   experiments.

   Whereas Operations provides notification of device faults, each
   group leader is responsible for maintaining his own group’s
   equipment.  If maintenance work is required, group leaders must
   obtain a work permit to authorize the work.  A work permit must
   name the person(s) allowed to perform maintenance, and must also
   indicate how long it will take, and what, if any, safety lockout
   procedures need to be followed.  All safety information pertaining
   to both radiation and industrial hazards (e.g.  radiation-survey
   maps and dose acquired during repairs) is also recorded on the work
   permit, and the worker must comply with these requirements and any
   others as specified by the area supervisors,  Operations and the
   TRIUMF Safety group.  After the work is completed, these forms are
   returned to Operations and reviewed by the shift supervisors. 
   Completed work permits are filed in each group'’s section in the
   Operations manuals.  Work permits may also be filed as “deferred”
   if further action is required.  Repairs that can be completed in a
   short time are usually carried out on weekly maintenance days,
   while larger jobs are postponed until the semi-annual shutdown
   periods.  After maintenance or shutdown periods, the shift
   supervisor reviews the maintenance done to ensure safety and
   completion of necessary tasks before cyclotron start-up.

   The semi-annual shutdown period is an opportunity for maintenance
   that cannot be carried out during the regular operating periods. 
   This includes procedures that would be impossible to do because of 
   scheduling or hazards (e.g. radiation).  When a fault report, work
   permit, defeat form, or general maintenance request is returned
   deferred, these items are then entered into the scheduling process
   for the next upcoming major shutdown.  Group leaders may enter
   items into the schedule as necessary.  The shutdown coordinator
   produces a daily shutdown schedule well before each shutdown, but 
   revisions will occur during the shutdown itself.  Each new revision
   to the shutdown schedule is discussed at weekly shutdown meetings
   and is distributed to the different groups (e.g. Operations, 
   Vacuum, RF, etc.) via e-mail, the Cycinfo system, and general
   postings.  Near the end of each shutdown period, checklists
   maintained by the Operations coordinator are consulted by the group 
   leaders to ensure that all necessary procedures have been carried
   out.  This includes ensuring that all deferred forms have been
   examined and processed, and that all recently created work permits,
   fault reports and defeats have been dealt with properly. Finally,
   anything that requires future consideration (i.e. during the next
   shutdown period) is reported to the group leaders and Operations,
   and noted for the next shutdown schedule.