Shane AO Startup Procedures

Shortcuts to various procedures:
Cabling/Power | Ethernet | Servers | Power GUI | Motor GUI | SAO UI | Peeko | TT Peeko | SAO Initialization | ShARCS Setup |ShARCS GUIs | Laser Shutdown Monitor | Startup Checklist

Cabling and powering up the instrument.

Assuming that everything was properly cabled up in the lab and that it has been mounted properly to the Shane 3-m Telescope, there are only five cables that need to be connected for the Shane AO (SAO) instrument:

  1. Ethernet Cable - This cable must be connected to the gigabit network.
  2. Power Cable 1 - Plugged into the UPS protected outlet on the TUB labeled AO Only. SAO Pulizzi B and C are powered from this outlet.
  3. Power Cable 2 - Plugged into the non-UPS protected PG&E power outlet on the TUB. SAO Pulizzi A is powered from this outlet.
  4. Two regular BNC cables from the PI uplink tip/tilt mirror power supply to the ShaneAO instrument. X-axis is a black BNC cable, Y-axis is a green BNC cable.

When the Pulizzis are powered on they are set to turn on the following items immediately:

Pulizzi A: Pulizzi B: Pulizzi C:

Checking Ethernet Communications

You should be able to ping the following computers, Lantronixes, etc. that are on the SAO instrument:

The following are not used in regular operations, but are in the AO bench:

The following computers are not on the instrument but are used for SAO and ShARCS operation:

Starting Server Software

There are lots of pieces of software that do many different things. The servers are generally already running, most starting at boot time. For the most part you don't need to start most servers individually, but can start them all by typing the following command (if a service is already running, the script will do nothing for that item):

/usr/local/lick/bin/ao start

on hosts shark, shadow, and covert. The services on real (saocon) and musher (woofer) should start at boot time or when starting the realtime code. More details about starting individual components are below.

Power Server

The power dispatcher runs on host shadow and can be started with the command:

/usr/local/lick/bin/ao start saopower

Motor Servers

The motor dispatchers run on host shadow and can be started with the command:

/usr/local/lick/bin/ao start saomot

If you need to only start a particular motor dispatcher, the command is:

/usr/local/lick/bin/ao start saomot1

or saomot2, saomot3, saomot4 (or rio), saomot5, or saomot7.

Motors controlled from each dispatcher (in Galil channel order ABCDEFGH):

Calibration samp state is monitored by saomate and runs on shadow. It is started with the command:

/usr/local/lick/bin/ao start saomate

ShARCS World Coordinate System

The ShARCS World Coordinate System server, shawcs, also runs on shadow and can be started with the command:

/usr/local/lick/bin/ao start shawcs

Instrument Monitoring Servers

There are two monitoring programs, saomon1 and saomon2. saomon1 is the Primary Monitor (everything except the heartbeat monitor) and saomon2 is the Heartbeat monitor for the MEMS DM. They are started on host shadow with the command:

/usr/local/lick/bin/ao start saomon

or saomon1 or saomon2 if you wish to start them individually.

Keyword History Server

The AO keywords are written to accession so a history of state and changes can be saved. The status of the aohistory service can be checked by logging onto accession and executing the following command:

/usr/local/lick/etc/init.d/ao_keygrabber status

To start the service type:
/usr/local/lick/etc/init.d/ao_keygrabber start

ShARCS Services

There are a number of services that help with the operation of ShARCS. On host: covert, the relevant programs are sharcspump, ao_keyheader, and sharcs. All are started on host covert with the command:

/usr/local/lick/bin/ao start

or individually by typing:

/usr/local/lick/bin/ao start sharcspump

or ao_keyheader or sharcs.

sharcspump controls the ion pump for the ShARCS dewar. ao_keyheader facilitates adding information to ShARCS FITS headers, and sharcs governs the communications between stealthie and covert for ShARCS detector operation.

Additional ShARCS services run on host shark. mako, which governs communications between ShARCS and SAO. mako is started with the command:

/usr/local/lick/bin/ao start mako

Temperature monitoring, sharcstherm, runs on shanevnc and be started with the following command:

/usr/local/lick/bin/ao start sharcstherm

Starting AO Software User Interfaces

Power GUI saopower_ui

The power control GUI is started with the command

/usr/local/lick/bin/saopower_ui

After starting the GUI, make sure that the items turned on by default when the Pulizzi are powered up are on and that the others are off.

Motor Control GUI saomot_gui

To start the motor control GUI, type

/usr/local/lick/bin/saomot_gui

Once the GUI is up, the stages will all need to be calibrated. Stages can be calibrated one at a time by clicking the Calibrate button in the motor's detail panel or all at once using the Actions - Calibrate - All Stages menu item. Calibrating all at one time is not recommended at this time because the TT Cam X, TT Cam Y, Sci filter, and Waveplate motors are currently not installed.

Problems can occur with homing and there are known ways to deal with them as follows:

Once stages are calibrated, they should be moved to the nominal, on-axis positions shown in Figure 1.

GUI showing nominal on-axis positions
Figure 1: Nominal on-axis motor positions.

Also be sure to set the Orientation to "OnTelescope" when mounted on the Shane or "Stow(Z)" when in the lab.

Starting Shane AO UIs

At this time there are few UIs for SAO operation, so many things need to be started and run from command line interfaces. Short list of items is:

Message Log

Before starting the shaneao realtime control code, start monitoring real's message log so you can monitor the state of the system. To view the message log, log into real as user and type the following:

tail -f /var/log/messages

Pay close attention to this log as important errors about DPIO2 will be shown there.

Realtime Code shaneao

Now the realtime SAO control code can be started on real from the user account with the following command:

/usr/local/lick/bin/shaneao

Once the program starts you will see the following information if all is normal. 

DPIO2:  DPIO2 Device Driver - Release: 2.4.13
        Copyright (C) 2005 - VMETRO, Inc.  All rights reserved.
        Creation Date: Apr 21 2014, Time: 16:07:22
DPIO2:  Scan PCI for DPIO2 modules. Found 1
DPIO2:  Device #1 (bus 55, device 9) is a DPIO2-LB with FPGA code BO514

The software should automatically check to make sure things are normal and can start. If not or if you get a vastly different message, either there is already a shaneao running (only one can run at a time) or there may be a problem with communicating with the DPIO2 MEMS driver (in which case a reboot of real might be necessary). In either case you'll want to exit shaneao at the prompt.

Another error that shows up in the Message Log is 

Aug  1 11:05:59 real kernel: last message repeated 14476 times
Aug  1 11:07:00 real kernel: last message repeated 14475 times
Aug  1 11:08:01 real kernel: last message repeated 14476 times

which indicates a problem and will result in poor performance of the SAO system. The recommended solution is to exit shaneao at the prompt and restart it.

SAO realtime control GUI - saocon_gui

The SAO realtime loop control GUI, saocon_gui, runs on shanevnc from the user account. To start it type

/usr/local/lick/bin/saocon_gui

WFS Display peeko

From the user account on karnak, the WFS and DM actuator displays are started with the following command:

/home/user/peeko_start

TT Camera Display ttpeeko

From the user account on karnak, the TT Camera display is started with the following command:

/home/user/bin/tpeeko_start

Field Steer software

From user@shanevnc, the field steering GUI is started with the following command:

/usr/local/lick/bin/fieldsteer.tcl

Initializing ShaneAO

Now that all the software is started, it is ready for use. Issue the following command to initialize shaneao and move the lenslet array:

init(16) for the 16x lenslet, or

init(8) for the 8x lenslet.

If the above init command fails (which is extremely unlikely), you need to do the following:

  1. Start the WFS camera:
    camstart()
  2. Start the TT camera:
    ttcamstart()
  3. Read frame rate of WFS camera:
    rate()
  4. Read frame rate of TT camera:
    ttrate()
  5. Examine the state to see what might need to be changed:
    state()

After initializing, a common issue is the WFS Camera will not initialize, and you'll see a pdv error every second in the shaneao window and the test pattern will be shown in the Peeko display. First try pushing the "WFS Camera Reset" button. If that fails to initialize communication, power cycle the WFS camera (leaving it off ~20 seconds). It sometimes requires more than one power cycle and "WFS Camera Reset" button press to resolve the issue.

At this point the realtime code is ready to use and the woofer and tweeter DMs can be turned on. The woofer is turned on via the saopower_ui GUI outlet B5. Before turning on the tweeter, check the message log on real to make sure there are no SW-IOMMU error messages. The tweeter MEMS mirror is turned on via the saopower_ui outlet B6.

At this point you can turn on the woofer and MEMS in the shaneao software with the following commands:

  1. Turn on the woofer communications:
    woof('on')
  2. Turn on the tweeter (MEMS) communications:
    mems('on')

Starting ShARCS

Before starting the ShARCS detector control software, you'll want to make sure stealthie has crossmounted covert's data disk by doing the following procedure:

  1. rdesktop stealthie and log in as user
  2. Click on the "My Computer" icon.
  3. Click on "public on 'Samba... Network Drive" and make sure it lists the data directories on covert

Information on starting ShARCS detector control software on stealthie is available in: HxRG_CustomManual_WeissMod_v3.pdf. Note: Replace HxRG2_3 with Test in step 10 of these directions to select the Test.soln file.

Start temperature and pressure monitoring through saomon with the following commands if not already set:

modify -s saomon SH_BASICDIS=0

modify -s saomon SH_COMBODIS=0

Starting ShARCS GUIs

There are a few user interfaces for ShARCS:

Detector Controller sharcs_fe

The main GUI for taking exposures, sharcs_fe is started on covert with the following command:

sharcs_gui

or if that fails, one can type java -jar /usr/local/lick/java/jar/sharcs_fe1.jar

Once the GUI is started, open Windows - Setup and do the following:

  1. Make sure AsicCfg is C:\startup\SHARCSColdControlConfig.xml
  2. Make sure UnixRootDir is /data
  3. If Connected is NO or if the upper right corner of sharcs_fe is red and says "Lost Server", press the Connect button.
  4. Initialize system by pressing the Init button. Initing status will change to Yes during the initialization and SideCarInit should say Success! when done (and Initing status will change to No).

Note, there is no harm in pushing the Connect button if you are already connected, but if you do, you must then press the Init button.

Image Display sharcsdisplay

The image display software is started on covert with the command

/usr/local/lick/bin/sharcsdisplay

Filter Wheel Controller saowheels_gui

The wheel and aperture wheel motor controller is started on covert or shanevnc with the command

/usr/local/lick/bin/saowheels_gui

Once the GUI is up, calibrate each of the three wheels by clicking Calibrate on its detail panel. Now the filters can be moved to the desired positions.

Script Tool scriptproc.tcl

The script running GUI is started on covert with the command

/usr/local/lick/bin/scriptproc.tcl

Sound GUI eventsounds

The eventsounds GUI can be started on any of our user computers (e.g. shanevnc) with the following command:

/usr/local/lick/bin/eventsounds -a sao

Ion Pump GUI sharcspump_gui

ShARCS ion pump control GUI is started with the command

/usr/local/lick/bin/sharcspump_gui

Once the GUI is started, the high voltage power supply, HV needs to be turned on.

Starting Laser Shutdown Monitor

The laser shutdown monitor runs on shanevnc and is started with the command

/usr/local/lick/bin/LSM.tcl

Once started, select the proper directory and laser target.

Starting the Laser Offload GUI

The laser offload monitoring GUI runs on shanevnc and is started with the command

/usr/local/lick/bin/laserOffload.tcl

Starting AO telescope position GUI

The AO version of the telescope position GUI run on shanevnc and is started with the command

/usr/local/lick/bin/aopococ

Startup Checklist

  1. Mount SAO on TUB.
  2. Connect 2 power cables.
  3. Connect 1 ethernet cable.
  4. Connect 2 laser uplink BNC cables.
  5. Turn on ion pump HV supply.
  6. Ping real, stealthie, musher, covert, shadow, and karnak computers. Assume other devices (lantronixes, routers, etc.) are also up if these few computers ping.
  7. Do ao start on covert, shadow, and shark.
  8. Start saopower_ui on shanevnc.
  9. Check that appropriate items are turned on.
  10. Start saomot_gui on shanevnc.
  11. Calibrate motors.
  12. Set motors to nominal positions (choose the most recent default setup on shanevnc: /u/user/observers/lgs/saomot_setups).
  13. Start ShARCS detector software on stealthie.
  14. Start tail -f /var/log/messages on real.
  15. Start shaneao
  16. Start saocon_gui on shanevnc.
  17. Start peeko on karnak or shimmynova.
  18. Start ttpeeko on karnak or shimmynova.
  19. Initialize shaneao: Type init(16) (or 8) in shaneao, or click Actions - Init 16 (or 8) in saocon_gui.
  20. Turn on power to Woofer (B5) and Tweeter (B6) if there are no SW-IOMMU errors in the message log.
  21. Start sharcs_fe on covert.
  22. In sharcs_fe Windows - Setup, check parameters, connect, and initialize.
  23. Start sharcsdisplay on covert.
  24. Start saowheels_gui on shanevnc or covert.
  25. Calibrate wheels if necessary and set to desired positions.
  26. Start scriptproc.tcl on covert.
  27. Start eventsounds on shanevnc.
  28. Start sharcspump_gui on shanevnc.
  29. Start fieldsteer.tcl on shanevnc.
  30. Start aopococ on shanevnc.
  31. Start LSM.tcl on shanevnc if using laser.
  32. Start laserOffload.tcl on shanevnc if using laser.
  33. start saoWFSFocusTrack.tcl on shanevnc if using laser.
  34. Start saomon_gui on shanevnc.