Remote Operation of The Two Keck
10-meter Telescopes
Robert Kibrick, Director of Scientific
Computing, University of California
Observatories / Lick Observatory
Remote Instrumentation Discussion
Spring 2002 Internet2 Member Meeting
Overview of Presentation
• Background
– The Keck Telescopes
– Motivation for remote operation
• Remote observing with the Keck Telescopes
– From remote control room at summit (30 meters)
– From Keck Headquarters in Waimea, Hawaii (32 km)
– From Santa Cruz, California via Internet2 (3200 km)
• Operational models and issues
10/3/2015
2
The Keck Telescopes
10/3/2015
3
The Keck Telescopes
10/3/2015
4
Keck Telescope Facts
• Twin, 10-meter optical/infrared telescopes
• Largest telescopes of this type in the world
• Construction funded by W. M. Keck Foundation
• Observing time shared between 4 institutions
– California Institute of Technology (Caltech)
– University of California (UC)
– National Aeronautics and Space Adminstration (NASA)
– University of Hawaii (UH)
• Located atop 4,200 meter summit of Mauna Kea
10/3/2015
5
Mauna Kea summit on the island of
Hawaii
10/3/2015
6
Mauna Kea Summit
• Dormant volcano
• Premier astronomy site in N. Hemisphere
–Above 90% of water vapor in atmosphere
–Non-turbulent airflow over the summit
–Sub-arcsecond atmospheric seeing
• Home to many international observatories
10/3/2015
7
Mauna Kea Optical Observatories
• UH 0.6 meter
• UKIRT
• UH 2.2 meter
• Gemini North
• CFHT
• NASA IRTF
• Keck-2
• Keck-1
• Subaru
10/3/2015
8
Common Facilities Operated by
University of Hawaii
• Dormitories at
Hale Pohaku
• Altitude is
2,800m
• All water must
be trucked in
• Acclimatization
required before
ascent to the
4,200m summit
10/3/2015
9
Mauna Kea Observatories Serve
An International Community
• Argentina
• Australia
• Brazil
• Canada
• Chile
• France
• Japan
• Netherlands
• Taiwan
• United Kingdom
• United States
10/3/2015
10
Keck Telescopes use Classical
Scheduling
• Kecks not designed for queue scheduling
• Schedules cover a semester (6 months)
• Approved proposals get 1 or more runs
–Each run is between 0.5 to 4 nights long
–Gaps between runs vary from days to months
–Half of all runs are either 0.5 or 1 night long
• Separate schedules for the two Kecks
10/3/2015
11
Keck Telescope Schedule
A P R IL 2001
K eck I S em ester 2001A
Su
1 3 1 W olfe, P roch aska
HQ
H IR E S r6
U C SD
M o 2 2 2 V ogt
"
H IRES r7
UC SC
Tu 3 1 3 "
"
"
HQ *
We 4
5 S egm en t E xch an ge
Th
5
0 "
Fr
6
Engineering/Butler,
0 M arcy, V ogt*
Sa
7
0 B u tler, Marcy, V ogt*
Su
8
I-E n g/B u tler, Marcy,
0 V ogt*
Mo 9
9 S tephens, M arley*
RG
U 9H
W olfe
"
"
U 1H
V ogt
"
C W /M W /B S
RG oc
"
PCS
KECK
"
KECK
"
PC S*
"
"
"
HQ *
P C S /H IRES r8*
K EC K /N A S A *
"
H IR E S r8*
"
S /H Q * D S M/H IR E S r8* K E C K /N A S A *
CW
B S /M W *
M W /G S *
GS*
RC
R C oc
R C oc*
S egm en t E xch an ge
"
K EC K /N 6H * Engineering/Butler*
"
B u tler*
K E C K /N 6H * I-E n g/B u tler*
"
N IRC 2
N ASA
"
RC
N 43N
Tu 1 0 1 9 "
"
"
"
"
G W oc*
"
Ellis, Brinchmann,
W e 1 1 2 8 S antos*
"
N IRC 3
C IT
RC
C 23N
Th 1 2 3 8 "
"
"
"
CS*
RC oc
"
F r 1 3 4 7 H arrison, M ao*
"
LRIS 17
"
"
GW
C 35L
Sa 14 55 "
"
"
"
"
G W oc
"
S u 1 5 6 3 F ilippen ko
"
L R IS p18
UCB
C S /M W *
GW
U 25L
G S /C S *
"
S tephens
"
Ellis
"
H arrison
"
F ilippen ko
10/3/2015
12
From 1993 to 1995, all Keck
observing was done at the summit
Observers
at the
summit
work from
control
rooms
located
adjacent
to the
telescope
domes
10/3/2015
13
Conducting observations involves
coordinated effort by 3 groups
• Telescope operator (observing assistant)
– Responsible for telescope safety & operation
– Keck employee; normally works at summit
• Instrument scientist
– Expert in operation of specific instruments
– Keck employee; works at summit or Waimea
• Observers
– Select objects and conduct observations
– Employed by Caltech, UC, NASA, UH, or other
10/3/2015
14
Keck 2 Control Room at the Mauna
Kea Summit
Telescope
operator,
instrument
scientist,
and
observers
work side
by side,
each at
their own
computer.
10/3/2015
15
Observing at the Mauna Kea
summit is both difficult and risky
• Oxygen is only 60% of that at sea level
• Lack of oxygen reduces alertness
• Observing efficiency significantly impaired
• Altitude sickness afflicts some observers
• Some are not even permitted on summit:
–Pregnant women
–Those with heart or lung problems
10/3/2015
16
Initiative to support remote
observing from Keck Headquarters
• 1995: Remote control rooms built at Keck HQ
• Initial tests via 1.5 Mbps (T1) link to the summit
• 1996: Videoconferencing connects both sites
• Remote observing with Keck 1 begins
• 1997: >50% of Keck 1 observing done remotely
• Link to the summit upgraded to 45 Mbps (DS3)
• 1999: remote observing >90% for Keck 1 and 2
• 2000: remote observing is now the default mode
10/3/2015
17
Keck 2 Remote Control Room at
the Keck Headquarters in Waimea
Observer and
instrument
scientist in
Waimea use
video
conferencing
system to
interact with
telescope
operator at
the summit
10/3/2015
18
Videoconferencing has proved vital
for remote observing from Waimea
• Visual cues (body language) important!
• Improved audio quality extremely valuable
• A picture is often worth a thousand words
• Troubleshooting: live oscilloscope images
• “Cheap” desktop sharing (LCD screens)
• Chose dedicated versus PC-based units:
–Original (1996) system was PictureTel 2000
–Upgrading to Polycom Viewstations
10/3/2015
19
Keck 2 Remote Observing Room
as seen from the Keck 2 summit
Telescope
operators at the
summit converse
with astronomer at
Keck HQ in
Waimea via the
videoconferencing
system.
10/3/2015
20
The Remote Observing Facility at
Keck Headquarters in Waimea
• Elevation of Waimea is 800 meters
• Adequate oxygen for alertness
• Waimea is 32 km NW of Mauna Kea
• 45 Mbps fiber optic link connects 2 sites
• A remote control room for each telescope
• Videoconferencing for each telescope
• On-site dormitories for daytime sleeping
10/3/2015
21
The Keck Headquarters in Waimea
Most Keck technical
staff live and work in
Waimea. Allows
direct contact
between observers
and staff. Visiting
Scientist’s Quarters
(VSQ) located in
same complex.
10/3/2015
22
Limitations of Remote Observing
from Keck HQ in Waimea
Most Keck observers
live on the mainland.
Mainland observers
fly > 3,200 km to get
to Waimea
Collective direct
travel costs exceed
$400,000 U.S. / year
10/3/2015
23
Remote Observing from Waimea is
not cost effective for short runs
• Round trip travel time is 2 days
• Travel costs > $1,000 U.S. per observer
• About 50% of runs are for 1 night or less
• Cost / run is very high for such short runs
• Such costs limit student participation
10/3/2015
24
Motivations for Remote Observing
from the U.S. Mainland
• Travel time and costs greatly reduced
• Travel restrictions accommodated
–Sinus infections and ruptured ear drums
–Late stages of pregnancy
• Increased options for:
–Student participation in observing runs
–Large observing teams with small budgets
• Capability for remote engineering support
10/3/2015
25
Fast and reliable network needed
for mainland remote observing
• 1997: 1.5 Mbps Hawaii -> Oahu -> mainland
• 1998: 10 Mbps from Oahu to mainland
• 1999: First phase of Internet-2 upgrades:
–45 Mbps commodity link Oahu -> mainland
–45 Mbps Internet-2 link Oahu -> mainland
• 2000: Second phase upgrade:
–35 Mbps Internet-2 link from Hawaii -> Oahu
–Now 35 Mbps peak from Mauna Kea to mainland
10/3/2015
26
Mainland remote observing goals
and implementation strategy
• Goals:
–Target mainland facility to short duration runs
–Avoid duplicating expensive Waimea resources
–Avoid overloading Waimea support staff
• Strategy:
–No mainland dormitories; observers sleep at home
–Access existing Waimea support staff remotely
–Restrict mainland facility to experienced
–Restrict to mature, fully-debugged instruments
10/3/2015
27
Mainland remote observing facility
is an extension of Keck HQ facility
• Only modest hardware investment needed:
–Workstations for mainland remote observers
–Network-based videoconferencing system
–Routers and firewalls
–Backup power (UPS) – especially in California!!!
–Backup network path to Mauna Kea and Waimea
• Avoids expensive duplication of resources
• Share existing resources wherever possible
–Internet-2 link to the mainland
–Keck support staff and operational software
10/3/2015
28
Keck software is accessed the same
regardless of observer’s location
• The control computers at the summit:
–Each telescope and instrument has its own computer
–All operational software runs only on these computers
–All observing data written to directly-attached disks
–Users access data disks remotely via NFS or ssh/scp
• The display workstations
–Telescopes and instruments controlled via X GUIs
–All users access these X GUIs via remote displays
–X Client software runs on summit control computers
–Displays to X server on remote display workstation
10/3/2015
29
Why did we choose this approach?
• Operational Simplicity
• Operational control software runs only at the summit
• All users run identical software on same computer
• Simplifies management between independent sites
• Allowed us to focus on commonality
• Different sites / teams developed instrument software
• Large variety of languages and protocols were used
• BUT: all instruments used X-based GUIs
10/3/2015
30
Overall Topology
10/3/2015
31
Focus effort on X standardization
and optimization over long links
• Maintain consistent X environment between sites
• Optimize X performance between sites
• Eliminates need to maintain:
• Diverse instrument software at multiple sites
• Diverse telescope software at multiple sites
• Coordinate users accounts at multiple sites
• Fewer protocols for firewalls to manage
10/3/2015
32
Accessing Keck software and data
from Keck HQ in Waimea
Telescope
operator
uses display
workstation
at summit.
Instrument
scientist and
observers
use display
workstations
in Waimea.
10/3/2015
33
Accessing Keck software and data
from the mainland
Telescope
operator uses
display
workstation at
summit.
Instrument
scientist uses
display in
Waimea
Observers use
display on
mainland
10/3/2015
34
Remote observing differences:
Waimea versus the mainland
• System Management:
–Keck summit and HQ share a common domain
–Mainland sites are autonomous
• Remote File Access:
–Observers at Keck HQ access summit data via NFS
–Observers on mainland access data via ssh/scp
• Propagation Delays:
–Summit to Waimea round trip time is about 1 ms.
–Summit to mainland round trip time is about 100 ms.
10/3/2015
35
Increased propagation delay to
mainland presents challenges
• Initial painting of windows is much slower
• But once created, window updates fast enough
• All Keck applications display to Waimea OK
• A few applications display too slowly to mainland
• System and application tuning very important
–TCP window-size parameter (Web100 Initiative)
–X server memory and backing store
–Minimize operations requiring round trip transactions
10/3/2015
36
Simulating long propagations
delays in the lab
• Instruments are designed and built on mainland
• Software is debugged on local area network
• Testing on LAN does not reveal delay problems
• Must measure delay effects before deployment
• Simulate WAN delays using NIST simulator
–Requires Linux PC with dual Ethernet interfaces
–Can select specific packets delays, losses, jitters
–http://www.antd.nist.gov/itg/nistnet
10/3/2015
37
Shared access and control of
instruments
• Most software for Keck optical instruments
provides native multi-user/multi-site control
• All users have consistent view of status and data
• Instrument control can be shared between sites
• Multipoint video conferencing key to coordination
• Some single-user applications can be shared via
X-based application sharing environments:
– XMX http://www.cs.brown.edu/software/xmx
– VNC http://www.uk.research.att.com/vnc
10/3/2015
38
Tradeoffs from this approach to
remote observing
• Disadvantages:
–X protocol does not make optimal use of bandwidth
–Long propagation delays require considerable tuning
• Advantages:
–Minimizes staffing requirements at mainland sites
–Only “vanilla” hardware and software needed there
–Simplifies sparing and swapping of equipment
–Simplifies system maintenance at mainland sites
–Simplifies authentication/access control
10/3/2015
39
End-to-end reliability is critical to
successful remote operation
• Keck Telescope time is valued at $1 per second
• Each observer gets only a few nights each year
• Observers won’t use facility if not reliable
• What happens if network link to mainland fails?
–Path from Mauna Kea to mainland is long & complex
–At least 14 hops crossing 7 different network domains
–While outages are rare, consequences are severe
–Even brief outages cause session collapse & panic
–Observing time loss can extend beyond outage
10/3/2015
40
Mitigation plan: install end-to-end
ISDN-based fallback path
• Install ISDN lines and routers at:
–Each mainland remote observing site
–Keck 1 and Keck 2 control rooms
• Fail-over and fallback are rapid and automatic
• Toll charges incurred only during network outage
• Lower ISDN bandwidth reduces efficiency, but:
–Observer retains control of observations
–Sessions remain connected and restarts avoided
–Prevents observer panic
10/3/2015
41
Summary of ISDN-based fallback
path
• Install 3 ISDN (6 B channels) between sites
• Install Cisco 2600-series routers at each end
–Dual auto-sensing Ethernet interfaces
–Quad BRI interfaces
•
•
•
•
Inverse multiplexing
Dial-on-demand (bandwidth-on-demand)
Caller ID (reject connections from unrecognized callers)
Multilink PPP with CHAP authentication
• Uses GRE tunnels and OSPF routing
• No manual intervention needed at either end
10/3/2015
42
Current status
• Prototype mainland facility operational at UCSC
in temporary quarters
• Enabled continuity of operation during week of
September 11, 2001 (no flights to Hawaii)
• Moving to larger, permanent facility in June
• Similar facilities being assembled at other sites
in California:
–California Institute of Technology
–UC San Diego
–UC Berkeley
10/3/2015
43
Similar efforts at other
observatories
• IRAM 30-meter radiotelescope in Spain
–Remote operation from Granada, Spain and
Grenoble, France since 1998
• NASA Infrared Telescope Facility on Mauna Kea
–Remote observing to observers’ home institutions
–IRTF instruments successfully operated from
Observatoire de Paris-Meudon during January 2002
• SOAR Telescope on Cerro Pachon in Chile
–Remote Observing Center at Michigan State
–High speed link via Internet-2 and REUNA
10/3/2015
44
Recent and upcoming conferences
• Advanced Global Communications Technologies
for Astronomy I – Munich, March 2000, SPIE
Proceeding Vol 4011
• Advanced Global Communications Technologies
for Astronomy II – Waikoloa, Hawaii, August
2002
• http://spie.org/info/as
10/3/2015
45
Summary
• Internet-2 makes mainland operation feasible
• Proposed model should be affordable:
–Mainland sites operate as satellites of Keck HQ
–Leverage investment in existing facilities and staff
–Leverage investment in existing software
–Share existing resources wherever feasible
–Maintain Keck HQ as focal point for remote operation
–Avoid wasteful duplication of resources
–Avoid expensive and inefficient travel for short runs
10/3/2015
46
Acknowledgments
• U.S. National Science Foundation
• U.S. Department of Defense
• University of Hawaii
• Gemini Telescope Consortium
• University Corp. for Advanced Internet
Development (UCAID)
• Corporation for Education Network Initiatives in
California (CENIC)
10/3/2015
47
Author Information
Robert Kibrick, UCO/Lick Observatory
University of California, Santa Cruz
California 95064, U.S.A.
E-mail: [email protected]
WWW: http://www.ucolick.org/~kibrick
Phone: +1-831-459-2262
FAX: +1-831-459-2298
10/3/2015
48
www.internet2.edu
Descargar

Remote Observing with the Keck Telescopes