The Laser Interferometer
Gravitational-Wave Observatory
Supported by the United States National Science Foundation
Use of Condor by the LIGO Scientific Collaboration
Gregory Mendell, LIGO Hanford Observatory
On behalf of the LIGO Scientific Collaboration
LIGO-G080280-00-W
Sources Of Gravitational Waves
Black Holes
Supernovae
Dense Stars
LIGO-G080280-00-W
Stochastic Background
Photos: http://antwrp.gsfc.nasa.gov; http://imagine.gsfc.nasa.gov
Worldwide Interferometers
GEO
Virgo
LIGO
TAMA
AIGO
LIGO-G080280-00-W
Worldwide Data Analysis using
The LIGO Data Grid (LDG)
LSC: Cardiff
LDAS: MIT
LDAS: LHO
LSC: Birmingham
LSC: Golm
LDAS: CIT
LSC: SYR
LDAS: LLO
LSC: UWM
LSC: PSU
3000+ CPUs with a typical clock speed of 2.6 GHz
LIGO-G080280-00-W
The LIGO Data Grid
LDG Client/Server
Distribution
packageName( 'Server' )
version( '4.5' )
pacmanVersionGE('3.18.5')
package( 'Server-Environment' )
package( 'VDT_CACHE:Globus' )
package( 'VDT_CACHE:CA-Certificates' )
package( 'VDT_CACHE:CA-Certificates-Updater'
)
package( 'VDT_CACHE:Condor' )
package( 'VDT_CACHE:GSIOpenSSH' )
package( 'VDT_CACHE:KX509' )
package( 'VDT_CACHE:MyProxy' )
package( 'VDT_CACHE:UberFTP' )
package( 'VDT_CACHE:EDG-Make-Gridmap' )
package( 'VDT_CACHE:Globus-RLS')
package( 'VDT_CACHE:Globus-Core')
package( 'VDT_CACHE:Globus-Condor-Setup' )
package( 'VDT_CACHE:PyGlobus' )
package( 'VDT_CACHE:PyGlobusURLCopy' )
package( 'VDT_CACHE:Pegasus' )
package( 'VDT_CACHE:VOMS-Client' )
package( 'VDT_CACHE:Globus-WS' )
package( 'VDT_CACHE:Tomcat-5.5' )
package( 'VDT_CACHE:TclGlobus' )
package( 'Server-FixSSH' )
package( 'Server-RLS-Python-Client' )
package( 'Server-Cert-Util' )
LIGO-G080280-00-W
package( 'Server-LSC-CA' )
The LIGO Data Grid
•
Users
 500+ scientist in the LIGO Scientific
Collaboration
 200+ doing data analysis on the LIGO Data Grid
Grid middleware
 LDG Client/Server
•

Virtual Data Toolkit (VDT)
– Globus Toolkit
– GSI and X.509 certificates
– pyGlobus, tclGlobus, Pegasus, etc…

In house packages


•
Glue: LSC Data Location & Pipeline Tools
LDR: LSC Lightweight Data Replication
 GridFTP for moving data and files
 Replica Location Service (RLS)
 Onaysis: LSC Online Analysis System
High throughput computing
 Condor for most analyses
•
 BOINC for [email protected]
LSC Analysis Software


LAL, Matapps,
DMT, etc..
LIGO-G080280-00-W
Use of Condor by the LIGO
Scientific Collaboration
• Condor handles 10’s of millions of jobs per year
running on the LDG, and up to 500k jobs per DAG.
• Condor standard universe checking pointing
widely used, saving us from having to manage this.
• At Caltech, 30 million jobs processed using 22.8
million CPU hrs. on 1324 CPUs in last 30 months.
• For example, to search 1 yr. of data for GWs from
the inspiral of binary neutron star and black hole
systems takes ~2 million jobs, and months to run on
several thousand ~2.6 GHz nodes.
LIGO-G080280-00-W
Inspiral Analysis Pipeline
•
•
Multidetector pipeline
 Pipeline topology is same for all
inspiral searches
 BNS, PBH, BBH, spinning BBH
 Different template/filtering code
used for different searches
 Can be used for LIGO-GEO and
LIGO-VIRGO analysis
Pipeline Description
 Inspiral search run on each IFO
 Look for triggers coincident in time
and mass between detetors
 Follow up with signal-based vetoes
 Perform coherent analysis of
surviving triggers
 Follow up candidate events
LIGO-G080280-00-W
Example of a LIGO Inspiral
DAG
LIGO-G080280-00-W
Example DAG Within Dag
DAG: finds data
and generates
Fourier Tranforms
used by other
DAGs
DAG: outputs
spectra of power
supply data:
LIGO-G080280-00-W
Example Test DAG for Condor
Regression Testing
• Makes fake data for each detector (same code is used in Monte Carlo simulations).
• Run the fully-coherent multi-detector continuous-wave search code, used to search
for GWs from rotating neutron stars.
• Compares the output with reference data.
LIGO-G080280-00-W
The LIGO/Condor Success Story
•
•
•
•
•
•
Condor handles most of our searches and is vital to the success of LIGO.
Condor and LIGO have a biweekly telecon to discuss issues & enhancements.
In approximately the past year, Condor successfully
•
enhanced scaling to support non-trivial O(1M) node DAGs,
•
implemented option to prioritize nodes, e.g., depth-first traversal of DAGs,
•
added categories to limit on number of resource intensive nodes in a DAG,
•
handling of priorities and staggered start of jobs.
Condor is working on a list of enhancements to, e.g.,
•
speed up of starting DAGs by O(100x),
•
automate finding of rescue DAGs, e.g., when there are DAGs within DAGs,
and merging of sub-DAGs.
•
Add standard universe support on RHEL/CentOS and Debian.
Condor is compatible with BOINC and can run backfill [email protected] jobs on the LDG
clusters when there are idle cycles.
For the future:
•
Our offline/online high throughput computing needs will continue to grow.
•
Online jobs moving towards low latency; need to think about computing
needs for realtime detection when Advanced LIGO comes on line.
LIGO-G080280-00-W
Descargar

The R-modes: New Theoretical Results