Web Services for Computing Portals Marlon Pierce Community Grids Lab Indiana University What Are Web Services? • Web services framework is an XML-based distributed object/service/component system. – SOAP, WSDL, WSIL, UDDI • Basic ideas is to build a distributed invocation system out of existing Web standards. – Most standards defined by W3C, Oasis (IP considerations) • Very loosely defined, when compared to CORBA, etc. – Take what you need – Ignore what you don’t like – Develop what’s missing. XML Overview • XML is a language for building languages. • Basic rules: be well formed and be valid • Particular XML “dialects” are defined by an XML Schema. – XML itself is defined by its own schema. • XML is extensible via namespaces • Many non-Web services dialects – RDF, SVG, MathML, XForms, XHTML • Many basic tools available: parsers, XPath, and XQuery for searching, etc. XML and Web services • XML provides a natural substrate for distributed computing: – Its just a data description. – Platform, programming language independent. • So let’s describe the pieces. • Web Services Description Language (WSDL) – Describes how to invoke a service (compare with CORBA IDL). – Can bind to SOAP, other protocols for actual invocation. • Simple Object Access Protocol (SOAP) – Wire protocol extension for conveying RPC calls. – Can be carried over HTTP, SMTP. • Web Services Inspection Language (WSIL) – A very simple “discovery” mechanism for locating WSDL service definitions. Web Services in Action • The following examples illustrate how Web services interact through a browser interface. • The browser interface could easily be changed to any other client. Client Browser (1) Client Requests (3) Bind and Use Service Service Provider User Interface Server (HTTP) (2) Find Service (SOAP/HTTP) Information Repository (0) Publish Services Service Provider Service Provider Service Provider Browser Interface HTTP(S) UI Server has stubs for all services (data base access, job submission, file transfer, etc.) A particular server has several service implementations. Backend is a database, application code plus operating system. User Interface Server + Client Stubs SOAP/HTTP(S) Server plus Service Implementations Local invocation, JDBC connection or Grid Protocol Backend Resources User Interface (1) (6) (10) (15) Client Stubs (2) (11) (5) (4) DB Service 1 JDBC (3) DB Host 1 (7) (9) (14) Job Sub/Mon And File Services (8) (12) (13) Operating and Queuing Systems Host 2 (19) (16) (18) DB Service 2 JDBC (17) DB Host 3 Web Services Description Language Defines what your service does and how it is invoked. WSDL Overview • WSDL is an XML-based Interface Definition Language. – You can define the APIs for all of your services in WSDL. • WSDL docs are broken into four major parts: – abstract interface definitions (including messages and data definitions). – protocol bindings (to SOAP, for example) – service point locations (URLs) • Some interesting features – A single WSDL document can describe several versions of an interface. – A single WSDL doc can describe several related services. WSDL Example: Job Submission • The example is a simple service that can executes local (to the server) Unix commands. • Service implementation (in Java) has a single method – ExecLocal takes a single string argument (the command to exec) and returns a 2D string array (standard out and error). • The WSDL maps to a Java interface in this case. <?xml version="1.0" encoding="UTF-8"?> <wsdl:definitions> <wsdl:message name="execLocalCommandResponse"> <wsdl:message name="execLocalCommandRequest"> <wsdl:portType name="SJwsImp"> <wsdl:operation name="execLocalCommand" parameterOrder="in0"> <wsdl:input message="impl:execLocalCommandRequest" name="execLocalCommandRequest"/> <wsdl:output message="impl:execLocalCommandResponse" name="execLocalCommandResponse"/> </wsdl:operation> </wsdl:portType> <wsdl:binding name="SubmitjobSoapBinding" type="impl:SJwsImp"> <wsdlsoap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/> <wsdl:operation name="execLocalCommand"> <wsdlsoap:operation soapAction=""/> <wsdl:input name="execLocalCommandRequest"> <wsdl:output name="execLocalCommandResponse"> </wsdl:operation> </wsdl:binding> <wsdl:service name="SJwsImpService"> <wsdl:port binding="impl:SubmitjobSoapBinding" name="Submitjob"> </wsdl:service> </wsdl:definitions> WSDL Elements I • Types: describes custom XML data types (optional) used in messages. – For OO languages, types are a limited object serialization. • Message: abstractly defines the messages that need to be exchanged. – Conventionally messages are used to group requests and responses. – Each method/function in the interface contains 0-1 request and 0-1 response messages. – Consists of part elements. Usually you need one part for each variable sent or received. Parts can either be XML primitive types or custom complex types. Types for Job Submission • Recall that the job submission service sends a string (the command) and returns a 2D array. • Strings are XML Schema primitive types, so we don’t need a special definition. • Arrays are not primitive types. They are defined in the SOAP schema, so we will import that definition. Example: WSDL types for Custom Data Definition <wsdl:types> <schema targetNamespace="http://.../GCWS/services/Submitjob" xmlns="http://www.w3.org/2001/XMLSchema"> <import namespace="http://schemas.xmlsoap.org/soap/encoding/" /> <complexType name="ArrayOf_xsd_string"> <complexContent> <restriction base="soapenc:Array"> <attribute ref="soapenc:arrayType" wsdl:arrayType="xsd:string" /> </restriction> </complexContent> </complexType> <element name="ArrayOf_xsd_string" nillable="true" type="impl:ArrayOf_xsd_string" /> </schema> </wsdl:types> Message Elements for Job Submission Service • Our service implementation has one method of the form (in Java) public String execLocalCommand(String cmd) • This will require one “request” message and one “response” message. • Each message has one part: – Request message must send the String cmd. – Response must get back the String array (defined previously as a custom type). • If we had to pass two input variables, our “request” message would need two part elements. • Note the name attributes of messages are important! Message Examples for Job Submission Service <wsdl:message name="execLocalCommandResponse"> <wsdl:part name="execLocalCommandReturn" type="impl:ArrayOf_xsd_string" /> </wsdl:message> <wsdl:message name="execLocalCommandRequest"> <wsdl:part name="in0" type="xsd:string" /> </wsdl:message> portTypes • portType elements map messages to operations. • You can think of portType==class, operation==class methods. • Operations can contain input, output, and/or fault bindings for messages. • An operation may support of the following message styles: – – – – One-way: request only Two-way: request/response Solicit-response: server “push” and client response Notification: one-way server push portType for JobSubmit • We previously defined the messages and types needed. Now we bind them into the portType structure. • PortType names are important – Will be referenced by binding element. • Note names of previously defined messages are used as references in the operations. Example WSDL Nugget <wsdl:portType name="SJwsImp"> <wsdl:operation name="execLocalCommand" parameterOrder="in0"> <wsdl:input message="impl:execLocalCommandRequest" name="execLocalCommandRequest" /> <wsdl:output message="impl:execLocalCommandResponse" name="execLocalCommandResponse" /> </wsdl:operation> </wsdl:portType> PortType Bindings • portTypes are abstract interface definitions. – Don’t say anything about how to invoke a remote method. • Remote invocations are defined in binding elements. • Binding elements are really just place holders that are extended for specific protocols – WSDL spec provides SOAP, HTTP GET/POST, and MIME extension schema examples. SOAP Bindings for JobSubmit Service • Note that the binding element contains a mixture of tags from different namespaces (wsdl and wsdlsoap). • WSDL child elements for binding element are operation, input, and output. • WSDLSOAP elements are from a different XML schema (a new one, neither WSDL or SOAP). – This is how you extend WSDL bindings: define a new schema that gives mapping instructions from WSDL to the protocol of choice. • The binding element name is important, will be used as a reference by the final port binding. <wsdl:binding name="SubmitjobSoapBinding" type="impl:SJwsImp"> <wsdlsoap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http" /> <wsdl:operation name="execLocalCommand"> <wsdlsoap:operation soapAction="" /> <wsdl:input name="execLocalCommandRequest"> <wsdlsoap:body encodingStyle=http://schemas.xmlsoap.org/soap/encoding/ namespace="http://.../GCWS/services/Submitjob" use="encoded" /> </wsdl:input> <wsdl:output name="execLocalCommandResponse"> <wsdlsoap:body encodingStyle=http://schemas.xmlsoap.org/soap/encoding/ namespace=http://.../GCWS/services/Submitjob use="encoded" /> </wsdl:output> </wsdl:operation> </wsdl:binding> SOAP Binding in Detail <wsdlsoap:body encodingStyle=http://schemas.xmlsoap.org/soap/encodi ng/ namespace=http://.../GCWS/services/Submitjob use="encoded" /> • All this really means is “encode the message by the rules in encodingStyle and put it in the SOAP body.” Service and Port Definitions • So far, we have defined the class method interfaces (portTypes) and the rules for binding to a particular protocol. • Port elements define how the bindings (and thus the portTypes) are associated with a particular server. • The service element collects ports. Service and Port Elements for the Job Submission Service <wsdl:service name="SJwsImpService"> <wsdl:port binding="impl:SubmitjobSoapBinding" name="Submitjob"> <wsdlsoap:address location="http://.../GCWS/services/Submitjob" /> </wsdl:port> </wsdl:service> Explanation • Note the port element’s binding attribute points to the appropriate binding element by name. • The only purpose of the port element is to point to a service location (a URL). This is done by extension (SOAP in this case.) • Ports are child elements of the service element. A service can contain one or more ports. – Note the value of multiple ports: a single portType may correspond to several ports, each with a different protocol binding and service point. WSDL Trivia • The schema rules allow all of the elements we have discussed to appear zero or more times. • A single WSDL file may contain many portTypes (although this is not usual). – You may want to do this to support multiple interface definitions of a service for backward compatibility. • Multiple ports may also be used to provide different views of a service – One portType defines the interface. – Another provides access to metadata about the service. – Yet another may define how the service interacts with other services via notification/event systems. • This is being exploited by OGSA. Open Grid Services Architecture in One Slide • Conventions for metadata about services – serviceData: extends portType elements to provide additional metadata needed to describe the particular service. • When is the service available? – serviceDataDescription: defines serviceData element rules. • Is an element a string? An int? • How may times can it occur? • Various supplementary portTypes – GridService: allows you to query serviceData – Notification ports: communicate state changes. Simple Object Access Protocol A message format for exchanging structured, typed information SOAP Basics • SOAP is often thought of as a protocol extension for doing RPC over HTTP. • This is not completely accurate: SOAP is an XML message format for exchanging structured, typed data. • It may be used for RPC in client-server applications but is also suitable for messaging systems (like JMS) that follow one-to-many (or publish-subscribe) models. SOAP Structure • A SOAP message is contained in an envelop. • The envelop element in turn contain (in order) – An optional header with one or more child entries. – A body element that can contain one or more child entries. These child entries may contain arbitrary XML data. SOAP Headers • Headers are really just extension points where you can include elements from other namespaces. – i.e., headers can contain arbitrary XML. • Header entries may optionally have a “mustUnderstand” attribute. – mustUnderstand=1 means the message recipient must process the header element. – If mustUnderstand=0 or is missing, the header element is optional. SOAP Body • The SOAP body contains any number of entries that must be immediate children. • Body entries are really just placeholders for arbitrary XML from some other namespace. Example Messages • Recall the WSDL interface for “SubmitJob” – Sends one string command – Returns array of strings for standard out and error. • The envelop is decorated with a few useful namespaces: – soapenv defines the version – xsd is the Schema definition itself – xsi defines some useful constants. • The body is just an arbitrary XML fragment. – Assumes the recipient knows what this means. – Recipient must looks up the ExecLocalCommand operation in the JobSubmit service and passes it one string argument. – The ns1 namespace tells the recipient the WSDL namespace that defines the service. – xsi:type lets the recipient know that the arbitrary XML element in0 is in fact a string, as defined by the XML Schema. SOAP Request <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <ns1:execLocalCommand soapenv:encodingStyle ="http://schemas.xmlsoap.org/soap/encoding/" xmlns:ns1 ="http://.../GCWS/services/Submitjob/GCWS/services/Submitjob"> <in0 xsi:type="xsd:string">/usr/bin/csh /tmp/job.script</in0> </ns1:execLocalCommand> </soapenv:Body> </soapenv:Envelope> Example Response • The structure is the same as the request. • The interesting thing here is that the request returns a 2-element array of two strings. – Arrays not defined by XML schema – SOAP encoding does define arrays, so use xsi:type to point to this definition. – <item></item> surrounds each array element. • Note that arbitrary XML returns can likewise be encoded this way. – Use xsi:type to point to a schema. SOAP Response <soapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/ xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <ns1:execLocalCommandResponse soapenv:encodingStyle= http://schemas.xmlsoap.org/soap/encoding/ xmlns:ns1= "http://.../GCWS/services/Submitjob/GCWS/services/Submitjob"> <execLocalCommandReturn xsi:type="soapenc:Array“ soapenc:arrayType="xsd:string" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"> <item></item> <item></item> </execLocalCommandReturn> </ns1:execLocalCommandResponse> </soapenv:Body> </soapenv:Envelope> Developing Web Services Using Apache Axis to develop Java implementations of Web services. Web Service Development Tools • Web service toolkits exist for various programming languages: – C++,Python, Perl, various Microsoft .NET kits. • We’ll concentrate on building Java Web services with Apache Axis. • Language and implementation interoperability is addressed through WS-I. – http://www.ws-i.org/ Apache Axis Overview • Apache Axis is a toolkit for converting Java applications into Web services. • Axis service deployment tools allow you to publish your service in a particular application server (Tomcat). • Axis client tools allow you to convert WSDL into client stubs. • Axis runtime tools accept incoming SOAP requests and redirect them to the appropriate service. Developing and Deploying a Service • Download and install Tomcat and Axis. • Write a Java implementation – Our SubmitJob is a simple example but services can get quite complicated. – Compile it into Tomcat’s classpath. • Write a deployment descriptor (WSDD) for your service. – Will be used by Axis runtime to direct SOAP calls. • Use Axis’s AdminClient tool to install your WSDD file. • That’s it. – Axis will automatically generate the WSDL for your service. Sample WSDD <deployment name="Submitjob" xmlns="http://xml.apache.org/axis/wsdd/" xmlns:java="http://xml.apache.org/axis/wsdd/providers/java"> <service name="Submitjob" provider="java:RPC"> <parameter name="scope" value="request"/> <parameter name="className" value="WebFlowSoap.SJwsImp"/> <parameter name="allowedMethods" value="execLocalCommand"/> </service> </deployment> Explanation • Use Axis’s command-line AdminClient tool to deploy this to the server. • Axis will create a service called – http://your.server/services/SubmitJob • WSDL for service is available from – http://your.server/services/SubmitJob?wsdl • A list of all services is available from – http://your.server/services Check your Tomcat Server for a list of deployed services. WSDL generated by inspecting the Java implementation. Can be download from the server. (XML was shown in earlier slides) Building a Client with Axis • Obtain the WSDL file. • Generate client stubs – Stubs look like local objects but really convert method invocations into SOAP calls. • Write a client application with the stubs – Can be a Java GUI, a JSP page, etc. • Compile everything and run. Sample Java Client Code /**Create SubmitJob client object and point to the service you want to use */ SubmiJob sjws = new SubmitJobServiceLocator().getSubmitjob(new URL(http://your.server/services/SubmitJob)); /** Invoke the method as if local. */ String messages = sjws.execLocalCommand(command); Two Notes On Client Stubs • Axis stubs convert method calls into SOAP requests but WSDL does not require the use of SOAP. – Web Service Invocation Framework (WSIF) from IBM allows flexibility of protocols. (Alek Slominski, IU) • Client stubs introduce versioning problems. – We are developing dynamic (stubless) clients that construct SOAP messages by inspecting WSDL at runtime. Web Service URLs • Java – http://xml.apache.org/axis/ • XSOAP: C++ and Java toolkits for WS – http://www.extreme.indiana.edu/xgws/xsoap/ • gSOAP: C++ SOAP toolkit – http://www.cs.fsu.edu/~engelen/soap.html • Python Web Services: – http://pywebsvcs.sourceforge.net/ • Perl: – http://www.soaplite.com/ GEM Portal Services and user interfaces for our earthquake modeling portal. Solid Earth Research Virtual Observatory Grid • A number of simulation methods for studying earthquakes are being developed by GEM consortium including: – Simplex, Disloc (JPL) – Virtual California (UC-Davis) – PARK codes (Brown) • As codes become more robust and accepted, problems emerge: – Need to manage information about distributed data sources: multiple databases, sensors, simulated data. – Need to organize, manage information about multiple code installation sites. – Need to simplify access to data, use of codes, and use of visualization/analysis tools for broad range of users – Need to link together • NASA funded activity to develop SERVOGrid Interoperability framework SERVOGrid Architecture Repositories Federated Databases Database Loosely Coupled Filters Sensor Nets Streaming Data Database Closely Coupled Compute Nodes Analysis and Visualization Grid Web Service and Portlet based Portal Architectures • Web services are part of a broad industry and academic initiative to build distributed computing infrastructure around existing standards (HTTP, XML, etc). • Web services can be used to provide lightweight service descriptions and send messages between components in XML. • OGSA (Open Grid Service Architecture) builds on basic Web Services – Component model for Grid replacing/building on CORBA, COM, Enterprise Javabeans etc. • Portlets provide component model for user interfaces Computing Portal Grid Web Services • We have built a suite of general purpose Grid Web services for managing distributed applications. • Core Computing services define general purpose functions: – Ex: job submission, file transfer, job monitoring, management of jobs and results – Described as a GridShell as plays same role to Grid that Shell does for UNIX on a single machine • Application Grid Web services include metadata about applications. – Built on top of core services. – Original application NOT changed • We have developed a toolkit that allows one to convert general software packages into Grid Web Services and manage application collections User Services System Services Grid Computing Environments Portal Services System Services Application Application Metadata Service Middleware System Services Actual Application System Services System Services Raw (HPC) Resources “Core” Grid Database Application Grid Web Services • AGWS are designed to make scientific applications (i.e. earthquake modeling codes) into Grid Resources • AGWS services are described by two XML Schemas: – Abstract descriptors describe application options. Used by the application developer to deploy his/her service into the portal. – Instance descriptors describe particular user choices and archive them for later browsing and resubmission. Portals Aggregate Access to Distributed Services Client Browser (1) Client Requests (3) Bind and Use Service Service Provider User Interface Server (HTTP) (2) Find Service (SOAP/HTTP) Information Repository (0) Publish Services Service Provider Service Provider Service Provider Browser Interface JSP + Client Stubs DB Service 1 Job Sub/Mon And File Services JDBC DB Host 1 DB Service 2 JDBC Operating and Queuing Systems Host 2 DB Host 3 GEM Portlets and Portal Stacks Aggregation Portals User facing Web Service Ports Application Grid Web Services Core Grid Services Message Security, Information Services • User interfaces to GEM services (Code Submission, Job Monitoring, File Management for Host X) are all managed as portlets. • Users, administrators can customize their portal interfaces to just precisely the services they want. File Selector Screenshot • Users can view files on remote hosts – Shown is user file list for host noahsark. – Host solar is also shown. • Files can be uploaded or downloaded between PC and remote host. • Files may be crossloaded between two remote hosts. • Same interface definition may be used to access databases. File management Tabs indicate available portlet interfaces. Lists user files on selected host, noahsark. File operations include Upload, download, Copy, rename, crossload Application Selection Screen Shots • Users can select from available codes (Simplex, Disloc, various VirtualCalifornia) • Users also select from available hosts for particular code (grids, noahsark, solar shown). • Selecting solar (Sun 64 Node E10000) prompts user for information needed to use PBS script generating service. User Application Selection and Submission Generate script for job submission Select desired application and host Application Administrator Interfaces • Application Administrators deploy and manage applications • Update screen shows various application and host parameters that are set by the user. • These are used to generate user interfaces for the selected codes. Administer Grid Portal Provide information about application and host parameters Select application to edit Futures I: General Capabilities • System builds on Gateway Portal developed for NCSA Alliance and DoD HPCMO. This project intends to: • Integrate portlets from other groups in collaborations (common interoperability framework for all tiers) • Continue developing message-based security system (currently available as demonstration) that signs SOAP messages. • Integrate AGWS and Results Manager with arbitrary Schema Wizard/Storage/Browser system used in Indiana XML newsgroups. • Add negotiation capabilities to Web Services for version control • Track evolving OGSA service standards and implementations. • Add dynamic information service and monitoring capabilities for transient Web Services. • Distributed, ant-based tools to simplify service upgrades of distributed servers (and simplify our lives). Futures II: SERVOGrid • Assist with design of common data format and associated tools for GEM codes. – i.e. SERVOGrid Object Model defined in XML • Add support for service linkage (workflow) for specific GEM applications – e.g. Disloc<->Simplex interoperation). • Add support for Web-based visualization with portlet interface. • SERVOGrid portal will be publicly available Web Services for Computational Web Portals Marlon Pierce Community Grids Lab Indiana University Grid Computing Environments • GCEs are a general name for both Grid clients and middleware. • GCEs aim to bridge the gap between users and Grid infrastructure developers. • GCE-RG: Official GCE forum in the GGF. – Geoffrey Fox, Dennis Gannon, Mary Thomas are cochairs. • The work presented here is an outgrowth of several GGF and workshop meetings. • See Grid Computing: Making the Global Infrastructure a Reality " edited by Fran Berman, Geoffrey Fox and Tony Hey – http://www.grid2002.org XML and Web Services • Web services use Internet standard protocols and technologies for distributed computing. • Basic components – Wire exchange of XML messages (SOAP) • Extensible message wrapper capable of carrying general XML, binary data, remote procedure call instructions. – XML descriptions of service interfaces (WSDL) • Can express detailed description of how to invoke service – But the architecture is loosely connected • Web services are very loosely coupled – Distributed development – Distributed deployment What is Missing? • Transient services – Web services are not permanent or static. – May need to be created dynamically and registered. – One of the issues being addressed by the Open Grid Service Architecture/Infrastructure • Still must address numerous other issues – – – – – – – Metadata information services. Aggregation of web services for scientific applications. Aggregation of clients into component environments. Message security. (Workflow). (Quality of service). (Federation of services). Problems with Traditional Portal Architecture • • Web browser Portals accesses heterogeneous back ends and grids through a particular middle tier. Most portal projects are not interoperable – – • Middle tier software incompatible Wide range of protocols. – – Portal developers don’t have to reinvent every single important service. Users will have access to more services than any one project can provide. Users will be able to pick up the best available implementation of a service. Web browser ? services services … Why do we need the portal interoperability aspects of services? – … Back end resources … Back end resources Portals + Web Services Client Browser (1) Client (HTTP) Requests (3) Bind and Use Service Service Provider User Interface Server (2) Find Service (SOAP/HTTP) Information Repository (0) Publish Services Service Provider Service Provider Service Provider Core Web Services • Let’s get started writing services! • Given WSDL and SOAP, what can you build? • Site Dependent Services – – – – Job Submission File Transfer Job Monitoring Host Monitoring • Site Independent Services – Context Management – Batch/RSL Generation – Archiving Services • These core services are simple, stateless. • Many others (from Gannon group, for example) Example Core Service: Context Management • Common problem of portals is to store all of the metadata associated with user sessions. • Context Management service provides simplest possible data model – Context manager provides an easy interface to data trees. – Context data nodes are defined by recursive schema that hold optional, unbounded name/value pairs and child nodes. • We use CM to store locations of job scripts, miscellaneous file URIs, etc. • CM metadata stored on file systems, XML-native databases, …. – Actual data may be anywhere. • Searched with XPath queries. Application Codes as Web Services • Scientific applications consist of several core Web services. – Get files to right place, script submission instructions, submit the job, get notified at various states. • We need a meaningful metadata model for applications – Describe application-specific requirements – Describe bindings of applications to host environments and to Web services Application Web Services • (From discussions with Tomasz Haupt) • We divide application lifecycles into several stages: – Abstract state: describes optional choices and configurations that are available. – Ready state: Specific choices are made – Submitted: Application is running (and OGSI takes over) – Completed: Application is finished, but we need to archive information about it. AWS Schema Structure • Two sets of XML schema: – Application Descriptors: describe abstract state. – Application Instance Descriptors: describe particular instance states (ready, running, archived). • Schema sets are arranged hierarchically – Applications contain hosts – Schema are designed to be pluggable • Don’t like my queue description schema? Plug in your own. General Metadata Web Services (MDWS) • Application web services are based around schema. • They are part of a larger class of XML schema-based metadata services. • We need to automate the deployment of these services. • Current process is time consuming and fragile – An affront to any virtuous programmer. • Deployment Steps – Develop XML schema – Bind schema to data classes (Castor) – Write Façade classes to simplify use of fine-grained interface. – Compile and deploy service – Throw WSDL to client – Create client stubs – Write Web interface or GUI client with stubs. – Scream when schema changes. Automating Metadata Web Services (MDWS) • Metadata services are extremely important, should be very wide spread. – Should just create your schema and go. • Need to drastically simplify process of redeploying services when schema are updated. • Need to keep metadata schema creation democratic – Keep standards at the right level: standardize around schemas and a few XML languages Schema Wizard: First Steps for Automating MDWS • The Schema Wizard reads in XML schema and maps these to JSP pages. – Nice mapping between HTML visual widgets and schema elements • It also ties form actions to appropriate data objects. • Use the Schema Wizard to create a user interface for your schema • Use the generated User Interface to create XML instances SchemaWizard Architecture Annotated XML Schema Castor Schema Unmarshaller Castor Source Generator Web Application Template JavaBeans Castor SOM Schema Parser Velocity Templates Java Compiler Libraries Classes JSPs XML Form Wizard created as a Web Application XML Schema location is given to SchemaWizard. XML Form Wizard is generated. XML instance is marshaled. Next Steps for Schema Wizard • Schema Wizard currently used to automate local (to the server) XML message creation for publication. – More about this in a minute • We also want to use the SW create WSDL wizards for metadata services – Use Castor to represent schemas as Java classes on a remote server. – Note that get/sets define the interface, so any schema expressible as WSDL. – Use WSDL (actually, convenient intermediate form) in the schema wizard. – Schema Wizard forms tied to classes that can discover and dynamically invoke methods to get/set remote XML instance. XML Metadata Nugget Management • All web service entities are described with metadata and given URIs – Ex: Application Web Services define metadata about applications. – Application instance records archive all portal sessions. • What do we need? – Schema wizards to simplify deployment of new schema services and creation of valid instances. – Federating messaging systems to deliver created nuggets. – Distributed persistent storage – Name based browsing and searching of XML nugget catalogs. • Such systems should be able to handle any metadata – Create, post, and browse application instance data, citations, news postings, etc. Wizard Publish with JMS or Narada URI directories cataloged with RSS, documents retrieved on demand. Blah blah blah Federated data storage (Files, XML, RDB) Nugget Browser Blah blah blah Single Publish/Subscribe Layer Computational Web Portal Stack Aggregate Portals User Interfaces Application Web Services and Workflow Core Web Services Message Security, Information • Web service dream is that core services, service aggregation, and user interface development decoupled. • Q: How do I manage all those user interfaces? • A: Use portlets. Portlets in One Slide • Portlets are pieces of (Java) code that run in the UI server. • Each portlet corresponds to one content provider (local or remote). • Portals aggregate portlets into a single, usercustomizable display. • Jetspeed is a free, open source portlet implementation. • Jetspeed portlets need to be extended to handle our requirements. Message Level Security Aggregate Portals User Interfaces Application Web Services and Workflow Core Web Services Message Security, Information • Multiple layers of security. • Transport level security provides point-to-point protection. • Also need message level security for endto-end protection. Message Signing with SAML and Kerberos • SAML expresses security assertions in XML. • Demonstration Steps – Establish both servlet session and GSS context between the UI and AS. – UI signs SAML assertion and SOAP Body message with GSS Context’s wrap method. – Service extracts SAML assertion and SOAP Body message with GSS Context’s unwrap method from AS and verifies it. Web Browser HTTPS User Interface Server Authentication Service Kerberos Client Kerberos Server HTTP(S) +SOAP +signed SAML SOAP Service An assertion-based authentication service for Gateway Web Services The authentication service Process the SOAP message Client login process for the user authentication Initialize the secure context to get the shared key. Client Generate the assertion such as SAML, WS-security. Unwrap the assertion. Test the user validity. Sign the assertion. Unwrap the SOAP Body message. Add it to SOAP Header. Sign the SOAP Body message. Check the assertion type such as SAML, WS-security and the security mechanism such as Kerberos, PKI. Internet (HTTP) cloud Rebuild the SOAP message. Add it to SOAP Body. Process the SOAP message. Send the SOAP request. Send the SOAP response. Acknowledgements • Mary Thomas, Steven Mock and the TACC/SDSC teams. • Alliance Portal Project (ask for demo) • Geoffrey Fox • Ozgur Balsoy (Schema Wizard, Metadata Management) • Choonhan Youn (Core and Application Web Services, SAML) • Galip Aydin, Ahmet Topcu, Ali Kaplan, Beytullah Yildiz (Metadata management) Information and Downloads • Web services: www.gatewayportal.org • Application Metadata Schemas: www.servogrid.org/GCWS/Schema • Schema wizard and metadata browser: www.xmlnuggets.org • Email me: [email protected] • See me at “Research in Indiana” booth. Web Services Overview • Web services use Internet standard protocols and technologies (XML) to build distributed services and applications. • Leverages XML’s strengths – General purpose language definition rules. – Extensibility through namespaces as well as specific schema elements. – Limited inheritance – Reusable software (parsers,Castor) • Limitations: – Not good for high performance messaging. – Most pieces still being developed Information Archival Web Service (TACC/SDSC) • Collects machine status info • Uses various mechanisms for collecting info • Queue info, job info, node info – MDS, Ganglia, Clumon, PBS qstat,…. • All info expressed in schema-based XML • Secure (SSL, password protected) • Info is archived – Some averaged – Others are snapshots SchemaWizard Architecture • • • SW consists of an empty Web application, the SchemaParser, and Velocity Macro template files. First, SW unpacks and deploys a predefined and empty Web application into a Web server’s application repository. A schema whose location is provided by a user is read in to create an in-memory representation (SOM) and also to create Java files. – SOM=Castor’s Schema Object Model • • • Java files are compiled, and binaries are placed into the new project’s classes directory. Each schema element is mapped to a self-contained JSP nugget. JSP nuggets are generated from templates. – One template for each element type (simple, complex, enumerated, unbounded,….). – Velocity is used for convenient scripting of JSP. • • The final JSP page is an aggregate of the JSP nuggets files (using <%@:include>). Complex schema elements are mapped to JavaBeans generated from the schema with Castor. – Scripting templates set up the imports XML Form Wizard Project Page • create a session • initialize the environment • retrieve an XML instance list Complex Types New command list XML instance XML • create a complex type bean • include sub elements • if submitted, validate the content Index Page • create a root element bean as new or from existing XML • include sub element pages • if submitted, validate the bean content • if editing is complete, generate XML includes Simple Types • if parent bean has data, display content for this type (or property) using a form element • if form is submitted, update the parent bean for this type includes SchemaParser Data Flow and Actions Traverse schema for types Castor SOM Schema object Collect type information, create a context Individual types Decide template: JavaBeans info Velocity context with type info Templates Context, template Project page Index page Simple type Enumerated simple type Unbounded simple type Complex type Unbounded complex type Velocity Template Engine JSP SchemaWizard and XML A Mapping of XML to SchemaWizard Parts SchemaWizard Annotated XML Schema input of SchemaParser generates follows outputs XML Instances edited XML Form Wizard SAML example <?xml version="1.0" encoding="UTF-8"?> <ns1:AssertionSpecifier xmlns:ns1="http://www.oasis-open.org/committees/security/docs/draft-sstc-schema-assertion-27.xsd"> <ns1:Assertion MajorVersion="1" MinorVersion="0" AssertionID="22.214.171.124.1037385546507" Issuer="Gateway Web Portal" IssueInstant="2002-11-15T13:39:06.507-05:00"> <ns1:Conditions NotBefore="2002-11-15T13:34:06.518-05:00" NotOnOrAfter="2002-11-15T13:49:06.518-05:00"> <ns1:AudienceRestrictionCondition> <ns1:Audience>http://www.gatewayportal.org/agreement.xml</ns1:Audience> </ns1:AudienceRestrictionCondition> </ns1:Conditions> <ns1:AuthenticationStatement AuthenticationMethod="urn:ietf:rfc:1510" AuthenticationInstant="2002-11-15T13:39:06.558-05:00"> <ns1:Subject> <ns1:NameIdentifier SecurityDomain="www.gatewayportal.org" Name="cyoun"/> <ns1:SubjectConfirmation> <ns1:ConfirmationMethod>urn:ietf:rfc:1510</ns1:ConfirmationMethod> <ns1:SubjectConfirmationData>A Kerberos Ticket</ns1:SubjectConfirmationData> </ns1:SubjectConfirmation> </ns1:Subject> <ns1:AuthenticationLocality IPAddress="126.96.36.199" DNSAddress="grids.ucs.indiana.edu"/> </ns1:AuthenticationStatement> <ns1:AuthorizationDecisionStatement Resource="AccessLevel" Decision="Permit"> <ns1:Subject> <ns1:NameIdentifier SecurityDomain="www.gatewayportal.org" Name="cyoun"/> <ns1:SubjectConfirmation> <ns1:ConfirmationMethod>urn:ietf:rfc:1510</ns1:ConfirmationMethod> <ns1:SubjectConfirmationData>A Kerberos Ticket</ns1:SubjectConfirmationData> </ns1:SubjectConfirmation> </ns1:Subject> <ns1:Actions> <ns1:Action>5</ns1:Action> </ns1:Actions> </ns1:AuthorizationDecisionStatement> </ns1:Assertion> </ns1:AssertionSpecifier> Discovering Deployed Services • Our model deploys core services on distributed servers. – Need to find servers, their deployed services, WSDLs, service points. • This model has usual problems of distributed systems. – Network failures, machine crashes, software failures, topological defects. • Information services must be dynamic, up-to-date. – UDDI has had problems with data aging. • Such problems can be addressed by peer-to-peer and/or messaging technologies. – Let’s try using NaradaBrokering for distributed events – Narada has more extensive capabilities then used here. – Shrideep Pallickara, www.naradabrokering.org Integration of Security into Web Services • Authentication through single sign-on. Users – Kerberos, PKI – Distributed ticket system – Getting assertions about authentication, authorization, user attribute • • Authenticate Generating Assertions SOAP security should be provided through standard interfaces to specific mechanisms. General methods are Signing Assertions – Message signing. – Message integrity. – Message encryption. • • Kerberos, PKI are specific mechanisms. Assertion is an XML document describing the information about authentication acts performed by subjects, attributes of subjects and authorization decisions, Created with a specific mechanism. Security Mechanism Encryption Assertions SOAP HTTP Web Service HTTP Web Service …… …… HTTP Web Service Technical resources • Modified Apache Axis 1.0: SOAP Engine • Security assertion – SAML being standardized at OASIS is an XML-based security standard for exchanging authentication and authorization information. – SAML schema: draft-sstc-schema-assertion27.xsd • Kerberos: Version 5, Release 1.2.2 Bridging Between Client-Server and Messaging Services Browser Peers register themselves to Aggregator Blah blah blah Tomcat Server Tomcat Server Broker Aggregator HTTP Tomcat Server SOAP Dynamic User Interface Component Web service request for information Tomcat Server Servers run Narada Notifiers Tomcat Server The message structure of the SOAP request on the client-side. <soapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/ xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance> <soapenv:Header> <ns1:Saml xmlns:ns1=http://www.gatewayportal.org/sign.xsd> <ns1:SignedAssertiion> secure SAML Assertion message </ns1:SignedAssertion> <ns1:SecurityMechanism>Kerberos</ns1:SecurityMechanism> </ns1:Saml> </soapenv:Header> <soapenv:Body> <ns2:SignedBody xmlns:ns2=http://www.gatewayportal.org/signbody.xsd> secure SOAP body message</ns2:SignedBody> </soapenv:Body> </soapenv:Envelope> The client-side process • Convert SAML schema to Java classes – • Develop utility classes for creating assertions, marshalling them back and forth between Java and SAML. – • • • • • Castor can be used to easily convert between XML and Java data objects. Assertion attributes filled in by the appropriate mechanism. Login process: the authentication and getting the Kerberos ticket. Establish the security context with the server for getting the shared key. Generate user’s SAML security assertion. Sign the user assertion and SOAP Body messages. Rebuild the SOAP messages. The server-side process • • Establish the security context with client for getting the shared key. Handle the SOAP message. – – – – • • Unwrap the secure assertion It checks the validity of the assertions. – – – – • • Secure assertion message. Secure body message. Security mechanism name such as Kerberos, PKI. Message format such as SAML, WS-security. Issuer name “Conditions” time limit Subject name Authorization for accessing resources Unwrap SOAP body message Rebuild the SOAP message. Schema Wizard for MDWS (1) (2) (3) Maps Schema to Java classes, binds to WSDL Retrieves WSDL and generates specific interface for data Requests form pages Browser Blah blah blah HTTP Gathers user requests (4) Dynamic User Interface Component SOAP Maps entries to WSDL expression of schema and invokes dynamically (no stubs) (5) Metadata Service Assigns elements to Castor classes (6) Context Manager Architecture Client SOAP/HTTP Shared WSDL Interface Axis Servlet Context Manager Internal Communication Context Data FS XMLDB Miscellaneous Info • WSDL-to-Java mappings are defined by JSR 172. See http://www.jcp.org/en/jsr/detail?id=172. • Other language mappings for WSDL depend on appropriate standards body.