Virtual Reality Movies Real-Time Streaming of 3D Objects
Stephan Olbrich, Helmut Pralle
RRZN/RVS, University of Hannover
TERENA-NORDUnet Networking Conference (TNNC)
Lund, Sweden, 07.-10.06.1999
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
1
Overview


Introduction
Requirements
 Tele-immersive, virtual reality environment
 Series of complex 3D scenes: result of scientific visualization
 High-performance online presentation

Solution
 Generation of 3D scenes: optimized „DVR” file format
 Streaming server: play-out 3D animations
 Efficient „DVR“ viewer: Inline-Plugin for Netscape

Evaluation
 3D streaming application: „Oceanic Convection“
 Local gigabit ethernet configuration

Future work
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
2
Introduction

Distributed multimedia information systems
 WWW - Based on Internet Standards (RFCs), e. g.:
— Protocols - TCP/IP
— Addressing Schemes - URL
— Services - HTTP
— Hypermedia Content - MIME, HTML, GIF, JPEG, MPEG, VRML, etc.
 Client - Generic browser
— Netscape Communicator, Microsoft Internet Explorer
— Extension techniques - Helper applications, Plugins, ActiveX Control, Java

3D/Virtual Reality Technology - application areas
 Ergonomic user interfaces
 Method to navigate in an information space
 Online presentation of virtual 3D scenes, for example:
— Reproduction of real objects
— Artificial scenes, such as Scientific Visualization of HPC results
 ISO Standard: Virtual Reality Modeling Language (VRML97)
— Container format for multimedia content in the Web
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
3
Requirements (1)
3D/Virtual Reality Technology - Applications

Information Spaces / 3D User Interfaces
 Intuitive human-machine-human interface
 Ergonomic navigation

Visualization of geometric objects
(modeling / reconstruction)
 Design: „virtual prototyping“
 Arcitecture, environment planning

Visualization in scientific computing
 Experiment / Theory / Simulation
 Virtual experiments
— Exploration and demonstration of results
— Increase of insights / Teleteaching / Telelearning
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
4
Requirements (2)
Visualization in Scientific Computing
Supercomputer
e.g. Cray T3E
Simulation
Raw data,
e.g. temperature
Postprocessing
3D / VR
visualization
systems
Symbolic
representation
VR System
Multimedia
representation
VE - Virtual
Environment
e.g. AVS on
SGI Onyx2
Viewer
Display
Visual
Acoustic
Haptic
2D, 3D
Interaction
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
5
Requirements (3)
Tele-Immersive Distributed Virtual Environment
Visualization B
Visualization A
Synchronized
navigation,
annotation,
conferencing, etc.
Request
Request
3D objects,
HPC results,
etc.
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
data source(s):
WWW, Supercomputer
3D objects,
HPC results,
etc.
TNNC ‘99
08.06.1999
6
Requirements (4)
Constraints of available Internet-based 3D
systems


Several limits of VRML format and viewers
Especially in the context of typical scientific
and industrial research environments with
 High quality application requirements, e. g.
handling objects with high complexity (> 100,000 polygons)
 Network infrastructure offering high bitrates, such as local networks
 High performance server and client systems

Constraints regarding performance, quality, and
functionality aspects prohibit useful application





unacceptable delays
no progressive presentation
low frame rates
low rendering quality
little support of virtual reality presentation and interaction techniques
 no 3D streaming capabilities
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
7
Requirements (5)
Analysis and Classification
Pre s e n t a t io n
Re p re s e n t a t io n
 N e t w o r k In f r a st r u ct u r e
 Tr a n sp o r t P r o t o co ls
 Se r vice s, e . g . W W W :
 H TTP , U RL




 Sh o r t la t e n cy
 St a r t u p
 N a vig a t io n
 H ig h f r a m e r a t e s
 Re n d e r in g r a t e




 Clie n t -sid e co m p u t a t io n
 D e co d e
 D e co m p r e ss
 P a r se
 Bu ild sce n e g r a p h
 D a t a vo lu m e s
 Q u a lit y o f se r vice
Q u a lit y
 Re so lu t io n
 A n t ia lia sin g
 Co lo r m a n a g e m e n t
 Re so lu t io n
 Co m p r e ssio n m e t h o d
 Co lo r m a n a g e m e n t
Fu n ct io n a lit y
 O n -t h e -f ly
p r e se n t a t io n
 VR p r o je ct io n
syst e m s
 N a vig a t io n co m f o r t
 3 D se n so r s
 St r e a m in g
 H TTP , RTSP , RTP
 Syn ch r o n iz a t io n
 M e d ia -sp e cif ic sca lin g
 A t t r ib u t e s f o r VR
p r e se n t a t io n
 Se q u e n ce s o f 3 D o b je ct s
 Vo o m ie (Vir t u a l M o vie )
S t a n d a rd s
Pe rf o rm a n ce




Co m m u n ica t io n
W W W Br o w se r
P lu g in -A P I
Vie w e r , P lu g in s
G r a p h ics-/G U I-A P Is
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
H ig h b it r a t e s
Sh o r t la t e n cy
Lit t le jit t e r
O p t im iz e d
im p le m e n t a t io n
W W W Se r ve r
CG I-, P r o xy-Te ch n iq u e s
St o r a g e
M M /VR File a n d St r e a m
Fo r m a t s
TNNC ‘99
08.06.1999
8
Requirements (6)
Complexity of typical rendering primitives


N independent triangles
with normals
Volume: N * 72 byte




N triangles as triangle strips
with normal vectors
Volume: (N+2) * 24 byte
4,3 Mio. triangles/s  800
Mbit/s
(SGI Onyx2 Infinite Reality)
Per vertex data:
 3D coordinates: 3 float values per vertex  3 * 4 byte / vertex
 Normal vector: 3 float values / vertex  3 * 4 byte / vertex
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
9
Solution (1a)
Partitioning/Distributing a Virtual Reality system
Local
Scenenavigation
update loop
(a) local file
Select
access
new
scene
Parse
2D
Render
3D
HighLevel
Traverse/
Optimize
LowLevelTraverse
Navigation
Loop
Position,
Orientation,
Button states
Scene repr. Scene graph Display list
File
Memory
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
Memory
TNNC ‘99
08.06.1999
10
Solution (1b)
Partitioning/Distributing a Virtual Reality system
Scene update
(b) WWW integration: VRML viewer
Helper application
get URL
HTTP
Select
Render
exec
WWW
Server
WWW
Browser
Scene repr.
Scene repr.
File
File
HighLevel
Traverse/
Optimize
Parse
LowLevelTraverse
Scene graph
Display list
Memory
Memory
Navigation
Loop
Position,
Orientation,
Button states
VRML
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
11
Solution (1c)
Partitioning/Distributing a Virtual Reality system
Scene update
(c) Preprocess VRML to „DVR“ format
Helper application
get URL
HTTP
exec
WWW
Server
WWW
Browser
Scene repr.
Scene repr.
File
File
VRML
Select
Render
VRML-to-DVR
HighLevel
Traverse/
Optimize
Parse
LowLevelTraverse
Scene graph
Display list
Memory
Memory
Navigation
Loop
Position,
Orientation,
Button states
„DVR“ format
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
12
Solution (1d)
Partitioning/Distributing a Virtual Reality system
Scene update
(d) Our „DocShow-VR“ Plugin for DVR format („thin client“)
Plugin (DVR)
NPN_GetURL
Select
Render
HTTP
WWW
Server
WWW
Browser
NPP_Write
Display list
Memory buffer
LowLevelTraverse
Navigation
Loop
Position,
Orientation,
Button states
Scene repr.
File
DV
R
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
13
Solution (2)
„Play-out“ of 3D animations: streaming system
Real-time streaming „DVRS“ Plugin, RTSP/DVRP protocols
Plugin (DVRS)
WWW
Server
HTTP
WWW
Browser
Render
Meta data
RTSP
Streaming
Server
Double buffer
Streaming
Client
Display list
LowLevel
Traverse
Navigation
Loop
DVRP
Play, Stop, Slow motion
3D scene repr.
DVR
Files
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
14
Solution (3)
Performance issues

Optimized 3D file format
 Binary representation - IEEE format / network byte ordering
 Preprocessing of VRML data, e. g.:
— ASCII cleartext to binary format
— Linearization of scene graph (objects, attributes, transformations, lights)
— Restructuring, precalculation for efficient and progressive rendering support




VRML: Coordinate lists / indexed references Store and transmit direct values
VRML: IndexedFaceSet - Polygons Triangles Triangle strip
Calculation and storage of normal vectors
High-performance viewer (Netscape inline plugin)
 Tightly coupled into data delivery
— Netscape Plugin API
— Real-time streaming, initiated by special meta data MIME type („DVRS“)
 Efficient 3D rendering implementation, based on OpenGL
 Real-time streaming of sequences of 3D scenes: „Virtual Reality Movies“
— Control protocol: RTSP (RFC 2326)
— 3D data transport: „DVRP“
— Multithreaded: communication, 3D rendering (multi-pipe)
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
15
Solution (4)
Other issues

Quality of Presentation
 Multisampling antialiasing support (HW-accelerated OpenGL extension)
 Colour management support (device-independant presentation)

Functionality
 Stereoscopic viewing with shutter glasses or stereo projection
 Tracking devices
 Multi-platform support (Win32, UNIX)

Interoperability to ISO Standard VRML97
 Preparing files via VRML-to-DVR converter
— VRML97 subset supported: static scenes
— Command line tool / Web service via HTML form, VRML upload, DVR download
 Caching proxy/gateway technique
— transparent access to any VRML file
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
16
Evaluation (1)
Testbed setup




Client: SGI Onyx2 Infinite Reality (2x à 2 RM), 4xR10000, 195 MHz
Server: SGI Origin200, R10000, 225 MHz via Gigabit-Ethernet
Network: Gigabit Ethernet (back to back), Alteon Jumbo Frames
Application scenario:
„Oceanic convection“
 Supercomputer
simulation results
— 161x161x31 grid
— 740 time steps
 Visualization of
temperature, velocity
—  100,000 primitives
— 8 Mbytes / scene
 Frames 720-739 used
for measurement
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
17
Evaluation (2)
Measurement
[M b p s]
Clie n t
SG I O n y x 2
Se r v e r 1
SG I O n y x 2
Se r v e r 2
SG I O r ig in 2 0 0
Re a d D VR f ile s
(ca ch e d )
—
250—350
(9 8 0 — 1 0 2 0 )
109—110
(1 0 9 0 — 1 1 7 0 )
TCP /IP
t r a n sm issio n
—
790—950
(lo o p b a ck )
560—670
(g ig a b it e t h e r n e t )
3 D r e n d e r in g
(O p e n G L)
380—420
—
—
St r e a m in g
— M a x. f r a m e r a t e
—
280—320
324—326
St r e a m in g
— 4 fra m e s / s
—
251
251
St r e a m in g
— 2 fra m e s / s
—
126
126
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
18
Evaluation (3)
Illustration

Configuration





Client: SGI Onyx2 Infinite Reality
Server: SGI Origin200
Communication: Gigabit Ethernet, Jumbo Frames
Dataset: „Oceanic Convection“, frames nr. 720-739
1 Graphic Pipe
 Maximal frame rate
 4 frames per second
 2 frames per second

2 Graphic Pipes
 Maximal frame rate
 4 frames per second 
 2 frames per second
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
19
Future work

Preparation of scenarios in wide-area networks
 Presentation (Prepared 3D scenes, as described)
 Exploration (Steering / High-Performance Computing - HPC)
 Discussion (Computer-Supported Cooperative Work - CSCW)

Support of an exploration scenario
 Steering of a computation on a supercomputer
 Dynamic scene generation
 Low-latency, on-the-fly transmission and presentation

Support of cooperative working




Synchronization of navigation
Telepointer, annotations
Synchronized play-out of 3D animations
Streamed, spatial video and audio integration
— Video texture mapping on a rectangle in 3D space
— e. g. SGI Onyx2 IR: DIVO/GVO - ITU-R 601 „D1“ digital video interfaces
— e. g. SGI Onyx2 IR: ADAT - 8-channel digital audio interface
Virtual Reality Movies - Real-Time Streaming of 3D Objects
Stephan Olbrich
© 1999 RRZN/RVS, University of Hannover
TNNC ‘99
08.06.1999
20
Descargar

Virtual Reality Movies - Real-Time Streaming of 3D Objects