Exchange Server 2010 Planning and Sizing Exchange Deployment Planning Services Exchange 2010 Planning Goals The Exchange 2010 Planning module has the following goals: Introduce planning tasks and recommended configurations for Exchange 2010 Server View typical role placement scenarios for Exchange 2010 Server Exchange 2010 Planning Audience Ideal audience for this workshop Messaging SME Networking SME Security SME Exchange 2010 Planning Agenda In this module focus on the following: Overview of performance can scalability testing and direction Guideline and ratios Role specific details Toolkit for planning and sizing Exchange 2010 Planning After this module you should have: Basic planning knowledge for Exchange 2010 Basic understanding of Exchange 2010 Server role placement Overview of Testing and Guidance Process Agenda Research Evaluation Implementation • Understand hardware • Business requirements to technical requirements • Understand Exchange • Solution sizing, configuration, validation • Hardware selection • Migrate & grow • Exchange sizing and performance tuning can be complex − Use planning & sizing toolset to simplify − Take advantage of hardware advances to get the most out of Exchange Understand Hardware Driving Factors In The Technology Landscape • • • • • • Major changes in server hardware technology in the last decade have influenced changes in Exchange architecture Processor advances − 64-bit: massive amounts of addressable memory − Core density: cores per processor continues to grow Storage advances − SAS/SATA/SSD − Topologies: SCSI, FC, iSCSI, DAS − Growth in drive capacity (and areal density) Power consumption & efficiency Virtualization Cloud computing Processor Advances • • Overall available megacycles per processor (socket) increasing rapidly − Per-core megacycles constant or decreasing to maintain power requirements Processor technology improvements make GHz comparison somewhat meaningless Storage Advances Hardware • Since 2003, disk capacity has grown dramatically • • Workload • Mailbox sizes rapidly increasing (1-10GB desired) − 2TB desktop class SATA (and midline SAS) disks available, • Knowledge workers (and larger sizes available shortly IT) want everything online Sequential throughput and instantly searchable increasing linearly based − Reduced mailbox on areal density management effort − 2010 SATA =~ 250MB/sec Random I/O performance not expected to improve substantially − 15k RPM is the ceiling • − Data accessible from everywhere (incl. mobile client) − Increased knowledge worker productivity Average message size increasing Storage Terminology • Three classes of drive types − Enterprise (ENT) − − − − Dual port SAS interface 10K, 15K RPM 146, 300, 400, 450, 600GB + Large Form Factor (LFF) & Small Form Factor (SFF) − Midline (MDL) − − − − SAS – dual port SATA – single port 7200 RPM + LFF & SFF 500GB, 750GB, 1TB, 2TB + − Entry (ETY) – Not suitable for Exchange Business Requirements Business Requirements • • • • • • Information retention (size and duration) − Limited by Restore SLA − Regulatory requirements − Growth, mergers etc. Backup/restore strategy Site level disaster recovery − RPO/RTO Administrative model Consolidation and virtualization Power consumption Understand Exchange Scale Out vs. Scale Up • • • • Scale out is a strategic choice made by the product group Scale out provides the following at low cost: − Large mailboxes − High availability − Rich feature set Scaling up increases risk that an outage or failure affects more users Scaling up usually costs more, and can force feature decisions due to hardware choices − Consider all factors in the equation, particularly storage Scale Up Options • • • Multiple Role Servers (“brick” deployments) − Likely the best option for big hardware (> 2 socket) – best hardware utilization overall − Be aware of recommendations for max processor & memory Virtualization − Evaluate whether potential added complexity and monitoring challenges make this a win Single role − Product not engineered for single role high scale (> 2 socket) Extreme caution necessary – validate carefully in a test lab Supported vs. Recommended • • • • Supported usually means well tested Support statements define strict boundaries Recommendations define the “best case” or the state that we want our customers to achieve Understand risks of going outside of recommendations or support boundaries Processor Core Scalability • Single Role Servers − Recommend a 2-socket platform − − 4-core processors = 8 total cores 6-core processors = 12 total cores − Expect diminishing returns moving to 16+ cores on >= 4 socket platform − Known issues updating memory across cores • • − − Not NUMA aware or optimized for scale around data locality Code can take longer to execute; transaction costs rise Multiple Role Servers − Recommend 24 cores maximum for high-scale “Enterprise Multiple Role Server” − Multiple processes from different roles help us scale better Hyperthreading − Disable on production Exchange servers − Causes monitoring and capacity planning challenges Role Ratio Guidelines • Processor core ratios − CAS : Mailbox − =3:4 − HUB : Mailbox − = 1 : 7 (no A/V on Hub) − = 1 : 5 (with A/V Hub) − GC : Mailbox − = 1 : 4 (32-bit GC) − = 1 : 8 (64-bit GC) Processor and Memory Configuration Recommended Configuration Role Maximum Processor Cores Optimal Memory Hub & Edge Transport 12 cores 1GB per core (or 8GB minimum) Client Access Server 12 cores 2GB per core (or 8GB minimum) Mailbox 12 cores 4GB plus 3-30MB per mailbox Unified Messaging 12 cores 2GB per core (or 4GB minimum) Multiple Role Server 24 cores 8GB plus 3-30MB per mailbox Network Load Balancing • • • Exchange 2010 requires load balanced CAS for internal connections − Consider HA needs − Size for connection count spikes Windows NLB − Not recommended above 8 nodes Hardware Load Balancer − Recommended for larger environments − Multiple Role Server HA scenarios Sizing The Mailbox Role • Proper sizing is key across all resources: Resource Key Considerations Storage I/O and capacity requirements Memory Database cache requirements (reduces I/O) CPU Required for RPC operations, content indexing, mailbox assistants, replication operations Network • • Log replication and RPC operations consume bandwidth Storage & memory are most critical – ensure proper sizing for performance, capacity, reliability In depth detail available at http://tinyurl.com/262cpg9 Mailbox Role Sizing Rules Of Thumb • • • 2-socket platform best for performance and TCO User profile determines resource requirements for IOPS, memory, CPU Don’t forget about high availability Mailbox Storage Sizing • • • • • Storage must be sized for − Performance (IOPS) − Capacity (GB) Performance sizing based on user profile (message throughput) Capacity sized based on user mailbox size − See TechNet for details on required overhead (whitespace, dumpster, etc.) Design will either be performance- or capacity-bound IOPS guidance based on production observations and internal testing Estimated IOPS Per-Mailbox Messages Sent+Received per mailbox per day Database cache per mailbox (MB) Estimated IOPS: Single database copy Estimated IOPS: Multiple database copies 50 3 .060 .050 100 6 .120 .100 150 9 .180 .150 200 12 .240 .200 250 15 .300 .250 300 18 .360 .300 350 21 .420 .350 400 24 .480 .400 450 27 .540 .450 500 30 .600 .500 (~75KB average message size) Mailbox Memory Sizing • • • Services need base memory for ongoing operations: − Basic overhead for servicing user requests − Content indexing − Mailbox assistants Store process needs per-user memory for database cache, based on user profile − Properly sized database cache memory required for IOPS reduction Deep checkpoint depth + 32KB pages allow E2010 to benefit from larger memory configurations than E2K7 Mailbox Role Cache Memory Sizing Messages Sent+Received per mailbox per day (~75KB average message size) Database cache per mailbox (MB) 50 3 100 6 150 9 200 12 250 15 300 18 350 21 400 24 450 27 500 30 Mailbox Memory Sizing • • • Cache size defaults based on installed RAM − Size per-mailbox memory, then map to fit in default cache − Remaining memory reserved for base service requirements Nehalem platform has new rules for memory configuration − Haven’t seen a need to optimize for memory speed, so optimize for memory size For example: − 4000 users with the 200 profile (12MB per mailbox): 4000*12MB = 48GB − 48GB fits in 53.6GB default cache − Deploy 64GB server Default Mailbox Database Cache Sizes Server Installed Physical Memory Database Cache Size (Mailbox Role Only) Database Cache Size (Multi-role) 2GB 512MB Not supported 4GB 1GB Not supported 8GB 3.6GB 2GB 16GB 10.4GB 8GB 24GB 17.6GB 14GB 32GB 24.4GB 20GB 48GB 39.2GB 32GB 64GB 53.6GB 44GB 96GB 82.4GB 68GB 128GB 111.2GB 92GB Mailbox CPU Sizing • • • • Proper CPU sizing is critical: sizing of other roles depends on it Megacycle values provided are based on a particular reference platform, newer CPUs differ − Megacycle adjustment based on SPECint may be required Sizing process & calculation can get somewhat complex − Use calculator tools to simplify this process − See TechNet guidance for details on megacycle adjustments Recommend disabling hyperthreading − May cause capacity planning & monitoring challenges Estimated Per-Mailbox CPU Consumption Messages Sent+Received per mailbox per day Megacycles for active or stand-alone mailbox Megacycles for passive mailbox (~75KB average message size) (increase by 10% for each passive copy) 50 1 .15 100 2 .3 150 3 .45 200 4 .6 250 5 .75 300 6 .9 350 7 1.05 400 8 1.2 450 9 1.35 500 10 1.5 Sizing The Client Access Server Role • CPU and memory are key for CAS: Client Access Server Role Sizing Rules Of Thumb • • • • • 2-socket platform best for performance and TCO CPU is typically the bottleneck, memory sizing is key as well 3 CAS CPU cores for every 4 Mailbox CPU cores (servicing active users) Load balancing is important for performance and high availability 2GB RAM per CPU core is optimal Resource Key Considerations CPU Required for handling client workload transactions, content conversion, garbage collection Memory Memory required for ongoing transaction processing Network All clients connect to CAS, network bandwidth and latency are important for client experience, load balancing likely required Storage Utilized for content conversion, logging Client Access Server Workload Sizing • • • Workload impact on CAS server is variable depending on user CAS Workload Relative Cost profiles & mix of workloads Comparison CPU & memory scale guidance CPU Cost Network Cost Workload for CAS based on assumptions (MHz/user) (Kbytes/sec/user) of a mixed-protocol heavy Outlook 0.35 0.37 information worker profile Outlook Anywhere 0.80 0.44 − Consider other workloads and adjust Exchange ActiveSync 1.60 1.04 − Remember all MAPI traffic (delta from Outlook) now affects CAS Exchange Web Use Windows Server 2008 R2 Services (Microsoft 0.71 0.54 for best CAS scale Entourage) − Major improvements in Outlook Web App 0.86 0.88 rpcproxy (Outlook IMAP4* 0.86 0.14 Anywhere), potentially scaling to 15k Outlook POP3* 0.33 0.79 Anywhere users on 8-core CAS OWA SP1 Performance • Major SP1 investment on improving OWA performance − Pre-fetch: get data to the client before they ask for it − Async: make long running operations happen in the background − Defer: wait to download scripts when possible OWA Perf Improvement Results 100ms/5Mbps Scenario 200ms/512Kbps 300ms/50Kbps E2010(ms) SP1(ms) E2010(ms) SP1(ms) E2010(ms) SP1(ms) Logon (PLT1) 3,720 3,499 16,099 14,557 156,363 133,981 Logon (PLT2) 2,406 1,981 3,116 2,429 8,191 6,071 307 139 548 122 1,035 131 Preview a conversation 1,130 79 1,398 82 3,143 92 Mark as read or unread 257 35 467 22 1,125 30 Move or copy 340 123 622 145 1,029 150 Empty a tree folder 185 134 375 142 792 138 Attach 2.13 MB file 16,067 < 150 < 150 454,019 <150 Delete a message Sizing The Hub Transport & Edge Role • • • CPU and memory are key for transport: Resource Key Considerations CPU CPU required for message processing, hygiene activities, custom agents Memory Database cache requirements (reduces I/O), messages in queue represented in memory for perf Transport Roles Sizing Rules Of Thumb • Storage Low to moderate I/O requirement for relay & delivery activity, queuing needs more I/O + capacity • Network Bandwidth utilized to relay/deliver messages – scales with message volume, latency can cause queuing • Size storage capacity for queue requirements Use battery-backed write cache disk controller − Disk I/O can be a bottleneck on an untuned Hub − Log I/O becomes virtually free with a BBWC controller • 2-socket platform best for performance and TCO CPU is typically the bottleneck, memory sizing is key as well 1 Transport CPU core for every 7 Mailbox CPU cores (no A/V) or 1 Transport CPU core for every 5 Mailbox CPU cores (with A/V) 1GB RAM per CPU core is optimal Transport Write Cache Observations • • • Battery-backed write cache not absolutely necessary If workload is low enough, sequential nature of log I/O & ESE write behavior may allow running w/o BBWC Lab test can confirm if HW will sustain desired load Sizing The Unified Messaging Role • Unified Messaging Role Sizing Rules Of Thumb • • • • • 2-socket platform best for performance and TCO 2GB RAM per core is optimal CPU is typically the bottleneck, particularly when Voice Mail Preview is being used Default 100 concurrent calls per server (inbound or outbound) Voice Mail Preview is CPU intensive: ~1 message/min/core • • CPU and network are key for UM: Resource Key Considerations CPU CPU used for media operations and Voice Mail Preview transcription – UM is typically “CPU heavy” Network Network bandwidth used for calls as well as communication with Mailbox role. Minimize latency for best user experience Memory Memory required for ongoing transaction processing Storage UM doesn’t have significant storage requirements Scale out UM servers based on concurrent call requirements Size CPU based on requirements for Voice Mail Preview Sizing Multi-Role Hub/CAS Servers • • Potentially optimal hardware utilization Multi-Role Hub/CAS Sizing Rules Of Thumb − Server consolidation – minimize physical servers • Simplified sizing − Hub and CAS roles are relatively well balanced given resource requirements − Virtualization – simplify server configuration with Hyper-V 4-core VMs (8-core physical = 1 4-core Mailbox, 1 4-core Hub/CAS) CAS/HUB CAS/HUB CAS/HUB CAS/HUB CAS/HUB CAS/HUB Mailbox MailboxMailbox MailboxMailboxMailbox • • • 2-socket platform best for performance and TCO CPU is typically the bottleneck, memory sizing is key as well 1 Hub/CAS CPU core for every 1 Mailbox CPU core 2GB RAM per CPU core is optimal Sizing Multi-Role “Brick” Servers • Multi-Role “Brick” Servers Sizing Rules Of Thumb • • Recommend maximum 4socket platform for multirole deployment Use 8GB RAM plus 330MB per mailbox (see Mailbox role sizing details) • • • Mailbox, CAS, and Hub Transport roles recommended − Excellent solution for high core configurations Half of cores for Mailbox, half for CAS+Hub Use 8-24 cores − • UM supported, but not recommended 8GB RAM plus 3-30MB/mailbox recommended (follow mailbox database cache sizing guidance) Typical deployment scenarios: − Simple unit of scale (brick) model − − − Each multi-role server represents a building block Servers with on-board SATA storage (10-16 disks) are optimal Small organization/branch office – server consolidation − Minimize the number of physical servers, operating system instances, and Exchange server instances to manage Sizing Virtualized Server Roles • • • • Exchange isn’t virtualization “aware” – VM is just a different hardware platform TechNet is the best source of support guidance and best practices http://tinyurl.com/26k4g5j http://tinyurl.com/5abmlh Server Virtualization Validation Program (SVVP) Support Policy Wizard helps to determine supported configurations Virtualized Server Roles Sizing Rules Of Thumb • http://tinyurl.com/lyko6t Be aware of major support limitations − Root clustering + DAG − Unified Messaging role − Snapshots & differencing disks • • • Size for physical resources, add ~12% CPU overhead for hypervisor Avoid resource oversubscription Don’t co-locate Mailbox databases on a root server CAS+Hub combination can make scale calculations easy Capacity Planning Tools 38 Capacity Planning Tools • • • Profiling − EPA 2010 Sizing − Mailbox Server Role Requirements Calculator Validation − Jetstress 2010 − Exchange Load Generator 2010 “Loadgen” Exchange Profile Analyzer 2010 • • • • • Generates statistical profile of user actions − Messages sent and received/day − Rule counts − Item size and counts Inputs − Crawls mailboxes with MAPI (previously DAV) − OWA log analysis tool and “summarizer” Accuracy somewhat dependent on how users manage their mailbox Updated version expected Q3 CY10 Download existing version from http://go.microsoft.com/fwlink/?LinkId=80471 Mailbox Role Requirements Calculator • • • Follows Product Group recommendations on: − Storage − Memory − Mailbox sizing Goal of the calculator is to output: − I/O requirements − Capacity requirements − LUN design Available today via the Exchange team blog: http://msexchangeteam.com/ Jetstress 2010 • • • • • Exchange I/O simulator − Uses Jet (ESE) database engine Analyzes server I/O performance for Exchange requirements What can Jetstress be used for? − Storage performance validation − Storage reliability testing − End-to-end testing of storage components What can’t Jetstress be used for? − Validation of client experience − Integration testing with third party software solutions Download from http://www.microsoft.com/downloads/en/details.aspx?Fami lyID=13267027-8120-48ed-931b-29eb0aa52aa6 Jetstress 2010 What’s new • • • • • Updated with Exchange 2010 Mailbox I/O Profile − This profile is not yet final and is subject to change between now and Exchange 2010 release Database duplication is now multi-cast − Dramatically reduces the time to prepare databases for testing Now using MSExchange Database I/O counters for I/O measurement − Allows placing databases and logs on the same volume Log replication I/O is simulated based on Exchange 2010 HA architecture Background Database Maintenance (Checksum) is now simulated Exchange Load Generator 2010 • • • • • • The only supported multi-protocol load generator for Exchange − Replaces Loadsim and ESP Windows UI interface as well as a command-line interface Both task-based and scripted simulation modes Consumed both internally at Microsoft and externally Existing modules include: Outlook 2003/2007 (online and cached), POP, IMAP, SMTP, OWA, ActiveSync Download from http://www.microsoft.com/downloads/en/details.aspx?Fami lyID=cf464be7-7e52-48cd-b852-ccfc915b29ef Exchange Load Generator 2010 What’s new • • • • • • Requires Windows® Vista, Windows® 7 or Windows 2008 OS (SP2/R2) No longer requires Exchange Management Tools ActiveSync Module Dynamic mail generator − No need for message files, available in 5 languages, supports attachments NSPI connections UI enhancements Tools Process Flow User Profile User Profile Exchange Tested Solutions Exchange Tested Solutions • • • • • A program designed to showcase well-designed, well-tested, and cost effective Exchange 2010 solutions Includes latest and greatest server and storage products from our partners Highlights key Exchange 2010 features alongside key IHV enabling technology Can be used to accelerate hardware purchasing decisions Targeting Q4CY10 for release to TechNet Exchange Solution Reviewed Program • • • • • ESRP encourages storage vendors to properly test storage solutions with Exchange workloads Results are published in the form of solution whitepapers for a particular user count & profile Overall test process validated by Microsoft prior to posting link to completed whitepaper Currently 24 Exchange 2010 solutions posted from Dell, EMC, HDS, HP, IBM, NetApp, and Xiotech http://tinyurl.com/yhlmbhg Migrate & Grow My System Is Stable…Now What? • • • Monitor, measure, analyze − Consider System Center Operations Manager to monitor performance − Keep an eye on user profile – changes may require additional hardware − Watch performance data trends for capacity planning Keeping the system balanced is key to optimal hardware utilization − Balance heavy users across databases − Ensure that DAG active copies are balanced across mailbox servers − CAS workloads should be load balanced across array − Storage should be balanced for capacity and I/O throughput Consider eliminating hard boundaries − Cloud services may help reduce cost/complexity – evaluate TCO Design Session Architectural Design Session End of Exchange 2010 Planning and Sizing Module For More Information • Links to follow… © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.