Patch Warfare & Security Incident Response Microsoft Corporation Presented by Robert Hensing - PSS Security Specialist Agenda Situation Solution Components Roadmap Security Incident Response Customer Feedback Inadequate Communications, Guidance, and Training Inconsistent Patching Experience Reduce Frequency, Quantity of Patches Multiple, Incomplete Patch Management Tools Inconsistent Patch Quality Addressing The Situation Security and patch management priority #1 – bar none – at Microsoft Microsoft problem Industry problem Ongoing battle with malicious hackers Microsoft taking a comprehensive, tactical and strategic approach to addressing the situation Patch Management Initiative Progress to Date (July 2004) Informed & Prepared Customers Rationalized patch severity rating levels Better security bulletins and KB articles Security Guidance Kit; Patch Management guidance, etc. Security Mobilization Initiative – 500K IT Pros trained Consistent & Superior Update Experience Standardized patch and update terminology Standardized patch naming and installer switch options* Installer consolidation plan in place – will go from ~8 to 2 Reduced patch release frequency from 1/week to 1/month Superior Patch Quality Improved patch testing process and coverage Expanded test process to include customers Reduced reboots by 10%; reduced patch size by up to 75%** Best Patch & Update Management Solutions Released SMS 2003 which delivers expanded patch and update management capabilities Released MBSA 1.2 which integrates Office inventory scanning Windows Update Services in development More on the deliverables of the Patch Management Initiative in the Roadmap Section of this presentation… *Update.exe now using standardized switches; Windows Installer will use these in MSI 3.0 **75% for Windows Update installs, more than 25% for other patches Terminology Name Description Distribution Private Fix An unofficial fix which may not be fully tested or packaged. It is released to the customer to verify that it solves the problem before final testing & packaging. Limited to the customer who reported the problem. Hotfix A single cumulative package composed of one or more files used to address a defect in a platform. Limited to customers who contact Microsoft Product support services and are experiencing the specific problem. Update A broadly released fix for a specific problem addressing a non-critical, non-security related bug. Publicly available for download. Critical Update A broadly released fix for a specific problem addressing a critical, non-security related bug. Publicly available for download. Security Patch A broadly released fix for a specific platform addressing a security vulnerability. Publicly available for download. Update Rollup A cumulative set of hotfixes, security patches, critical updates and updates packaged together for easy deployment. A rollup targets a specific area such as "security" or component of the platform such as "IIS". Publicly available for download. Service Pack A cumulative set of all hotfixes, security patches, critical updates, and updates created and fixes for issues found internally since the release of the platform. Service packs may also contain a limited number of customer requested design changes or features. Service packs are broadly distributed and therefore tested heavily. Publicly available for download. Naming Standards 824685 - Description of the File Names That Are Used for Microsoft Product Updates, Tools, and Add-ins http://support.microsoft.com/?kbid=824685 The standardized file naming schema that Microsoft is adopting for packages that contain product updates, tools, and add-ins uses the following format: ProductNameKBArticleNumber-Option-Language.exe WindowsXP-KB123456-IA64-ENU.exe - An update for the English (US)-language version of Microsoft Windows XP for computers with 64-bit Intel processors. The update is associated with Microsoft Knowledge Base article 123456. OfficeXP-KB123456-Client-ENU.exe - An update for the English (US)language version of Microsoft Office XP. The update is associated with Knowledge Base article 123456. SQL2000-KB123456-8.00.0000-JPN.exe - An update for the Japanese-language version of Microsoft SQL Server 2000 Build 8.00.000. The update is associated with Knowledge Base article 123456. Bulletin Severity Rating System Rating Definition Customer Action Critical Exploitation could allow the propagation of an Internet worm such as Code Red or Nimda without user action Apply the patch or workaround immediately Important Exploitation could result in compromise of the confidentiality, integrity, or availability of users’ data, or of the integrity or availability of processing resources Apply patch or workaround as soon as is feasible Moderate Exploitability is mitigated to a significant degree by factors such as default configuration, auditing, need for user action, or difficulty of exploitation Evaluate bulletin, determine applicability, proceed as appropriate Exploitation is extremely difficult, or impact is minimal Consider applying the patch at the next scheduled update interval Low Revised November 2002 More information at http://www.microsoft.com/technet/security/policy/rating.asp Prioritizing and Scheduling the Release A Serious Problem Decreasing time in which to deploy a patch Decreasing Time To Patch (Blaster) July 1, 2003 Vulnerability reported to us / Patch in progress July 16, 2003 July 25, 2003 Bulletin & patch available No exploit Exploit code in public Report Bulletin Vulnerability in RPC/DDOM reported MS activated highest level emergency response process Exploit MS03-026 delivered to customers (7/16/03) Continued outreach to analysts, press, community, partners, government agencies X-focus (Chinese group) published exploit tool MS heightened efforts to get information to customers Blaster shows the complex interplay between security researchers, software companies, and hackers Aug 11, 2003 Worm in the wild Worm Blaster worm discovered –; variants and other viruses hit simultaneously (i.e. “SoBig”) Decreasing Time To Patch (Sasser) April 13 April 24-29 Bulletin & patch available No exploit Bulletin April 30 Worm in the wild Exploit code in public Exploit MS03-026 delivered to customers (7/16/03) Continued outreach to analysts, press, community, partners, government agencies Worm Reverse shell code posted to various web sites Sasser worm discovered. Multiple variants hit simultaneously Sasser shows the continually shrinking window between the time a patch is released, exploit code is generally available and a worm is written to exploit it. Solution Components Solution Components Prescriptive Guidance Analysis Tools Online Update Services Content Repositories Management Tools Microsoft Guide to Security Patch Management Patch Management Using SUS Patch Management Using SMS Microsoft Baseline Security Analyzer (MBSA) Office Inventory Tool* Windows Update Office Update Windows Update Catalog Office Download Catalog Microsoft Download Center Automatic Updates (AU) feature in Windows Software Update Services (SUS) Systems Management Server (SMS) *Office Inventory Tool is no longer needed – MBSA 1.2 (released in January 2004) includes Office scanning functionality Update Management Guidance Implementing a consistent, high quality update management process is the key to successful update management Microsoft delivers best practices prescriptive guidance for effective update management Assess Deploy Identify Evaluate & Plan Uses Microsoft Operations Framework (MOF) Based on ITIL* (defacto standard for IT best practices) Details requirements for effective update management: Technical & operational pre-requisites Operational processes & how technology supports them Daily, weekly, monthly & as-needed tasks to be performed Testing options Three update management guidance offerings Microsoft Guide to Security Patch Management** Patch Management using Software Update Services*** Patch Management using Systems Management Server*** *Information Technology Infrastructure Library **Emphasizes security patching & overall security management ***Comprehensive coverage of patch management using the specified technology MBSA Helps identify vulnerable Windows systems Scans for missing security patches and common security mis-configurations Scans various versions of Windows and other Microsoft applications New Update Assess Identify Evaluate & Plan Deploy Scans local or multiple remote systems via GUI or command line invocation Generates XML scan reports on each scanned system Runs on Windows Server 2003, Windows 2000 and Windows XP Integrates with SUS & SMS MBSA: How It Works* 1. Run MBSA on Admin system, specify targets Microsoft Download Center 2. Downloads CAB file with MSSecure.xml & verifies digital signature MSSecure.xml MSSecure.xml contains • Security Bulletin names • Product specific updates • Version and checksum info • Registry keys changed • KB article numbers • Etc. 3. Scans target systems for OS, OS components, & applications 4. Parses MSSecure to see if updates available 5. Checks if required updates are missing 6. Generates time stamped report of missing updates MBSA Computer SUS Server *Only covers security patch scanning capabilities, not security configuration detection issues Windows Update (WU) Microsoft online update service (windowsupdate.microsoft.com): Identifies missing Windows OS* patches / updates on accessing computer Generates targeted list of missing updates Installs user selected missing updates Provides update installation history New Update Assess Identify Evaluate & Plan Deploy WU content can be automatically downloaded via Automatic Updates Supplemented by Windows Update Catalog site which provides: Comprehensive repository for all Windows and ‘Designed for Windows’ logo device driver updates Search – to find desired update Manual download of desired updates Download history for accessing computer *Windows 98 and later versions. Note: also updates 64-bit editions of Windows Server Windows Update: How It Works Scenario 1: User Initiated Access Scenario 2: Access via Automatic Updates (AU) 1. User points browser to WU site & selects ‘Scan for updates’ or AU automatically checks for new updates (every 17-22 hours) Windows Update 2. Client side code (CC) in browser (or AU) validates WU server & gets download catalog metadata 3. CC (or AU) uses metadata to identify missing updates 4. WU (or AU -- if so configured) lists missing updates and user selects updates to download 5. CC (or AU) downloads, validates, & installs updates. AU downloads using BITS, and can be configured to allow user to select updates to install 6. CC (or AU) updates history & statistics information* *Note: No personally identifiable information is collected. See http://v4.windowsupdate.microsoft.com/en/about.asp#privacypolicy SUS 1.0 Deploys Windows security patches, security rollups, critical updates, and service packs only Deploys above content for Windows 2000, Windows Server 2003 and Windows XP only Provides patch download, deployment, and installation configuration options New Update Bandwidth optimized content deployment Assess Provides central administrative control over which patches can be installed from Windows Update Identify Provides basic patch installation status logging Evaluate & Plan Deploy SUS 1.0: How It Works Windows Update Service Windows Update Service 1. SUS Server check for updates every 24 hours* 2. Administrator reviews, evaluates, and approves updates 3. Approvals & updates synced with child SUS servers** 4. AU (the SUS client) gets approved updates list from SUS server Parent SUS Server 5. AU downloads approved updates from SUS server or Windows Update Child SUS Server Child SUS Server 6. AU either notifies user or autoinstalls updates 7. AU records install history *Configurable 1/day or 1/week **SUS maintains approval logs & download, sync, & install statistics SUS Client Component: Automatic Updates Centrally configurable to get updates either from corporate SUS server or Windows Update service Can auto-download and install patches under admin control Consolidates multiple reboots to a single reboot when installing multiple patches Included in Windows 2000 SP3, Windows XP SP1, and Windows Server 2003 Localized in 24 languages SUS Server Component: SUS Server Downloads updates from Windows Update Web based administration GUI Specify server & update process configuration options View downloaded updates Approve updates & view approved updates Security by design and default Requires NTFS; Installs IIS Lockdown and URL scanner* Supports secure administration over SSL Digital signatures on downloaded content validate authenticity Uses HTTP for content synchronization – only port 80 needs to be open Server side XML based logging on Web server Patch deployment & installation statistics Supports geographically distributed or scale-out deployments with centralized management for content synchronization & approvals Localized** in English & Japanese *If not already installed **Note: Delivers updates for all 24 supported client languages SMS 2003 Identifies & deploys missing Windows and Office security patches on target systems Can deploy any patch, update, or application in Windows environments Inventory management & inventory based targeting of software installs New Update Assess Install verification and detailed reporting Flexible scheduling of content sync & installs Central, full administrative control over installs Bandwidth optimized content distribution Identify Evaluate & Plan Deploy Software metering and remote control capabilities SMS 2003 Patch Management: How It Works Microsoft Download Center 1. Setup: Download Security Update Inventory and Office Inventory Tools; run inventory tool installer 2. Scan components replicate to SMS clients 3. Clients scanned; scan results merged into SMS hardware inventory data 4. Administrator uses Distribute Software Updates Wizard to authorize updates SMS Distribution Point SMS Site Server 5. Update files downloaded; packages, programs & advertisements created/updated; packages replicated & programs advertised to SMS clients SMS Clients SMS Distribution Point SMS Clients 6. Software Update Installation Agent on clients deploy updates 7. Periodically: Sync component checks for new updates; scans clients; and deploys necessary updates SMS Clients SMS 2003 Patch Management: Functionality System scanning & patch content download Content from Microsoft Download Center MBSA & Office Inventory plug-ins scan for missing patches Supports updating of remote & mobile devices Updates various versions of Windows, Office, SQL, Exchange, and Windows Media Player without need for update packaging / scripting Administrator control Update targeting based on AD, non-AD groups, WMI properties; additional options via scripting Patches content is downloaded from a central SMS repository only when the deployment process is initiated by the SMS administrator Specific start and end times (change windows); multiple change windows Easily move patches from testing into production Reference system patch configurations can be used as a template to verify or enforce compliance of systems that must mimic reference system configuration SMS 2003 Patch Management: Functionality (2) Patch download & installation Delta replication (site-site, server-server) of patches Uses BITS* for mobile / remote client-server Uses SMB* for LAN / priority situations Reminders and rescheduling of install / reboot & enforcement dates Optimized graceful reboots, but forced when enforcement date arrives Per-patch reboot-needed detection to reduce reboots Status & Compliance Reporting Deployment status as patches are attempted Standard and customized reports through read-only SQL queries Determine actual baselines in the environment before changing the environment SLA measurement and rate-of-spread *Requires SMS Advanced Client Choosing A Patch Management Solution Needs-Based Selection Adopt the solution that best meets the needs of your organization Core Patch Management Capabilities Capability Windows Update SUS 1.0 SMS 2003 Supported Platforms for Content NT 4.0, Win2K, WS2003, Win2K, WS2003, WinXP WinXP, WinME, Win98 NT 4.0, Win2K, WS2003, WinXP, Win98* Supported Content Types All patches, updates (including drivers), & service packs (SPs) for the above Only security & security rollup All patches, SPs & updates for the patches, critical updates, & above; supports patch, update, & SPs for the above app installs for MS & other apps Targeting Content to Systems No No Yes Network Bandwidth Optimization No Yes Yes Patch Distribution Control No Basic Advanced Patch Installation & Scheduling Flexibility Manual, end user controlled Admin (auto) or user (manual) Administrator control with controlled granular scheduling capabilities Patch Installation Status Reporting Assessing computer history only Limited Comprehensive Granularity of Control (for patch deployment) (client install history & server based install logs) (for patch deployment & server sync) (install status, result, and compliance details) Additional Software Distribution Capabilities Deployment Planning N/A N/A Yes Inventory Management N/A N/A Yes Compliance Checking N/A N/A Yes *MBSA does not support scanning Win98 – Win98 can be updated using SMS2003 inventory management and software distribution capabilities Choosing A Patch Management Solution Typical Customer Decisions Customer Type Large or Medium Enterprise Small Business Consumer Scenario Customer Chooses Want single flexible patch management solution with extended level of control to patch & update (+ distribute) all software SMS Want patch management solution with basic level of control that updates Windows 2000 and newer versions* of Windows** SUS Have at least 1 Windows server and 1 IT administrator** SUS All other scenarios All scenarios *Windows 2000, Windows XP, Windows Server 2003 **Customer uses Windows Update or manual process for other OS versions & applications software Windows Update Windows Update What could be better than patching? Not having to patch . . . Introducing Slipstreaming! Slipstreaming “Slipstreaming” – Integrating a patch into a product installation directory Windows, Internet Explorer, and Office support “Slipstreaming” It’s so simple! An example . . . Copy Windows 2000 CD to network share “Slipstream” Service Pack 4 into the share “Slipstream” all post-SP4 critical security updates into the share Perform network / RIS installation of Windows 2000 from that share Fully patched after setup completes! Slipstreaming For instructions on “slipstreaming” service packs – consult the deployment guide for the service pack you are deploying http://www.microsoft.com/windows2000/dow nloads/servicepacks/sp4/default.asp For instructions on “slipstreaming” hotfixes and udpates – consult the hotfix deployment guide http://www.microsoft.com/windows2000/dow nloads/servicepacks/SP4/HFDeploy.htm Finding critical security updates to slipstream Subscribe to the Security Alert Notification Service We’ll tell you when critical updates are available! http://www.microsoft.com/security/security_bulletins/alerts2.asp Visit the Security Bulletin Search site to view security bulletins for all products http://www.microsoft.com/technet/security/current.aspx Under Product/Technology choose the product you are interested in finding updates for Under Service Pack choose the SP level you are using Check “Show only bulletins that have not been superseded” and press ‘Go’ Roadmap Informed & Prepared Customers New Security & Patch Management workshops Regular web casts on security patch management* Updated roadmap, whitepapers, and guidance Q1 ‘03 Improved KB Articles Security Bulletin Teleconferences Q2 ‘03 Q3 ‘03 Patch Management Guides Q4 ‘03 Q1 ‘04 Q2 ‘04 H2 ‘04 H1 ‘05 Updated Patch Management Guidance for SMS 2003 SP1 Bulletin Search Page GTM Partnership Deliverables Patch Management Workshops Revised Patch Management Guides Informed and Prepared Customers Clearer Severity Rating Levels Patch Management Guides Security Readiness Kit (Guides, Tools, Best Practices) Patch Management Roadmap Security Guidance Kit Sustaining Engineering Practices White Paper Patch Management White Paper *See http://www.microsoft.com/usa/webcasts/upcoming/default.asp for upcoming web casts Patch Management Guidance for Windows Update Services Consistent & Superior Update Experience Q1 ‘03 Q2 ‘03 Q3 ‘03 Q4 ‘03 Q2 ‘04 Q3 ‘04 Add/Remove Program improvements in XP SP2 Standard installer switches defined Naming & signing standard defined Q1 ‘04 Standard terminology for documentation defined MSI 3.0 Q4 ‘04 Standard Detection Manifest 2 Installers: MSI, Update.exe Consistent & Superior Update Experience Patches & Security Bulletins released once a month Standard Titles* defined Standard Registry Entries defined Product teams compliant with SE Baseline standards MSI 3.0 supports uninstall, binary delta patching, etc. Converge to two installers -- end of 2004 Consistency standards implemented in all new updates -- end of 2004 *For Add/Remove Programs, Windows Update, and Download Center Superior Patch Quality Up to 75% reduction in patch size* 10% reduction in patch reboots Patch test process extended to include customers Q1 ‘03 Q2 ‘03 Q3 ‘03 Q4 ‘03 Q1 ‘04 Q2 ‘04 H2 ‘04 H1 ‘05 Installer restarts services when possible 75% Reduction in Patch Size* 90% Reduction in Patch Size 25% Reduction in Patch Size Superior Patch Quality 10% Reduction in Patch Reboots Patch test process includes participating customers *For Windows Update installs, more than 25% reduction for other patches **For Windows Server 2003 patches using HotPatching (in-memory patching) technology, delivered in SP1 30+% Reduction in Patch Reboots** MBSA Update Scanning Futures Overall direction Microsoft will have a single scanning engine for detecting missing updates The scanning engine will be part of the Windows Update Services / Automatic Updates client MBSA and other product that need to detect or report on missing updates will request this information from the Windows Update Services / Automatic Updates client MBSA becomes Windows vulnerability assessment & mitigation engine Near-term plans MBSA 2.0 (H1 2005) Initial integration with Windows Update Services / Automatic Update client for update scanning Further deprecation of native MBSA scanning occurs on an ongoing basis as Microsoft Update continues to add support for updating additional Microsoft software over time WU and XPSP2 AU Improvements New release of Windows Update (v 5) Improved homepage design and navigation Implements download throttling for dial-up and low bandwidth connections Will not recommend updates that have already been installed Download regulation feature reduces amount of data transmitted per update Improved ability to update systems with latest critical updates Customer offered choice during Windows XP SP2 install to have AU automatically download and install critical updates New version of Automatic Update client Uses BITS 2.0 to enable restart of interrupted download and improved bandwidth throttling Ability to delay reboot to next system shutdown Microsoft Hosted Update Services Microsoft Update H1 2005 Today Download Microsoft Update Center Office Windows Update Update Microsoft Update Online service and update repository for updating all Microsoft software Microsoft Update: superset of Windows Update Initially supports Windows XP, Windows 2000, Windows Server 2003, Office XP, Office 2000, SQL Server 2000, MSDE 2000, and Exchange 2003. Support for additional Microsoft products will be added on an on-going basis Built on Windows Update Services (formerly SUS 2.0) infrastructure Includes automated scanning, update install, and reporting capabilities Windows Update maintained for legacy reasons WUS SMS Patch Management Products Future Direction Near-term milestones Windows Update Services (H1 2005) SMS 2003 / WUS Phase 1 Integration (H1 2005) Leverages Windows Update Services for update scanning Longer-term (Longhorn time frame) Windows Update Services (WUS) becomes core update management component of Windows Server WUS updates all Microsoft corporate software SMS / WUS Phase 2 integration – SMS builds on WUS infrastructure to deliver advanced patch management WUS infrastructure can be used to build patch management solutions for 3rd party and in-house built software Windows Update Services* The update management component of Windows Server that enables IT administrators to more easily assess, control and automate the deployment of Microsoft software updates Update management solution for all Microsoft products Initially supports Windows XP Pro, Windows 2000 Pro, Windows 2000 Server, Windows Server 2003, Office XP, Office 2003, SQL Server 2000, MSDE 2000, Exchange 2003, + additional products over time** Support for additional update types – security, critical and non-critical updates, update rollups, service packs, feature packs, and critical driver updates Core update management infrastructure in Windows Data Model - supersedence, update dependency & bundle relationships Built-in update scanning engine to detect missing updates Server APIs (.NET) and remoteable Client APIs (COM) Enhanced bandwidth optimization Uses BITS for client-server and server-server communication ‘Binary delta compression’ technologies dramatically reduce data transfer needs Configurable update subscriptions -- specify subset of content to be downloaded *WUS is currently in beta. Microsoft does not guarantee that all capabilities listed will be in the released version. Datasheet and sign up for the Open Evaluation Program at: www.microsoft.com/wus **Without the need to upgrade or redeploy WUS Windows Update Services (2) Expanded administrative control Scanning: Pre-deployment scan for missing updates Download & approval: Specify only metadata be downloaded, rules for autoapproving updates, etc. Targeting: Install or uninstall to systems grouped via enumerated lists or Group Policy Scheduling: Set new update detection frequency*, specify install deadline**, etc. Implementation: Options to use specified communication port, work with Internet proxy, deploy in hierarchical replica or independently managed server topologies, support update management for networks not connected to the Internet, etc. End-user experience: Options to notify users of new updates, reboot, etc. Status reporting Deployment status aggregation per machine/per update/per group Download / install success, failure, and error info Logs statistics to SQL Server or MSDE Improved ease of administration New, intuitive Web administration console simplifies ongoing administration and provides detailed information on new updates Command line utilities and scriptability to enable scalable, efficient administration *Max. frequency 1/hour. Can use command line option or script to trigger new update checks on demand **Deadlines also enable enforcement of update installs (re-installation of required updates removed from the system at a later date) Comparing Microsoft Update, Windows Update Services, and SMS 2003 Adopt the solution that best meets the needs of your organization Capability Microsoft Update Windows Update Services SMS 2003 Supported Software and Content Supported Software for Content Same as Windows Update Services + WinXP Home Win2K, WS2003, WinXP Pro, Office Same as Windows Update Services + 2003, Office XP, Exchange 2003, NT 4.0 & Win98* + can update any SQL Server 2000, MSDE other Windows based software Supported Content Types for Supported Software All software updates, critical driver updates, service packs (SPs), and feature packs (FPs) All updates, SPs, & FPs + supports All software updates, critical driver update & app installs for any Windows updates, SPs, & FPs based software Update Management Capabilities Targeting Content to Systems N/A Simple Advanced Network Bandwidth Optimization Yes Yes Yes Patch Distribution Control N/A Simple Advanced Patch Installation & Scheduling Flexibility Manual & end user controlled Simple Advanced Install errors reported to user. Patch Installation Status Lists missing updates for Reporting accessing computer Simple Advanced Deployment Planning N/A Simple Advanced Inventory Management N/A No Yes Compliance Checking N/A No – status reporting only Advanced *MBSA does not support scanning Win98 – Win98 can be updated using SMS2003 inventory management and software distribution capabilities Choosing A Patch Management Solution Typical Customer Decisions Customer Type Large or Medium Enterprise Small Business Consumer Scenario Want single flexible update management solution with extended level of control to update (+ distribute) ALL Windows OSes and Applications, as well as an integrated asset management solution Customer Chooses SMS 2003 Want update management-only solution that provides simple updating for Microsoft software and initially supports Windows (Win2K & later versions), Office (2003 & XP), Exchange 2003, SQL Server 2000, and MSDE 2000 Windows Update Services* Have at least 1 Windows server and 1 IT administrator Windows Update Services* All other scenarios Microsoft Update* All scenarios Microsoft Update* *Customer uses Windows Update, another update tool, or manual update process for OS versions & applications not supported by Windows Update Services or Microsoft Update Consolidated Solutions Roadmap Longhorn Time frame H1/2005 Q4/2003 Update Content Repositories and Online Services Windows Update Download Center Office Update Download Windows Center Update Microsoft Update Windows Update Microsoft Update Standalone Update Scanning Tools Office Inventory Tool MBSA 1.2 MBSA 2.0 (includes OIT) MBSA 1.1.1 SMS 2003/ WUS phase 1 integration SMS 2.0 with Feature Pack SMS 2003 SUS 1.0 Manual / Script Based Updating WUS Server WUS Client 3rd party apps update repository In-house developed apps update repository 3rd Party / In-house Tools WUS N.0 SMS v4 Windows Server Longhorn Update Management Products Adopt a Patch Management Solution At Microsoft, our #1 concern is the security and availability of your IT environment If none of the Microsoft patch management solutions meet your needs consider implementing a solution from another vendor Partial list* of available products: Company Name Product Name Company URL Altiris, Inc. Altiris Patch Management http://www.altiris.com BigFix, Inc. BigFix Patch Manager http://www.bigfix.com Configuresoft, Inc. Security Update Manager http://www.configuresoft.com Ecora, Inc. Ecora Patch Manager http://www.ecora.com GFI Software, Ltd. GFI LANguard Network Security Scanner http://www.gfi.com Gravity Storm Software, LLC Service Pack Manager 2000 http://www.securitybastion.com LANDesk Software, Ltd LANDesk Patch Manager http://www.landesk.com Novadigm, Inc. Radia Patch Manager http://www.novadigm.com PatchLink Corp. PatchLink Update http://www.patchlink.com Shavlik Technologies HFNetChk Pro http://www.shavlik.com St. Bernard Software UpdateExpert http://www.stbernard.com *Microsoft does not endorse or recommend a specific patch management product or company Note: Enterprise Systems Management products such as IBM Tivoli, CA Unicenter, BMC Patrol, and HP OpenView may also provide patch management functionality Summary Addressing the patch management issue is a top priority Taking a comprehensive, tactical & strategic approach Made progress, but much more work to be done Microsoft focused on: Reducing the number of vulnerabilities & associated patches Improving customer preparedness, training & communication Simplifying & standardizing the patching experience Improving patch quality Unifying and strengthening patch management offerings Key Recommendations: Implement a good patch management process – it’s the key to success Adopt a patch management solution that best fits your needs Make use of the resources referenced in these slides Security Incident Response Trends – 2003 CSI / FBI Survey Of 532 respondents, 92% detected attacks Only 251 organizations were able to quantify losses 25% of respondents suffered attacks on WWW servers Only 50% of intrusions were reported to law enforcement www.gocsi.com for complete results Case Study – Edge Server Symptoms Admin shares deleted repeatedly New service / security patch installed Server reboots unexpectedly Bandwidth consumption / server sluggish Low disk space Findings Malware “hidden” (+H) in subdir of system32 Malware “hidden” (+H) in c:\recycler Malware really hidden in “c:\System Volume Information” directory FTP / Backdoor Server installed to run as SYSTEM service Case Study – Intranet DoS Symptoms High CPU utilization on affected systems (DC’s may have high CPU in LSASS) Account lockouts Increased TCP 139/445 network traffic RPC / LSASS crashing, machines rebooting AV stops working on some machines Can’t access AV web sites on some machines Findings You’ve got bot like Gaobot.AFW or Agobot.JF, Phatbot, SDBot, Randex There is no spoon . . . In the last century, organizations relied upon firewalls / perimeter defense as the basis for protecting the Intranet This has created a hard crunchy shell with a soft chewy center for most organizations In the 21st century with blended threats, firewalls alone do not effectively stop worms Did your firewall stop Slammer or Blaster? Will it stop bots like Gaobot / Phatbot / Agobot? VPN connections from home machines blur the ‘perimeter’ and increase the threat of automated attacks Threats – Modus Operandi Fact: Most intrusions are not accomplished via awe-inspiring skill. Fact: It is much harder to secure than it is to hack. Most intrusions involve: Weak administrator passwords!!! Un-patched security vulnerabilities in underlying software products (OS and applications) Weak out of box security settings that were never hardened Lack of secure coding in custom applications Recommendations Normal operations staff trained to recognize symptoms of security incidents Escalate cases to security incident response team to: Determine time / date intrusion occurred Determine how the intrusion occurred Develop ‘signature’ for the intrusion Scan nearby machines for ‘signature’ Make changes to security posture to prevent future incidents Preparing a Security Incident Response Plan Processes should be put in place before an incident has occurred that will facilitate: Detection Determining whether an incident has occurred Investigation Determining how an incident has occurred Containment Isolating affected hosts Resolution Restoring service / lessons learned Escalating the Incident Define symptoms or behaviors that become triggers that will kick off an investigation Ensure admins and helpdesk staff understand and can recognize them! Security Incident Response team should Compare current ‘state’ to previous ‘state’ Look for new processes, files, folders, network connections, listening ports, services Not possible if you don’t know what the previous state was Baseline and catalog your servers! Run a live response IR toolkit to collect data Have trained IR specialist analyze output Suspicious Symptoms, Behaviors Suspicious event log data Suspicious server reboot (no admins remember rebooting) Admin shares disappearing Security patches installed mysteriously News processes / services / files / folders Abnormal process termination (i.e. IIS crashes) A blue-screen occurs Sluggish system performance Suspicious network traffic to/from an IP address Things You Need To Know Why you need an Incident Response team within your organization Because it’s not a matter of ‘if’ but ‘when’ Auditing is everything Sufficient auditing is not usually enabled by default! Proper business continuity planning facilitates successful incident response If business isn’t down – more likely to have time to do a proper investigation Building the Security Incident Response Team Overview Training – Staying Current Tracking Security Incidents Live Response vs. Offline Response Assembling a Live Response Toolkit Microsoft PSS Security Incident Response Toolkit Training Know your adversary Strongly recommend reading security and hacking related books Attend security conferences (Blackhat, RSA etc.) Subscribe to managed security service (ISS, TruSecure, LUHRQ etc.) Learn Incident Response Read books Attend specialized incident response training Training Recommended resources Hacking Knowledge Hacking Exposed series of books Security Warrior Stay abreast of security vulnerabilities and exploits as they are released by subscribing to managed security services and monitoring Full-Disclosure mailing list Exploit web sites Incident Response Knowledge Windows Security Resource Kit: http://www.microsoft.com/mspress/books/6418.asp Foundstone: Ultimate Hacking Incident Response / Forensics Incident Response & Computer Forensics 2nd Ed. SANS: Track 4 – Incident Handling CERT Incident Response Handbook: http://www.cert.org/archive/pdf/csirt-handbook.pdf Tracking Incidents Tracking incidents is extremely important Historical data can be used to spot trends Central repository for keeping case notes during an investigation (encrypted?) Can be used for reporting progress to upper level management as incidents are resolved Options Literally Hundreds of Help Desk software solutions Request Tracker IR (Best Practical) Request Tracking software specifically for CERT teams Track-IT! (Intuit) CRM / CIM Solutions – Not always a great fit here Home grown solution may be best? Live Response vs. Offline Response Live Response vs. Offline Response Two different approaches to IR Offline response involves imaging disks and using specialized software to look for clues and evidence ProDiscover IR EnCase NOT mutually exclusive Create disk image first for use with ProDiscover / EnCase if necessary Then perform live response using automated IR toolkit Live Response: Risks Rootkits Introduced for Windows, publicly, circa 1997 They modify operating system behavior to hide files, folders, processes, registry entries, and network connections to avoid detection by live response tools Kernel mode drivers, usermode processes By observing the system, you alter its state Sort of like Schroedinger’s cat theorem. Placing output on target system overwrites free space / slack space etc. Altering time stamps and files may invalidate collected evidence if pursuing litigation Assembling a Live Response Toolkit Purpose Offline forensic analysis not always possible, needed or timely Technical barriers, unacceptable downtime etc. Not always able to respond, in person to remote locations Live response toolkit facilitates consistent data collection from remote systems for offline analysis by an IR specialist Can be used as a first response tool to triage and investigate reported security incidents Systems can remain online during investigation Very important when an intrusion has not been confirmed positively Microsoft Incident Response Toolkit Design Goals Trustworthiness (anticipate that a rootkit is installed) Run in automated fashion on NT4 or later Collect volatile data from a live system Compress collected data into a .CAB file for submission to an IR specialist Not designed to Create or preserve evidence for use by law enforcement for use in legal proceedings Image a drive for offline analysis and response Microsoft Incident Response Toolkit Two tools Data collection agent (The “IR toolkit”) Batch file that automates dozens of .EXE’s zipped up in a zip file with a readme.txt Data analysis tool (The “IR Viewer”) C# application, runs on examiners workstation Utilizes custom-built tools designed for incident response Utilizes free 3rd party tools Had to work with legal team and get written permission from authors to redistribute their tools! Be aware of EULA’s and licensing fee’s associated with ‘free’ tools when used in a business environment Microsoft Incident Response Toolkit Randomized filenames Gets local system / Internet time kernel profiler Netstat / arp / ipconfig / routing table DIR commands (hidden, modified, accessed, created) Rootkit detection Dumps registry as text Saves event logs as TSV Enumerate NULL session information Get patch status Scan for ADS’s Enumerate running processes Get file versions of all loaded modules / key directories Get audit policy Dump security policy information (policy, users, rights, etc.) Map processes Ports Enumerate installed services several ways Enumerate ACL’s (if specified) Generate hashes for executables (if specified) Run ‘net’ commands Dump scheduled tasks Copies all .log, .bat, .cmd, .vbs, .js files from system32 Microsoft Incident Response Toolkit Takes anywhere from 10 to 20 minutes to run Can be used to identify signs of an intrusion (some rootkits, suspicious processes, services, files, folders, registry entries, event log entries, suspicious accounts in the administrator group, missing security patches etc. Areas for improvement Better approach to rootkit detection (in progress) Run file system commands as SYSTEM (in progress) Registry last write times (in progress) Security Incident Response Team Objectives Incident Response Objectives Confirm whether an intrusion has actually occurred By analyzing the contents of the IR toolkit output for a specific server(s) Determine when the intrusion occurred Based on a lead like an event ID or a suspicious files or folders creation date Determine how the intrusion occurred Based on implicit or explicit evidence (absence of a critical security update at the time the intrusion occurred etc.) Identifies weakness in security posture and leads to corrective action being taken If new malware identified – submit samples to the antivirus partners PSS Security team in partnership with most leading antivirus vendors To rebuild or not, that is the question! Microsoft’s stance It’s a risk assessment really We provide evidence (or lack thereof) of an intrusion. Sometimes we find no evidence of a compromise Most of the time it’s pretty straightforward We provide case notes for malware we’ve identified Submit to the AV partners so they can update signatures Customer usually cleans manually or waits for new sigs Other times, when a rootkit is known to be installed and hiding software, who knows what else is on the machine Facilitating Effective Incident Response How to avoid common mistakes . . . Common Mistakes Companies Make When helping organizations investigate security incidents we see the same mistakes being made over and over again. The following slides detail the most common mistakes that are usually made and give guidance on how to avoid making these mistakes. Common Mistakes Companies Make No formal, documented policies Server security hardening policy Acceptable Use policy Auditing policy Password complexity requirements Secure operating system builds Security patch deployment policy No formal change management process Many systems are shared between groups with many user accounts in the administrators group No process for tracking changes to the system back to a group or person No documentation about what should be installed on a system vs. what actually is installed on a system Common Mistakes Companies Make No baseline data If you don’t know what ‘normal’ looks like – how can you spot abnormal behavior Perform software inventory updates Perform period port-scans of the network Know the normal operating thresholds for your servers Know the normal traffic patterns for your network Inability to ‘scale out’ during an investigation Suppose after the initial response you confirm that a group of servers were successfully attacked? How do you scale out the investigation to the neighboring servers / networks? Common Mistakes Companies Make No formal security incident response team Why? Usually lack of budget and planning? Use some form of risk assessment and threat modeling to make a business case for a team! (STRIDE / DREAD) Incident Response team is old-school So you have an IR team but they aren’t up to date? Do they know about rootkits? Do they know about the latest worms and bots? Consider performing a penetration test of the environment to see how they do. Play with malware and study it in undoable isolated virtual machines! Common Mistakes Companies Make Lack of a business continuity plan Some security incidents can be investigated while the systems are on-line, others require off-line analysis How long can you afford to be down? Lack of a trusted IR toolkit An automated toolkit should be created to facilitate the process of gathering information off of live systems The output of the toolkit should be known and well understood! Tips for Responding To Security Incidents Advice from the front line . . . Incident Response Tips Decide as quickly as possible whether or not to involve law enforcement They have their own evidence collection process and procedures Anything you do before law enforcement is involved potentially hinders the investigation and collection of evidence Interview the person reporting the incident thoroughly What’s the behavior being reported, how are things different? What day / time did you first notice something was wrong? Write everything down and keep accurate time / date stamps Identify Symptoms of a Rootkit If a rootkit is installed, the output of the IR toolkit should be considered trustworthy It is imperative to identify whether a rootkit is possibly installed right away Consider using rootkit detection tools like VICE http://www.rootkit.com/vault/fuzen_op/VICE_ Bin.zip Identify Symptoms of a Rootkit Port scan the server remotely from a known good machine (all TCP and UDP ports) Look for any ports that show up on the network but not in local netstat, portqry or fport output Sure sign that a rootkit is hiding a backdoor listening on a port Boot the system into safe mode and examine installed services Look for services that show up in safe mode but not normal mode (rootkit may not load in safe mode) Locally list the files in the %windir% directory and all subdirectories and then do it again from a mapped network drive Look for files that don’t show up locally but that do remotely (again, rootkit) Identify Symptoms of a Rootkit Configure Device Manager to show ‘hidden’ devices and view them Look for suspicious device drivers under ‘NonPlug and Play Drivers’ IR Toolkit Data Analysis Determining a Date / Time gives you something to search on Look for leads that will yield a date or a time Suspicious processes, services, event log entries or files created on or around the date / time of the reported incident Once you have a ‘lead’ (i.e. a suspicious process or service) get the creation date of the file on the file system Perform a search for other files created on or around that time Build a Time-Line of Events Once you have found some ‘leads’ build a chain of events that paint the picture Example leads from the System Event log System mysteriously rebooted on 4/20/2004 at 2:41am Just before that a Microsoft Security update was installed by the ‘SYSTEM’ account Could be a remote-shell, attackers often install the security patch they used to compromise a system to prevent others from stealing it Look for files created on that date / time Build a Time-Line of Events Example Suspicious service identified in Services snap-in That’s your ‘lead’ Identify the process backing that service (double click the service) Find the creation date of that file Look for other files created on that date Look for account logons on that date at around that time Determine when security patches were installed relative to that date time (before or after?) Look In The Right Places Miscreants often hide their malware in the c:\recycler\<SID> folder (where SID is a real or fictitious security identifier) Miscreants are increasingly turning to hiding their malware in the hidden SYSTEMonly “c:\system volume information” folder Grant admins access to the folder and look in there as well. Laws and Legal Issues What you don’t know can hurt you . . . Laws and Legal Issues Decide early on whether you might want to prosecute or not There are usually laws surrounding the collection of evidence and surveillance In litigious investigations you will be much more successful if you involve law enforcement immediately Laws and Legal Issues Most companies have a lack of knowledge about “Cyber crime” laws Acceptable Use Policies Search and Seizure Laws Reasonable Expectation of Privacy Is it lawful to monitor an employees e-mail / network traffic / or search their hard drive? Due Diligence Laws Can you be held liable for personally identifiable information that was stolen? Always involve proper legal counsel at the onset of a security related incident response investigation! Laws and Legal Issues List of Worldwide Cyber Crime Law Links http://www.ccmostwanted.com/LL/global.htm U.S. Laws www.cybercrime.gov European Laws http://conventions.coe.int/ http://www.epic.org/privacy/intl/ http://www.europa.eu.int/index_en.htm Australian Laws http://www.aph.gov.au/house/ http://parlinfoweb.aph.gov.au/piweb/search_main.as px http://www.ntu.edu.au/faculties/lba/schools/Law/apl /Cyberspace_Law/articles1.htm © 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.