Exchange 2007 Performance Troubleshooting

Exchange 2007 Performance Troubleshooting

the RPC Counters – these counters will show you if the clients are “feeling” a resource issue

• MsExchangeIS\RPCAveraged Latency – should be under  50 (100 if in cached)
  • RPC Operations/Sec – Relative (Baseline\Trending
  • RPC Requests – Rec  under 70
• If you see RPC ops go at around time of latency may be adding too much load

Windows::Server Performance::Troubleshooting::Citrix

Troubleshooting Server Performance
The discussion of a specific issue below is perhaps useful in a more general sense for troubleshooting and performance monitoring topics.

Problem: After upgrading to Citrix Presentation Server 4.5 a higher average cpu utilization is observed as well as a high rate of context switches. Previously we have often received warnings in Citrix Performance Monitor for %interrupt -- this issue continues and is perhaps seen more often in 4.5 servers as well.

Background: Running PS4.5 using published applications and desktops on a Microsoft Windows 2003 SP2 server on a physical machine. Running several "high maintenance" accounting applications on two PS4.5 as published applications on virtual machines on VMWare Virtual Infrastructure 3.0 cluster. These all exhibit the symtoms above just since the upgrade to 4.5. Also, we are still running 4.0 on several other servers in the same Citrix Farm and various versions of PNA are in use by client machines (predominantly 8.x)

Investigation regarding context switches
A lot of good resources turned up:
Intel: Using Windows Performance Monitor
Sysinternals
www.thomaskoetzing.de
MSDN-Context Switches
Analyzing Processor Activity
Since this issue occurs on both physical and virtual servers it is not a VM problem, but will investigate this avenue as well to ensure correct and optimal configuration.
VMware: improving scalability for Citrix PS
http://redmondmag.com/features/article.asp?editorialsid=718


- definition: CPU's share their time between all threads according to priority. When the CPU stops working on one thread and starts working on another that is a context switch.
- monitoring: A ballpark rule of thumb is "normally" there should be no more than 28000 context switches per CPU on a system.
- What to look for
- Page file - too small, or is allowed to dynamically grow - recommendation: set to larger fixed size.
- Consider write cache on RAID controller
- insufficient hardware
- poorly designed device drivers or applications

Tools
- PerfMon - system/context switches
- SysInternals - Process Explorer - View > select columns > Process Performance > context switches, context switch delta
- pstat.exe (windows resource kit or support tools

VMWare
Some asides that came up during this investigation explained some issues we have had with virtualizing citrix servers. We needed to keep 2 cpu's in the VM after we converted them. That is the opposite of the VMWare recommendations we have seen.
- The multiprocessor HAL had not been downgraded to single processor HAL.
- Hidden devices in device manager had not all been removed.

1. Click Start, click Run, type cmd.exe, and then press ENTER.
2. Type set devmgr_show_nonpresent_devices=1, and then press ENTER.
3. Type Start DEVMGMT.MSC, and then press ENTER.
4. Click View, and then click Show Hidden Devices.
5. Expand the Network Adapters tree.
6. Right-click the dimmed network adapter, and then click Uninstall
uninstall any other physical devices not needed

Investigation
- Interesting - on the VM servers when looking at Task Manager the %cpu listed individually for all the processes for all users did not appear to add up to what was showing up on the Performance tab (at least 50% discrepency.) This was not observed on the physical server
- For both VM's and physical servers: Citrix Performance Monitor was showing warnings and intermittent error conditions on %cpu, %interrupt, context switches/sec.
- The VM's cpu utilization on the host machine is extremely high. On the server with the greatest number of users it maxed out the host cpu for much of the time I watched it.
- Watching performance monitor a few minutes showed context switches/sec to be in the hundreds of thousands.
- Opened Process Explorer and set view to show context switches and context switch deltas. I observed that at times it reported up to 50% cpu was due to hardware interrupts (this was not as dramatic when I checked it on the physical machine so I wonder if this is a reporting issue related to vmware's magic behind the scenes.) Also, the highest context switch delta was for hardware interrupts so Process Explorer was no help to further isolate it.
- To isolate what driver or program might be causing this issue, I piped the output of pstat.exe to a file and looked for the highest count of context switches. I took the memory address of that thread and looked it up in the bottom section to find what address range it fell in. In this case it was CDM.SYS
- google search of CDM.SYS turned up multiple articles about Citrix servers. I think CDM stands for Client Data Mapper. Of greatest interest is an article about a hotfix for PS4.5:
http://support.citrix.com/article/CTX114121 (and I see a lot of other post FR1 hotfixes out there too.)
The issue resolved in this hotfix is:

"Winlogon.exe shows higher than average CPU consumption on the server. The issue occurs because the server refreshes the smart card reader state more frequently than necessary. This occurs even if smart cards are not being used. With this fix, the reader state is refreshed only once per noticeable event."

Performance::MSSQL

Quote from: Buck WoodyThese objects and counters serve as a demonstration of the types of things I monitor for a SQL Server application I use:

Object	Counter	Meaning	Notes
Cache	Data Maps Hits %	NTFS Processing Efficiency	Should be better than 90% or may need to defrag Hard Drive
Cache	MDL Read Hits %	Cache Hits for IIS	Should be better than 90%
Memory	% Committed Bytes	Shows memory use	Should be under 70
Memory	Available MBytes	How much unallocated RAM is left on the server	Should be greater than 50MB
Memory	Cache Faults/Sec	Physical Disk Read I/O Operation	The fewer the better
Network Interface	Bytes Total/sec	Shows network activity	Network Monitor Driver must be installed for accurate measurement of networkitems - Measure against network bandwidth availability
Network Segment	% Network Utilization	Total Network segment use - not from this server alone	Depends on many networking variables, but can be useful to a Networkspecialist
PhysicalDisk	Current Disk Queue Length:_Total	Shows physical drive activity	Less than 3 per physical disk is acceptable
PhysicalDisk	% Disk Read Time	How much time spent doing reads	Combine with Writes to see if Index usage is correct - may need to adjustfillfactor
PhysicalDisk	% Disk Time -- _Total	Shows drive activity	diskperf -yv is required for proper logical disk counters to operate. Shouldbe less than 55% - watch for increase
PhysicalDisk	% Disk Write Time	How much time spent doing writes	Combine with Reads to see if Index usage is correct - may need to adjustfillfactor
Process	% Processor Time	Pick Specific Object	Will explain how much that object is taking on the processor
Processor	% Total Processor Time	Shows the CPU activity being taken by all processes	Should not exceed 80% for continuous periods with high Proc Queue Length.NOTE: W2K measures non-busy time and subtracts it from 100%
Server	Bytes Received/Sec	Data received by the server NIC
Server	Bytes Transmitted/Sec	Data sent by the server NIC
SQL Server:Access Methods	Full Scans/Sec	Table scans	For the entire server - not just one database
SQL Server:Access Methods	Page Splits/Sec	Splits happen when data or indexes span a page (8k)	Fillfactors may not be correct if this is high
SQL Server:Buffer Manager	Cache Size (pages)		Multiply x 8192 for RAM amount, should be close to the RAM in yoursystem.
SQL Server:Locks	Average Wait Time (ms)	Time processes spend waiting on a lock release	Should not be high
SQL Server:Locks	Number of Deadlocks	Number of processes deadlocking each other	Measurement is by the second
SQL Server:Memory Manager	Target Server Memory	How much RAM SQL wants
SQL Server:Memory Manager	Total Server Memory	How much RAM SQL is using
SQL Server:SQL Statistics	Batch Requests/Sec	Bow many batches sent to SQL Server	Over 1000 indicates a busy SQL Server - May indicate CPU bottleneck. A 100Mbnetwork card can handle 3000 per second.
SQL Server:SQl Statistics	SQL Compilations/Sec	How many compiles SQL has to do	Over 100 may indicate a SQL problem
SQLServer:Buffer Manager	Buffer Cache Hit Ratio	Shows how much data is found in the Buffer	Should be 99% or greater
SQLServer:General Statistics	User Connections	Shows the number of connections taken on the SQL Server	Varies
System	% Total Privileged Time	Kernel mode operations	Should be less than 20% or may be IO bound. Pair with %Disk time counter tosee if greater than 50%. Can also indicate driver or NIC.
System	Context Switches/Sec	Server thread switches	Should not exceed 8000 per second, per processor
System	Processor Queue Length	Shows the amount of processes waiting for CPU time	Pairs with % Processor Time counter - Should not be greater than 2 perproc.
Web Service	GET Requests/Sec	Number of GET requests	Each GET connection attempt
Web Service	POST Requests/Sec	Number of POST requests	Each POST connection attempt
Web Service	Total Method Requests/Sec	Hits on the Web Server	Service Request rate

CITRIX::VMWARE

BriForum 2006 Video: Ron Oglesby on the feasibility of running Citrix in a VM: "Video: Ron Oglesby on the feasibility of running Citrix in a VM"

Network Monitoring, Free Network monitor, Woodstone Servers Alive, POPPager the best networking tools and services: "Servers Alive is an end-to-end network monitoring tool running on Windows 2000/2003/XP. It works agentless and across operating systems. Its checks and alerts are on the cutting edge of technology. "

Physical Address Extension - PAE Memory and Windows: "Physical Address Extension - PAE Memory and Windows
Updated: February 9, 2005"
Fixing the 4GB memory limit issue

VisualRoute traceroute server: trace IP address, trace route, IP trace, IP address locations : "Test Internet Connectivity and Trace IP Addresses with VisualRoute "


Online demo

Citrix::Performance Metrics

Percent Interrupt Time
The processor is spending a large amount of time responding to I/O rather than user processing. A large value for interrupt time indicates a hardware problem or a very busy server.

Context Switches per Second
A large # of threads and/or processes are competing for processor time.

Links::Printing::TIFF::Acceleration

Graphic printing acceleration

http://www.ggx.com/downloads/datasheets/XipPRINT.pdf

Mark's Sysinternals Blog: "Mark's Sysinternals Blog"

Windows::Networking::Optimize TCP

SpeedGuide.net :: Download Cable/DSL Speed Patches: "SpeedGuide Programs"

TCP/IP and NBT configuration parameters for Windows XP
PREVENT FRAGMENTATION
set MTU size to the same or lesser than the router along the tracert with the smallest MTU allowed.

Find the smallest MTU allowed in the route by process of elimination using:
ping -l ____ server.domain.com -f

(-l = lower case L and means to use ____ packet size, -f means do not fragment.)

I found:
1413 and greater => "needs fragmented" response
1412 - 1395 => timeout response
1394 and lower => response received.