SYSPREP on cloned Windows Server 2008 R2 Fails

Trouble with sysprep not running when vmware runs customization after deploying a Win2K8R2 template. -> SID for all the clones is the same. Supposedly this matters much less these days but some odd stuff happened that we couldn't explain when we attempted to join to AD domain (NEWSID doesn't work past Win2003) SYSPREP logs are located at: c:\windows\system32\sysprep\panther log files: setupact.log, setuperr.log Apparently sysprep will not run when it thinks Windows has been upgraded in place. This particular template they were copying had several applications installed on it for which we don't know the owner so rebuilding fresh was not an option. The following worked to allow sysprep to run: - Remove the machine from the domain - Registry export: HKLM\SYSTEM\Setup (as backup) - Delete from the registry: HKLM\SYSTEM\Setup\Upgrade - Run: c:\windows\system32\Sysprep\Sysprep.exe /oobe /generalize

Magic Quadrant for Blade Servers

Magic Quadrant for Blade Servers: Magic Quadrant for Blade Servers

How to fix a messed up domain controller

Just get rid of it!

• Disconnect the DC from the network
• Run dcpromo /forceremoval
• From 2003, use NTDSUTIL to do a "metadata cleanup" to clean the DC out of your AD.  (See below) If you have at least one Windows Server 2008 DC, then open Active Directory Users and Computers from a 2008 DC, find the bad DC, right-click and delete.

Metadata Cleanup using NTDSUTIL
MS Article 216498 explains how to forcibly remove a domain controller account from AD using NTDSUTIL.
http://support.microsoft.com/kb/216498

Time to Check Sysvol
Sysvol is a share you can find on every domain controller, a share that contains files needed by DCs -- the big ones are the file components of group policy objects (GPOs), pieces called "group policy templates" or GPTs, as well as login scripts. Sysvol is a neat, built-in implementation of DFS (Distributed File Services) that is multi-master, meaning that if you have four DCs named DC1, DC2, DC3 and DC4, then you can drop a file into any one of those four Sysvol folders, and eventually DFS will ensure that there's a copy of that file in each of the other three Sysvols. The fact that you can introduce a new file into the family of Sysvols is why it's said to be "multi-master." On 2003 SP3 and later, there's a command that lets you force replication between a source DC (DC3, in this example) and a destination DC (DC2, in this example) that looks like this:

ntfrsutl.exe forcerepl DC2 /r "Domain System Volume (SYSVOL share)" /p DC3

Check SYSVOL before upgrading
Go to a DC. In its Sysvol, create a file, such as [dcname].txt. After you have created a small text file on each DC whose name reflects the DC that you created it on. Wait a while and go to each DC and look in its Sysvol... there should be a file for each domain controller. If, for example, DC4's dc4.txt shows up nowhere, then DC4 probably has the problem. To fix it, run DCPROMO to demote it and, if the rest of the network doesn't see that you've demoted it, remove the DC's account from the Domain Controllers OU and remove its metadata. Once it is successfully removed, test again by creating another unique file in each DC's Sysvol and be sure everything is cool.

Thanks to:
http://www.minasi.com/newsletters/nws1205.htm

Automated Remote Install of SCOM client

###############################################################################################################
#  SCOM-INST.PS1
# Install SCOM Agent on a list of machines.
# 
$rootMS = "RMS01"
$list = "c:\server-list.txt"
clear-host
$Servers = Get-Content $list
#attach SCOM SnapIn
    Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client" -ErrorVariable errSnapin;
    Set-Location "OperationsManagerMonitoring::" -ErrorVariable errSnapin;
    new-managementGroupConnection -ConnectionString:$rootMS -ErrorVariable errSnapin;
    set-location $rootMS -ErrorVariable errSnapin;
#
#install SCOM client
    $DiscoverCnfg = New-WindowsDiscoveryConfiguration -computername:$servers -performverification:$true
-computertype:"server"
    $ms = Get-rootManagementServer
    $DiscoverResults = Start-Discovery -managementserver $ms -windowsdiscoveryconfiguration:$DiscoverCnfg
$DiscoverResults.CustomMonitoringObjects
    if ($DiscoverResults.CustomMonitoringObjects) {
        install-agent -ManagementServer $ms -AgentManagedComputer: $DiscoverResults.custommonitoringobjects
        }
    else {
        write-host "ERROR:  Machine(s) not found"
        }

#
Write-Host "Press any key to continue ..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")

Powershell: list servers that respond

##################################################################################
#
# Create list of all servers with machine accounts in AD that respond to PING
#
##################################################################################
# Create $list of AD machine accounts for Windows Servers
$ServerList = ".\SERVERLIST.TXT"
New-Item $ServerList -Type file -Force >$nul
"Gathering Server List..."
$strCategory = "computer"
$strOS = "Windows*Server*"
$objDomain = New-Object System.DirectoryServices.DirectoryEntry
$objSearcher = New-Object System.DirectoryServices.DirectorySearcher
$objSearcher.SearchRoot = $objDomain
$objSearcher.Filter = ("OperatingSystem=$strOS")
$colProplist = "dnshostname"
foreach ($i in $colPropList){$objSearcher.PropertiesToLoad.Add($i)}
$colResults = $objSearcher.FindAll()
foreach ($objResult in $colResults) {
$objComputer = $objResult.Properties;
$Server = $objComputer.dnshostname
$Server = $Server -replace "\s{2,}", ""
$Server = $Server -replace "\.usa\.domain\.com", ""
if ($Server) {#skip null value
$Server
if (Test-Connection -ComputerName $Server -quiet -count 1) {#ping test
write-output $Server | out-file -encoding ASCII -filepath $ServerList -append
" .....OK"
}#if
}#if
}#foreach

Check services on all servers

##########################################################################################
#
# SVC-AUTO.PS1
# For all servers, find services with Automatic Startup that are not running
# Ignore SysMonLog("Performance Monitor Logs and Alerts")
#
##########################################################################################
#$startflag = $true
$startflag = $false
$inputfile = ".\dc-list.txt"
$names = Get-Content list.txt
$today = get-date
write "==========================================================================="
write " $today"
write " Automatic Service Status"
foreach($name in $names) {
write "---------------------------------------------------------------------------"
write-host " $name"
$svc = Get-WmiObject Win32_Service -ComputerName $name -erroraction SilentlyContinue
if ($svc) { #not null
foreach ($service in $svc) {
$svcname = $service.name
$svcdisplay = $service.displayname
$svcmode = $service.startMode
$svcstate = $service.state
if ($svcname -ne "SysMonLog") {
if ($svcmode -eq "Auto") {
if ($svcstate -eq "Stopped"){
write-host " $svcdisplay = DOWN"
if ($startflag) { #start it up
write-host " starting $svcdisplay"
$service.StartService() >$nul
sleep(10)
$svc = Get-WmiObject Win32_Service -ComputerName $name -Filter "name='$svcname'"
write-host " " $svcdisplay "..." $($svc.State)
} #if start flag
} #if stopped
}#if Auto
}#if
}#foreach service
}#if not null
else { write-host " ERROR: $name - No Response" }
}#foreach server
write "==========================================================================="

Checkuptime.ps1

#powershell to check uptime for list of servers after test for PING
$list = ".\wsus-groups\wsus-pri-group-1.txt"
$Servers = Get-Content $list
foreach($Server in $Servers) {
if (Test-Connection -ComputerName $Server -quiet -count 1) { #responds to PING
$wmi=Get-WmiObject -class Win32_OperatingSystem -computer $server
$LBTime=$wmi.ConvertToDateTime($wmi.Lastbootuptime)
[TimeSpan]$uptime=New-TimeSpan $LBTime $(get-date)
Write-host $server "Uptime: " $uptime.days "Days" $uptime.hours "Hours" $uptime.minutes "Minutes" $uptime.seconds "Seconds"
}
else {
Write-host $server "NO RESPONSE"
}
}#foreach

UCS server hardware manual

Including optimal memory module installation.
http://www.cisco.com/en/US/docs/unified_computing/ucs/hw/chassis/install/blade.html

Website Redirection

I seem to be looking this up every time. Say I have a website with an app in a subfolder. I want to redirect to website/appfolder/default.htm so I don't make the user type https://website/appfolder.
Take this and save it to default.htm

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>TITLE</title>
<meta http-equiv="REFRESH" content="0;url=https://info.domain.com/reaction"></HEAD>
<BODY>
Please wait while you are being redirected.
Click<a href="https://website/appfolder">here</a> to go there immediately.
</BODY>
</HTML>
If you want to redirect from unsecure (http) url to a secure (https) url then it is necessary to customize the 403.4 error message text to be the HTML code above.
Do this for IIS by replacing C:\WINDOWS\help\iisHelp\common\401-4.htm with above.
You can revert to the default anytime:

right-click the website and choose properties

Click Custom Errors tab

Click on the 401;4 entry and hit "Set to Default"

Windows::Server Performance::Troubleshooting::Citrix

Troubleshooting Server Performance
The discussion of a specific issue below is perhaps useful in a more general sense for troubleshooting and performance monitoring topics.

Problem: After upgrading to Citrix Presentation Server 4.5 a higher average cpu utilization is observed as well as a high rate of context switches. Previously we have often received warnings in Citrix Performance Monitor for %interrupt -- this issue continues and is perhaps seen more often in 4.5 servers as well.

Background: Running PS4.5 using published applications and desktops on a Microsoft Windows 2003 SP2 server on a physical machine. Running several "high maintenance" accounting applications on two PS4.5 as published applications on virtual machines on VMWare Virtual Infrastructure 3.0 cluster. These all exhibit the symtoms above just since the upgrade to 4.5. Also, we are still running 4.0 on several other servers in the same Citrix Farm and various versions of PNA are in use by client machines (predominantly 8.x)

Investigation regarding context switches
A lot of good resources turned up:
Intel: Using Windows Performance Monitor
Sysinternals
www.thomaskoetzing.de
MSDN-Context Switches
Analyzing Processor Activity
Since this issue occurs on both physical and virtual servers it is not a VM problem, but will investigate this avenue as well to ensure correct and optimal configuration.
VMware: improving scalability for Citrix PS
http://redmondmag.com/features/article.asp?editorialsid=718


- definition: CPU's share their time between all threads according to priority. When the CPU stops working on one thread and starts working on another that is a context switch.
- monitoring: A ballpark rule of thumb is "normally" there should be no more than 28000 context switches per CPU on a system.
- What to look for
- Page file - too small, or is allowed to dynamically grow - recommendation: set to larger fixed size.
- Consider write cache on RAID controller
- insufficient hardware
- poorly designed device drivers or applications

Tools
- PerfMon - system/context switches
- SysInternals - Process Explorer - View > select columns > Process Performance > context switches, context switch delta
- pstat.exe (windows resource kit or support tools

VMWare
Some asides that came up during this investigation explained some issues we have had with virtualizing citrix servers. We needed to keep 2 cpu's in the VM after we converted them. That is the opposite of the VMWare recommendations we have seen.
- The multiprocessor HAL had not been downgraded to single processor HAL.
- Hidden devices in device manager had not all been removed.

1. Click Start, click Run, type cmd.exe, and then press ENTER.
2. Type set devmgr_show_nonpresent_devices=1, and then press ENTER.
3. Type Start DEVMGMT.MSC, and then press ENTER.
4. Click View, and then click Show Hidden Devices.
5. Expand the Network Adapters tree.
6. Right-click the dimmed network adapter, and then click Uninstall
uninstall any other physical devices not needed

Investigation
- Interesting - on the VM servers when looking at Task Manager the %cpu listed individually for all the processes for all users did not appear to add up to what was showing up on the Performance tab (at least 50% discrepency.) This was not observed on the physical server
- For both VM's and physical servers: Citrix Performance Monitor was showing warnings and intermittent error conditions on %cpu, %interrupt, context switches/sec.
- The VM's cpu utilization on the host machine is extremely high. On the server with the greatest number of users it maxed out the host cpu for much of the time I watched it.
- Watching performance monitor a few minutes showed context switches/sec to be in the hundreds of thousands.
- Opened Process Explorer and set view to show context switches and context switch deltas. I observed that at times it reported up to 50% cpu was due to hardware interrupts (this was not as dramatic when I checked it on the physical machine so I wonder if this is a reporting issue related to vmware's magic behind the scenes.) Also, the highest context switch delta was for hardware interrupts so Process Explorer was no help to further isolate it.
- To isolate what driver or program might be causing this issue, I piped the output of pstat.exe to a file and looked for the highest count of context switches. I took the memory address of that thread and looked it up in the bottom section to find what address range it fell in. In this case it was CDM.SYS
- google search of CDM.SYS turned up multiple articles about Citrix servers. I think CDM stands for Client Data Mapper. Of greatest interest is an article about a hotfix for PS4.5:
http://support.citrix.com/article/CTX114121 (and I see a lot of other post FR1 hotfixes out there too.)
The issue resolved in this hotfix is:

"Winlogon.exe shows higher than average CPU consumption on the server. The issue occurs because the server refreshes the smart card reader state more frequently than necessary. This occurs even if smart cards are not being used. With this fix, the reader state is refreshed only once per noticeable event."

Microsoft Clustered Print Server

Creating and Configuring a Highly Available Print Server

Print Services

KB278455

KB197046

Windows::Management

Find how long since you last restarted:

systeminfo | find "Up Time"

The 4GB Windows Memory Limit: What does it really mean?: "The 4GB Windows Memory Limit: What does it really mean? "
and
http://www.brianmadden.com/forum/tm.aspx?m=16032