Calculating IOPS Requirements

IOPS = input output operations per second
A measure of demand and a measure of capability.

IOPS Demand
Servers - perform monitoring, refer to os & app vendor information on requirements
Users (virtual desktop) - about 25 iops for a typical user running multiple apps at once, 2GB RAM, single CPU.

IOPS Capability
IOPS per disk = Rotational latency + Seek Latency / 1000

Disk Speed     Est IOPS
7200 rpm            75
10000 rpm          125
15000 rpm          175
SSD                    6000 (?)

Read vs Write
Typical average:
40% Read, 60% Write

RAID "Penalty"
Write operations to RAID disk arrays require additional io operations to write parity data.
see more at theithollow.com

RAID Level          Write i/o Penalty
     0                              1
     1                              2
     5                              4
     6                              6

Calculation of required capability to meet demand:
IOPS Required =
(IOPS Demand * Read i/o%) + (Target IOPS * Write i/o% * RAID Penalty)

Unfortunately, I don't find such a scientific way to factor in the affect of caching/etc.

For example
Demand = 25 iops
read% = 40, write% = 60
RAID 5

(25 * 0.40 + 25 * 0.60 / 4) = 70
** Nearly triple!

So for 1000 users generating 25000 iops, we need 70000 iops on the "back end."

70000/175 = 400 15K disks would be required - holy moly also see yellowbricks.com

Powershell: Count datastores on VMWare Hosts

Powershell: Count datastores on VMware Hosts

All the LUN mappings & datastore names need to match on all hosts in a cluster. I hope to someday script a more comprehensive comparison of datastores & names. However, a quick and dirty confirmation is to count the datastores that are connected on every host in each cluster. If they match, it at least gives me a warm feeling.

#####################################################################
#
#   
#        Gather count of LUNs/Datastores connected to all hosts
#

$vcs = Read-Host "vCenter"
$user = Read-Host "userid"
$pw = Read-Host "Password for $user" -AsSecureString
#convert $pw to plain text
    $pass = [Runtime.InteropServices.Marshal]::PtrToStringAuto(
        [Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw))

connect-viserver -Server $vcs -User $user -Password $pass

$pass = " "

$today = get-date
$day = $today.Day
$mth = $today.Month
$year = $today.Year
$hour = $today.Hour
$min = $today.Minute
$sec = $today.Second
$date = "$year-$mth-$day-$hour$min$sec"
$outfile = ".\datastores-"+$vcs+"-"+$date+".csv"

clear-host

"Host-Name,Datastore-Count" | out-file $outfile -encoding ascii
foreach ($VMHost in (Get-VMHost -Location $Cluster)) {
"    $VMHost"
    $dstores = $VMHost | Get-Datastore
    $ds = $dstores.count
"        $ds"
    "$VMHost,$ds" | out-file $outfile -encoding ascii -append
}

disconnect-viserver -confirm:$false

Powershell: Compellent SAN configuration

Add Servers, Create boot volumes, map volumes to servers. Requires the right version of Compellent Storage Center and the Compellent plugin for Powershell

$user = Read-Host "userid"
$pw = Read-Host "Enter Password for $user" -AsSecureString
$san1 = get-scconnection -HostName san1 -User $user -Password $pw
$san2 = get-scconnection -HostName san2 -User $user -Password $pw
#$pass ='' #erase plain txt pw
#remove-scserver -connection $san1 $server

$inputfile = ".\test.csv"
$profiles = get-content $inputfile

foreach ($line in $profiles) {
$line
 $line = ($line -split',')
 $profile = $line[0]
 $wwn1 = $line[1]
 $wwn2 = $line[2]
 
 #Create Server
 $s1server = new-scserver -connection $san1 -name $profile
 $s2server = new-scserver -connection $san2 -name $profile

 #Set WWNs
 add-scserverport -connection $san1 -scserver $s1server -worldwidenames $wwn1
 add-scserverport -connection $san1 -scserver $s1server -worldwidenames $wwn2
 add-scserverport -connection $san2 -scserver $s2server -worldwidenames $wwn1
 add-scserverport -connection $san2 -scserver $s2server -worldwidenames $wwn2
 #Set Server OS Type 
 $s1ostype = get-SCOSType -index 35 -connection $san1
 $s1server = get-SCServer -connection $san1 -name $profile
 $s2ostype = get-SCOSType -index 35 -connection $san2
 $s2server = get-SCServer -connection $san2 -name $profile
 set-scserver $s1server -connection $san1 -SCOSType $s1ostype
 set-scserver $s2server -connection $san2 -SCOSType $s2ostype

 #Create Boot LUN
 #use "Boot LUNs" storage profile
 $storageprofile = get-scstorageprofile -connection $san2 -name "Boot LUNs"
 $volname = $profile+"_boot"
 $folder = get-scvolumefolder -connection $san2 -name "BOOT LUNS"
 $volume = new-scvolume -connection $san2 -name $volname -parentfolder $folder -scstorageprofile $storageprofile -size 10g
 #map volume
 $map = new-scvolumemap -scvolume $volume -scserver $s2server -connection $san2
}#end foreach profile
Remove-SCConnection $san1
Remove-SCConnection $san2

Powershell: Cisco MDS Fibre Channel Switch Zone Configuration Builder

Create Commands to configure zones on Cisco MDS 91xx switch

####################################################################################################
#
#  fc-cfg-bldr.ps1
#
#  Create cmd file for Cisco MDS fibre channel switch to create zones for new servers
#  INPUT:  CSV file containing list of server names and WWN's.
#  OUTPUT:  2 TXT files containing commands to create zones in Fabric A and Fabric B.
#

$inputfile = ".\test.csv"
#$inputfile = ".\servers.csv"
$outfile = ".\fc-cmds.txt"
$tempA = ".\\configA.txt"
$tempB = ".\\configB.txt"

$profiles = get-content $inputfile

$today = get-date
$day = $today.Day
$mth = $today.Month
$year = $today.Year
$hour = $today.Hour
$min = $today.Minute
$sec = $today.Second
$date = "$year-$mth-$day-$hour$min$sec"

clear-host

"------------------------------------------------------------------------------"
write "! $date Fabric A Configuration" | out-file $tempA -encoding ascii
write "! $date Fabric B Configuration" | out-file $tempB -encoding ascii

$zonesetA = @"
    zoneset name SAN1-SAN2-FAB-A vsan 2
"@
$zonesetB = @"
    zoneset name SAN1-SAN2-FAB-B vsan 3
"@

foreach ($profile in $profiles) {
    "!------------------------------------------------------------------------------" | out-file $tempA -encoding ascii -append
    "!------------------------------------------------------------------------------" | out-file $tempB -encoding ascii -append
    $profile
    $server = ($profile -split',')
    $name = $server[0]
    "    $name"    
    $wwn1 = $server[1]
    "    $wwn1"
    $wwn2 = $server[2]
    "    $wwn2"
    
    $zoneA1 = $name+"_hba_A_to_cmp1"
    $zoneA2 = $name+"_hba_A_to_cmp2"
    $zoneB1 = $name+"_hba_B_to_cmp1"
    $zoneB2 = $name+"_hba_B_to_cmp2"
      
    #create commands
    "    Generate Commands"


    #Create Zones in Fabric A
    "        Fab A Zones"
    $configA = @"
    zone name $zoneA1 vsan 2
        member pwwn 50:00:d3:10:00:0c:80:03
        member pwwn 50:00:d3:10:00:0c:80:09
        member pwwn 50:00:d3:10:00:0c:80:11
        member pwwn 50:00:d3:10:00:0c:80:17
        member pwwn $wwn1
    
    zone name $zoneA2 vsan 2
        member pwwn 50:00:d3:10:00:0c:82:03
        member pwwn 50:00:d3:10:00:0c:82:0b
        member pwwn 50:00:d3:10:00:0c:82:13
        member pwwn 50:00:d3:10:00:0c:82:1b
        member pwwn $wwn1
"@
    write $configA | out-file $tempA -encoding ascii -append
    
    #Add to zonesetA
    "        Add to ZonesetA"
    $configA = @"
    
        member $zoneA1    
        member $zoneA2
"@
    $zonesetA = $zonesetA + $configA
    
    #Create Zones in Fabric B
    "        Fab B Zones"
    $configB = @"
    zone name $zoneB1 vsan 3
        member pwwn 50:00:d3:10:00:0c:80:0d
        member pwwn 50:00:d3:10:00:0c:80:05
        member pwwn 50:00:d3:10:00:0c:80:1b
        member pwwn 50:00:d3:10:00:0c:80:13
        member pwwn $wwn2
    
    zone name $zoneB2 vsan 3
        member pwwn 50:00:d3:10:00:0c:82:0f
        member pwwn 50:00:d3:10:00:0c:82:05
        member pwwn 50:00:d3:10:00:0c:82:1f
        member pwwn 50:00:d3:10:00:0c:82:15
        member pwwn $wwn2    
"@
    write $configB | out-file $tempB -encoding ascii -append
    
    #Add to zonesetB
    "        Add to ZonesetB"
    $configB = @"
    
        member $zoneB1    
        member $zoneB2
"@
    $zonesetB = $zonesetB + $configB
    "-------------------------------------------------------------------------------"
}#end foreach
    "!------------------------------------------------------------------------------" | out-file $tempA -encoding ascii -append
    "!------------------------------------------------------------------------------" | out-file $tempB -encoding ascii -append


#Config zonesets
    write $zonesetA | out-file $tempA -encoding ascii -append
    write $zonesetB | out-file $tempB -encoding ascii -append

#Activate & Save
"        Complete Configs"
    $configA = @"

        zoneset activate name SAN1-SAN2-FAB-A vsan 2
    
        zone commit vsan 2
    
        copy run start
"@
    write $configA | out-file $tempA -encoding ascii -append
    
    $configB = @"

    zoneset activate name SAN1-SAN2-FAB-B vsan 3
    
    zone commit vsan 3
    
    copy run start
"@
    write $configB | out-file $tempB -encoding ascii -append    

"-------------------------------------------------------------------------------"
"    COMPLETE - Configuration commands saved to $tempA, $tempB"
"-------------------------------------------------------------------------------"

HP Virtual Connect Boot From SAN

HP Virtual Connect Boot From SAN Configuration

Only allows entry of 2 target WWPN's:



Month of Chaos

Month of Chaos

Keeping head barely above water is all the progress made in the month of September.
- budget prep crunch for my bosses so I was hopping fulfilling requests for information
- swine flu - one employee and other possibly related absences for care of sick child, pick up from child-care with fever, etc. 2 staff with vacations, 1 staff with training absence, and a regional flood which resulted in multiple staff absences in a 2 day period, and I had a 2 day family emergency as well.
- I still have outstanding the "critical" "urgent" projects from over 3 weeks ago: extranet infrastructure, Windows 2008 Domain upgrade, Database cluster hardware replacement, Shanghai vpn concentrator setup, transition to new Exchange server in Hong Kong, migrate mailboxes out of NYC offices asap to get off rickety hardware platform as well as make progress toward centralization. Progress has been made in fits and bursts, but most of my time and that of my staff has been in dealing with crises almost on a daily basis.
- The Highly Visible Crises: in LIFO order: NYC circuit flap resulting in no failover, Centralized exchange server mysteriously bogged down for 2 minutes, multiple mysterious losses of connectivity of Gig/E link connecting our stretched vlan to secondary data center, various weekend power outages from flooding/storms, security door cardscanner controllers down, new remote contractor with vpn phone problem dragging on for over 1 week must be fixed now, 18 hour total loss of connectivity to branch office awaiting local operations visit to restart router, urgent planning and discussions for replacing database cluster server hardware asafp, two SAN issues in a single day, revelations about free disk requirements for our Compellent SAN, HP virtual connect interface disconnect from HP Onboard Administrator and hurried deployment of new servers to migrate VM's to in order to fix virtual connect, HP iLO bug remotely mapping A: does not work -- the only stinking way to deploy a server is via HP RDP -- pita.
Lessons Learned
- HP iLO firmware bug prevents remotely mapping to A: disk. So there is just about no way to do a manual install and hit F6 during install to load the boot from SAN disk drivers when you are using a blade server. Perhaps this will be fixed with an upgrade to firmware, however that's a house of cards I'd rather not disturb right now. I thought I had made some progress with nLite build of windows installation, but it keeps telling me that I don't have the right disk driver for 64 bit windows 2003. Before we do a remote blade enclosure around the world we *have* to get a windows install working without having to first implement an RDP infrastructure (chicken and egg thing...)
- Duh to Local Operations: Just the fact that somebody can connect to the internet but their phone can't connect to the VPN that doesn't mean there is a problem with the phone itself requiring re-requesting the phone and building in a 3 day delay before this problem gets abdicated on account of lack of gumption to figure it out.
- IPSec vpn - phone or not - *really* doesn't work well when a user buys a router and sticks it in to his 8 year old dsl modem. This was another really phun day of telling a 60+ guy how to unplug his PC from his router into his dsl modem and guess at how to logon to it and browse for possible options for turning off NAT. At one point he got his ISP tech support on his other phone, put it on speaker and put me on speaker and had me try to talk to her. English was her second language (or third) and that didn't work out so he held the phone up to the speaker then to his ear and I would translate her question to him and coach him through our answer back and then explain her instructions to him on what to open, etc.
- Although it may be technically possible to add a tray to a Data Domain Restorer appliance it didn't really work that way for us. A graceful shutdown is much more advisable first.
- Compellent SAN would work best with 15-20% *unallocated* free space to allow for faster rebuilds of the RAIDsets of failed disks, etc. We are pushing 15% of "free" but allocated space. And we had a failing SATA drive around the time when a software bug in 4.3.2 code resulted in it "losing" the preferred paths to the SATA disks. This resulted in the "unlucky" i/o requests coming down the wrong path to pass over the backplane instead and that resulted in 4500ms (yes, 4.5 s) disk latency for the pool overall. This resulted in over 12 hours of troubleshooting and cussing and a conference call and some more stuff before this bug was isolated and resolved. That was an extraordinarly bad day in a totally crappy month.