Good Monday

- standard Monday morning stuff: timesheet corrections, review of schedules,
- short staffed: one guy broke his foot over the weekend, another going to the doctor for strained back. another left early because his wife was sick and his child got sick at school and had to come home.
- Help calls and emergencies were not excessive -- it's a good thing, I only have two hands and three keyboards.
- First day of school today and I spent my final moments on my way out the door this morning "encouraging" my children to stop whining and fighting. This will be a fun year of homeschooling if this morning lasts all week.....
- meeting with vendors
- on licensing.microsoft.com downloading enterprise and standard versions of Windows 2008 R2 -- nearly 5GB!! Had to delete some old stuff off my IT server to fit these ISO's in.
- signed up for Laura Chappel free WireShark class this week.
- trying to cleanup notes and other work knowledge that is scattered around several places. Deleting old stuff.
- familiarizing myself with lightweight access points/wireless lan controller wireless solution.

Finicky QOS

Laura's Blog
The link to Laura Chappel's blog is really a teaser she posted to make us interested in her "top 10 reasons why the network is slow" online training session.
If "training" was not a bad word around here, I'd recommend a couple of us attend that session and all the rest of them too. They are just $99 and they are awesome. I was able to watch one of them some time ago and it was extraordinary.
Anyway, her blog has a link to a sample capture file. I wish I had gathered some captures a while back on our network because we had this very issue at one of our WAN sites.
That office moved and got a new phone setup: all IP phones pointed to an enterprise phone system back in the HQ datacenter. That was all fine and good except the voice contractor doesn't listen to anything we say and we spent several days coming to some kind of half understanding of what the setup was and how we needed to make adjustments to QOS.
Basically, we just need to dedicate half our circuit to the EF queue and then stick everything in it. :) Having work now with both Avaya and Cisco VOIP people I find there is a really big gap in Avaya contractors giving a care about communicating with network people.

Finally Friday

This has been a long week. My boss was away and gladly it was fairly quiet. My network guy was out at the beginning of the week and broke his foot last night so was out today too.
- fighting with moving backups off one of the backup to disk appliances so we can get rid of that physical unit. It has been a long road but we are finally where we need to be on backup storage, at least for now. Along the way it took a lot of acrobatics and some loaner equipment from the vendor. Once we get all the data off it we can return it. Most of the data "aged off" as designed, but some backup data we wanted to keep long term needed AUXCOPY'd off. This took all week and we had to reschedule moving the equipment until next week because it wasn't done.
- just finished paperwork for performance appraisal for our best guy. Once I review it with him next week I will be done with administrative BS for a while.
- speaking of administrative BS, our HR dept just forced all the managers to read a lame book and attend discussion sessions. It was what you might expect.
- Planning more for upcoming extranet work and it's starting to shape up to the point I actually think we can get it done and it will work. Got a design drawing done and listed off what we need to buy. still need to do a formal project plan.
- Scheduled weekly meetings with all my staff. I had tried to save time and frustration by just having a big group meeting with everyone there. This meeting is very helpful to everyone and we get actual work done in them. However, it didn't really accomplish the same thing as personally reviewing status with each one individually.
- spent some time talking with some guys from another firm like ours who aren't quite as far along as we are with Exchange 2007. I saw them on a list serve asking how well failover and failback worked with SCR replication. We had worked for a while under the delusion that we could fail back from an SCR failover. It was gratifying to help somebody out by sharing that wouldn't work, but I also got to compare notes about how they were doing VMWare and other stuff. I need to focus on "networking" in the social sense much more like this.
- got a bit done this week in the way of catching up and cleaning up. and left on time more days than usual.
- will need to begin planning for either adding tier2 storage or finding a way to age more stuff into tier 3. We aren't nearly in as bad a shape as we were at the beginning of the year. that was a true crisis, we came close to a disastrous situation. I've been summarizing and reporting free space weekly and have a much better handle on what we have in use. It is just going to be hard to push for adding 2 more trays of storage after already adding disk this year.

Exchange 2007 Client Access Server: Proxy

This is an unbelievable pain.
Exchange 2007 CAS works fine with Exchange 2003 mailbox servers in other AD sites. Why can't it connect with a Exchange 2007 server in another AD site and not require proxying which means we have to used integrated authentication.
That really sucks because all our users a lulled into not having to put their domain in front of the userid when they logon -- all thanks to forms based authentication.

Day in my I.T. life

Today I changed the name of this blog to reflect my endeavor to post more than just little snippets of notes and references for problem solving. I will continue to post those as always, but in addition I am going to attempt to chronicle my life as an IT Manager at a professional services firm.

I wear many hats even though I work as part of a management team in a fairly large IT organization. I get to be "specialized" to the point of being focused on enterprise-wide infrastructure operations and projects. Although I tend to hate my job, I love the technical work that I do and am privileged to work with a diverse team of fun people. Aside from that, like most organizations, I think my greatest challenge is the organization of this business, even down to the organization of the IT department. Part of being in this sub-department includes being a catch-all place for unknowable questions and the final destination of superstitious help requests that nobody else can figure out and deem them "network issues." My documented responsibilities are enterprise wide design and support of server side systems including physical servers and storage, messaging, server workloads(vmware guests), and infrastructure such as DNS, DHCP, etc. The enterprise team is unfortunately without a network manager and hiring is frozen indefinitely. That means I get to split network responsibilities with my boss. This adds some aspects to my job that I enjoy, but it makes for busy days.

Today's activities:

- arrive at work, urgently needing the restroom after a long commute.
- before visiting the facilities I got pulled into troubleshooting an odd problem
with a system supporting our cardkey system. Long story on background, but a
recovery operation conducted on a large number of VM's to fix a storage
issue, somehow resulted in the software being "unregistered" since it didn't
see the server as the same hardware. Delegated someone to work with this
totally useless vendor to re-register or reinstall or whatever.
- discussed with my guys some alerts we received overnight and a helpcall
regarding access to our intranet from a specific office. Suspected an issue
with Riverbed (much more on this thing in the future...) However this
problem has gone away by the time we troubleshoot it.
- restroom break - finally.
- respond to stupid e-mails from stupider people about a project that has been a goat rodeo since it's botched initiation phase. As the non-stupid person on the project team it is my task to do all useful work and answer every single idiotic question they come up with now that the vendor seems to be checking out since the poorly defined
requirements are 50-85% complete. (I find I cope better when I have low expectations and don't try to be the superhero of every project, especially the more stupid ones.)
- received update that completely uncooperative local tech in one of our asia offices failed to follow instructions for connecting a new physical server to the network switch and we will not be able to move forward with installing it until another day is wasted on asking for her to correct the situation.
- one on one meeting with my most experienced guy. Discussed many little items that had cropped up while he was out of the office on leave. And walked through the security architecture of a get-it-done-yesterday extranet project that has long been whined for but at numerous times when we get into details and ask questions about requirements the effort has fizzled out.
- Got update on cardlock server which is dragging on half the day to do what the retard vendor said would take 5 minutes
- Get coffee and run into extranet developer, successfully dodge questions and avoid
standing there all day getting grilled about stuff that isn't approved.
- research into Scrutinizer netflow aggregation software. briefly compare with
Orien plus SolarWinds with a colleague who had used them in the past.
- use Scrutinizer to further investigate an issue with qos for an office that was
recently expanded and reconfigured. [Yesterday spent watching the service policy
report dropped packets from priority queue (over 2000 in 24 hours.) Tried using IP Accounting to sort out if something strange was going on or if there is just too much traffic.] Scrutinizer is a great utility and I am going to recommend
purchasing it. The free version is useful on it's own -- I gained several
insights today -- but actually buying the real product will give us tons more
features/options and the ability to store the data longer than 24 hours and get
trends reports/etc.
---
An aside here -- even though the telephone system is under the responsibility of our department it is very tighly siloed and many systems details are hidden deep in documents that nobody knows are there or are on screens in systems that we don't have a logon to access. Paranoia and limited networking skills on the part of our one primary resource are a big headache as we integrate more with voice systems. It's been 18-24 months that we have been in the same department and yet we are still dragging details out about how IP trunks have been configured and what systems a
voice call passes over.
---
- Get more coffee. It's already after 2 and our cut-to-the-bone staff is light today due to vacations and sickness and I'd probably get called back if I tried to go to lunch so I'll just eat this coffee cake that somebody dumped here in the kitchen after
nobody ate it at some meeting or something.
- After finding some interesting things in the netflow data I have some questions to ask of the voice person/vendor. I also see some odd things hitting the priority queue that shouldn't. I check the router configuration and see an obvious error and correct
it.
- after seeing over 2000 drops again today we shall see what tomorrow
brings. However I am still seeing packets getting dropped.
- Talk to yet another useless network service provider about a chronic problem with a circuit on a backup WAN link to one of our domestic offices. This circuit went down
shortly after activation and went unnoticed for a week or so since it was a redundant T1 on a backup WAN router. When it was reported it spent a week getting passed from the service provider to the LEC and back only for them all to report it was fine, even though nobody ever called us about it and it was still down. Ticket was reopened and then mysteriously got closed without any action. After over a month we are escalated to a service manager and this guy was so scatterbrained. He put me on hold about 5 times while he talked to people about my issue and got paged by his boss about something else, etc. Every time he gets back apologizes again that this ticket got closed for no reason last time. He made lame small talk while his slow system came up and expressed a sentiment that I share but I would never tell one of
my customers: "I love routers, I'd rather work with routers than people..." ROTFL ... So anyway while I'm on the phone trying to get him to call and yell at the LEC again to get them to finally come onsite to check this out apparently the LEC is doing testing on this circuit for the 3rd time. And by the time I get off the phone with this crazy guy the circuit is actually working. Then I get to call this guy back and go through a long process of convincing him that it's now working after he reads the number wrong pulls up the wrong interface and swears to God and me that it's still down. Finally he is convinced and will call the LEC and tell them not to go onsite and charge us a fortune.
- Get a negative response to a backchannel request to get the RTM
of Windows 2008 R2 now, instead of 2-3 weeks from now, so that we can proceed
with upgrading our domain prior to another big project that is likely to affect
AD. Realize I've never gotten a response to my official request from our TAM
that seems to spend his life e-mailing everybody patch notices and e-mails full
of links to stuff on the MS website.