DPC triangle…

So once upon a time, a PFE had a problem with his video game stuttering.  The game would be playing fine and then suddenly, the sound would skip until he rebooted his PC.  Nothing in the event log, device manager reports all good…

So we collected an xperf trace using the following flags:

xperf -on Latency+DISPATCHER -f kernel.etl -stackwalk CSwitch+ReadyThread+ThreadCreate+Profile -BufferSize 1024 -MaxBuffers 1024 -MaxFile 1024 -FileMode Circular

Reproduced the problem..(-FileMode Circular, you see, makes xperf collect a black box trace, auto-overwriting itself, oldest data drops…)

Then post repro, run

xperf –d results.etl

Easy as pie!

image

 

DPCs are way high here.  It’s a 4 processor box with very high DPCs on one processor and spikes in other CPUs…

(DPCs are discussed a bit here and here.)

So I was interested…

image

 

So here we have a gap of time where CPUs were ‘busy’ doing a DPC operation, interrupt, or processing a stack.  For something like .8 seconds, long enough to be a sound gap noticeable to the human ear…

An analysis of the stacks revealed we were grinding in DPC\USBPort.sys, but in the stack it was referencing USBAudio.sys.

A quick search found he was running an old version of USBAudio.sys, and this hotfix was available:

The audio applications stop responding in Windows 7 or in Windows Server 2008 R2 after you resume the computer from the S3 sleep mode

http://support.microsoft.com/kb/2122063

The symptoms didn’t match exactly, but once he updated that driver, the problem vanished…

BPAs, not just for Exchange anymore…

So for those who don’t know, I used to be mainly an Exchange dork…ESE flowed in the veins don’t cha know?

Anyway, one of the tools that rocked (and still does) is the ExBPA and the family of Exchange Analyzers.  But did you know in Windows Server you get analyzers as well?  And some are even built into the OS?

Active Directory

AD

The ADBPA is easily accessible in the Server Manager console in Window Server 2008 R2.  It is not, I repeat, NOT, an ADRAP, by any stretch.  But, it’s free…

DNS

has its own as well…

DNSFree

What else has a built-in BPA? 

It’s a secret…just kidding, I’m going to list them right here for you…

· Active Directory Certificate Services

· Active Directory Domain Services

· Domain Name System

· Internet Information Services

· Remote Desktop Services

(Above links stolen from WSiX blog here)

Basically, some of the core infrastructure parts of Server now have health monitors built in, and you can powershell them!  Why not run them once a week and dump the xml results to a directory archive?

But what, what else has a BPA?  You might ask…

GPOBPA

http://blogs.technet.com/b/askds/archive/2008/04/11/group-policy-best-practice-analyzer.aspx

Lync 2010

http://blogs.technet.com/b/ucedsg/archive/2011/02/18/lync-2010-best-practice-analyzer-is-now-available.aspx

SharePoint

http://www.microsoft.com/downloads/en/details.aspx?familyid=cb944b27-9d6b-4a1f-b3e1-778efda07df8&displaylang=en

SBS

http://support.microsoft.com/kb/940439

Hyper-V

http://support.microsoft.com/kb/977238

TMG

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8AA01CB0-DA96-46D9-A50A-B245E47E6B8B&displaylang=en

SQL 2000

http://www.microsoft.com/downloads/en/details.aspx?FamilyId=B352EB1F-D3CA-44EE-893E-9E07339C1F22&displaylang=en

SQL 2005

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=DA0531E4-E94C-4991-82FA-F0E3FBD05E63

SQL 2008 R2

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=0FD439D7-4BFF-4DF7-A52F-9A1BE8725591

 

All of this is free, as in beer, so check it out.  Script the collections of your core infrastructure services (or other services for that matter) and store that output.  It’s a monitor for un-documented changes Smile).

Yes, it matters

Often at customer engagements when I encourage them to use things like SCCM and SCUP along with HP or Dell SCUP integration to keep drivers (and firmware!) up to date, I’m told it isn’t worth it.  If the drivers from 2006 work, it doesn’t matter if an update is out.

More often than not, the reason I’m there in the first place is to analyze and solve slow boot/client performance….

These two statements above are connected, I promise.

Lets take a walk through my Dell e6500’s life cycle for example.  When I started in PFE I was issued a Dell e6500 laptop with 8 gig of RAM and a 7200 RPM drive.  BIOS rev A08.

Lets look JUST at BIOS as an example:

image

Line item 4.  Slow boot performance (a user isn’t going to understand it’s a PXE issue, they just complain it boots slow)

But wait, there’s more…

 

image

Line 1 is interesting.  If you were rolling out Windows 7 to this machine, it MIGHT work with previous BIOS versions, but wouldn’t it be cool to be in a supported configuration from the company who made the hardware?

And look, Line 3, updated the Nvidia BIOS, either we’re fixing something or making it faster…

And hey, Line 6, access speed for PCMCIA cards.

image

Hey look, Ambient Light Sensor “improved support” for Windows 7.  Wonder if that fixes my slow boot issue I blogged about previously on ALSSVC64.exe adding 20 seconds to my boot time…

image

Ah, we remove, REMOVE, AAM on line 2.  Remember I blogged about this, the feature to slow your hard drive so it boots your system slow but doesn’t make noise…

But I digress.

image

Hey look, line 4.  Nvidia BIOS update again.  Fixes problems or improves performance (or why was it written?).

Am I picking on Dell here?  NO!  Does Dell make bad hardware and this is why there are so many fixes?  NO!  Every vendor with a brain makes stuff that can be upgraded.  Does anyone recall back when the old Pentiums had the divide by 0 bug?  And hardware was replaced/RMA’d?  Yeah.  Updates = good.  It means the vendor is servicing the product line, taking feedback and aggregating service call data and improving their product for you, the end user/company.

Note that none of these BIOS improvements cost anything except the time to download and apply them.  Free performance gains.

My laptop took a nose dive off a desk last week, so I am lacking in good solid pictures to prove the gains, but they are appreciable.

 

Ok great, lets look at something you paid for, Anti-Virus!

I am NOT naming this product, it’s a picture example of what a simple update from one version to another can do to the disk IO at boot.  Note that AV engine updates are usually pretty simple to roll out in an enterprise.

PRE update:

cleanmachine.disk

POST update:

cleanmachine.disk (2)

 

Tell me, which sytem would you rather want to be on?  Given we’re looking at the disk activity from 0 seconds to 230 seconds, and more disk activity is more likely going to mean a lack of responsiveness, I’ll go with POST myself. Smile

Ok, what was the point of this rant?

Glad you asked.  Computers are like anything else.  Our bodies, our cars, our homes, our loved ones, all require maintenance and care.  Give your computer some love today, go to your vendors site, see if any updates are out there, and apply them if they are appropriate, if they make sense, you know?

SCUP and System Center are a great way to keep things current, with an approval process, in a large environment, and I think they are worth investigating.  Or why not, when rolling out that new image, include a BIOS update as part of the task sequence?

XPERF, from the WPT, is the way to analyze the impact.  Like in the screen scrapes above.  The ONLY change made, was an AV engine update.  Easy to look at this and say “Yeah, that’s an improvement”.  If its such a subtle thing in a test that you can’t tell, chances are it’s not worth rolling it out in the enterprise you manage.

Food for thought.

Jeff