Part 1: Taking the pain out of server procurement

One day, a few years ago a stack of HP equipment showed up for job we had just won.  Everyone was excited.  "Did you hear?  We just got [insert amazing high-visibility job X for key client Y]".  "That's great Eric... but did you see that stack of HP equipment sitting in receiving... what's that for?", I asked.  "That's for this new job, I ordered it yesterday as soon as we received the PO."  "Uh-huh... and who did the spec?", I asked... knowing the answer.  "Don't worry... you did... I just switched out the chassis, and left most of the other parts the same from our last job the same.  They're all low-utilization servers." (Says the person who has neither heard of esxptop, nor has ever used a performance monitoring solution).  I also replaced a couple of things with stuff I found on newegg, mostly just RAM and hard drives... they're not HP branded, but I think we've really been overpaying some of this stuff."  And so it goes...

If you don't have a vendor management office (VMO) doing this kind of stuff for you, then you're like just about every other SI out there.  Maybe your procurement process has grown organically as your company has grown, or maybe Bob in Accounting always does it... or maybe you're fortunate enough to have a Systems Engineer who inherited the responsibilities from someone else, and actually does know what he/she is doing (at least from a technical standpoint)... and that enables the ship of procurement to sail unhindered until something stupid happens, and then everything is a wreak.  So even if you think what you're doing works, you might want to go back and compare your process with mine, and see if there are any gaps.

Over the course of the next few posts, I'm going to talk about my server procurement process.  I suspect that most SIs have to deal with this, so I'm going to go over what I've picked-up in the time since Eric stopped doing procurement.  I can't say that I've covered everything, or that you can beat a VMO at their own game, but I've picked-up a few things that help prevent scenarios like the above from happening, while at the same time enabling you to be more competitive (both in terms of pricing, and in terms of eliminating labor), and ultimately help you find the hidden server margin.  Because, like you... I primarily care delivering the scope, on schedule, under budget at the level of quality that I committed to.  Hardware and software are just necessary to insure we can deliver the scope, so let me help you take the pain out of procurement, by giving you my process.

SMB: Margin on Hardware?

I know most SMB IT service providers are fans of Dell hardware pricing – and I’m not knocking it, we still do quite a bit of volume with them. But if you do have a consistent volume of hardware purchases, you might be surprised by HP’s pricing. Check out HP Smartbuys with your distributor, and spend some time learning about the HP PartnerONE program (I know it’s no fun – but take a look ). I’m not saying that HP is the solution, but in my experience there's more opportunities to find margin. Some of the partner pricing and quarterly internal benefits are interesting too.

Another thing to consider – if you’re consumed with running your business - hire someone to do the vendor management piece. When it was just me ordering everything though Dell – we got by. But since adding someone to manage the vendor piece, we’ve easily covered their cost, and now someone is incentivized to work the pricing.

Dell Pricing via the reseller Premier Site

If you source hardware from Dell, have you checked out the Dell Premier Site yet?

As a quick spot check, I priced thee identical business-class workstations… one from the regular site, one via our Dell rep, and one via the Premier site. Now, granted, I didn’t talk to our rep and tell him that I needed some extra margin on the order, but the Premier site had the edge beating our rep by 4-5%, and both beat the regular web-site (even with web-site discounts applied).

Not earth-shattering, but not bad either. And no, you don't need to provide your customer information when ordering via the Premier site. For details, check with your rep - he/she can tune up your account.

Hardware: Adding drives to a PE2650 with hot-swappable bays takes how many reboots?

The answer is 6.

Okay, I’ll be fair… 4.

So we have a new client that initially involved us because one of their Dell PowerEdge 2650’s with 2003 was “running slowly”…, after an initial review we could better phrase that as… “out of drive space”… but otherwise at first glance it was none too interesting.

So after verifying that we have good backups of the box, and freeing up enough space to start working we checked the server’s application stack to make sure it was supportable, applied SP1 and rebooted. Then with the intention of configuring two new drives in a RAID1 configuration, we installed the Dell Open Manage Administrator (OMSA) 5.1.0 with the Storage Manager component, and rebooted. After logging into OMSA interface, we checked on the Storage component, clicked the Information/Configuration tab, and got… nothing. Under Global Tasks, it said “No Task Available”, and no storage controllers were visible. Which isn’t’ good. So we talked to Dell Open Manage Support, and they knew exactly what it was… the server wasn’t using the Dell driver for the PERC3/Di controller… fair enough. After talking to support, we came up with a path-forward …

1) Download latest driver then reboot
2) Update the firmware then reboot
3) Apply a patch to “make the warning icon in device manager go-away” (uh-huh…), reboot…

So that’s pretty reasonable for the most part, right?

Well, because the Dell PE2650 isn’t ours, and isn’t a box that we’ve standardized on, we talked to tech support a bit more about the process of adding the new drives. Well, the process really wasn’t what I expected. It turns out that even after doing the driver, firmware, and patch updates, we still won’t be able to configure the drives using the web-based OMSA. Which sounds a bit out of sorts… isn’t the whole point in having OMSA and hot-swappable bays to minimize reboots, keep people out of the RAID controller BIOS, and meet SLAs? Now maybe I’m not adding drives to a PE2650 everyday… but that just doesn’t sound right to me.

Well, if we take the Dell recommended approach we’re going to have to do the following:

1) Shutdown the server
2) Remove the existing drives, and add the new drives
3) Use the RAID controller BIOS to configure the new drives
4) Turn off the server. Add the original drives back in.
5) Enter the RAID controller BIOS, and accept the notification that “Changes have been made”
6) Reboot. The server should then boot normally.

So all things considered that’s more risk than we really anticipated, not to mention time and reboots. Granted, we probably wouldn’t had been in this situation if whomever built the box had used the Dell drivers, and installed OMSA during the initial install, but we don’t want to point fingers. Further, we could have noticed the driver issue before going to Dell support… ultimately, we want to show-value to this new client, and start building trust. With the server about to go out of warranty, we’ve decided to use the drives that had been purchased in a different server, move a couple of shares around, and recommend that the server be repurposed out of the business critical role that it currently occupies.

How to handle a similar situation better in the future? If this were an existing client, we probably would have managed the issue differently. Certainly if we envision working with the PE2650 line more, or the Dell PERC controllers more, we need to come up with some internal procedures for working in the BIOS. Otherwise, we’d be accepting too much risk on behalf of our client – which we really shouldn’t be doing in the first place.

Let me know if you have any thoughts on this process, and if we had any major misses that we could have addressed.

Network Administration: Backup Exec 9.1, Dell Ultrium LTO 2 issue

This past weekend I added an external Dell PowerVault 110T tape drive (Certance Ultrium LTO 2) to a server, in order to replace an aging drive. The new server is a Dell PowerEdge 2850, with an Adaptec 39160/3960D – Ultra 160 card cabled to the external tape drive.

After getting the machine built, installing Veritas Backup Exec 9.1, including the latest firmware for the tape drive, I ran into an interesting problem. The tape drive appeared to be recognized by Backup Exec; it came up under the devices tab and was identified as “CERTANCE 1”. However, when a tape was inserted, Backup Exec 9.1 acted as though there was no media, displaying a message of .

Test backup jobs would just fail with a status of “failed”, having no error code or additional explanation. Running an inventory on the tape would result in the following error message: “a0008101 - Invalid Physical Volume Library Argument.”

I found a number of references to running “tapeinst.exe”, but this really didn’t apply to a new installation – though I did go through the recommended steps. Considering this is a new model drive, and that backup exec 9.1 isn’t the latest version available, this was leading me to believe that the latest driver I was using from the manufacturer wasn’t something that Backup Exec liked, as BE9.1 really prefers you use Backup Exec provided drivers.

Fortunately, Veritas does provide a driver update pack, which I found here. Worth noting is that it doesn’t specifically say my drive is supported; though on installing the driver pack, it does detect the drive and installs a new driver, which seems to resolve the issue.

WMI: Using WMI to Find Dell Express Service Tags

Need an easy way to determine the express service tag of a Dell machine, or a series of machines? Just copy and past the following code into a *.vbs file, and run it using the following command line: “scriptname.vbs Workstation1 Workstation2…”.

On Error Resume Next
If Wscript.Arguments.Count = 0 Then
Wscript.Echo "Usage: scriptname.vbs workstation1 [ws2] [ws3] ..."
Wscript.Quit
End If

For Each strComputer In Wscript.Arguments
Set objWMIService = GetObject("winmgmts://" & Computer)

If Err.number <> 0 then
Wscript.Echo Computer & ", " & Err.Description
'Err.clear Resets the error code
Err.Clear
else

Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * from Win32_BIOS",,48)
End If

For Each objItem In colItems
Wscript.Echo strComputer &amp; ": " & objItem.SerialNumber
Next
Next