Open-Mesh APs... just-work

It surprises me that I haven't heard more about Open-Mesh within the context of the SMB community... as it enables you deploy a convenient wi-fi infrastructure quickly, and cost-effectivly. What's more - it really is just-work easy. So... if not open-mesh, perhaps you've heard of Meraki? If not, you probably know some of their handiwork... they took over one of the semi-failed attempts at providing municipal free wifi in San Francisco and essentially turned it into their live lab. I won't bother rehashing most of the history, since there is a lot stuff out there on them, so I'll just leave it at this... Meraki was a really interesting company that evolved from MIT's Roofnet project... unfortunatly some business model changes resulting from round 2 VC funding appear to negativly impacted their relationship with a formerally loyal community.

So, flash forward to the more recent past, and you'll find a growing community over at open-mesh.com. Open-Mesh uses the many of the same bits found on the old Meraki products, expect they now support the OSLR protocol, as well as the BATMAN protocol. What makes Open-Mesh so neat is that all of the complexity and routing intelligence is built into the open-mesh firmware. Open-Mesh wifi consists of what appear to be normal AP's wired to Ethernet, or some high-speed Internet gateway... except these AP's can also serve as repeaters. In other words, if you don't have an Ethernet drop... but can see your wifi signal, just plug in another Open-Mesh AP and it will detect the signal of the other AP, and start functioning as a repeater. Obviously - this useful because you can just put them anywhere near another Mesh signal and they'll 'just work' by auto-configuring themselves.

SMB: Margin on Hardware?

I know most SMB IT service providers are fans of Dell hardware pricing – and I’m not knocking it, we still do quite a bit of volume with them. But if you do have a consistent volume of hardware purchases, you might be surprised by HP’s pricing. Check out HP Smartbuys with your distributor, and spend some time learning about the HP PartnerONE program (I know it’s no fun – but take a look ). I’m not saying that HP is the solution, but in my experience there's more opportunities to find margin. Some of the partner pricing and quarterly internal benefits are interesting too.

Another thing to consider – if you’re consumed with running your business - hire someone to do the vendor management piece. When it was just me ordering everything though Dell – we got by. But since adding someone to manage the vendor piece, we’ve easily covered their cost, and now someone is incentivized to work the pricing.

How does your bandwidth utilization look?

For most of us in the SMB segment, clients predominantly use either DSL or cable connections to the internet. Why? It’s inexpensive for one… and in general it seems to meet or exceed their needs. But how do you know when it ceases to meet those needs? Do you know how much bandwidth they’re utilizing on a daily (or hourly) basis? What about traffic spikes- how often do they occur? Have you considered the upstream limitations?

All of these things require data. As IT service providers, we make data-based recommendations to our clients – we use measurements, and performance metrics to justify these recommendations – and we proactively notify clients when things change about their network. We don't talk about how fast or slow things subjectively feel.

Great… so how do you go about getting this data?

Well, they are lots of different ways to approach this… We like to use ntop. Why? It’s pretty straightforward to install, configure, and get working – even for those new to Linux (apparently W32 binaries do exist). It’s also open-source, and free. Most importantly though, it produces graphs depicting bandwidth utilization over time. And graphs are great for visualizing problems. It does lots of other things too… it eanbles you to track down the biggest bandwidth users at a client site, it helps identify broadcast traffic where it shouldn’t be, and it enables you track down things like unexpected p2p traffic.

Of course, there are lots of other options to getting at bandwidth data… MRTG comes to mind, as does GFI Web Monitor for those looking for an ISA-based solution. You can even do some neat stuff with LogParser, ISA, and IIS. All that being said, ntop is convenient, free, and it produces results that your clients can clearly envision when you discuss utilization with them.

Reduce Your IT Workload... where it makes sense

I saw an interesting post over on Daily Cup of Tech yesterday about identifying application “champions” throughout your organization. The objective is to help manage IT workload in smaller organizations, and to get the right people doing the right jobs (i.e. your receptionists doing Word templates, your CAD people owning AutoCAD, etc.). I'm a big proponent of this - under the right circumstances - but you have to be aware of how IT is perceived by the organization if you want this to be received in the manner that it's intended.

So I posted a comment that ran a bit long (okay… ran too long to be a comment), and I just wanted to link it up here. Reason being, that while I really like the ideal of identifying non-IT people as the application “champions” or “owners”, and I’ve seen it work to varying degrees, I also wanted to post an excerpt to point out the other side of that coin when wearing the leadership/owner hat…

(Putting on the ownership/leadership hat now)If you’re in IT serving in an overhead capacity, and other departments are billable… “you’ve got a problem. It’s your job to be the technology catch-all (just like the receptionist/secretary is a catch-all). Further, I’d argue that if you think you’re perceived as “more valuable” by organizational leadership relative to the receptionist who makes 1/3 (or whatever) of what you make… you’ve probably got your head screwed on wrong. You’re 3x the cost, and your value probably isn’t perceived day-to-day. So from an ownership/ leadership perspective… if IT isn’t pulling their own weight, why keep them around? I kid you not… for the vast majority of small and midsized organizations that I’ve been involved with, the value-add of IT isn’t perceived beyond the break-fix of the day-to-day. So forget all of your big-dreams and interesting Exchange 2007 migration projects. No one really cares besides the IT group. And why does no one care? Because your IT managers consistently (and repeatedly) overpromise and under-deliver. Beyond that, every three years you go and complain about not having new equipment, not having good training, not making enough money. You’re just a cost-center that someone hasn’t gotten around to cutting…”

(Taking the ownership hat off) Harsh? Maybe… but that’s the perspective of ownership in some organizations. And the reason that I point it out is because it’s extremely important for you to understand the context for success inside that type of organization. So, if you can get buy-in for application “owners”, do it… it’s great, and it makes life more manageable in a smaller organization. But keep in mind, that while you’re trying to get buy-in on having application owners, consider the other side of the coin, and make certain that ownership/leadership’s take away isn’t that “IT’s too lazy to do their jobs”.

What’s your value proposition?

Since my last post, I’ve been reflecting on what others have been posting in comments and elsewhere about partnering with Microsoft, Software without service, and the like. One thing that strikes me about the “infrastructure” side of the work we do is that from a business perspective, delivering infrastructure is really just a necessary evil. If you think about it, infrastructure is certainly the biggest “risk” in terms of the relationship (managing expectations, delivering services, and closing the project), the most capital-intensive (outlays for equipment), and the most difficult to manage from a labor perspective (preventing overruns and the like). Now we certainly do business on the infrastructure side, and we still deliver equipment and infrastructure to our client’s “server rooms”. Are we profitable on a project-basis? – maybe… but I liken it to selling razors. We deliver infrastructure (the razor) at or above our cost, but we make our real margins as we work-up the services stack. What I mean by the services stack, is going from a break-fix shop, to having some type of “managed services” offering, to eventually becoming the trusted advisor and/or virtual IT Director for the businesses that our clients own. As we work up the stack, it’s these “softer skills” that are the most profitable… getting infrastructure on-site is the just the hook that we use to deliver everything else that we do… and the more I think about it, the less I care if I’m delivering via the cloud, or via equipment. To use another analogy… think about gold rush of the 1850’s… sure maybe guys like Hurst (think Microsoft, Google, and Dell) made vast fortunes, but most of the miners (SBSC’s and IT Pros) just scraped by, or worse, went bust. In between, were the guys and gals that moved up the stack, and sold mining tools and equipment. It’s these types of businesses that thrived as people went out seeking their fortune. Look at the SMB IT segment, who’s moving up the stack today, and what exactly does the stack look like?

So I’m looking at the cloud and thinking about IT infrastructure… be it OWN’s offerings, or the various platforms that you can use to build-out solutions… think 3TERA, Amazon, and Sun (and to a lesser extent VMWare, and Virtual Server). The utility/grid computing folks are in business to deliver scalable infrastructure that “just works”, so that they can insert their “infrastructure-tax” (just like the Google advertising tax, or the Microsoft Windows tax) into tomorrow’s solutions. Why? Because it’s zero-touch (and has higher margins than selling boxes). You design a really good scalable “grid”, and you sell computing as a utility to companies that want to offer some type of application or service (this should all sound very familiar). The ASP (for lack of a better term) then treats the utility as an incremental cost of doing business, and can factor it into the solution they’re delivering. What’s great for the grid-folks is that they make money as their customers are successful, but they can offer the utility virtually zero-touch, and as we all know… zero-touch and scalability is where the money is.

Taking into account services via the cloud, where are we – as SBSC’s and IT Pros – going? We’re probably not going to go out and build our own grids (the grid-folks of tomorrow will be the Dell’s and HP’s of today), because grids already exist in some form today; they’re inexpensive, highly complex, and evolving. The one-man-shops and network plumbers who just deliver infrastructure are the businesses at the greatest risk – because they’re going to get squeezed from both sides (i.e. the cloud, and service providers with a niche further up the stack). Next up are the better-run small companies that just do break-fix – it will be slower in coming, but any one of us who can present a reasonable value-based argument is going to be better able to deliver services than that bunch. Next up are the vast majority of us who deliver some mix of managed services, break-fix, and block-hours. We’re the ones who need to be seriously considering how we deliver services – either partnering to deliver infrastructure via the cloud, building our own clouds, and/or moving further up the services-stack. To go back to the mining analogy… if we as SBSC’s, and IT Pros are mining gold… what do we need to do? Well, selling our services to each other is one good option, and moving up the stack to become virtual IT directors, developers, and the like for our clients is another. But as we look at some of the niches within the stack, there’s only so much room… how many different OWNs can the SMB community support… or how many MSPUs can provide really valuable mentoring to the SMB IT community? Sure, maybe we have a rising tide effect going on – but I’d venture that there will only be so many players further up the stack. Now maybe those companies have businesses that do work throughout the services stack… but they each have their own niche and value proposition that is unique and that not everyone in the community can readily deliver. So they’re better miners than us within their niches, and as a result are less vulnerable to a Microsoft or Google.

The future is unwritten.

That's true... and we don’t know how the grid computing game is going to play out just yet. But we can make some informed guesses based on past technology cycles, and I think we really need to be prepared in order to compete and stay relevant. Does that mean I won’t be selling infrastructure tomorrow? No, I probably will be. But I’ll probably be delivering infrastructure via the cloud and via equipment, focusing on service as a subscription, and I’ll keep moving up the stack delivering “IT Director” services to my clients. At the end of the day, it’s about the relationship that you have with your clients, and how good of job you’re able to do at scaling your offering, while remaining competitive.

Security: Thoughts on Security in the SMB space

Ross and Andy both make a good points about the depth of security; or rather, that just doing the basics when it comes to security leaves you predictable, and as a result… vulnerable. In other words, you should be doing more than checking your perimeter logging, looking at your IDS/IPS logs, and making sure your workstations are up-to-date.

If we are only talking about the security of one organization, or of a medium-to-large organization – I think that would be perfectly reasonable. But when I think about our SMB clients… and the SMB market in general… I don’t know of anyone in this market that’s doing much more than playing “night watchman” (using Ross’s terminology). I don’t even know if the SBS Diva Susan Bradley - who is one of the biggest champions of security in the SBS space - is talking penetration testing.

You always need to balance real risk, the client’s perception of risk, and real costs of the meeting what you consider to be valid security objectives. Now maybe, in the SMB-space you don’t have as many valid security objectives… certainly smaller clients will have difficulty justifying the type of security Ross is talking about – and perhaps these activities are ignored, and ignored rightfully so.

I work in a medium-sized organization… but even little things like software licensing, re-purposing servers that are out-of-warranty and trying to justify basic security considerations all represent uphill battles. Coupled with the fact that so much of security sounds like insurance when you’re trying to sell someone (your boss, a client, whomever), and not only is security an up-hill-battle that requires rocking the boat from time-to-time, but it’s something that you need to approach carefully. Closely consider if the risk that you’re attempting to mitigate justifies the time, energy, and opportunity-cost to you’re committing to. Make certain that you’ve got your other bases covered - like your equipment, software licenses, backup/recovery plan, etc. – before trying to tackle new commitments.

Now, none of this is to say that you shouldn’t be improving security for yourself, and/or the organizations that you support. And this is NOT an excuse to start ignoring your logs, or not doing port scans, and the like. But you do need to evaluate the attack surfaces that you’re exposing, and what the risks are to your organization, and your own career as well as the perception of your role in an organization - when you start talking security.

Bottom line… rock the boat when it needs rocking - but don’t rock just for the sake of rocking.

Backups: Rotation of backups

What’s your approach? I’m asking... because for SMB clients, I’ve heard literally every possible combination of tactics, all with the same strategic aim - protecting the business.
Here’s what I do - not saying it’s perfect... certainly not saying it’s the only way... but it’s working for us internally, and is basically what we do/recommend for clients (depending on their risk profile, budget, and expectations).

1. Nightly full backups of everything that’s "business critical", with a two-week rotation.
2. At the beginning of each week, the last completed backup is taken off-site, and the previous one is returned.
3. At the end of every month, the most recent completed tape is taken out of rotation and stored off-site.
4. At the end of the year, a tape is taken off-site and stored permanently.

This gives us the ability to restore data from any day in the past two weeks, end-of-month for the past 12 months, and end-of-year for the past N years. The reason that I chose nightly full backups is because relying on the weekly tapes and daily incrementals introduces more risk. Now, there are some other components not covered here - archive shares, web servers, and so on... those are all covered by a separate monthly policy for non-business-critical assets.

What do you think? Obviously the more tapes we include the more options we have... but in general terms do you have any suggestions for improvement?

SMB: Tracking down a rogue DHCP server

Rogue DHCP servers can be a real pain to track down on an SMB LAN. You’d think that with a small network you’d be able to readily identify just about any unknown devices on a LAN, right? It’s just a matter of wandering around the office, maybe? In my experience that ceases to be the case as a network grows… at 50 or so employees and maybe 80 or so devices things tend to get a bit harder to track down.

Real-life example:

So an email alert pops up on my cell last night indicating that the DHCP service had stopped on one of our client’s systems. Because they’re a new client, I make sure that I get copied on all of these alerts regardless of who is actually on-call at the time. So we called the client and left them a message to make them aware of the issue, and then followed-up this morning.

The system log on the SBS server revealed event IDs 1053 and 1054, with 1053 being as follows:

The DHCP/BINL service on this computer running Windows Server 2003 for Small Business Server has encountered another server on this network with IP Address, 192.168.2.1, belonging to the domain:

Unfortunately we didn’t have a way to talk to 192.168.2.x remotely, so I went on-site to track this down (yes, we probably could have done most of this remotely, but you’ll see why it was beneficial to be on-site in this case). After firing up my laptop I received a 192.168.2.x address – which was wrong; it should have been something in the 192.168.1.50-200 range.

So the next thing I did was ping what appeared to be a rogue gateway.

c:\>ping 192.168.2.1

I received responses, so that’s a good starting point.

c:\>Tracert 192.168.2.1

No hops… so nothing was between myself and the device.

>arp –a

So IP is a layer 3 protocol, and it doesn’t know what’s going on at the physical level. Using the Address Resolution Protocol (ARP) we can get the physical address of the destination host once we’ve sent a ping and the ARP has cached the results of its broadcast. Thus, “ARP –a” showed me the MAC address of the rogue device.

Once we had the MAC address, it was just a matter of finding it on the managed switch.
So I telneted to the managed switch (after configuring the switch, but that’s a different story… just remember that they’re a new client).

# sh ?

This displayed help for the show command.

#sh mac-address

This displayed a list of MAC addresses that the switch has learned. From there, I could tell that the MAC in question was associated with the physical port number 1 on the switch.

Port 1 was uplinked to a 48-port unmanaged switch. My first thought was great… now I’ve got to trace all of these active connections. But before doing that I figured it would be worth trying try to talk to the rogue device… so after connecting to the IP via telnet and web browser it turned out to be a cheap router, with the default admin password configured.

Well, that’s an improvement. So I logged in and turned off DHCP. Next it’s was just a matter tracing the connections. The downside was that there wasn’t much of an up-to-date wiring diagram, so it took some time to figure out exactly where the box was located.

After tracking the device down to the particular drop and removing it (it was located under and behind a desk), I talked to our client. No one was able to pinpoint just exactly how this device ended up on the network. It was connected to a printer, so my guess is that someone had tried to use it as a switch. It wasn’t configured to do anything particularly malicious – beyond handing out IP’s that were inappropriate, and that was just a side effect.

The client was pleased with our responsivness and approach. So with that issue resolved, it’s time to start talking to the client about a network upgrade project that will bring them up to spec with our support-matrix.

Business: The Guide to a Successful Managed Services Practice… Or, how a $100 book saved me $12,000

So maybe you don’t trust Vlad’s opinion of Eric’s book… I mean, really… how do we know he’s not just plugging the book for his buddy, right?

Having read though the book myself last weekend, I have to say it’s a great resource for any SMB IT service provider. It doesn’t matter if you’re a 1-man break-fix shop looking to get out of the hole that you’re in, or if you’re already doing good work in a business that can scale, you will learn something. Oh, and if you ONLY come away with $100 worth of value, then you missed the point entirely.

I’m not going to write another review… that’s already been done effectively elsewhere. What did the book do for me? Like the title says… one week after having read it, the $100 book saved me $12,000. Don’t believe me? Okay… we’ve been looking at advertising options and working on the business plan for a while now… essentially, we’re trying to figure out a way to generate more good leads. The book talks about that… and we already have the numbers to drill down though for our ROI calculations… the $12,000 would have been better spent on… just about anything else. So instead of blowing 12K and hoping, we have a better plan. Are we where we want to be yet? NO! But 12K in savings isn’t bad for a casual read on the weekend… and it’s already paid for my trip to SMB nation next year.

SMB: Acquiring a new midsized customer

We recently acquired a new mid-sized customer on the SMB-side of the business. At 130 users, 80 computers, and 7 servers, I would say they fit in the medium-camp. Apparently, they’ve really been without any type of dedicated IT staffing for the better part of a year, and it looks like they’ve gone without any notable hardware or software upgrades for considerably longer.

From an operational standpoint, we really have an opportunity to make a difference for this customer. Given that things have essentially run untouched for quite some time, virtually anything we do is going to improve their situation. Viruses, spam, Windows NT 4.0 servers, Exchange 5.5, and a mix of aging workstations… you name it, there’s plenty of low-hanging fruit.

From a sales and revenue standpoint this is also a real opportunity. The current ownership/management is relatively new and frustrated with the current situation. Furthermore, it sounds we’re going to have a sizable budget for at least this first year, in which we can accomplish quite a bit.

Being involved pre-sales is always an interesting experience; and one that I think every technical person should make an effort to be part of.

That’s going to be it for this post. As this project progresses, I plan to provide more technical details, and address some of the day-to-day things I run up against. Also, I have it in mind to post something about technical-sales, which I just briefly touched-on in this post. It’s by no means the focus of what I do, but it really offers allot of insight and brings a new appreciation for the role of sales in an organization.

SBS compromise response

This must be an on-going debate over the “SBS compromise” because I’ve heard it frequently… and perhaps there are a few other places where this hits closer to home than in the organization I work for… but not many. As a matter of function, I tend to have one foot in the enterprise, and one foot in the SMB market. So yes, I can appreciate first hand how SBS tends to end up being the punch line instead of the solution.

Among the enterprise circles of the IT community, I think Chad is correct. Assuming the enterprise admin you’re talking to even knows of SBS, they’re probably going to hate it. “A DC, Exchange, ISA, RRAS, IIS, Tape backup… etc… all on one box… [insert punch-line here]”… I know, I’ve heard it. Heck, when I was first introduced to the SBS platform, I had a similar position.

But now, a few years out, I have a whole new appreciation for SBS. Why? Because SBS is the right tool for the right job.

“What do you mean, right tool – look at the security issues!”

Yeah, I know… just look at them. You know what the small business owners I talk to think about “security”? They demand it… that is, they demand it until they start to get an understanding of the cost/benefit. When we start talking about dedicated purpose servers… servers, which I might add, that are going to run at about 1% utilization on a busy day. And then we talk about licensing. And then we talk about firmware maintenance… and then the patch management lifecycle on each of these boxes… you get the idea. That somewhat reasonable 10k-15k initial investment for a complete SBS-solution, all of a sudden turns into something literally impossible for your average SMB-sized customers. And if by some chance you were to get an SMB customer to buyoff on your “secure” dedicated server solution, how do you think they’re going to pay to maintain it? So when no one has patched those boxes in two years, how secure do you think they’re going to be?

For your average SMB customer, you need to understand what you’re trying to protect. For the most part, they’re not banks, or financial institutions… you’ll see some HIPAA compliance issues, and a few with some type of trade-secrets. But at the end of the day, you’re trying to come up with an acceptable cost/benefit for the customer. Understand the need for security, but appreciate the fact that if they need some type of IT infrastructure, an SBS solution beats paper, or a workgroup hands-down.

The managed services model

I’m right with you Susan. And thanks for linking to my post here on Addicted to IT! I think Susan is right on with the managed services concept. It’s about more than just being “proactive”, as I described in my recent healthmon post.

One of the other things we’re doing at my company is taking some of the learnings from integration work with our “enterprise” customers, and we’re starting to apply those learning’s to our SMB customers in the form of managed services. Why the lag time?

Economies of scale.

What you do in the enterprise might make sense, and have an acceptable cost/benefit attached to it for that environment. But to be successful in the SMB market you really need to have your customer base reach that “critical mass” so that cash-flow catches up with, and exceeds your existing model. Then managed services start to get really exciting.

Oh, you wanted examples?

Well, Susan already mentioned VoIP, and CRM (two things which I think partnering on makes a lot of sense). But just think about all of the things that have to happen in the enterprise to be successful… it’s more than just the technical side… there’s all of that “management stuff” to consider. And as much as maybe some technical people don’t like to think about it, management adds value too. And your customers should be paying you for that value.

Does your customer call you on the phone when there’s a problem?
That’s break-fix.

Do you call them?
That’s proactive.

Or do your customers have an online reporting tool that let’s them generate service requests online, check their account-balance, and review completed activities ? Oh, and does that tool also assign service-requests based on priority to your operations team? Don't have an Ops-team? Well, does it page your cell phone?
That’s part of the managed-services model.

So what am I doing with managed services?

The online reporting tool outlined above for one thing. But it doesn't stop there. From business intelligence, to staff-augmentation and sharing, to remote-outsourcing, single-point cost models, and application development... all kinds of neat stuff. But we're in transition. We didn't wake up one day and say "hey we now offer managed services", it's a transition. You have to work with your customers. And most of all, you have to offer real business-value.

Expect some follow-up posts on some more examples of the managed-services model. I’ll also be adding to my healthmon series.