- The Fat Finger Factor
- Vendors and Change
- The Risk/Benefit Ratio
- Summary
- Q&A
- Workshop
This chapter is from the book
This chapter is from the book
This chapter is from the book
Vendors and Change
Your vendors, whether consultancies, ISPs, ISVs, VARs, and so on, get repeat business based on how well you are treated, and so, if they're smart, they do have your best interests at heart. Still, vendors can be a major source of unintended consequences at the worst possible times.
Several factors will affect the types of changes—and the timing of changes—that vendors cause to your network. Although vendors do want to do a good job for you, they have several other warring desires at work. Remember, like people, there are good vendors and there are bad vendors. (Of course, this is also true of internal staff.)
The following are some factors that might affect the way a vendor performs on a job on your network. The really good ones don't let any of these factors stand in the way of performing well for you. Still, you should at least consider that vendors might
Have a schedule to keep and other jobs to do.
Have a bottom line that is affected by the hiring of top talent, and thus often will deploy less-than-top talent unless they feel that top talent is necessary.
Have the sense that there is no short-term consequence for poor quality (doing a bad job might mean losing you as a customer, but there are a lot of customers out there); and providing high quality is an expensive proposition, requiring an investment in training, lab gear, and so on.
Have a vested interest in selling you new software, services, and hardware.
For example, because your vendor has various projects scheduled, this can mean that someone might show up on your doorstep at a random time (not necessarily when it's convenient for you or your organization). When someone shows up randomly, perhaps the correct staffer at your location is not available to give this vendor proper information—which might cause this vendor to make mistakes; or, it might be that a critical, uninterruptible process is in process when the vendor staffer shows up—which might force the vendor to cut corners, like making changes but not restarting a service.
The bottom line is to give your vendor the benefit of the doubt, but keep a watchful eye for these things.
I know it sounds obvious, but you do have a right to insist that changes to your network— no matter how necessary for a given project—must be scheduled. You wouldn't let your plumber walk into your house at any old time and fiddle with your water heater, would you?
Project Panic
When projects are large, of course, the cost of undoing a bad change is also large. For large projects, you should always insist that a sensible rollout and rollback plan be built into the project; the gradual rollout lets changes be introduced slowly; the rollback plan is how to undo the changes in case of problems.
For example, I was called into a medical office where the character-based medical office system was being upgraded to a graphical, windows-based system. Not a lot of planning or testing had gone into the upgrade; basically, someone had thought to himself, "It loads and runs on my 50-user test database, so surely it will work with the 5,000-user database with hundreds of thousands of detail records." Apparently, there was a real problem with network-based record access in the graphical version, and when faced with database sizes several orders of magnitude larger than the test system, the system became excruciatingly slow. At first, the vendor would not confirm this; but when faced with a packet trace (see Hour 20, "In-Depth Application Troubleshooting," for more information on when a packet trace is appropriate) that proved it, the vendor backed down.
Because there had been no gradual rollout, all the many workstations had to be rolled back until the vendor could fix the problem. Clearly, this cost a significant amount of labor, and thus money.
TIP
Moral of the story: No matter who makes changes, you or your vendor, an incremental rollout of changes followed by real-life testing, with a rollback plan in the wings (in case testing does not point out the problems) is a sure recipe for success!
NOTE
Incremental rollout means that after a limited deployment to key individuals (who are perhaps good testers or expert users of the system), you start giving the application or system to batches of users in increasing quantities as you proceed. That is, you do the rollout in small chunks that get bigger as the rollout becomes more successful. For example, you might give five people a new application. Later, you give the application to 10 more people; then 15, 20, and in your final increment, you might be rolling out 30 people a week (once you're sure that things are working fine). Using an incremental rollout ensures that if you have a problem early on, the least number of folks are affected.
When possible, always roll out changes or new systems incrementally.
It pays to negotiate up front with your vendor about a rollout/rollback plan for a major project. The cost for "making it work in a disciplined way and putting it back if it doesn't work" might be a different price than "throwing it in and then dealing with it later if it's a problem." If you're dealing with an expert and reputable vendor, he won't mind if you bring up these types of things during project negotiation.
NOTE
Don't be a pushover, but don't be a pit bull either. Having a good relationship with your vendor is really important; your vendor can either be a cause of trouble or a troubleshooting partner—you want to shoot for the latter.
Remote, "Invisible" Change
Vendors responsible for ongoing maintenance of your network devices might tend to use remote access to accomplish some of these maintenance tasks, such as patching or enacting manufacturer-supplied security recommendations. As such, you will have little or no idea that these things are being done, short of catastrophic failure.
As I mentioned in Hour 3, you might want to investigate the possibility of an intranet application that allows various folks to key in their network changes. You could make this available to outside vendors as well.
Patching and tweaking, particularly security tweaking, are good things to do, and if your vendor is on the ball about this, this is awesome. The important thing is to make sure that you get a phone call prior to or just after the change, or you will not have complete information to go on when you are troubleshooting a problem.
Remember, ISPs and the telephone company don't need you to give them a remote access account to make changes that will affect your network. It is a fact that ISPs and telephone companies make changes on the weekend or over holidays when utilization is low, so it's smart to call your provider and politely ask about possible network or equipment changes if you come back from the weekend or a holiday, and things aren't working quite right.
Unfortunately, when you ask "What's changed?" the answer will likely be "Nothing" unless you have a very together provider. To make matters more aggravating, even after you verify that nothing has broken over the weekend at your end, the problem might mysteriously vanish around lunchtime.
But the problem might not vanish around lunchtime.
For problems that don't, you might have to convince your provider that nothing's wrong with your computer equipment (or figure out that there is something wrong and apologize for having doubted them).
You can set up an alternate path between the two networks in question. One way to do this is to create a VPN between the two networks; another way is to simply dial up from one network to another. There are several ways to skin this cat.
For example, on one fine Monday, my brother had remote access cease functioning at one site. After tearing around and verifying that his servers were okay, he was able to ascertain that, in fact, using a different path, he was able to get in to his remote access. Although he could have dialed in, he simply changed the parameters of his remote access server to use a different TCP port (see more about TCP ports in Hour 1, "Understanding Networking: The Telephone Analogy"), and was able to work. He then found out that the provider in question had blocked the original remote access port.
When the line is totally down, obviously, you can't set up a VPN (but you still can do dialup). When you are totally down, you actually have some more options: As long as you can't communicate anyway, you can truck the equipment from site No.1 to site No.2, and see if they are functioning when connected directly. If this works, you have pretty compelling evidence for the telecomm provider that nothing has changed with your equipment and that something has changed between the two sites.
Here's one warning, though. Although there are instances when you can connect two devices together with no additional equipment or configuration (for instance, ISDN routers with RS-232 ports can be connected with a crossover RS-232 cable), there are many other instances when you cannot do this. (Using a crossover cable with two cable modems is not going to work because there isn't a server in the middle to automatically configure them.)