The Age Old Question to Host or not to Host

October 11, 2010

The choice of a Hosted Solution vs. an Enterprise Install is a big one with many moving parts. The choice is up to you but here are a few reasons why installing, managing, deploying and hiring around an Enterprise solution isn’t the way to go.  Hosted Solutions, Cloud Computing and Virtualization offer flexibility that is infinitely scalable so as your agency grows your hardware footprint doesn’t need to. If you look at hosting and analyze the capital costs and time you are risking when you go the enterprise route it is will become clear that Hosted solutions can deliver far more value then you can ever achieve with an enterprise set up.  Here are the five things to look at when choosing your overall solution.

The first thing to consider is Traffic Diversity inbound and outbound. Diversity simply means that you have more than one way of sending data in to your facility and out of your facility.  If you have one T-1 or one Cable Connection you are vulnerable and for most agencies this is how they connect to the internet.  You can’t do true back up and redundancy without bringing in two truly diverse circuits and from two different entrance points and carriers into your facility.  What this really means is that to be truly diverse you need to be physically diverse not virtually diverse. Let’s assume that you have two internet lines coming into your office maybe one from the north and one from the south. Let’s says at a top level this can be done where your office is.  Here is an example of how most offices are connected to the internet and what many companies think they are diverse without realizing that they haven’t made things any better.  We will assume that your office can afford this and you have the onsite skill sets to manage this type of environment. You put a plan together and you have Company A install one circuit into your facility from the north and Company B install a circuit from the south. In theory you have diversified your internet access which means in theory that if one connection fails the other will take the load. The problem with this scenario is that in almost all cities in the United States the last mile access or the last mile to your facility is managed by one single company, lets say Company C.  More than likely Company A and Company B Pier or rent space on Company C’s line.  If Company C goes down both Company A and B are down.  Your mitigation plan is a failure. With a hosted solution and a Tier 3 or enterprise level hosting solution and facility this will never happen.  A hosted solution or provider brings in multiple circuits from multiple locations that use more than one carrier to access the hosting center.  This means that if a Tractor, an Explosion or an attack on any one carrier or line will not lead to a disruption of your service. The other carrier will handle the load and the agency experience will not be impaired at all, more than likely the agency won’t even realize the failure.

The second item to consider is Server Mirroring, Back up and Redundancy.  You’ve gotten past step one and somehow set up true diversity in and out of your facility, now what? Now you need to ensure that your data is always on, protected and recoverable. Now that you have your two inbound circuits each of these circuits must be routed to two separate servers or server farms and if it is in your office it is probably two closets.  For the purposes of this article we will skip why you shouldn’t use a closet to house your servers. These two closets in order to work as true back up must be configured identically, with the same operating system, same applications and same hardware configurations to make mirroring and back up work correctly. The cost of your initial build out has now doubled because you need twice the amount of servers, software and space to keep this architecture up and running. This takes a lot of skill and one change in an update or service pack can crash both systems.   The main issue with this approach is that a loss in power or damage to your office may not be recoverable. If you lose power both servers are down and if there is a fire or flood both servers will surely be ruined.  A fire or flood will not only destroy the servers but it will destroy all of your data. Once the data is gone it is game over.  In hosted environment there are three critical systems that are implemented to make sure that loss is mitigated.  The first is gas powered generators that can run indefinitely, the second is fire suppression systems that don’t use water but use powders and other non liquid fire fighting chemicals so as not to damage the hardware any further and the third is raised flooring so that in the event of water damage water can be routed out of the facility without touching electrical circuits.  In addition to those protections a hosted provider will back up your data across geographically diverse regions. For example your hosted solution provider may have one server installed in New York and the other in California this type of back up scenario ensures that even if the gas powered generator fails in one location it won’t fail in the other, if fire or flood knocks out one facility it won’t knock out both and if both diverse circuits at one facility are cut the other facility will continue to function without any service interruption to the agency.  In a hosted environment back up and redundancy are built into the infrastructure so that down time is at best completely eliminated.

The third item which is often thought of as the first item is Scalability or controlling and managing the exponential growth of your data and processing requirements as you add clients and accounts to your system. For the purposes of this article we will assume you have not mirrored or created any type of replication architecture because if you did please double all of the hardware requirements outlined below.    If you start with two servers an Application and Database server each server has limited storage space so at some point you will need to add either servers or hard drives to support your company’s growth. If add hard drives you will need to upgrade the processor speed or add processors so that the user experience remains consistent.  The more data you need to manage or more importantly access means a proportional increase in time to access that same amount of data. More hard drives and storage will always mean more processing power is needed.   Once you max out your current servers you will want to add two new servers to handle the demand.  If you decide to add severs you need to create two new servers instances every time you scale and they must be identical. The effect of Moore’s Law on hardware capabilities relative to speed would mean that if you needed to add a server every 12 months that every 18 months your prior environment is obsolete so six months after your first upgrade your original environment will need patches and security updates and in some cases hardware updates to keep current.  Managing this type of environment even for a seasoned professional is cumbersome at best and at worst a disaster waiting to happen.   In a hosted   carrier class environment this is managed month after month and as updates are needed and new servers are added the skill set to implement and the access to upgrades are all core to their product offering. Lastly it is something that can be implemented with zero down time.

The Forth item is Human Resources or people allocation and understanding how to manage IT resources. This is probably one of the biggest underestimations that agencies make when deciding to go Enterprise instead of Hosted.  The theory makes sense; if I hire and train my own team and they are working where I can see them I will have more control and quicker turnaround time for custom reporting and overall system customization.  Here is the hidden problem.   Let’s assume that you have budgeted $100,000 per year in salary for the programmer that you will need to get this done, assuming you are going with only one person and you can get them for $100,000.  If you go with an enterprise solution you are beholden to your programmers for every single thing you need, customization, reporting and data cleansing to name a few. Let’s assume that you ask your programmer to do something and they state it can’t be done. How do you know if something can be done or can’t be done; is it a limitation of the system or a limitation of your developer, you will never know. Then there is the intangible aspect of hiring a programmer what happens when they are sick, need personal time, vacation and what happens when they quit.  You are at their mercy and they know it.  Now painfully your collection agency who simply hired a programmer to help them become a more efficient agency has now becomean IT development firm. If you look back to business school and the SWOT analysis one of the things that you do in your SWOT analysis is you try to find out what business are you in, what business do you think you are in and what business you should  be in? Many of the best CEO’s in the world went to work for one of the big three automakers in Michigan. All opined that they thought they were becoming Car Makers and would create the next great Mustang or Corvette but in reality they were becoming administrators of a massive health care system. Don’t get caught up in making your agency into something it shouldn’t be.  With a hosted provider you have developers, trainers and customer service agents available around the clock. The Company doesn’t call in sick. Your phone call never goes unanswered and there is always a customer service agent, a designer and a programmer available for your questions and request for programming.  Because their architecture is standardized and documented if a programmer leaves the provider it has no bearing on what can be delivered because there is a team of developers and an entire organization ready to help.  You are never beholden to the whims and capabilities of one single person.

The last item is Opportunity Cost. To really look at this, how much does it cost the average agency to build out servers, get a desk for a programmer and manage the computing environment vs. using that same space to add a collector or collectors to your office, why would you use a perfectly good collector’s desk for a programmer?  What my talking points don’t take into account and which you must understand before embarking on the enterprise journey is the amount of collection revenue that will be lost the day your agency uses collection square feet for a programmer instead of a collector. Let’s say you have the choice to hire three collectors or hire a programmer. The three collectors for argument’s sake will collect $500.00 per day and roughly $33,000 per month. If it takes you six months to get your infrastructure set up you could have collected $198,000 in that same period of time.  This time frame makes a huge assumption that the person you hire understands everything you require and they understand the collection software that they are working on or enhancing.  So the IT person you hired needs to hire at least one consultant to help with the set up for the six months. Let’s say you are a great negotiator and everything is on your side and you get both of these resources for $10,000 per month.  If this doesn’t work and this doesn’t launch on month seven you are out $258,000($198,000 in collection revenue and $60,000 in development costs) and if you do it right and you launch you are only out $60,000 plus you now need to recoup the opportunity costs, so your collectors for the next six months must collect enough money to compensate for the $33,000 per month that was lost over the last six months.  It is clear from the points outlined in this article that you need to be in your area, be an expert at collections. Find the resources, tools and companies that have experts in their fields so you can focus on collecting.  It is clear that a Hosted Solution is the way to go.