• Home
• Development
• Labs

• Contact

BLOGS

• Allowing node publishing/unpublishing for a non-node admin
• Patch submitted for activitystream
• jQuery tab background template
• SMART Project Management
• New Training: Do It With Drupal
• Market opportunity for Drupal? Document Management Systems and Digital Asset Management
• S3Saver notes from Sacramento Drupal User's Group
• Announcing 24x7 availability: 866-WorkHabit
• Finally IE testing for all versions...
• Subversion Merge: the easy way. Copying changes from trunk to a prod tag.

Fifteen tips for capacity planning and initial deployments for startups

by jonathan on Tue, 06/05/2007 - 21:21

I don't mean to single anyone out here, but this is an excellent opportunity to single out why it's important to back up your strategies with a good technology plan.

Most of the time, I talk to people who are scaling websites, and they are either running on a technology platform that is advanced enough to still have relevance (and probably still bein the process of getting paid off) when we actually do complete our first round trip to mars.

This is ok, but you probably overbuilt, and that can actually be a danger to the business because effectively you are paying for something you cannot actually put to use and spending money you could spend elsewhere. (And remember, you need qualified people to keep things working as well as implement! This means expense!).

The other majority are running on DreamHost, BlueHost, 1&1, or similar shared hosting.

Bad idea. See above example.

So, here are some tips for people who are just starting out:

1. When you start working with intensive, database-backed websites, you need to do capacity planning for about six months out. Don't plan for more - that's overplanning - unless you have budget constraints or are otherwise limited, in which case try to keep it to a year. For a site with user generated content, you're practically gambling with your money at that point.

2. If you have to save money, start with a cheap hosting account, but move to a non-Virtuozzo based (VMWare, or preferably XEN-based) Virtual server before you deploy (you can't oversell these as much as Virtuozzo, they have "hard" limits)
3. Remember, Virtual Servers inherently have some I/O problems, so make sure you plan your big scaling directly on hardware. Virtualization will work for the first 2/3 of your sites life however, and will ease management measurably.
4. When given the choice to lease monthly, lease. When you go to buy equipment, lease it as well - your "now money" is worth more than your "later money" in a startup, even if it's incrementally more expensive.
5. When starting out, don't fool with infrastructure. Hire it out, as it will distract your most valuable developers from driving your core value. If you choose to implement your own infrastructure, hire someone who doesn't do development to maintain it.
6. Don't use RackSpace for Social Networking sites. Astoundingly (yes, I know, this is hard to believe) they do not have a High Availability SAN solution that handles file locking, so you can't support multiple web servers on a SAN. And they're $6500/mo for 2TB of San storage, which isn't that much. And your only other option is Windows, single server NFS, or rolling your own HA NFS). Besides, they're expensive and have no special relationships with CDN networks. I do like Rackspace, but they're not a good place to run a social network. You can't even buy a Dell MD1000 or MD3000 from them, despite their massive relationship with Dell. This is true for a lot of companies, so listen up: just because a place is expensive doesn't has good support doesn't mean it's an answer to your technology problems.
7. When you do buy storage, buy upmarket with less capacity so that you can use the same system to scale out, or risk losing your entire initial investment. Stick with Open Source (ATAoE (the backup network solution), GFS (no, not Google)) when possible.
8. When working with intensive database-backed applications like Drupal and others, it's paramount that you also load test your application at least once before putting it into production. If you have to, do it in VMWare on your desktop computer, but have some idea of how much traffic you'll be able to support with a given memory profile (a ballpark is better than nothing), as you might just find your application won't be able to handle traffic - it happens more than you think.
9. Higher a good technology advisor, and pay them well. Try to get someone who's not necessarily a superstar (some really are superstars, many are just the star part without the super), but has a track record of completed projects. This person will save you money and prevent you from stepping into commitments you will regret later.
10. Don't be afriad to refactoring your infrastructure. It isn't as big a deal as refactoring your application, so make sure your application supports things like a split M/S backend server for MySQL if it uses it, etc. Building flexibility into the application isn't expensive you think about it from the beginning, then you won't mind moving things from your host to Amazon down the line to save a buck.
11. Remember, also, that Drupal is quite database intensive. You can get a lot of miles out of a second virtual server running a dedicated database server, and save a bundle in the process.
12. Learn the different between High Availability, Performance, and Scalability (in fact, read this).
13. When buying servers, buy just off the high end models. You'll save 30% or more, and get 95% of the capacity. But don't buy too low end, you want to be able to standardize on one model for most horizontal scaleabilty models.
15. Automate your builds. If you want to make all of your capacity planing for nought, keep hand deploying your applications during 80 hour weeks and see what happens.

Tags: botchedcapacityDrupalhosthostingload testingplanningrackspaceshared hostingstartupssysadminsystemsvirtual serversvirtualization

jonathan's blog