At long last, some of the actual implementers of the advanced systems we built at IMVU for <a href="http://startuplessonslearned.blogspot.com/2008/09/just-in-time-scalability.html">rapid deployment and rapid response</a> are starting to write about it.
I find these on-the-ground descriptions of the system and how they work so much more credible than just theory-type posts that I am excited to share them with youI can personally attest that these guys know what they are talking about; I saw them do it first-hand.
I will always be full of awe and gratitude for what they accomplished.<br /><blockquote><a href="http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/">Continuous Deployment at IMVU: Doing the impossible fifty times a day by Timothy Fitz</a><br />Continuous Deployment isn’t just an abstract theory.
At IMVU it’s a core part of our culture to shipIt’s also not a new technique here, we’ve been practicing continuous deployment for years; far longer than I’ve been a member of this startup.<br /><br />It’s important to note that system I’m about to explain evolved organically in response to new demands on the system and in response to post-mortems of failures.
Nobody gets here overnight, but every step along the way has made us better developers.<br /><br />The high level of our process is dead simple: Continuously integrate (commit early and often).
On commit automatically run all testsIf the tests pass deploy to the clusterIf the deploy succeeds, repeat.<br /><br />Our tests suite takes nine minutes to run (distributed across 30-40 machines).
Our code pushes take another six minutesSince these two steps are pipelined that means at peak we’re pushing a new revision of the code to the website every nine minutesThat’s 6 deploys an hourEven at that pace we’re often batching multiple commits into a single test/push cycle.
On average we deploy new code fifty times a day.<br /></blockquote>We call this process continuous deployment because it seemed to us like a natural extension of the continuous integration we were already doing.
Our eventual conclusion was that there was no reason to have code that had passed the integration step but was not yet deployedEvery batch of software for which that is true is an opportunity for defects to creep in: maybe someone is changing the production environment in ways that are incompatible with code-in-progress; maybe someone in customer support is writing up a bug report about something that's just being fixed (or worse, the symptom is now changing); and no matter what else is happening, any problems that arise due to the code-in-progress require that the person who wrote it still remember how it works.
The longer you wait to find out about the problem, the more likely it is to have fallen out of the human-memory cache.<br /><br />Now, continuous deployment is not the only possible way to solve these kinds of problems.
In <a href="http://timothyfitz.wordpress.com/2009/02/08/continuous-deployment/">another post</a> I really enjoyed, Timothy explains five other non-solutions that seem like they will help, but really won't.<br /><div></div><blockquote><div>1.
More manual testing.</div> <div>This obviously doesn’t scale with complexityThis also literally can’t catch every problem, because your test sandboxes or test clusters will never be exactly like the production system.</div> <div> </div> <div>2.
More up-front planning</div> <div>Up-front planning is like spices in a cooking recipeI can’t tell you how much is too little and I can’t tell you how much is too muchBut I will tell you not to have too little or too much, because those definitely ruin the food or product.
The natural tendency of over planning is to concentrate on non-real issuesNow you’ll be making more stupid mistakes, but they’ll be for requirements that won’t ever matter</div> <div> </div> <div>3.
More automated testing.</div> <div>Automated testing is greatMore automated testing is even betterNo amount of automated testing ensures that a feature given to real humans will survive, because no automated tests are as brutal, random, malicious, ignorant or aggressive as the sum of all your users will be.</div> <div> </div> <div>4.
Code reviews and pairing</div> <div>Great practicesThey’ll increase code quality, prevent defects and educate your developersWhile they can go a long way to mitigating defects, ultimately they’re limited by the fact that while two humans are better than one, they’re still both human.
These techniques only catch the failures your organization as a whole already was capable of discovering.</div> <div> </div> <div>5Ship more infrequently</div> <div>While this may decrease downtime (things break and you roll back), the cost on development time from work and rework will be large, and mistakes will continue to slip through.
The natural tendency will be to ship even more infrequently, until you aren’t shipping at allThen you’ve gone and forced yourself into a total rewriteWhich will also be doomed.</div></blockquote><div>What all of these non-solutions have in common is that they treat only one aspect of the problem, but at the expense of another aspect.
This is a common form of sub-optimization, where you gain efficiency in one of the sub-parts at the expense of the efficiency of the overall processYou can't make these global efficiency improvements until you get clear about the goal of your development process.<br /><br />That leads to a seemingly-obvious question: what is progress in software development? It seems like it should be the amount of correctly-working code we've written.
Heck, that's what it says right there in the <a href="http://agilemanifesto.org/principles.html">agile manifesto</a>But, unfortunately, startups can't afford to adopt that standardAs I've <a href="http://startuplessonslearned.blogspot.com/2008/10/lean-startups-vs-lean-companies.html">argued elsewhere</a>, my belief is that startups (and <a href="http://startuplessonslearned.blogspot.com/2008/11/principles-of-lean-startups.html">anyone else trying to find an unknown solution to an unknown problem</a>) have to measure progress with <span style="font-style: italic;">validated learning about customers</span>.
In a lot of cases, that's just a fancy name for revenue or profit, but <a href="http://mashable.com/2009/01/02/how-to-raise-money/">not always</a>Either way, we have to recognize that the biggest form of waste is building something that nobody wants, and continuous deployment is an optimization that tries to shorten this <a href="http://startuplessonslearned.blogspot.com/2008/09/ideas-code-data-implement-measure-learn.html">code-data-learning feedback loop</a>.<br /><br />Assuming you're with me so far, what will that mean in practice? Throwing out a lot of code.
That's because as you get better at continuous deployment, you learn more and more about what works and what doesn'tIf you're serious about learning, you'll continuously learn to prune the dead weight that doesn't work.
That's not entirely without risk, which is a lesson we learned all-too-well at IMVULuckily, Chad Austin has recently weighed in with an excellent piece called <a href="http://aegisknight.livejournal.com/138346.html">10 Pitfalls of Dirty Code</a>.<br /><p></p><p> </p><blockquote><p>IMVU was started with a particular philosophy: We don't know what customers will like, so let's rapidly build a lot of different stuff and throw away what doesn't work.
This was an <a href="http://www.amazon.com/gp/product/0976470705?ie=UTF8&tag=aegisknightor-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0976470705"> effective approach</a> <img src="http://www.assoc-amazon.com/e/ir?t=aegisknightor-20&l=as2&o=1&a=0976470705" alt="" style="border: medium none ! important; margin: 0px ! important;" border="0" height="1" width="1" />to discovering a business by using a sequence of product prototypes to get early customer feedback.
The first version of the 3D IMVU client took about six months to build, and as the founders iterated towards a compelling user experience, the user base grew monthly thereafter</p> <p> This development philosophy created a culture around rapid prototyping of features, followed by testing them against large numbers of actual customers.
If a feature worked, we'd keep it If it didn't, we'd trash it</p> <p> It would be hard to argue against this product development strategy, in general However, hindsight indicates we forgot to do something important when developing IMVU: When the product changed, we did not update the code to reflect the new product, leaving us with piles of dirty code.
</p></blockquote><p></p>So that you can learn from our mistakes, Chad has helpfully listed ten reasons why you want to manage this dirty-code (sometimes called "technical debt") problem proactively.
If we could do it over again, I would have started a full continuous integration, deployment, and refactoring process from day one, complete with <a href="http://startuplessonslearned.blogspot.com/2008/11/five-whys.html">five why's for root cause analysis</a>.
But, to me anyway, one of the most inspiring parts of the IMVU story is that we didn't start with all these processesWe hadn't even heard of half of themSlowly, painfully, incrementally, we were able to build them up over time (and without ever having a full-stop-let's-start-over timeout).
If you read these pieces by the guys who were there, you'll get a visceral sense for just how painful it was.<br /><br />But it workedWe made itSo can you.<br /></div>
<p><a href="http://feedads.googleadservices.com/~a/63_t4w55ymDVOt7gzsMVzv7awxQ/a"><img src="http://feedads.googleadservices.com/~a/63_t4w55ymDVOt7gzsMVzv7awxQ/i" border="0" ismap="true"></img></a></p><div class="feedflare">
<a href="http://feeds2.feedburner.com/~f/startup/lessons/learned?a=eGY0K6DZ"><img src="http://feeds2.feedburner.com/~f/startup/lessons/learned?d=41" border="0"></img></a> <a href="http://feeds2.feedburner.com/~f/startup/lessons/learned?a=cRlqrBKw"><img src="http://feeds2.feedburner.com/~f/startup/lessons/learned?i=cRlqrBKw" border="0"></img></a> <a href="http://feeds2.feedburner.com/~f/startup/lessons/learned?a=WGQvlO13"><img src="http://feeds2.feedburner.com/~f/startup/lessons/learned?i=WGQvlO13" border="0"></img></a> <a href="http://feeds2.feedburner.com/~f/startup/lessons/learned?a=6VsK1bIp"><img src="http://feeds2.feedburner.com/~f/startup/lessons/learned?d=52" border="0"></img></a> <a href="http://feeds2.feedburner.com/~f/startup/lessons/learned?a=ik396021"><img src="http://feeds2.feedburner.com/~f/startup/lessons/learned?i=ik396021" border="0"></img></a>
</div><img src="http://feeds2.feedburner.com/~r/startup/lessons/learned/~4/oBSlJnTAwMw" height="1" width="1"/>.