markitechture

Turn Off Your App

I want you to do something, and this is not a joke.

Turn off your web application. Seriously, pull the plug - don’t put up a ‘site maintenance’ page. Wait an hour. Turn it back on.

Miraculously, the world continued to rotate, didn’t it?

If you are reading this, chances are you are not publishing/developing a product that people rely on for, say, the functioning of their internal organs or upper-atmospheric trajectory adjustment of ballistic entities.

Most likely you are interested in the world of consumer web applications, centered around the collection (programmatic and user/form input) of data, and the visualization of this data in interesting ways. This is not rocket science, and no one is going to die if your app goes down.

Remember this. No one will die if your app goes down.

So, what happened when you turned your app off? (You didn’t, by the way, and that proves a bit of a point that we will get back to. In the meantime, go turn your damn site off, just do it.) Were you inundated with a flurry of angry support tickets, phone calls, IMs, twitter @-replies? Probably not, but if you were, that’s okay too - in fact, it’s better.

If people missed your site in the hour it was unavailable, congratulations; users hating not having something is the only true form of users loving something. Oh, and guess what? Those users who love your app will come back, and continue to use the shit out of it. Who complains about the Twitter fail? Twitter fanatics who would probably re-adopt the service if it were, for some reason, to disappear for 6 weeks.

If people did not miss your app, well, you’ve learned something very valuable. You have some work to do on your product, don’t you?

Now we’re getting somewhere.

———————

Time-to-market is a self-fullfilling prophecy
A process that fears publishing creates the root of this fear - Remove the fear, remove the reason it exists.
A release engineer recommended to me yesterday morning that we wait 5 (!!!) days to push a new build* (of a language search application we built) live. His fear was simple, and ultimately rational: if something went wrong, it was going to take us days to fix it, so we better get it right the first time.
What if it didn’t take us days to push a build live? Well, then everything would be cool. When we see a bug in production, we fix it and update production. This is not rocket science.
The trap is aiming for perfection out of fear. When one stops being afraid, that there is far less to be afraid about.
*It’s worth mentioning that our release candidate build had one known bug, and scores of resolved bugs and new features. Which would the user prefer?

———————

How often does new code appear in your live app?

For people who don’t hear a peep when their site is offline for an hour, here is a challenge: explain why every commit you make should not go live to production. Cue the crickets.

For people who have a loyal userbase, consider the power of the ‘oh like this?’ moment. You get awesome feedback from an early adopter or power user. Is your inclination to file a ticket, hope a PM schedules it for the next release, and forget about it, or do you have the agility to make the change (perhaps even just for that one user), push it live, tell the user to refresh and say the magic fucking words: ‘Oh, like this?’ That’s one hell of a ‘this’ to the user, I might add. If you are a pretty sophisticated web publisher, you may be thinking about how to generate more evangelism from your early adopters and power users; implementing their best ideas right before their eyes is pretty much unbeatable to this end.

———————

No release of a web application is perfect.

And none will ever be. The question should always be: is my currently-deployable release candidate better or worse than what is in production? I hope you are rolling your eyes and saying that pretty much every time you commit changes you could push a better build than what you have live in production. Kudos to you, I fully assume you are right. So put it in production.

But … No buts.

But what if there is a bug in the build, we haven’t put it through QA!

Tell QA you’re pushing a build live, let them poke at it in production. Testing outside of production is often a waste anyway; many bugs appear only when moved to production. There is a good best-of-both-worlds solution that you might consider, and we’ll get to that in the conclusion**.

But what if a potential customer sees an issue and gets a bad first impression of our product?

Ever the Irishman, I will answer that question with a question: is your development build better or worse than what’s in production? Do you want your potential customer to see those 10 cool things you added this morning, or not? Don’t you know of many bugs that are already in production that a customer might see anyway, and aren’t many resolved in your RC?

But, what about if 

Sorry, gotta cut you off here. Time is of the essence, and you have code to push live. 

———————

Practical Considerations

This is not meant to simply be a polemic position paper of the perfect pretty push pattern. Let’s think about how to do this in real life.

For very new products, consider the straight-up hot deploy approach. I learned this seemingly insane method from Aaron White, and we used it and continue to use it on Habits, a simple, (hopefully) useful Facebook app we built in 24 hours because, well, we are huge dorks and that’s a fun thing to do for people like us. Here’s the rig for that app: Commit changes to version control » hook triggers deploy script » new code hits cloud. The time from new code working locally to being in front of users is measured in seconds.

This works incredibly well for brand-new products, for one- or two-person teams. You are in synch with your teammates, and only ever slightly ahead of the production environment. We launched a Minimum Viable Product in 24 hours, the product continued to work well after we moved our attention to other things, and when we go back to make minor changes, fix bugs, etc, we are delighted that there is absolutely 0 overhead in maintaining our code and deploying changes. Build and deploy is simply invisible, commit === deploy.

For larger teams working on adding features that don’t exactly come together overnight, it might seem that the logical idea is for a developer to work locally until the whole feature or feature set is working, then check it in, QA it in a test environment, fix bugs, and eventually push the whole thing live, maybe with a fancy email announcement to the userbase.

Scary, huh? Pushing big new features tends to give us that little tingle in our sphincters, and we immediately go through worst-case scenarios of the app failing just as the rush of users comes to check it out. Yeah, that approach sucks and it’s broken.

Take the feature, maybe write your tests first, at least outline all the work you’ll need to do. Probably you do this already, so take your kanban board or stickies or whatever, and try to spot the foundational work that needs to be done to support the feature. Do that first, and push it live, piece by piece.

Now, do the user-facing work, and maybe even push that in front of some users. Watch and listen. Featured in production do not all have to be exposed! A feature in production != a feature available to all users.

Afraid of sending that announcement now? The prospect of exposing a ‘new’ feature becomes purely exciting.

This is not a new concept. What we are talking about here is minimizing time from order/idea/innovation/execution to delivery. Unreleased code is surplus inventory in the warehouse, it’s waste, and it’s costly.

How can I make this happen?

Remove manual steps. You probably aren’t interested in the full (automatic) hot deploy. What is worth considering, though, is examining the value added by each manual step in your build and deploy process. If something is done manually that could be done better automatically, do it.

Provide rollback. To live free of fear, you need to be able to revert to a previous release with one click. Consider building hooks so you can do this from email or from your phone. This alone will probably encourage you and your team to push way faster than without rollback.

**Test in beta/production. A glorious build/deploy rig has the capability to deploy a new release to specific subdomain, where it interacts with the same live machinery and technology as the live app, including the datastore. Test here, and push here often. Feeling spicy? Let some of your users know about this ‘edge’ build and let them hammer on it as well; they get the hottest, newest features and you get usage and interaction metrics, as well as early pre-public-release qualitative feedback.

Push small changes. Self-explanatory.

Associate commits with tickets. A nifty and useful piece of housekeeping. Many systems allow synchronization between commit notes and issue tracking, find out how to do it with your system.

Get a kickass build engineer. Consider having a dedicated development resource who knows your environment inside and out, especially your continuous build and integration tools and version control. A good candidate would be someone already focused on system administration whose life would probably be easier if build/deploy were more automated anyway. Arguably the only person who knows the state of the product better than the PM would be the build/release engineer. As a team grows and more branches are introduced into the codebase, hiring a dedicated build engineer makes more and more sense.

Postlogue

I’d be remiss not to mention that Google App Engine offers a remarkable number of relevant features in this regard. Version rollback is stellar, as is the ability to run different versions of your app on different subdomains. It rules, consider checking it out.

OUT!

/@markitecht