At Acts As Conference last week an Engine Yard engineer briefly took questions from the audience on performance and scaling Rails. Some of his suggestions seemed fairly obvious but I realized that might be because I have been working on high trafficked Rails apps for over a year so I have gone through all the pains of trying to scale a Rails app. I took some notes in the conference and over the last couple of days on some of the guidelines and principles that I try to follow in my Rails apps to keep it from becoming a kludge. These may seem obvious to most seasoned Rails developers, but I am really writing this for the Rails developer about to face her first heavily trafficked Rails app.

No Yak-shaving here.

A lot has been talked about on the topic of scaling Rails. There have been more than one can rails scale? dustups over the past year. Every couple of the months the meme revives itself like a phoenix (no, not THAT Phoenix). Although sometimes it's fun to get into yak-shaving debates it's usually just a waste of time (that's why it's called yak-shaving). Instead of spending time arguing over whether something is scalable, let's spend some time on making our applications scale by focusing on the part of your app you have the most control over: your code. We'll save the debates for another time.

A 'problem' with the Mongrel/Rails platform

First lets just look at a simplified picture of a classic Rails production architecture. Nginx, Apache, or a similar web server sits atop a cluster of Mongrel servers, each with their own Rails app loaded into memory waiting for requests. A request comes in to the web server, and if it's dynamic, falls down to a waiting mongrel process. Mongrel calls Rails which wraps a big lock around most of the request/response cycle (see Ezra's talk for more information), so the mongrel thats serving the request has to wait for a response from Rails to unlock and start serving other requests. This is the Blocked Thread stability anti-pattern that Michael Nygard talks about in Release It!. The problem is that Nginx or Apache doesn't know that a mongrel is blocked and keeps blindly sending requests. Thats means that even a single blocked mongrel will result in some slow responses for requests that come in to that same mongrel. Here is a quick diagram of a simplified view of this architecture:

Cut the Request Fat

But this isn't a post about making the Rails deployment architecture scalable, its about making YOUR Rails code scalable. And one way to do that is to use the "Skinny Request, Fat Backend" principle. What that means in practical terms is pretty simple: Do as little as possible inside of the Request/Response cycle. That way you don't lock up all of your mongrels while your users sit in proxy land. Remember Skinny Controller, Fat Model? Well, instead I like to think "Skinny Request, Fat Queue" or "Skinny Request, Fat Cron", or "Skinny Request, Fat Client".

The Bad Cholesterol

Ok so by now your almost convinced that you should keep your requests skinny. (if not just read Release It!). Here are some common fat operations you can cut from your apps:

• Cache columns calculations. Most of the time your cache columns will be simple increment/decrement counters implemented using rails :counter_cache option on belongs_to. These are simple and don't add much overhead to your RR. Some times cache columns are complicated calculations that touch collection of objects. These can easily get very slow if performed on every object creation or save.
• Calls to other web services. Almost every app I have built in the past 3 years use at least 3-4 web services. Often these services are from powerful servers, such as Google, Amazon, and Yahoo. Other times they are from another startup that face scaling problems just like you and me (twitter anyone). In either case, one cannot expect to always have quick response times from external services. Hell even Amazon S3 went down for 2 hours last week. If you are lucky you will consume an API responds promptly with either a success or failure. Murphy's Law states you will not be so lucky, instead the response will never come, and your app will hang. If this API call lives within your Request/Response cycle, and you don't set extremely aggressive timeouts, you now have a Cascading Failure (Nygard 54) on your hands.
• Image manipulation. Most Web 2.0 apps these days have google looking Avatars that are nice to look at but tie your requests up with image resizing, redrawing, and re-*. Or they create 10 different sized thumbnails for use in different sections of your site since attachment_fu makes it so simple. Image manipulation (and video) should always be done outside of the request. If you are using RMagick, you may even have a bad memory leak on your hands, which will fill up a Mongrel's memory footprint quickly.
• Sending emails A lot of times your apps will need to send email. Sending an activation email is one thing, but mass emailing can quickly become something you don't want to do inside of the request. This may sound anal, but considering the ease of which they can be taken out of the request, why not avoid another integration point failure?
• High Demand Controllers Any controller that is used for an extraordinary amount of traffic compared to the rest of your application. This could be a controller being consistently polled by many ajax clients or a controller that does any kind of tracking. If you can isolate one controller/action that is taking large proportion of requests compared to the total requests to all other actions, it could make for a good case of extraction, especially if the controller has sessions turned off.
• Any other side effect. Side effects are things like sending email, image manipulation, cache column calculations, or anything that doesn't contribute to the response body and does not need to be performed in a timely manner.

Note about premature optimization: I don't believe in premature optimization, but I do believe in mature optimization. After seeing the same problems pop up time after time, I make a note to try and not repeat it again. What some will see as premature optimization I see as experience and problem avoidance. When building in the precautions is just a tiny bit or even equal amount of work, why not save yourself the hassle down the road? I use the following techniques even before releasing an app to the wild.

Bypass Surgery

• Cron This is a fairly obvious technique used by most apps I have run across. A operation that needs to happen periodically gets put in a rake task with access to the rails environment and an entry is put in the crontab. Don't use rails_cron, instead just use regular old cron. Cache Column Calculations and sending emails/broadcasts work well with this Fat Backend. Be mindful of long running cron tasks that are called frequently. Watch out of operations that run iterate through every record in a table. When that table starts growing exponentially your operation will do the same. I recently came around to a technique of using a Robot.rb module that makes it obvious which operations get run at specific intervals. Here is an example Robot.rb file:

  module Robot
  
  	def self.hourly
  		#hourly operations here
  	end
  
  	def self.daily
  		#nightly operations here
  	end
  
  	def self.five_minutely
  		#five minute operations here
  	end
  
  	def self.monthly
  		#monthly operations here
  	end
  
  end

  Then in your crontab:

  0 0	* 	* 	*		 /RAILS_ROOT/script/runner -e production "Robot.daily"
  0 *	* 	* 	*		 /RAILS_ROOT/script/runner -e production "Robot.hourly"
  0 0	* 	* 	0		 /RAILS_ROOT/script/runner -e production "Robot.weekly"
  0 0	* 	0 	*		 /RAILS_ROOT/script/runner -e production "Robot.monthly"
  

  This keeps all of your operations that are run with cron under one module that signifies to the developers exactly which operations get run when. I have to give credit to Dray Lacey (blogless, sorry, no link) for coming up with this technique.


• Messaging Queues are a powerful way to trim some fat off those requests. Its simple: In your app instead of performing the operation inside the RR cycle, you send a message to a queue that includes information on how to run the operation. Worker processes grab messages from the queue in a First In, First Out queue, perform the operation described by the message, and then pop the message off the queue. There are many Ruby gems and Rails plugins designed for this specific solution. I haven't used it, but from what I hear the BJ plugin has effectively replaced backgroundrb. Starling is another queue that was developed at Twitter and is their workhorse. SQS is Amazon's queue in the cloud service, which recently dropped in price (depending on your usage profile, too much polling == more money). We have used SQS at Izea and it has worked decently well, but has been prone to problems if there is a large queue backlog. For a much more exhaustive overview of messaging queue's read topfunky's post on the subject. UPDATE: also check out Tobi's just release DJ plugin extracted from shopify.


• Extract controller outside of rails This technique is mainly used on High Demand controllers. The reason this is effective is there are much more efficient ways to handle requests using something more lightweight than Rails, and that doesn't throw a huge lock around the RR cycle. This extraction is a Bulkhead Pattern (Nygard 108), partitioning systems to provide more capacity and isolate those partitions from harming each other. There are two easy ways that I have seen: creating a custom Mongrel Handler, or creating a Rack App. Seeing as Rack is being adopted by some major frameworks, it seems like a more viable and long lasting solution than a custom Mongrel handler. If you go down the Mongrel handler route, you'll have to re-implement a solution if your app ever switches off of Mongrel (don't think thats possible? check out thin and come back to me). Here is a quick and dirty tutorial on extracting out your high demand controller with a Rack app (i'll save the more lengthy tutorial for a future post).

  Set up your web server / proxy

  If your using nginx as your web server this is a simple couple line config file edit:

     upstream thin_rackapp {
        
          ip_hash;
          
          server 11.11.111.11:10000;
      }
  

  Then in your server block:

    server {
      //other nginx server config here
     	location ^~ /rackapp/ {
          proxy_set_header   Host             $host;
          proxy_set_header   X-Real-IP        $remote_addr;
          proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        
          proxy_redirect     off;
        
          proxy_pass  http://thin_rackup;
      }
    }
  

  This will pass any requests that start with rackapp (such as izea.com/rackapp/whatever) to route to port 10000 on 11.11.111.11.

  Create the Rack App

  Next you just have to create the rack app which consists of two main files, the config.ru file, and the Rack handler. For a short primer on creating the rack app see this post. I prefer keeping everything inside of the same source control repo, so I would create a directory in the top level of the rails app so my directory structure would look like:

   -- RAILS_ROOT
     -- app
     -- config
     -- spec
     -- lib
     -- vendor
     -- db
     -- script
     -- public
     -- rackapp
        -- config.ru
        -- rackapp.rb
        -- test
          -- rackapp_test.rb
          -- test_helper
        -- configs
          -- staging.ru
          -- production.ru
  

  config.ru holds the development environment config information. rackapp.rb is the rack app that implements .call(env). The test directory holds a Test/Unit class and a test_helper for loading ActiveRecord. configs holds the config files for each environment I want to deploy to (I'll be covering deployment in a bit). Here is an example config.ru file that loads ActiveRecord:

  require 'activerecord'
  require 'yaml'
  configuration_options = YAML::load(File.open("../config/database.yml"))["development"]
  
  ActiveRecord::Base.establish_connection(configuration_options)
  
  run Rackapp.new

  I'm only loading ActiveRecord here for database access. It uses your existing database.yml file so as to not duplicate database setup information. If you want to you could go ahead and just use the Mysql gem for higher performance if you are doing many inserts/reads. If you dont need database access then just go ahead and get rid of the AR code and just run Rackapp.new

  Deploy with capistrano

  If you've used the same setup with your rack app as above deployment will be super easy, and even easier with thin, since it knows rack out of the box. Here is a sample recipe:

  namespace :rackapp do
      desc "start the rackapp under the Thin webserver"
      task :start, :roles => :app do
        run "thin start -r #{current_path}/rackapp/configs/#{stage}.ru -d -P #{current_path}/tmp/rackapp.pid -p 10100 
            -l #{current_path}/log/rackapp.log"
      end
      after "deploy:start", "deploy:rackapp:start"
      
      desc "stop the rackapp handler"
      task :stop, :roles => :app do
        run "thin stop -P #{current_path}/tmp/rackapp.pid"
      end
      before "deploy:stop", "deploy:rackapp:stop"
      
    end

  Here, we want to start the thin server after deploy:start runs. The thin start command takes a -r option that takes a rack config file path as its argument. Notice how the correct config file is loaded by naming the config file after the environment. We pass the -d option to start thin in daemon mode, which writes the process id to #{current_path}/tmp/rackapp.pid for later use in stopping the thin server when running the deploy:stop task. Finally we specify the port we want to run it under with -p 10000 and the log file we want it to log to with -l #{current_path}/log/rackapp.log

  Perf stats

  A couple of httperf runs is enough to convince me that extracting a high demand controller out of rails can dramatically increase performance. Here are some perf stats (all in production, primed, with the same db access profiles, 1 route, no sessions, using AR::Base.connection.execute):
  
  1. thin server running rack app

           Request rate: 742.7 req/s (1.3 ms/req)
         

  2. Large 30000+ line Rails 1.2.3 app (actual app being developed for)

           Request rate: 24.2 req/s (41.3 ms/req)
         

  3. Clean Rails 1.2.3 app

           Request rate: 255.2 req/s (3.9 ms/req)
         

  4. Clean Rails 2.0.2 app

           Request rate: 269.6 req/s (3.7 ms/req)
         

  As you can see the rack app under the thin server can handle 2.7 times the req/s of the clean rails 2.0.2 app. Other than just the higher throughput, I get to relieve my mongrels of the extra traffic and Rails aggressive locking.

Just the beginning

This is really just the tip of the iceberg when it comes to web application performance and scalability. I still have a lot to learn, thats why I am reading Release It!. Here are some resources I have referred to and have found extremely useful:

• Youtube Scalibilty talk
• Ezra's Merb presentation at Ruby Hoedown
• High scalability twitter article
• 300 million page view per month rails app scaling talk
• Testing various configurations of Rails, Merb, Swiftiply, and Nginx
• Nginx config like woah
• Michael Nygard's blog, Wide Awake Developers
• Event Driven Mongrel and Swiftiply
• Merb the Rails-like framework that doesn't suffer from the RR lock problem

Feedback

This is really one of my first tutorial posts I have done and really put a lot of effort into. If you have criticisms, witicisms, or suggestions, please leave a comment or email me at rubymaverick AT the google email top level domain.