We’ve finally managed to track down and fix a bug in our system which has been bothering us for a while. It turns out to be related to a race condition in Ruby on Rails’ session management code. We would like to share our analysis of the problem, and our solution, with you.

Session management in Rails

Session management is one of the (many!) things which “just works” in Rails.

Until it doesn’t.

The session in a Rails app is a hash. In between actions, it’s stored persistently as a marshaled string. A number of different persistent stores are supported, but whichever you use, roughly speaking each action in a Rails controller does the following:

1. Load the current session, or create a new one if necessary
2. Run the code of the action
3. Save the session, including any changes made while processing the action

What could possibly go wrong?

Concurrent requests

What could go wrong is concurrency. Imagine that the user makes two requests at the same time (let’s call them request1 and request2). What’s going to happen is something like this:

1. The action servicing request1 (lets call it action1) loads the current session
2. The action servicing request2 (lets call it action2) loads the current session
3. Action1 makes some changes to its copy of the session
4. Action2 makes some changes to its copy of the session
5. Action2 completes and saves its changed session data
6. Action1 completes and saves its changed session data, destroying the changes made by action2 in the process

Depending on the precise order in which things happen, maybe the changes made to the session while serving request1 “win”, maybe those made during request2. But whichever, if two requests are made concurrently with a single session, somebody’s data is going to be lost.

What makes this even worse is that Rails saves the session even if all the action ever does is read from it. So in our example above, action2’s changes to the session would be destroyed even if action1 made no changes to the session at all.

Is this really a problem?

Short answer: yes.

Longer answer: it depends on your application. In a “traditional” non-AJAX application although this kind of thing might happen in theory (if the user has two browser windows open on your application and refreshes them both at the same time, for example), in practice its not going to happen very often.

In an application making use of AJAX, on the other hand, it’s increasingly common for pages to be constructed from multiple requests, and for these requests to “overlap”. And as with most race conditions, things will probably work just fine most of the time - but occasionally they will break in difficult to understand, difficult to reproduce and difficult to debug ways.

An example

The following isn’t a very realistic example, but it’s simple and demonstrates the problem. Imagine that you have a controller containing the following:

def long
  session[:foo] = "bar"
  sleep 15
  render :nothing => true
end
 
def short
  session[:short] = 1
  render :nothing => true
end
 
def status
  render :text => "session[:short] = #{session[:short]}"
end

To see the problem, you will need to run this on a webserver which allows more than one action to be handled simultaneously (i.e. not webrick). A couple of mongrels will do just fine.

In one browser window, run the long action. Before it completes, run the short action in another browser window and then the status action. You should see that the :short key contains the value 1. Run the status action again after the long action has completed, however, and you’ll see that the :short key goes back to being empty.

So, what to do?

The first thing to note is that this kind of thing is a fundamental issue with concurrent actions. Nothing we do is going to help if two different actions want to make conflicting changes to the session. But that doesn’t mean that we can’t improve things considerably.

One improvement we could make would be to avoid saving the session if an action makes no changes to it. That would help, but we can do better. The solution we’ve chosen is to modify Rails’ session handling so that each action performs the following steps:

1. Load the current session, or create a new one if necessary
2. Save a copy of the unmodified session for future reference
3. Run the code of the action
4. Compare the modified session with the copy saved previously to determine what has changed
5. If the session has changed:
   1. Lock the session
   2. Reload the session
   3. Apply the changes made to this session and save it
   4. Unlock the session

This approach means that two actions which modify different keys within the session hash won’t interfere with each other at all. Of course this doesn’t come for free; we’re doing quite a bit more work than the standard Rails code (although we should recover a little performance by not saving the session unless it’s changed). Depending on how much data you store in your session, this may or may not be acceptable. Speaking for ourselves, we’re prepared to trade a little performance off against correctness!

We’ve put together a plugin (based on SqlSessionStore) which implements these changes and is available from here. Please feel free to use it, and let us know how you get on!

Links

This problem is not unique to Rails. A discussion of the same problem and a (slighly different) PHP-based solution can be found here.

Credits

All of the work described above has been carried out by Frederick Cheung, as part of his work here at 82ASK.

This entry was posted on Tuesday, May 1st, 2007 by Paul Butcher and is filed under Wizardry. You can leave a response, or trackback from your own site. You can follow any responses to this entry through the RSS 2.0 feed.