Ruby/Rails Performance Links
June 21, 2008
Update: Yikes, I hit the Post button before I really wanted to! So you’ll get to see more of my revision process than normal. Sorry about that. It also might be incomplete when you read it, as it might take me a few days to finish it.
So I haven’t been posting here - or anywhere - a lot. That’s because I got a full-time contract doing some Ruby On Rails… while also trying to keep 2 other long-term clients happy.
Anyway, enough of that. One of my assignments has been performance optimization on part of the RoR website. A quick search will find you links… or you can read on to find mine.
First off, I was testing just one batch-type action in a specific controller, not anything massive. I bet my links will help you with that too, but some of that advice will be specific too my task, not yours.
First, I separated out the testable parts with the “un-testable” parts. Or rather, the stuff I wanted to test vs the view (which I didn’t want to test). Basically following the performance advice in the Agile Web Development With Rails book: create unit tests to make sure your performance doesn’t go backwards. This is OK advice, and with this model in place I was able to do profiling pretty easily… but there’s a catch.
Advice #1: Don’t trust Benchmark.realtime
I used Benchmark.realtime to test performance. Ok… except I was getting some pretty big differences between runs of a chunk of code - without having touched it! Then I remembered about Python’s timeit module, which says it “avoids common timing traps”, and makes mention of something Tim Peters wrote in the Python Cookbook. The money quote is :
… time.time measures wall-clock time [which I think Ruby’s Time.now does as well]. So, for example, it includes time consumed by the operating system when a burst of network activity demands attention…
So checking Benchmark.realtime once, like the code example in the Rails book does, is naive and probably won’t deliver consistent results. It’ll help if you run the test multiple times and take the mean, for example, or use maybe an even better statistical approach. Want to read more? Read The Ruby Programming Language or a blog post from the authors on profiling (where they suggest this).
But, there’s a catch! (Database Debris)
At the end of a test, Rails (at least to my understanding) rollbacks back the records etc you changed during the test. If your doing N test runs inside a test function cruft will build up if you do database manipulation. If you’re consistently creating record id 1 (or any other column that must be unique), your tests will fail after the first time. Which sucks. I don’t know of a way to rollback the transaction in the middle of the test.
See also: Unit Testing: Leave No Trace vs. Schmutz.
And another one (Timing Ranges, Again)
So when you have your mean values, it’s very (very) possible that you can pick a high threshold and say “it should never take longer than this” (Which is what the Ruby book does). There are two problems with this approach: that a simple assert MAX > ACTUAL only says “false is not equal to true” when it fails. Thanks guys. Secondly, it doesn’t account for variability. Yes, we (should have) gotten rid of some of this via our work above… but what if the computer has a full load while you’re running the entire test? Or if you’re testing on a slower computer than you develop on?
Both these problems can be avoided by using UnitTest’s assert_in_delta(). Let’s say we want our operation to take 3 seconds, give or take 1 second
expected_float = 3.0 delta_float = 1.0 assert_in_delta(expected_float, meantime_float, delta_float)
Apparently, assert_in_delta will pass if (meantime_float - expected_float <= delta_float) (Thanks WikiBooks!)
If this does fail (say with a mean_time_float of 5.0) we get something like: 1) Failure: test_it(MyTestCase) [test/my_test_case.rb:156:in `test_it' /home/ryan/my_test_case.rb:150:in `test_it']: "3" and "5" expected to be within "1" of each other.
Which is way better than “Hey, your test failed! That’s all!“)
Use A Profiler
Know What ActiveRecord Is Doing
Read Other Articles
Dominiek.com: Quick And Dirty Rails Optimization Guide- Making Rails Go Vroom