July 27, 2008

Book Recommendations: Geek Leisure

Filed under: Uncategorized — Ryan Wilcox @ 8:52 pm

The release of a new book from one of my favorite authors gave me incentive to write this list.

Techno-geekery Fiction

Because I read so quickly, for me to spend any length of time with a book requires the book to be pretty massive… as in around a thousand pages. These books fit that bill, in a very technological setting (or diving into very technical topics)

Geek “History”

  • The Cuckoo’s Egg: A hippy-ish astronomer turned sys admin/hacker tracks hackers through all kinds of early 1990s systems to… you guessed it… East Germany. All real.
  • Microserfs: Were you paying attention to the programming technologies coming out of Apple and Microsoft in say 1992-1993? (Ie: Do you remember yack about Pink or OpenDoc?) If so this book will bring back a lot of memories… and is just a good story about work, family, and companionship.

Geek Philosophy

  • Things A Computer Scientist Rarely Talks About: A lecture and question and answer format where one of computer science’s great writers talks about his Christianity, his experiences writing a book where he analyzes verse 3:16 of every book in the Bible, etc. Fascinating.


July 21, 2008

Book Recommendations: Python

Filed under: Uncategorized — Ryan Wilcox @ 10:35 pm

I’m starting something new on this blog – along with my normal content, I’ll give out book recommendations on topics or languages. I’ve got a pretty big bookshelf, and i pick the best of the best and list them here.

So, now on with my first topic: Python!

  • Python In A NutshellA BIG nutshell, a great reference book. The sections on performance profiling and testing are particularly excellent. When one of my alumni went to Hong Kong to do Python work, I gave him my first edition copy of this book.
  • Dive Into PythonThis is online for free at Dive Into Python.org. The book is meant to help experienced programmers ramp up into Python by taking examples and ripping them apart to see how they work
  • Python Cookbook15 minutes after cracking this book open I learned so much about what Python can do, Python idioms, etc. Excellent if you’re coming from another language too and just need to learn the lay of the land

Hope this helps speed people on their Python adventures!

July 9, 2008

Everything you ever wanted to know about debugging backgrounDRb in Rails… before it eats your soul

Filed under: Uncategorized — Ryan Wilcox @ 8:20 pm


BackgrounDRb is both a blessing and a curse. First, it allows you to farm off work to other Ruby processes, so that your main process can get back to the work of serving your client (before a proxy timeout). All kinds of things can be done: re-encoding data, doing long queries, importing data. You can query BackgrounDRb to see what the status of an item is: so giving a simple progress indicator.

That’s the promise. The curse is that BackgrounDRb is very hard to debug. Partially due to the fact that it’s like multithreaded programming (technically farmed out to other Ruby processes, presumably via fork)… but multithreaded programming with bad error messages is worse. BackgrounDRb is in this “worse” category.

So here are my tips for avoid having your soul eaten by debugging this beast. Or, at least, let you put up a good fight.

Zeroth up – Information

Go, read these articles. I’ll be waiting.

Don’t expect the RubyDocs to give you much information… there’s no actual documentation there. It just barely beats browsing the source files in your editor… but you may prefer this.

First Up – Know your versions

Install from trunk if you can. The December 2007 build of BackgrounDRb has an issue where you can’t pass more than let’s say 4K of data to or from a worker. I’m serious. Yes, that means register_status too. Now you might want to consider files as an interprocess communication method anyway… but I think BackgrounDRb is one of those projects where you should Trust Head.

Second Up – Architecture

Ok, so this stuff is hard. You want to be able to use regular old Ruby to test things out before you add BackgrounDRb to the mix. My suggestion: do all your work in a totally separate, isolated, and unit-tested class, and use your worker only to set that up and run it. Seriously: the more you can debug in unit tests, normally, the more hair you’ll have.

Third Up – The Pain

There are two ways to approach things at this point: the mock way and the interpreter’s way. I don’t care which you pick.

The Mock Way

Instead of inheriting from BackgrounDRb::MetaWorker, inherit from this sucker….

class FakeyWorker
attr_reader :logger
def initialize
@logger = logger
def register_status(data)
@logger.debug("FROM REGISTER_STATUS")

def self.set_worker_name(something)

def get_status
return @currdata

Import your worker file into your controller, and call it directly. Call get_status after the call to your worker method, and see what comes back. Since this is (still) happening synchronously, you’ll see exceptions in your browser. Fix errors. Rinse and repeat. Then lose the training wheels and use MiddleMan like you’re supposed to.

The interpreter Way

If that doesn’t appeal to you, (or you’ve done the above but still don’t trust) do the following:

  1. Fire up one console window with script/backgroundrb running
  2. Fire up another console window, and run script/console
  3. Trigger your worker from here. Here’s some example code:

    >>> MiddleMan.new_worker( :worker => :import_data_worker, :job_key => 41258 )
    >>> data = {....}
    >>> MiddleMan.worker(:import_data_worker, 41258).my_worker_method(data)
    >>> MiddleMan.worker(:import_data_worker, 41258).ask_status

If ask_status returns nil, you have a problem. If a traceback shows up in the script/backgroundrb window, you have a problem…. but don’t bother reading the traceback – it’s probably worthless.

If you’re having troubles, break it down: are you 100% sure your worker is not nil? (Answer: No). save the worker to a variable and check it! Also a worker’s worker_info method is nice.

If you did get nil from ask_status, I’d ask: is my_worker_method even getting called? Do something as obvious as possible: I like writing to a file in my_worker_method. No way to miss that, or have it go to the wrong log.

Fourth Up – BackgrounDRb STILL wants your soul

BackgrounDRb really wants to eat your soul. Really. Here are some suggestions:

  • Wrap your entire worker method in begin… rescue and register_status $!.backtrace (or, write it to a file)
  • Logging MiddleMan.all_worker_info.to_yaml in your controller action is brilliant. Verify you have the worker you’re looking for!
  • I’m not joking about checking to make sure MiddleMan.worker returns you a worker.
  • Is your worker method not getting called for “no reason at all”? Are you passing a lot of data to it and using the December 2007 version?
  • I’m assuming you’re copying off a worker and action that works already?
  • Some of these things could be full of crap. If so, let me know.

The End

With any luck this article helped your debugging, and you spent way less time than I did debugging your workers. Yes, BackgrounDRb sucks: the documentation isn’t good, the error messages are horrible, it’s a ton of infrastructure, insanely complicated, and sometimes it Just Doesn’t Want To Work. It’s also a fair tool. For all that, I might not be able to submit patches to help out the project, but I can write this article.