December 21, 2011

Using Fabric to import your Django models

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 9:49 pm

A client wants me to write an import task for their Django app. They already use Fabric to deploy their site, so I figured that writing this script as a Fabfile would work out well.

The script requires me to import classes from their Django app. Specifically, I’m doing queries against their domain models, and adding things to the database, and I’d like to reuse the Django ORM classes already defined.

I finally got it working, but it was non-obvious. Here’s how I did it.

Setting up the module import path

In this project all the fabfiles go in a folder named “fab” in the Django project’s directory. So, I need to tell Python to look outside the fab folder for what it is trying to import.

But that’s not enough – I also need to import the Django project by name, so I need to go one more folder out (to the parent folder of the Django project).

Actually importing the Django models

the django.project nonsense is to set up an environmental variable (pointing to the settings.py file) – Django requires this variable to be set, and will error without it.

Next we import the Django project, and start using entities from an app inside it.

Easy, Huh?

October 19, 2011

Announcing: delegate_presenter: the simplest Presenter Gem that could possibly work

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 8:19 pm

Lately there’s been a lot of buzz in the Rails community about using the Presenter Pattern to organize common view related code outside of the model.

Think of Presenters like Helpers: The Next Generation

I used Presenters on one project with great success. Today I was about to add presenters to a second project, when I said:

Self, you could copy and paste all this code from Project X to Project Y, or you could extract it into a gem

So, that’s what I did: introducing the delegate_presenter gem

Read more documentation on the Github page

August 5, 2011

Capistrano, system wide RVM and creating gemsets as part of deploy:setup

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 11:29 am

Introduction to the Problem

Capistrano is the standard way to deploy Rails apps. Yesterday I was using Capistrano to deploy to a machine where I had installed RVM (Ruby Version Manager) at the system level.

I manually set up Ruby 1.8.7 and Ruby 1.9.2, because I need to run two applications on that machine (one a Ruby 1.8.7 app and one a Ruby 1.9.2 app). Using RVM for production deploys is great for this.

My cap deploy:setup task, however, complained that Ruby 1.9.2 wasn’t installed on the machine.

That’s funny, because I did install it, I thought. After banging my head up against the problem for a few hours, I finally posted the question to Stackoverflow.com: (Capistrano deploying to system wide RVM not seeing installed Rubies)

I got my answer: the message about the Ruby not being installed was misleading, it actually meant that the gemset wasn’t installed. Which it wasn’t (I was planning on doing that as part of the cap deploy:setup task.

Creating gemsets in your deploy:setup step

Ideally I want cap delpoy:setup to take care of eveything for me: installing some rubies, creating the appropriate gemsets, you name it. Because automated deployments mean everything should be automated (amirite?).

But then I get errors like this when I’m trying to create the gemset I want to use!

It’s non-obvious how to do this – and in fact the obvious way will not work!

Background

You see, require 'rvm/capistrano' hooks into the low levels of Capistrano’s run function, meaning everything happens in the context of the ruby+gemset that you declared in your Capfile. (Technically rvm/capistrano uses a user shell called rvm-shell, instead of bash or sh. This shell knows enough to properly set your paths to Ruby etc etc.

Normally this is awesome – that means that Capistrano knows about your Gemset, and installs gems there etc etc. Capistrano’s run command just does the right thing.

However, there are two cases where you want things to happen outside of rvm-shell:

  1. Installing the Ruby
  2. Creating the Gemset

If you try to do these things using run, Capistrano will give an error about Ruby not being installed, like it gave me. Even if RVM is trying to say, “I don’t see that gemset”, the error message will be about a missing Ruby.

The obvious solution, and why it doesn’t work (as a conversation)

The obvious thing you might try in your Capfile is this command:

run "rvm install 1.9.2"

Except, as I explained above, that won’t work. Here’s what’s going on, as a conversation.

You, to Capistrano: Run this command for me

Capistrano, to remote machine: Hey, I want to log into this machine, using the rvm-shell command, using Ruby 1.9.2 and gemset MY_APP. When I’m logged in please execute rvm install 1.9.2

Remote machine, to Capistrano: Could not log you in, an error happened when firing up rvm-shell. I could not find the ruby/gemset you wanted, so I can’t set the Ruby paths appropriately. I’m giving up and stopping because I can’t possibly do whatever that command was that you wanted me to execute

Capistrano, to you: I couldn’t install that Ruby you wanted me to install because I can’t activate that Ruby you want me to use for the gemset you want me to use – I don’t think that Ruby is installed!

You: Le sigh.

The solution: avoid rvm-shell for Ruby installation AND gemset creation

You might think that you need to avoid rvm-shell just for the installation of your Ruby. In fact, you need to avoid rvm-shell for both the installation of Ruby and the creation of your gemset!

How to avoid rvm-shell

Define this method in your Capfile.


def disable_rvm_shell(&block)
old_shell = self[:default_shell]
self[:default_shell] = nil
yield
self[:default_shell] = old_shell
end

Now, in the context of that block, run will execute commands by using sh/bash as a shell, instead of rvm-shell.

In your Capfile, install Ruby and your gemsets by:


disable_rvm_shell do
run "rvm install 1.9.2"
run "rvm use 1.9.2@MY_APP --create"
end

This installation process must come before any other command in your deploy:setup chain.

Conclusion

And that’s that – I really hope this helps someone out there!

May 22, 2011

Rails, REST, and .js (TL;DR: We’re doing it wrong)

Filed under: ResearchAndDevelopment — Ryan Wilcox @ 10:32 pm

In my last blog article I talked about Returning HTML content from AJAX requests. However, I’ve been struggling with a question since then:

I asserted (implicitly) that the following (and canonical!) pattern for doing AJAX replacement is wrong. I felt this strongly, but why is it wrong?

First, the current pattern

Normal Rails pattern for HTML replacement is to make an AJAX call to the .js format, in which we return something like the following back to the browser:


<%# show.js.erb %>
var str = '<div id="<%= dom_id my_object -%>" >
Name : <%= my_object.first_name -%>
</div>';
$("#destination_div").replace(str);

To this I say Ugh.

Why Ugh? Why is it wrong to return this in the .js format?

Why? The DOM transform operation above isn’t the representation of the object

Let’s review what Rails actions do in the respond_to block

  • format.html returns a HTML representation of the object (for display on screen) — aka: it renders the view
  • format.json returns a JSON representation of the object
  • format.xml returns an XML representation of the object

These are all pretty normal in the Rails world. The last two take the object and serialize it (using .to_json or render :xml => object. Straight from rails generate scaffold

You might imagine (from DHH’s RailsConf keynote introducing Rails 2) that other formats existed, with mostly the same result:

  • format.icl might return the iCal representation of the object
  • format.csv might return a CSV representation of the object

But in the canonical Rails show.js.erb file at the beginning of this blog post, the .js is not returning the Javascript representation of the object.

Can I do this with your object?


$.ajax(
...
success: function (data, textStatus, xhr) {
var obj = eval(data);
alert(obj.first_name); /* attribute, like JSON */
alert(obj.api_function() ); /* and here's a function call! */

Answer: No, because your .js “representation” is some DOM injection of an unparsable HTML string.

A pattern for returning an actual Javascript Object

If you return an actual Javascript representation of the object, you have the same data attributes available to you on the JS side as you do on the Ruby side.

For example, if your .js format rendered a .js.erb template like this:

Yes, this smacks on the If you’re using to_json, you’re doing it wrong article by the Gomiso guys. I absolutely agree with their core idea of maybe you should be using views to construct your json objects!

Notice here, we are returning a Javascript object, with data and methods that encapsulate functionality … which is exactly what objects are.

I also know that you use a Ruby object in different contexts than you do a Javascript object. In Ruby I want to perform my business logic – computing priority of due dates, in the case of my Todo object here. In Javascript I care about different operations – perhaps mostly related to AJAX and DOM manipulation.

For example, I’ve always struggled with getting my Rails routes into my unobtrusive Javascript. In Ruby (or .erb code) I use routes and path helpers like I should… and on the Javascript side I hardcode the routes. Heaven help me if I ever have to change the routes – I’ve solved only half the problem!

Since I implemented a routes method in my JS object above, the route will always be correct – because I’m using the normal Rails routes helpers to generate it!

In Rails 3.0.x, it’s non-obvious to keep the Javascript representation of an object in show.js.erb. This location is certainly it’s not where I would look for the source code to the object! Rails 3.1, I believe, will save this day. Imagine a Rails 3.1 app that uses the assert pipeline to store the Javascript representation in the following hierarchy:


app/
assets/
javascripts/
models/
todo.js.erb

Much better!

To Review

  1. The normal Rails pattern of returning DOM manipulation functions as the Javascript representation of the object is not RESTful
  2. … and inconsistent, because every other format (.json, .xml) returns the representation of the object
  3. We should return actual Javascript objects – data and methods to act on that data

So, how do we do AJAX replacement then?

If we can no longer use the .js format to return a DOM manipulation function, how do we return chunks of HTML to the browser (for a standard “AJAX request and replace” type functionality)?

Answer: Request a partial HTML representation of your object

My Returning HTML content from AJAX requests previous article

covers the how very well, but there are some other advantages:

  • We’re returning proper representations for proper formats
  • The Javascript that originally makes this AJAX call knows more about the structure of the DOM – and where it wants the result – than a show.js.erb that lives on the other conceptional side of the app

May 12, 2011

Returning HTML content from AJAX requests – a pattern for Rails 3

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 5:25 pm

The problem: semantic formats for returned HTML for AJAX

In a previous Rails 2 project, we decided that Rails apps return 3+ kinds of content:

  • A complete HTML page, for user viewing
  • JSON (for JS/web API viewing)
  • A partial HTML page, for jQuery DOM swapping/

However, it was hard to know when to return a full HTML page, and when to :layout => false

So we invented a semantic format

Huh?

Rails actions typically go something like this

# from todo.rb

def show
@object = Todo.find(params[:id])
respond_to do |format|
format.html { render "show" } # being explicit here, render not really required
format.json {@object.to_json}
end
end

So, if the format.html gets called, how do you know if you should show the whole page, or just some partial page update (for example, to update the Todo item on the screen for some AJAX effect or another?

The solution

The solution was – for the Rails 2 project – to use a custom MIME type to identify when we wanted a snippet of HTML. Our AJAX requests looked like:


$.ajax({
url: "/todos/" + id,
beforeSend: function(xhr){ xhr.setRequestHeader("Accept", "text/html-partial") },
success: function(){....}

That Accept parameter set up the MIME type, and we added it to our config/initializers/mime_types.rb file and we were ready to rock

It’s not so simple in Rails 3

Setting up the MIME type

Rails, until about October 2010, didn’t respect MIME types that well. I believe that it would take this MIME type and return text/html

Rails 3 takes the MIME type, and returns it. So even if you got the rest of the example up, your browser would complain because it doesn’t know what to do with a text/html-partial MIME type.

So, instead of a MIME type, it’s best to use a format parameter, for Rails 3

Set up your config/initializers/mime_types.rb file to contain:


Mime::Type.register_alias "text/html", :partial

This is essentially just an alias for another MIME type (text/html).

So, we’re going to use a format

Because we don’t want the user to see “Unknown MIME type, save or open?” in their browsers, we are going to cheat a little bit and specify our format via an extension.

Our Javascript code should now look like


$.ajax({
url: "/todos/" + id + ".partial",
success: function(){....}

Your actions, knowing about full page HTML, and AJAX HTML

Now you need to set up your actions

# from todo.rb

def show
@object = Todo.find(params[:id])
respond_to do |format|
format.html { render "show" } # being explicit here, render not really required
format.partial { render "show", :layout => false }
format.json {@object.to_json}
end
end

Except we have a problem. If you actually try this out, you will get a Rails Missing Template error

Rails 3 takes the MIME type into consideration when constructing the template path/name to render.

format.html { render "show" }

— Rails looks for todo/show.html

format.partial { render "show", :layout => false }

— Rails looks for todo/show.partial

Now, ideally we want to share as much HTML as possible, so we have a problem

In my use case today, I was actually rendering a Rails partial – a partial that was also used in non Ajax situations. So renaming my file to be _something.partial.erb just wasn’t going to cut it

Actually, this behavior does introduce an interesting side effect: being able to further isolate snippets returned for AJAX vs full page reload requests, if the situation demands it.

But my situation demanded sharing. I’m sure separating things out will work for some requests, and it’s a handy thing to have… but most of the time I want Rails to read from the blah.html.erb file.

Setting up a Rails 3 ActionView::FileSystemResolver

So I decided to go monkey patching Rails, to provide the support I wanted.

Before I banged my head on my desk too hard, I found the following article on using FileSystemResolver for some other things:

Implementing A Rails 3 View Resolver

My resolver is very similar to that one:

Conclusion

  • We have a way to separate full page HTML requests from AJAX requests that only want a specific section of a page
  • We use the less magical extensions to provide formatting, instead of MIME types
  • We can provide AJAX specific templates if we want (blah.partial.erb
  • We fall back to .html if nothing specific exists, because we really want to share HTML code between a full page redraw and an AJAX redraw

April 28, 2011

CI Server for Rails Projects Poll Results

Filed under: ResearchAndDevelopment — Ryan Wilcox @ 6:56 pm

Today I did a poll, using PollEverywhere to get a feeling of the Ruby on Rails communities favorite CI server

PollEverywhere limits responses to 30 for the free version, and 50 for one pay level up. I spend the little bit of money to get the additional responses

And here they are:

Jenkins wins by a wide margin (50% of the participants).

Looks like I will be spending time with Continuous Cooking: a Vagrant project for Hudson, although Bamboo looks appealing.

April 27, 2011

Setting up Passenger Standalone, RVM and launchd on OS X

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 9:22 am

Installing Radiant with Passenger (Standalone), RVM and Launchd on OS X

Introduction – why Passenger Standalone

I like to deploy all my basic sites on the same host. For example, deploying a Radiant site along with a few small Rails sites on the same box. This keeps things easy for me, and no sense devoting a lot of resource if I know the site will be fairly small.

However, this poses some deployment challenges.

I want to use Passenger, like all the cool kids are doing. Just set up some Apache things and I’m ready to go.

I also always use RVM. It is so amazing being able to keep project gems separate from each other. This is especially important when you have multiple projects on the same machine.

This sounds like a dream team: use Apache/Passenger to host multiple Rails apps, with separate gemsets, end of story.

Except, the real world doesn’t work like that.

First, your rvm and Passenger must share a common Ruby. If they can’t, you must use Apache Reverse Proxy cleverness with other servers (like Passenger Standalone).

The easy way to avoid this mess is to make sure they match. We’re all using REE anyway, amiright?

Actually, usually not. If you’re on a shared host, or host multiple Ruby sites from one box, you don’t want to make everyone on that site use REE… even if it’s the best.

For example, what happens in a year when you’re working on a Rails 3 site in Ruby 1.9… you just forced yourself into yesterday’s version.

It is tempting to set up Passenger to use one Ruby (REE) and point Passenger to your gemsets. I really suggest you go to the extra work and set up this system instead

Setting up Apache as a Proxy

So, we want to use Apache as a front-end, and use reverse proxies to run all these sites on their own Passenger standalone servers.

In your Apache configuration (in /etc/apache2/sites, in a file that increments the leading count):


<VirtualHost: *:80>
  ServerName cms.example.com:80

  ProxyPass / localhost:3000/
  ProxyPassReverse / localhost:3000/
</VirtualHost: *:80>

Note: that trailing slash is important

If you have your Rails/Radiant site on the server, at this point you should be able to fire up Passenger (passenger start in your Radiant folder), visit your server name, and have it redirected.

This Passenger instance will run as long as you are connected, or as long as the machine stays up (whichever ends first). But that won’t do for a production webserver…

Making Your Passenger Get Up, And Stay Up (OS X)

Figuring out what command to give launchd

This goal is not as obvious as it might seem. We want a Passenger Standalone instance to be up all the time, including after machine restarts. So, how do we best do this?

In the past, for demonstration purposes, I’ve SSHed into a server, fired up GNU Screen, ran passenger, then detected the screen. GNU Screen will keep the SSH socket open (from the server’s perspective, anyway), and thus Passenger. This solution works, but it’s a hack.

Ideally I want to keep Passenger up without this hack, or other similar “poll based” monitoring methods.

Launchd to the rescue.

Here’s how to get that up:

  1. Create a rvm wrapper:

    $ rvm wrapper RUBY@GEMLIST radiant_site

    Will create ~/.rvm/bin/radiant_site_ruby

    So if you’re running ree with your gemset name “radiant_site”

    $ rvm wrapper ree@radiant_site radiant_site

    Which creates a Ruby named radiant_site_ruby.

  2. Find out where your passenger is:

    which passenger

Your command to start up your Rails site is an amalgamation of these three things and the path to your Rails root:

/Users/aias/.rvm/bin/radiant_site_ruby /Users/aias/.rvm/gems/ruby-1.8.7-p299@radiant_site/bin/passenger start PATH TO SITE -e production

Try this command on the command line before pouring it into a launchd script.

Pouring it into Launchd

Launchd has two interesting features which we’ll use here:

  1. KeepAlive: Launchd will make sure your progress stays up, restarting if neccisary
  2. UserName: run the command as a particular user

My Launchd file is here:

Save your version of this file to /Library/LaunchDaemons/ – you want this to launch even if your user is not logged in.

Load the file with: sudo launchctl load /Library/LaunchDaemons/FILE NAME

Launchd Debugging Notes

  1. launchd scripts must be owned by root:wheel. (Lingon does not save files this way)
  2. Use StandardErrorPath and StandardOutPath to debug these. Note that the files must exist first, and be writable by root. These are amazingly useful files to have to see what your job is going.
  3. Seriously, you do not want the -d flag for Passenger. (It makes the whole thing not work – launchd has rules about what you can and can’t do in KeepAlive/LaunchDaemon mode)

But what about Passenger Standalone on Linux?

I haven’t tried either of these out, but they look like they would fit the bill.

April 9, 2011

Installing Passenger Standalone — and solutions

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 12:10 am

Intro: What is Passenger, and why I want to use it

I seriously want to love Passenger Standalone, I really do. Certain client projects of mine could use the increased speed, and I’d love to use Passenger Standalone for production deploys too (more on that in a future article).

But it seems like I’m running into every problem under the sun installing this thing. Here are my solutions to these problems.

Problems & Solutions

I ran passenger start and it downloads and compiles things, then fails with sh: line 0: cd: ext/libev/: No such file or directory

Thanks to this issue on the Passenger Google Code site I found out my CDPATH needed to contain “.”.

Solution:

$ export CDPATH=.:$CDPATH and redo passenger start

I ran passenger start, it downloads and compiled everything, now I get: the following error during startup:
Unable to start the Phusion Passenger logging agent because its executable
(....passenger/standalone/3.0.6...-macosx-10.6/support/agents/PassengerLoggingAgent) doesn't exist

Thanks to this Google Group conversation the best way to deal with this issue is to:

  1. $ passenger package-runtime. This will create a passenger-standalone folder in your current directory, with a folder (3.0.6…macosx-10.6). This folder will have a support.tar.gz and a nginx-0.8.5.4.tar.gz file.
  2. Note where it says the file is missing from (standalone/3.0.6…-macosx-10.6, in the case above), and go ~/.passenger/(that directory). Trash the support folder that’s there, and extract the support.tar.gz file (from step 1) in this directory

Conclusion

That took a lot more debugging than it should have (or, to be fair, than it has in the past). But I eventually got Passenger Standalone running. Now that yack is shaved, I hope this blog entry helps others with similar issues

February 20, 2011

Introducing Scrum Status

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 3:07 pm

One of my projects has a problem: while we all try to be in the office for scrum, sometimes we can’t all be. Maybe one person is in the car and can’t talk, maybe another person is on the subway… maybe another person has the day off (but the team needs to know their status).

So I started working on a website to take care of these things, and used this team as guinea pigs. Today I announced this site to the world: Scrum Status

Now we have a centralized place to see the entire team’s statuses, easily. Likewise, it’s easy to make an entry for the day while on the go – scrum status works pretty well on iPhone (even though it hasn’t been specifically tweaked for it).

In the spirit of startup launches I’m launching early, to get feedback. Early testing with my team has ironed out the critical bugs, but now I’m announcing this to the world.

Scrum Status runs on Google App Engine, so I don’t have to worry about scaling issues: 1 person could hit the site, or 5,000 – Google App Engine should be able to handle it.

Interested in technical details? The entire project is open source, and the README has a lot of information about the tech involved: Scrum Status on Bitbucket.

July 5, 2010

Django app/sample

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 11:56 pm

So I decided to switch things around and write a sample application in Django. I started a simple project management / bug tracker / projects-have-many-todo-items app.

Hopefully in the near future I can spend some more time on this example app, for example actually finishing up more than just the project level screens :)

Find it at (in a Mercurial repository on bitbucket:) wd_project_dashboard.

« Previous PageNext Page »