July 30, 2012

Testing URLs in Django (like Rails route testing)

Filed under: General Information — Ryan Wilcox @ 9:10 pm

I’m doing more Django work and find myself contrasting how Rails does things and how things are done in Django.

Routing is one of those things.

Both Django and Rails want you to use their systems to dynamically create URLs to other places on the site, instead of hard-coding the path in the href part of the a tag. This makes life easier both now and in the future.

In Django routes are configured manually through matching regular expressions to view functions. In Rails routing happens automatically (by convention) by a domain specific language and suffixing and prefixing various parts of the object and call graph together.

Rails has this interesting feature called route testing. The idea being that you’re testing the rest of your application, you should make sure that Rails is handling your URL paths the way you expect them to.

Django doesn’t have a testing best practice for this, and this article attempts to create one.

First, let’s see what URL paths we have defined

The first time I played with Django I was confused. In Rails I’m used to running rake routes and getting a list of my routes and the URL paths they might match. I couldn’t find such a tool for Django at the time.

Now the Django community has the django-extensions app. Django-Extensions adds new commands to manage.py, one of which is show_urls.

Let’s see part of show_urls in action, for a simple Django app:


$ python manage.py show_urls

/admin/logout/ django.contrib.admin.sites.logout logout
/blogs home.views.blog_list home.views.blog_list
/blogs/<slug>/ home.views.blogs_show home.views.blogs_show

I’m only showing you the most interesting parts of show_urls, but yes I have the Django admin turned on and I have a blog app.

Next, let’s test against those URLs

The slightly annoying thing about Django is that since you’re building up your URLs by configuring regular expressions (which, by the way, are order specific as Django goes with the first expression found)… the match is dependent on the data fed into the path.

In our case we have a /blogs/SLUG route. But perhaps your regular expression forgets something (like perhaps it doesn’t handle URL escaped text, which your slug might be made up of). /blog/today+was+a+good+day should match the home.views.blogs_show route just the same as /blog/todaywasa

This seems like the thing automated testing was made for – making sure that a simple test URL path goes to the view we want, and testing a more complicated match, and testing that Django doesn’t accidentally pick the “wrong” view because us failable humans screwed up some regex or placement.

So, you want me to make a ton more client requests?!!!

We want to do this quickly – we don’t want to build up huge test cases to test obscure URL path names. Thankfully Django provides the tools we need to test our paths:

from django.core.urlresolvers import reverse, resolve

So, no – “just add URL related tests to your existing tests” is not the best answer here

Requirements for URL testing in Django

Let’s think about how we want to test URLs and their patterns:

  1. We want to have a hard coded URL path: as if a browser or a user had typed it in
  2. We, as humans, know which URL pattern name we expect that to match to
  3. We know what (keyword) arguments should be extracted from the URL string
  4. It has to be super fast – ideally without having to instantiate test data or make a single request to the Django application server.

We also know we want to test this backwards and forewards: first taking the URL path and seeing if we get our URL pattern name out, then trying to construct our URL (with Django’s automatic URL creation tools) and seeing if we get our hard coded URL path out again.

Defining an API

Let’s imagine for a minute and create a test:


class TestURLs(TestCase):
    def test_blog_routes(self):
        routes_to_test = (
            dict(url_path = "/blogs"pattern_name="home.views.blog_list"),

            dict(url_path="/blogs/my+wonderful+blog"pattern_name="home.views.blogs_show"kwargs={"slug""my+wonderful+blog"}),
            dict(url_path="/places/my%20wonderful+blog"pattern_name="home.views.blogs_show"kwargs={"slug""my%20wonderful+blog"}),

            dict(url_path="/blogs/my+wonderful+blog/"pattern_name="home.views.blogs_show"kwargs={"slug""my+wonderful+blog/"})
        )

        for stringOnestringTwo in test_paths(routes_to_test):
            self.assertEqual(stringOnestringTwo)

Here we have a list of routes to test and the attributes of each route: the url_path (what we would type into a browser address bar), the pattern_name (the name of the pattern / the pattern name we would use when creating our model’s get_absolute_url method, and lastly the kwargs we expect to be passed into our view by Django.

Implementing test_paths

test_paths ends up being quite simple – simple enough to put in a helper library!


from django.core.urlresolvers import reverseresolve

def test_paths(routes_to_test):
    for route in routes_to_test:
        path    = route["url_path"]
        pattern = route["pattern_name"]
        kwparams = route.get("kwargs")

        if kwparams:
            yield reverse(patternkwargs=kwparams), path
        else:
            yield reverse(pattern), path

        yield resolve(path).url_namepattern

Conclusion

Testing URLs in Django apps is simple with test_path!

July 14, 2012

Rails 3.2 attr_accessible, RailsAdmin, and “accessible by admins”

Filed under: ResearchAndDevelopment — Ryan Wilcox @ 10:59 pm

First of all, my blog is now back up! Instead of self hosting my WordPress blog, now the fine people at ZippyKid host it. These guys are awesome: taking a mess of an import and making it Just Work. A+++ WOULD DO AGAIN

Now, back to real news…

The Problem: Security in Rails: Say hello to the secure boss (different from the old boss)

Because of some high profile Rails sites being hacked, Rails 3.2 changed the default Rails model behavior to “only let users (and developers) update attributes in this list”. This list varies by model.

A refresher into the hack

Rails has this clever feature where you can say, “update this record with the form data passed in”. A semi-clever hacker could use this ability to change fields that the Rails developer didn’t intend to be changed (“shove this value into the POST params, even though there’s no field named that on the HTML for this form”)

What Rails 3.2 did about it

Before Rails 3.2 You used to have models that look like:

class User < ActiveRecord::Base

end

In Rails 3.2, now you have models that look like:


class User < ActiveRecord::Base
# Setup accessible (or protected) attributes for your model

attr_accessible :name, :email, :password, :password_confirmation
end

The attr_accessible block says, “hey, that ‘update this record with form data passed in’ feature? That’s only allowed to touch these fields”.

In this example the name, email, password, and password_confirmation, but is NOT allowed to edit anything else. Perhaps you store access keys, or middle name in the User model. You have to explicitly change these values, and not use the ‘update this record with form data’ shortcut.

Enough background: You said something about RailsAdmin?

RailsAdmin is a clever piece of software that automates creating an admin interface for your Rails site. You have simple access to create, read update or delete records in your site

… but how does it play with that attr_accessible thing?

Glad you asked

Normally it works very well. You can see all the fields in your model, and if the attribute is not attr_accessible, then RailsAdmin will display the value as read-only.

Read-only you say? But I have values I want admins to be able to edit, but I only want admins (not everyone) to edit them. How do I?

Returning to the access_keys example, you want admins to be able to edit this value in RailsAdmin. You don’t want to make that attr_accessible because then anyone can edit that setting (introducing a security hole).

The solution: attr_accessible + as (a user)

attr_accessible has an oft-forgot as parameter. This allows you say, “this is allowed, only if I I’m doing this as a ______________ user

Using this feature you can declare models like


class User < ActiveRecord::Base
# Setup accessible (or protected) attributes for your model

attr_accessible :name, :email, :password,
:password_confirmation, :as => [:default, :admin]

attr_accessible :access_key, as: :admin
end

access_key will only be changable when you’re doing something as the admin role, and the other attributes will be enable both for the default role and the admin role

Configuring RailsAdmin to use the admin role

In your config/initializers/rails_admin.rb file, add the following line in the RailsAdmin.config do |config| block


config.attr_accessible_role { :admin }

Conclusion

And that’s all there is to it: use as: :admin, and configure RailsAdmin to post things “as an admin”. Good to go!

March 10, 2012

A Rails Development Environment with Puppet

Filed under: Uncategorized — Ryan Wilcox @ 4:45 pm

Introduction

I really enjoy using Vagrant to do my development. I’ve even posted my Base Vagrant Package.

But I found I was still doing the same things over and over again: setting up postgres, and setting up the RVM. Yes, I had automated 80% of my “get a new Rails box up”, but that extra 20% eluded me.

Today I said, “No more”

I’ve improved on my package – now we set up postgres and RVM.

Now, my strategy

My strategy for new virtual machines is now

$ git clone git://github.com/rwilcox/vagrant_base.git rpw_NEWPROJECT
$ vi Vagrantfile # tweak project paths etc
$ vi manifests/lucid32. # search for PROJECT_NAME, start adding packages and editing the RVM name
$ vagrant up

(go get a sandwhich, because the first vagrant up will take 15 minutes)

Take a look

The best place to look at what I do is to dig into the manifest file.

I believe this provides one of the few examples on the net on how to do all these things together.

Notes

The RVM puppet manifests require Puppet 2.7.11, released Feb 23, 2012. You may need to update the puppet in your base boxes (sudo gem install puppet), then package them up again.

It’s important to remember that Puppet is a great tool for machine setup, but Capistrano (or your favorite deploy tool flavor) is better for deploying actual code.

Capistrano as a machine setup tool isn’t all that great. Even a simple task can get caught up in headbashing, as I found out when trying to set up RVM and gemsets. It’s so much easier when you’re using a tool (Puppet) that’s meant for those things.

Likewise, I wouldn’t want to use Puppet to deploy, bundle install, etc etc. the Capistrano community does all that stuff really well, thank you.

How to make your Rails deployments better with Puppet

Vagrant and virtual machines are great for development environments, but I’m also a big fan of reusing everything I can. Puppet was really meant to automate managing of servers… so why don’t we use the same Puppet files in managing our servers?

You can! Use the puppet modules and scripts in my vagrant base and Supply Drop.

If you’re using Vagrant to test this locally, do NOT set a provisioner – supply_drop will take care of that for us. (Normally you’d want a provisioner, because you don’t cap deploy:dev, but this is a simulated deploy test).

Your cap deploy:setup should call cap puppet:bootstrap: (appropriate OS here), in addition to creating a folder for your Git repo to land. Nothing else.

cap deploy should call cap puppet:apply. (You may want to create a cap deploy:pre task, which checks the syntax of the puppet files as well as doing anything else you need).

The thing to watch out for here is that your Puppet file must be named puppet.pp in current versions of Supply Drop.

Using Puppet both insures your development environments are consistent with your production environment, AND uses a tool that’s declarative instead of procudural (avoiding Capfile hell).

Supply Drop needs some improvements (for example, different machine roles in Capistrano should be able to have different puppet configs), but it’s a tool worth keeping an eye on.

Next Episode: A local QA stack with Vagrant

Over the upcoming weeks I hope to write about Puppet, Vagrant and Supply Drop to bring up a local QA stack for your application stack. Your application stack probably has multiple machines (web front end, application server, database server), and you can replicate the entire stack with Vagrant. For now, there’s an excellent starter article: Bootstrapping Vagrant with Puppet.

February 24, 2012

My Standard Estimating (and project work) Workflow

Filed under: Uncategorized — Ryan Wilcox @ 4:45 pm

Today I decided to record what my workflow is when I get a new potential project.

Maybe you’re a new client and what to see behind the scenes what goes on when you say, “Ryan, can you do this project for me?”. Especially if this is your first time working with me

For most of my client projects I use a tool (being released today, I believe) called Projector PM.

In the estimation stage I use Projector PM to create a list of behaviors (features, essentially) I see in the app. I’ll send you that list of behaviors and you can review them.

In the implementation stage I’ll invite you to projector PM. Here you can see some graphs: progress of the app vs allocated budget remaining. You’ll also get emails about what I worked on.

I ended up making two videos:

January 19, 2012

Whitepaper on early stage startup advice / “So, you wanna do a startup”

Filed under: Uncategorized — Ryan Wilcox @ 1:53 pm

Today I ended up writing a white-paper in which I collect links for early startup people.

Particularly, early, non-technical, startup people.

My standard operating procedure, next time I hear a pitch about some startup needing a technical co-founder for equity, is to send them to this whitepaper.

I’ve helped a lot of startups and small businesses launch products – sometimes even minimally viable products that only might cost a few thousand dollars in development time. I’d love to do even more of that in the future.

However, the current startup environment seems to contain a lot of people who think, “I’ll get some coder who will gladly develop my killer idea, for a cut of the millions of dollars we’ll make! Who could resist that!!!”

This whitepaper, I hope, serves an an education device: doing a startup doesn’t mean brainstorming all day (brainstorming about the app before noon, then taking a long lunch and brainstorming about which small island you should buy with all your startup IPO money until bed-time).

Startups, sadly, take actual work.

So, I present: Early Stage Startups: Advice for founding a startup

December 21, 2011

Using Fabric to import your Django models

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 9:49 pm

A client wants me to write an import task for their Django app. They already use Fabric to deploy their site, so I figured that writing this script as a Fabfile would work out well.

The script requires me to import classes from their Django app. Specifically, I’m doing queries against their domain models, and adding things to the database, and I’d like to reuse the Django ORM classes already defined.

I finally got it working, but it was non-obvious. Here’s how I did it.

Setting up the module import path

In this project all the fabfiles go in a folder named “fab” in the Django project’s directory. So, I need to tell Python to look outside the fab folder for what it is trying to import.

But that’s not enough – I also need to import the Django project by name, so I need to go one more folder out (to the parent folder of the Django project).

Actually importing the Django models

the django.project nonsense is to set up an environmental variable (pointing to the settings.py file) – Django requires this variable to be set, and will error without it.

Next we import the Django project, and start using entities from an app inside it.

Easy, Huh?

October 29, 2011

Whitepaper on node.js

Filed under: Uncategorized — Ryan Wilcox @ 9:02 am

I spent some time last night giving node.js a serious looking at. I was looking for best practices from the node community, and didn’t find any resources there.

So I dug and wrote my own.

Initially this was for internal research, but it seemed too good to keep secret.

So, I present: Node.js: research, analysis, and best practices

October 19, 2011

Announcing: delegate_presenter: the simplest Presenter Gem that could possibly work

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 8:19 pm

Lately there’s been a lot of buzz in the Rails community about using the Presenter Pattern to organize common view related code outside of the model.

Think of Presenters like Helpers: The Next Generation

I used Presenters on one project with great success. Today I was about to add presenters to a second project, when I said:

Self, you could copy and paste all this code from Project X to Project Y, or you could extract it into a gem

So, that’s what I did: introducing the delegate_presenter gem

Read more documentation on the Github page

September 21, 2011

Posting a Gist from the OS X Services menu

Filed under: General Information — Ryan Wilcox @ 2:15 pm

Today I wanted to post a Gist from the OS X Services menu.

Because I love Services. I use Services probably a dozen times a day. Call me crazy.

I found a Gist service, but it’s broken. It didn’t work for me, don’t really know why. Tried to write my own (using Ruby’s TempFile), but that didn’t work either. (Nothing was written in the temp file. I have no idea why

Then I noticed I could write my Automator Service with one line of shell code:
open `gist`

This takes advantage of the “Pass Input to STDIN” setting in your automator action.

Want to make your own? See my gist documenting it

August 5, 2011

Capistrano, system wide RVM and creating gemsets as part of deploy:setup

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 11:29 am

Introduction to the Problem

Capistrano is the standard way to deploy Rails apps. Yesterday I was using Capistrano to deploy to a machine where I had installed RVM (Ruby Version Manager) at the system level.

I manually set up Ruby 1.8.7 and Ruby 1.9.2, because I need to run two applications on that machine (one a Ruby 1.8.7 app and one a Ruby 1.9.2 app). Using RVM for production deploys is great for this.

My cap deploy:setup task, however, complained that Ruby 1.9.2 wasn’t installed on the machine.

That’s funny, because I did install it, I thought. After banging my head up against the problem for a few hours, I finally posted the question to Stackoverflow.com: (Capistrano deploying to system wide RVM not seeing installed Rubies)

I got my answer: the message about the Ruby not being installed was misleading, it actually meant that the gemset wasn’t installed. Which it wasn’t (I was planning on doing that as part of the cap deploy:setup task.

Creating gemsets in your deploy:setup step

Ideally I want cap delpoy:setup to take care of eveything for me: installing some rubies, creating the appropriate gemsets, you name it. Because automated deployments mean everything should be automated (amirite?).

But then I get errors like this when I’m trying to create the gemset I want to use!

It’s non-obvious how to do this – and in fact the obvious way will not work!

Background

You see, require 'rvm/capistrano' hooks into the low levels of Capistrano’s run function, meaning everything happens in the context of the ruby+gemset that you declared in your Capfile. (Technically rvm/capistrano uses a user shell called rvm-shell, instead of bash or sh. This shell knows enough to properly set your paths to Ruby etc etc.

Normally this is awesome – that means that Capistrano knows about your Gemset, and installs gems there etc etc. Capistrano’s run command just does the right thing.

However, there are two cases where you want things to happen outside of rvm-shell:

  1. Installing the Ruby
  2. Creating the Gemset

If you try to do these things using run, Capistrano will give an error about Ruby not being installed, like it gave me. Even if RVM is trying to say, “I don’t see that gemset”, the error message will be about a missing Ruby.

The obvious solution, and why it doesn’t work (as a conversation)

The obvious thing you might try in your Capfile is this command:

run "rvm install 1.9.2"

Except, as I explained above, that won’t work. Here’s what’s going on, as a conversation.

You, to Capistrano: Run this command for me

Capistrano, to remote machine: Hey, I want to log into this machine, using the rvm-shell command, using Ruby 1.9.2 and gemset MY_APP. When I’m logged in please execute rvm install 1.9.2

Remote machine, to Capistrano: Could not log you in, an error happened when firing up rvm-shell. I could not find the ruby/gemset you wanted, so I can’t set the Ruby paths appropriately. I’m giving up and stopping because I can’t possibly do whatever that command was that you wanted me to execute

Capistrano, to you: I couldn’t install that Ruby you wanted me to install because I can’t activate that Ruby you want me to use for the gemset you want me to use – I don’t think that Ruby is installed!

You: Le sigh.

The solution: avoid rvm-shell for Ruby installation AND gemset creation

You might think that you need to avoid rvm-shell just for the installation of your Ruby. In fact, you need to avoid rvm-shell for both the installation of Ruby and the creation of your gemset!

How to avoid rvm-shell

Define this method in your Capfile.


def disable_rvm_shell(&block)
old_shell = self[:default_shell]
self[:default_shell] = nil
yield
self[:default_shell] = old_shell
end

Now, in the context of that block, run will execute commands by using sh/bash as a shell, instead of rvm-shell.

In your Capfile, install Ruby and your gemsets by:


disable_rvm_shell do
run "rvm install 1.9.2"
run "rvm use 1.9.2@MY_APP --create"
end

This installation process must come before any other command in your deploy:setup chain.

Conclusion

And that’s that – I really hope this helps someone out there!

« Previous PageNext Page »