May 17, 2018

Jenkins, Groovy init scripts and custom Tools

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 10:56 pm

I’ve been working with Jenkins quite a bit lately.

When I set up a system I want it to be as reproducable as possible: you can never trust hardware, especially when it’s virtual.

I found Accenture’s Jenkins Docker configuration . It’s super good, especially as a basis of sample code. Based on this code I was able to install and configure plugins (ie I set up a Node.js Jenkins tool, etc etc).

My Jenkins installation also uses the CustomTool Plugin extensively, to provide CLI tools to my Jenkins pipelines. So I wanted to add my custom tool configuration to my Jenkins init scripts.

There’s plenty of documentation on installing tools based on plugins (even a section of my learning notes!) but the custom tools plugin seems to be left out of this flurry of documentation. No longer!

Installing custom tools is a bit different from installing tools that come explicitly as part of a plugin, and here is some code that worked for me:

import jenkins.model.*
import com.cloudbees.jenkins.plugins.customtools.CustomTool;
import com.synopsys.arc.jenkinsci.plugins.customtools.versions.ToolVersionConfig;


def installs = a.getInstallations()
def found = installs.find { == "gcc"

if ( found ) {
println "gcc is already installed"
} else {
println "installing gcc tool"

def newI = new CustomTool("gcc", "/usr/local/gcc/", null, "bin", null, ToolVersionConfig.DEFAULT, null)
installs += newI
a.setInstallations( (com.cloudbees.jenkins.plugins.customtools.CustomTool[])installs );


March 25, 2018

Gatsby.js: in his house and in my house

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 8:42 am


I’ve been playing with Gatsby.js. Gatsby is a static site render that uses React and GraphQL to make neat sites that a modern 2016+ web developer is gonna love… then render these sites to plain ol’ static HTML.

So I started building something…

My goal: render my reading notes into a website

I’ve kept a wiki for a long time, and have been keeping notes in books for a very long time. Previously I would buy PDF copies of books and highlight and write notes in the PDF book. Since O’Reilly stopped selling PDF books I’ve started keeping notes in markdown files.

So, thus my goal: using gatsby to render my markdown notes into a pretty website.

Challenge One: create a simple site that renders my markdown files

This turns out to be well documented on the Gatsby site. This worked super well – I followed that almost exactly and made something kind of nice.

Challenge accomplished pretty easily!

Challenge Two: “Hmmm… what if I want to embed a Gatsby site into another site?”

I’ve worked on so many Rails apps where I end up writing a simple blog component because the site needed a technical or marketting blog in the same style as the rest of the site.

Likewise, is a statically rendered site, and I if I used the learning / Gatsby site as a replacement for the Wiki then I’d want the styles to match. Which probably means recreating a pretty old design on old tech.

Soo…. what if I could:

  1. Get Gatsby to render my Markdown files
  2. Pull away enough React code to get the markup I care about
  3. Present it on my page

Because my Gatsby pages come from Markdown there’s no fancy React components or liveness on the site: just rendered text.

How Gatsby renders markup pages

Gatsby generates both a rendered React component for the file / path AND also generates the HTML statically.

Given the following structure:


And a path for that document at /learning/gatsby, the rendered path will be curl http://localhost:9000/learnings/gatsby/index.html

Problem solved? I can make an AJAX request for the index.html page, then cleverly insert it into the host page, right?

Inserting Markdown HTML content into a host page: simple insert

Again, I suspect this works because these pages are rendered markdown. No React components on my content pages: I just wanted the rendered HTML.

So I created a sample host page and busted out some jQuery.

In my gatsby template – what the Markdown walkthrough calls src/templates/blogTemplate.js, I gave a class to the returned div. <div className='learning-container'>.

In my host site I wrote the following Javascript function:

function htmlFromGatsby( url, callback ) {

    $.ajax( url, { complete: function( jqXHR, textStatus) {

        var info = $.parseHTML( jqXHR.responseText, document, { keepScripts: false } )
        var whereItIs = $(info).find("div.learning-container") // seek until we find the container in our learningTemplate

Tada! My AJAX function retrieves the statically rendered Gatsby page, breaks all the React stuff Gatsby added, goes into the container that has all my content, pulls it out, then adds it to the host document. Neat!!!

The index page of my notes site has a list of all my learnings. Because its Gatsby I construct that list with GraphQL and a custom React component that abstracts real HTML links.

My first tests started with a simple page of notes content. My next test was the index page: how would linking content work in a host website? And because the React Router takes care of all the page routing / location bar changing… well that’s a problem.

When I added my index page – chalked full of these links – my links just 404ed. OK, time to break out some more jQuery…

In my src/pages/index.html ( my Gatsby site) I added an ID to the div returned: div id="learning_site_index">, and my PostLink component I have create a <Link to={...} className="learning-link">.

Here’s the code I have on my host page:

$.ajax("http://localhost:8000/index.html", {complete: function( jqXHR, textStatus) {
    var info = $.parseHTML( jqXHR.responseText, document, { keepScripts: false } )

    var whereItIs = $(info).find("div#learning_site_index_content")
    $("#destination").html( whereItIs )

                    // gotta do it here, attaching to all, because can't use live events to override an event handled by the control itself. WD-rpw 04-24-2018
                    $("a.learning-link").on("click", function(evt) {
                        var defaultWhere = $("href") 

                        htmlFromGatsby("http://localhost:8000" + defaultWhere + "/index.html", function(outHtml) {
                          $("#destination").html( outHtml )
                        } )
                        return false

There: we display the index page, and clicking a link will fetch and display that content on the page.

Conclusions and Problems

So now I have a simple solution for embedding Gatsby content into other sites. There are some problems:

  1. It works for mostly static pages, like Markdown Gatsby rendered into HTML. Your fancy React components won’t work here.
  2. My sample code is 2006 style jQuery. There’s little (no?) SEO friendliness, the back button is broken, etc etc.

But it’s a great little proof of concept for when you need to make a simple blog on a simple site and want to get something out there quickly.

Wishes for the future

I understand the complexities involved, but I’d love to have a way to use my Gatsby created components as part of an already React-ified site. To be able to plop a <GatsbyRenderComponent> into my site somewhere and load data, or run a GraphQL query about something Gatsby knows about, inside and alongside my custom “host” site components would be super cool.

January 7, 2013

Develop For Good with Open Source (Sandy Disaster Recovery)

Filed under: General Information — Ryan Wilcox @ 10:03 am

A client of mine contacted me the other week. He lives in NYC, and was hit as part of Hurricane Sandy.

The trouble with all the relief efforts is that it’s hard to know what to do. What is your organization doing to help, and are there things you can do to get involved.

He explains it best:

Over the past several months a few developers and I have
created a collaborative work order system for disaster
recovery. We are making the
project open source, and providing it as a gift to the
disaster recovery community, for use in future disasters. The
platform implements a “Craigslist” philosophy to recovery
efforts-organizations that are aware of work orders enter them
into the system, and organizations with capacity to
help can claim and perform the work without any centralized
organization getting in the way. This should minimize
duplication and maximize efficiency.

Interested? He also Created a video

What’s awesome about this project is that it’s open source, on Google Code.

Or read the introductory blog post on the Disaster Recovery Work Order System

If you have some time, and Google App Engine experience, consider jumping in and helping!

July 30, 2012

Testing URLs in Django (like Rails route testing)

Filed under: General Information — Ryan Wilcox @ 9:10 pm

I’m doing more Django work and find myself contrasting how Rails does things and how things are done in Django.

Routing is one of those things.

Both Django and Rails want you to use their systems to dynamically create URLs to other places on the site, instead of hard-coding the path in the href part of the a tag. This makes life easier both now and in the future.

In Django routes are configured manually through matching regular expressions to view functions. In Rails routing happens automatically (by convention) by a domain specific language and suffixing and prefixing various parts of the object and call graph together.

Rails has this interesting feature called route testing. The idea being that you’re testing the rest of your application, you should make sure that Rails is handling your URL paths the way you expect them to.

Django doesn’t have a testing best practice for this, and this article attempts to create one.

First, let’s see what URL paths we have defined

The first time I played with Django I was confused. In Rails I’m used to running rake routes and getting a list of my routes and the URL paths they might match. I couldn’t find such a tool for Django at the time.

Now the Django community has the django-extensions app. Django-Extensions adds new commands to, one of which is show_urls.

Let’s see part of show_urls in action, for a simple Django app:

$ python show_urls

/admin/logout/ django.contrib.admin.sites.logout logout
/blogs home.views.blog_list home.views.blog_list
/blogs/<slug>/ home.views.blogs_show home.views.blogs_show

I’m only showing you the most interesting parts of show_urls, but yes I have the Django admin turned on and I have a blog app.

Next, let’s test against those URLs

The slightly annoying thing about Django is that since you’re building up your URLs by configuring regular expressions (which, by the way, are order specific as Django goes with the first expression found)… the match is dependent on the data fed into the path.

In our case we have a /blogs/SLUG route. But perhaps your regular expression forgets something (like perhaps it doesn’t handle URL escaped text, which your slug might be made up of). /blog/today+was+a+good+day should match the home.views.blogs_show route just the same as /blog/todaywasa

This seems like the thing automated testing was made for – making sure that a simple test URL path goes to the view we want, and testing a more complicated match, and testing that Django doesn’t accidentally pick the “wrong” view because us failable humans screwed up some regex or placement.

So, you want me to make a ton more client requests?!!!

We want to do this quickly – we don’t want to build up huge test cases to test obscure URL path names. Thankfully Django provides the tools we need to test our paths:

from django.core.urlresolvers import reverse, resolve

So, no – “just add URL related tests to your existing tests” is not the best answer here

Requirements for URL testing in Django

Let’s think about how we want to test URLs and their patterns:

  1. We want to have a hard coded URL path: as if a browser or a user had typed it in
  2. We, as humans, know which URL pattern name we expect that to match to
  3. We know what (keyword) arguments should be extracted from the URL string
  4. It has to be super fast – ideally without having to instantiate test data or make a single request to the Django application server.

We also know we want to test this backwards and forewards: first taking the URL path and seeing if we get our URL pattern name out, then trying to construct our URL (with Django’s automatic URL creation tools) and seeing if we get our hard coded URL path out again.

Defining an API

Let’s imagine for a minute and create a test:

class TestURLs(TestCase):
    def test_blog_routes(self):
        routes_to_test = (
            dict(url_path = "/blogs"pattern_name="home.views.blog_list"),



        for stringOnestringTwo in test_paths(routes_to_test):

Here we have a list of routes to test and the attributes of each route: the url_path (what we would type into a browser address bar), the pattern_name (the name of the pattern / the pattern name we would use when creating our model’s get_absolute_url method, and lastly the kwargs we expect to be passed into our view by Django.

Implementing test_paths

test_paths ends up being quite simple – simple enough to put in a helper library!

from django.core.urlresolvers import reverseresolve

def test_paths(routes_to_test):
    for route in routes_to_test:
        path    = route["url_path"]
        pattern = route["pattern_name"]
        kwparams = route.get("kwargs")

        if kwparams:
            yield reverse(patternkwargs=kwparams), path
            yield reverse(pattern), path

        yield resolve(path).url_namepattern


Testing URLs in Django apps is simple with test_path!

December 21, 2011

Using Fabric to import your Django models

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 9:49 pm

A client wants me to write an import task for their Django app. They already use Fabric to deploy their site, so I figured that writing this script as a Fabfile would work out well.

The script requires me to import classes from their Django app. Specifically, I’m doing queries against their domain models, and adding things to the database, and I’d like to reuse the Django ORM classes already defined.

I finally got it working, but it was non-obvious. Here’s how I did it.

Setting up the module import path

In this project all the fabfiles go in a folder named “fab” in the Django project’s directory. So, I need to tell Python to look outside the fab folder for what it is trying to import.

But that’s not enough – I also need to import the Django project by name, so I need to go one more folder out (to the parent folder of the Django project).

Actually importing the Django models

the django.project nonsense is to set up an environmental variable (pointing to the file) – Django requires this variable to be set, and will error without it.

Next we import the Django project, and start using entities from an app inside it.

Easy, Huh?

October 19, 2011

Announcing: delegate_presenter: the simplest Presenter Gem that could possibly work

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 8:19 pm

Lately there’s been a lot of buzz in the Rails community about using the Presenter Pattern to organize common view related code outside of the model.

Think of Presenters like Helpers: The Next Generation

I used Presenters on one project with great success. Today I was about to add presenters to a second project, when I said:

Self, you could copy and paste all this code from Project X to Project Y, or you could extract it into a gem

So, that’s what I did: introducing the delegate_presenter gem

Read more documentation on the Github page

September 21, 2011

Posting a Gist from the OS X Services menu

Filed under: General Information — Ryan Wilcox @ 2:15 pm

Today I wanted to post a Gist from the OS X Services menu.

Because I love Services. I use Services probably a dozen times a day. Call me crazy.

I found a Gist service, but it’s broken. It didn’t work for me, don’t really know why. Tried to write my own (using Ruby’s TempFile), but that didn’t work either. (Nothing was written in the temp file. I have no idea why

Then I noticed I could write my Automator Service with one line of shell code:
open `gist`

This takes advantage of the “Pass Input to STDIN” setting in your automator action.

Want to make your own? See my gist documenting it

August 5, 2011

Capistrano, system wide RVM and creating gemsets as part of deploy:setup

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 11:29 am

Introduction to the Problem

Capistrano is the standard way to deploy Rails apps. Yesterday I was using Capistrano to deploy to a machine where I had installed RVM (Ruby Version Manager) at the system level.

I manually set up Ruby 1.8.7 and Ruby 1.9.2, because I need to run two applications on that machine (one a Ruby 1.8.7 app and one a Ruby 1.9.2 app). Using RVM for production deploys is great for this.

My cap deploy:setup task, however, complained that Ruby 1.9.2 wasn’t installed on the machine.

That’s funny, because I did install it, I thought. After banging my head up against the problem for a few hours, I finally posted the question to (Capistrano deploying to system wide RVM not seeing installed Rubies)

I got my answer: the message about the Ruby not being installed was misleading, it actually meant that the gemset wasn’t installed. Which it wasn’t (I was planning on doing that as part of the cap deploy:setup task.

Creating gemsets in your deploy:setup step

Ideally I want cap delpoy:setup to take care of eveything for me: installing some rubies, creating the appropriate gemsets, you name it. Because automated deployments mean everything should be automated (amirite?).

But then I get errors like this when I’m trying to create the gemset I want to use!

It’s non-obvious how to do this – and in fact the obvious way will not work!


You see, require 'rvm/capistrano' hooks into the low levels of Capistrano’s run function, meaning everything happens in the context of the ruby+gemset that you declared in your Capfile. (Technically rvm/capistrano uses a user shell called rvm-shell, instead of bash or sh. This shell knows enough to properly set your paths to Ruby etc etc.

Normally this is awesome – that means that Capistrano knows about your Gemset, and installs gems there etc etc. Capistrano’s run command just does the right thing.

However, there are two cases where you want things to happen outside of rvm-shell:

  1. Installing the Ruby
  2. Creating the Gemset

If you try to do these things using run, Capistrano will give an error about Ruby not being installed, like it gave me. Even if RVM is trying to say, “I don’t see that gemset”, the error message will be about a missing Ruby.

The obvious solution, and why it doesn’t work (as a conversation)

The obvious thing you might try in your Capfile is this command:

run "rvm install 1.9.2"

Except, as I explained above, that won’t work. Here’s what’s going on, as a conversation.

You, to Capistrano: Run this command for me

Capistrano, to remote machine: Hey, I want to log into this machine, using the rvm-shell command, using Ruby 1.9.2 and gemset MY_APP. When I’m logged in please execute rvm install 1.9.2

Remote machine, to Capistrano: Could not log you in, an error happened when firing up rvm-shell. I could not find the ruby/gemset you wanted, so I can’t set the Ruby paths appropriately. I’m giving up and stopping because I can’t possibly do whatever that command was that you wanted me to execute

Capistrano, to you: I couldn’t install that Ruby you wanted me to install because I can’t activate that Ruby you want me to use for the gemset you want me to use – I don’t think that Ruby is installed!

You: Le sigh.

The solution: avoid rvm-shell for Ruby installation AND gemset creation

You might think that you need to avoid rvm-shell just for the installation of your Ruby. In fact, you need to avoid rvm-shell for both the installation of Ruby and the creation of your gemset!

How to avoid rvm-shell

Define this method in your Capfile.

def disable_rvm_shell(&block)
old_shell = self[:default_shell]
self[:default_shell] = nil
self[:default_shell] = old_shell

Now, in the context of that block, run will execute commands by using sh/bash as a shell, instead of rvm-shell.

In your Capfile, install Ruby and your gemsets by:

disable_rvm_shell do
run "rvm install 1.9.2"
run "rvm use 1.9.2@MY_APP --create"

This installation process must come before any other command in your deploy:setup chain.


And that’s that – I really hope this helps someone out there!

May 12, 2011

Returning HTML content from AJAX requests – a pattern for Rails 3

Filed under: General Information,ResearchAndDevelopment — Ryan Wilcox @ 5:25 pm

The problem: semantic formats for returned HTML for AJAX

In a previous Rails 2 project, we decided that Rails apps return 3+ kinds of content:

  • A complete HTML page, for user viewing
  • JSON (for JS/web API viewing)
  • A partial HTML page, for jQuery DOM swapping/

However, it was hard to know when to return a full HTML page, and when to :layout => false

So we invented a semantic format


Rails actions typically go something like this

# from todo.rb

def show
@object = Todo.find(params[:id])
respond_to do |format|
format.html { render "show" } # being explicit here, render not really required
format.json {@object.to_json}

So, if the format.html gets called, how do you know if you should show the whole page, or just some partial page update (for example, to update the Todo item on the screen for some AJAX effect or another?

The solution

The solution was – for the Rails 2 project – to use a custom MIME type to identify when we wanted a snippet of HTML. Our AJAX requests looked like:

url: "/todos/" + id,
beforeSend: function(xhr){ xhr.setRequestHeader("Accept", "text/html-partial") },
success: function(){....}

That Accept parameter set up the MIME type, and we added it to our config/initializers/mime_types.rb file and we were ready to rock

It’s not so simple in Rails 3

Setting up the MIME type

Rails, until about October 2010, didn’t respect MIME types that well. I believe that it would take this MIME type and return text/html

Rails 3 takes the MIME type, and returns it. So even if you got the rest of the example up, your browser would complain because it doesn’t know what to do with a text/html-partial MIME type.

So, instead of a MIME type, it’s best to use a format parameter, for Rails 3

Set up your config/initializers/mime_types.rb file to contain:

Mime::Type.register_alias "text/html", :partial

This is essentially just an alias for another MIME type (text/html).

So, we’re going to use a format

Because we don’t want the user to see “Unknown MIME type, save or open?” in their browsers, we are going to cheat a little bit and specify our format via an extension.

Our Javascript code should now look like

url: "/todos/" + id + ".partial",
success: function(){....}

Your actions, knowing about full page HTML, and AJAX HTML

Now you need to set up your actions

# from todo.rb

def show
@object = Todo.find(params[:id])
respond_to do |format|
format.html { render "show" } # being explicit here, render not really required
format.partial { render "show", :layout => false }
format.json {@object.to_json}

Except we have a problem. If you actually try this out, you will get a Rails Missing Template error

Rails 3 takes the MIME type into consideration when constructing the template path/name to render.

format.html { render "show" }

— Rails looks for todo/show.html

format.partial { render "show", :layout => false }

— Rails looks for todo/show.partial

Now, ideally we want to share as much HTML as possible, so we have a problem

In my use case today, I was actually rendering a Rails partial – a partial that was also used in non Ajax situations. So renaming my file to be _something.partial.erb just wasn’t going to cut it

Actually, this behavior does introduce an interesting side effect: being able to further isolate snippets returned for AJAX vs full page reload requests, if the situation demands it.

But my situation demanded sharing. I’m sure separating things out will work for some requests, and it’s a handy thing to have… but most of the time I want Rails to read from the blah.html.erb file.

Setting up a Rails 3 ActionView::FileSystemResolver

So I decided to go monkey patching Rails, to provide the support I wanted.

Before I banged my head on my desk too hard, I found the following article on using FileSystemResolver for some other things:

Implementing A Rails 3 View Resolver

My resolver is very similar to that one:


  • We have a way to separate full page HTML requests from AJAX requests that only want a specific section of a page
  • We use the less magical extensions to provide formatting, instead of MIME types
  • We can provide AJAX specific templates if we want (blah.partial.erb
  • We fall back to .html if nothing specific exists, because we really want to share HTML code between a full page redraw and an AJAX redraw

May 6, 2011

Subcontractors Wanted

Filed under: General Information — Ryan Wilcox @ 8:49 pm

Sometimes I find myself with more work than I can handle, or internal work I want done but can’t do myself.

Today, I decided to fix that problem

I’ve created a subcontractor survey on my website. If you do consulting work (mostly around the web sphere: Ruby on Rails, Django, front end design work using CSS frameworks and Javascript.)

How It Works

When I get a new project in I’ll evaluate if I have to pull in a subcontractor, and then look at my list of subcontractors for possible matches.

If I think your skillset and availability match with the needs of the project, I’ll send you an email asking about your availability. Evaluate carefully if you have time, or if it’s not going to work out, and respond.

If things work out, I’ll collaborate with you to pull together a quote for the project, or for a iteration worth of work.


I want to make sure I serve my existing clients my giving them work done to the best of my ability. I also (right now) have more things I’d like to do than I can actually get done.

Bidding things out on (or similar sites) piecemeal doesn’t suit me – I want to make sure I have high quality code going out of my company.

I need a list of subcontractors I can trust, thus this list

Why wait?

The survey takes 5 minutes – complete the subcontractor survey!

Next Page »