May 9, 2010

Announcing Email Form Relay

Filed under: Uncategorized — Ryan Wilcox @ 9:35 pm

The other week I created Email Form Relay: a Google App Engine app to make it really easy to implement a “Contact Us” style form.

As web developers we find ourselves coding “Contact Us” forms alot. In addition to having the form to write, we also have the backend logic to deal with: gather up the TO information, from, subject, body and send it off.

But that means configuring sendmail email servers, testing the sending capabilities, and making sure you don’t leave yourself open to spammers. Then making sure that email gateway stays up.

But we’ve made that easy: sign up with your Goggle API, register your site. Get an API key, put that in a hidden field, and point your form action to Email Form Relay’s server. On submit, Email Form Relay will do the email sending and redirect the user to a page of your choosing.

And that’s it: you provide the form, we provide the email sending.

So check it out: I hope it speeds up your contact form creating experience!

April 16, 2010

Turbogears2: Sprox and Formencode forms

Filed under: Turbogears,Uncategorized — Ryan Wilcox @ 9:37 pm

I find myself writing more and more sample projects for Turbogears 2. These sample projects are actually meant to be read by going through the commit history with GitX and reading the commit messages and reading the diffs.

Anyway, I’ve posted another one, on the form technology in Turbogears: Sprox/FormEncode. Clone it from its Github Repository

December 3, 2009

This is a test

Filed under: Uncategorized — Ryan Wilcox @ 11:11 pm

This is a test to make sure I have the blog up and working

March 18, 2009

New Open Source Project: PyMethodMatch

Filed under: Uncategorized — Ryan Wilcox @ 9:35 am

I’ve been doing a lot of Ruby on Rails Cucumber testing, and was inspired by the simplicity of controlling the flow of executing by matching on regular expressions.

The idea behind cucumber is you write English steps and assertions to test a web app. Plain English means just that: no “tell box 1 of window 1” almost-english-but-not-quite, and not nerd English like Rails makes popular “1.days”.

Plain English.

Here’s a (slightly modified) example from something I’ve been working on:


Given a logged in, registered user
And I have a blog post that I can edit
Then I should see "Edit"

This all happens by the parser matching those phrases with a bunch of regular expression code blocks elsewhere in the app, then executing one if one matches. Which is brilliant in its simplicity.

So I wrote this same kind of engine in Python. Just a generic regex matches calling functions, suitable for any purpose (not just test driven development like Cucumber).

I put it on on GitHub. Find it at PyMethodMatch’s home on GitHub

February 18, 2009

Product Launch: Experienced Man’s Guide to Cross-Platform Programming with wxWidgets

Filed under: Uncategorized — Ryan Wilcox @ 10:08 pm

Today I’m proud to announce a product launch: Experienced Man’s Guide To Cross-Platform Programming with wxWidgets.

When I buy a computer book I write notes in it. Sometimes these notes are back and forward references, sometimes I found a bug in something and I write the solution down, sometimes something behaves one way one place and another way another. Maybe time passes, things change, but the book has (now) outdated information.

So I write notes in the margin. When there isn’t any room in the margin I take out half a piece of paper and write my notes there.

Over the course of a few years my books become pretty note-ridden, especially if I was using the book every day.

I bought Cross-Platform GUI Programming with wxWidgets about 3 years ago, and during those 3 years I’ve been using wxWidgets almost non-stop stop clients. I have a lot of notes in this book.

So I copied all my notes from my book and put them in a handy dandy PDF, which I’m now selling as a DRM free PDF.

Since this is a PDF, and not a traditional printed book, I’m selling it like software: Version 1.0 is out today, but there will be upgrades and fixes and revision level releases so on and so forth. If you have knowledge to add, or ideas or comments, please contact me and we’ll see what we can do to make you happy.

So please support this product, and my effort to keep the wxWidgets book up to date, relevant and useful!

February 5, 2009

Running ReviewBoard Virtual Machine by rPath/rBuilder

Filed under: Uncategorized — Ryan Wilcox @ 6:31 pm

Today I got distracted by Review Board, a tool to help companies do code reviews. Because I was too lazy to deal with the dependancies I decided to look for a virtual machine for Review Board, all configured for me.

This was probably a stupid idea in retrospect.

I found a virtual image provided by rPath/rBuilder with (come to find out) no documentation. None at all. Which is annoying because to set up Review Board you need a “superuser” for the web app.

So, in order to set up Review Board on this box you need to do these things:

  1. Log into the machine. root with no password will log you into the shell
  2. rm /srv/reviewboard/data/reviewboard.sqlite
  3. /srv/reviewboard/manage.py syncdb. It’ll ask you if you want to create a superuser. Say yes.
  4. chmod 777 /srv/reviewboard/data/reviewboard.sqlite

Yes, the chmod 777 is a security risk. However, if you were to actually deploy this you’d want to use MySQL or something like that.

The settings files are in the /srv/reviewboard/ directory.

Also: the virtual machine hosts Review Board on port 80.

What is neat about the rPath/rBuilder virtual appliance is that they offer an EC2 virtual machine with Review Board too…

Hope this saves someone some time…

Update: And, having done all that, the machine doesn’t have PySVN on it, nor developer tools (gcc, easy_install) to bootstrap it myself. Arg.

July 27, 2008

Book Recommendations: Geek Leisure

Filed under: Uncategorized — Ryan Wilcox @ 8:52 pm

The release of a new book from one of my favorite authors gave me incentive to write this list.

Techno-geekery Fiction

Because I read so quickly, for me to spend any length of time with a book requires the book to be pretty massive… as in around a thousand pages. These books fit that bill, in a very technological setting (or diving into very technical topics)

Geek “History”

  • The Cuckoo’s Egg: A hippy-ish astronomer turned sys admin/hacker tracks hackers through all kinds of early 1990s systems to… you guessed it… East Germany. All real.
  • Microserfs: Were you paying attention to the programming technologies coming out of Apple and Microsoft in say 1992-1993? (Ie: Do you remember yack about Pink or OpenDoc?) If so this book will bring back a lot of memories… and is just a good story about work, family, and companionship.

Geek Philosophy

  • Things A Computer Scientist Rarely Talks About: A lecture and question and answer format where one of computer science’s great writers talks about his Christianity, his experiences writing a book where he analyzes verse 3:16 of every book in the Bible, etc. Fascinating.

Enjoy!

July 21, 2008

Book Recommendations: Python

Filed under: Uncategorized — Ryan Wilcox @ 10:35 pm

I’m starting something new on this blog – along with my normal content, I’ll give out book recommendations on topics or languages. I’ve got a pretty big bookshelf, and i pick the best of the best and list them here.

So, now on with my first topic: Python!

  • Python In A NutshellA BIG nutshell, a great reference book. The sections on performance profiling and testing are particularly excellent. When one of my alumni went to Hong Kong to do Python work, I gave him my first edition copy of this book.
  • Dive Into PythonThis is online for free at Dive Into Python.org. The book is meant to help experienced programmers ramp up into Python by taking examples and ripping them apart to see how they work
  • Python Cookbook15 minutes after cracking this book open I learned so much about what Python can do, Python idioms, etc. Excellent if you’re coming from another language too and just need to learn the lay of the land

Hope this helps speed people on their Python adventures!

July 9, 2008

Everything you ever wanted to know about debugging backgrounDRb in Rails… before it eats your soul

Filed under: Uncategorized — Ryan Wilcox @ 8:20 pm

Introduction

BackgrounDRb is both a blessing and a curse. First, it allows you to farm off work to other Ruby processes, so that your main process can get back to the work of serving your client (before a proxy timeout). All kinds of things can be done: re-encoding data, doing long queries, importing data. You can query BackgrounDRb to see what the status of an item is: so giving a simple progress indicator.

That’s the promise. The curse is that BackgrounDRb is very hard to debug. Partially due to the fact that it’s like multithreaded programming (technically farmed out to other Ruby processes, presumably via fork)… but multithreaded programming with bad error messages is worse. BackgrounDRb is in this “worse” category.

So here are my tips for avoid having your soul eaten by debugging this beast. Or, at least, let you put up a good fight.

Zeroth up – Information

Go, read these articles. I’ll be waiting.

Don’t expect the RubyDocs to give you much information… there’s no actual documentation there. It just barely beats browsing the source files in your editor… but you may prefer this.

First Up – Know your versions

Install from trunk if you can. The December 2007 build of BackgrounDRb has an issue where you can’t pass more than let’s say 4K of data to or from a worker. I’m serious. Yes, that means register_status too. Now you might want to consider files as an interprocess communication method anyway… but I think BackgrounDRb is one of those projects where you should Trust Head.

Second Up – Architecture

Ok, so this stuff is hard. You want to be able to use regular old Ruby to test things out before you add BackgrounDRb to the mix. My suggestion: do all your work in a totally separate, isolated, and unit-tested class, and use your worker only to set that up and run it. Seriously: the more you can debug in unit tests, normally, the more hair you’ll have.

Third Up – The Pain

There are two ways to approach things at this point: the mock way and the interpreter’s way. I don’t care which you pick.

The Mock Way

Instead of inheriting from BackgrounDRb::MetaWorker, inherit from this sucker….

class FakeyWorker
attr_reader :logger
def initialize
@logger = logger
end
def register_status(data)
@logger.debug("FROM REGISTER_STATUS")
@logger.debug(data)
end

def self.set_worker_name(something)
#ignore
end

def get_status
return @currdata
end
end

Import your worker file into your controller, and call it directly. Call get_status after the call to your worker method, and see what comes back. Since this is (still) happening synchronously, you’ll see exceptions in your browser. Fix errors. Rinse and repeat. Then lose the training wheels and use MiddleMan like you’re supposed to.

The interpreter Way

If that doesn’t appeal to you, (or you’ve done the above but still don’t trust) do the following:

  1. Fire up one console window with script/backgroundrb running
  2. Fire up another console window, and run script/console
  3. Trigger your worker from here. Here’s some example code:

    >>> MiddleMan.new_worker( :worker => :import_data_worker, :job_key => 41258 )
    >>> data = {....}
    >>> MiddleMan.worker(:import_data_worker, 41258).my_worker_method(data)
    >>> MiddleMan.worker(:import_data_worker, 41258).ask_status

If ask_status returns nil, you have a problem. If a traceback shows up in the script/backgroundrb window, you have a problem…. but don’t bother reading the traceback – it’s probably worthless.

If you’re having troubles, break it down: are you 100% sure your worker is not nil? (Answer: No). save the worker to a variable and check it! Also a worker’s worker_info method is nice.

If you did get nil from ask_status, I’d ask: is my_worker_method even getting called? Do something as obvious as possible: I like writing to a file in my_worker_method. No way to miss that, or have it go to the wrong log.

Fourth Up – BackgrounDRb STILL wants your soul

BackgrounDRb really wants to eat your soul. Really. Here are some suggestions:

  • Wrap your entire worker method in begin… rescue and register_status $!.backtrace (or, write it to a file)
  • Logging MiddleMan.all_worker_info.to_yaml in your controller action is brilliant. Verify you have the worker you’re looking for!
  • I’m not joking about checking to make sure MiddleMan.worker returns you a worker.
  • Is your worker method not getting called for “no reason at all”? Are you passing a lot of data to it and using the December 2007 version?
  • I’m assuming you’re copying off a worker and action that works already?
  • Some of these things could be full of crap. If so, let me know.

The End

With any luck this article helped your debugging, and you spent way less time than I did debugging your workers. Yes, BackgrounDRb sucks: the documentation isn’t good, the error messages are horrible, it’s a ton of infrastructure, insanely complicated, and sometimes it Just Doesn’t Want To Work. It’s also a fair tool. For all that, I might not be able to submit patches to help out the project, but I can write this article.

June 21, 2008

Ruby/Rails Performance Links

Filed under: Uncategorized — Ryan Wilcox @ 7:02 am

Update: Yikes, I hit the Post button before I really wanted to! So you’ll get to see more of my revision process than normal. Sorry about that. It also might be incomplete when you read it, as it might take me a few days to finish it.

So I haven’t been posting here – or anywhere – a lot. That’s because I got a full-time contract doing some Ruby On Rails… while also trying to keep 2 other long-term clients happy.

Anyway, enough of that. One of my assignments has been performance optimization on part of the RoR website. A quick search will find you links… or you can read on to find mine.

First off, I was testing just one batch-type action in a specific controller, not anything massive. I bet my links will help you with that too, but some of that advice will be specific too my task, not yours.

First, I separated out the testable parts with the “un-testable” parts. Or rather, the stuff I wanted to test vs the view (which I didn’t want to test). Basically following the performance advice in the Agile Web Development With Rails book: create unit tests to make sure your performance doesn’t go backwards. This is OK advice, and with this model in place I was able to do profiling pretty easily… but there’s a catch.

Advice #1: Don’t trust Benchmark.realtime

I used Benchmark.realtime to test performance. Ok… except I was getting some pretty big differences between runs of a chunk of code – without having touched it! Then I remembered about Python’s timeit module, which says it “avoids common timing traps”, and makes mention of something Tim Peters wrote in the Python Cookbook. The money quote is :

… time.time measures wall-clock time [which I think Ruby’s Time.now does as well]. So, for example, it includes time consumed by the operating system when a burst of network activity demands attention….

So checking Benchmark.realtime once, like the code example in the Rails book does, is naive and probably won’t deliver consistent results. It’ll help if you run the test multiple times and take the mean, for example, or use maybe an even better statistical approach. Want to read more? Read The Ruby Programming Language or a blog post from the authors on profiling (where they suggest this).

But, there’s a catch! (Database Debris)

At the end of a test, Rails (at least to my understanding) rollbacks back the records etc you changed during the test. If your doing N test runs inside a test function cruft will build up if you do database manipulation. If you’re consistently creating record id 1 (or any other column that must be unique), your tests will fail after the first time. Which sucks. I don’t know of a way to rollback the transaction in the middle of the test.

See also: Unit Testing: Leave No Trace vs. Schmutz.

And another one (Timing Ranges, Again)

So when you have your mean values, it’s very (very) possible that you can pick a high threshold and say “it should never take longer than this” (Which is what the Ruby book does). There are two problems with this approach: that a simple assert MAX > ACTUAL only says “false is not equal to true” when it fails. Thanks guys. Secondly, it doesn’t account for variability. Yes, we (should have) gotten rid of some of this via our work above… but what if the computer has a full load while you’re running the entire test? Or if you’re testing on a slower computer than you develop on?

Both these problems can be avoided by using UnitTest’s assert_in_delta(). Let’s say we want our operation to take 3 seconds, give or take 1 second


expected_float = 3.0
delta_float = 1.0
assert_in_delta(expected_float, meantime_float, delta_float)

Apparently, assert_in_delta will pass if (meantime_float – expected_float <= delta_float) (Thanks WikiBooks!)

If this does fail (say with a mean_time_float of 5.0) we get something like:

1) Failure:
test_it(MyTestCase)
[test/my_test_case.rb:156:in `test_it'
/home/ryan/my_test_case.rb:150:in `test_it']:
"3" and
"5" expected to be within
"1" of each other.

Which is way better than “Hey, your test failed! That’s all!”)

Use A Profiler

Know What ActiveRecord Is Doing

Read Other Articles

« Previous PageNext Page »