Introduction

Containers give us a way to package software and ensure that the software etc we expect installed is, in, fact, installed.

We are back to the days of a single binary to run your application, even though we all know there’s a language runtime, third party dependencies, and half a Unix userspace in that carefully crafted container.

But do we know it’s carefully crafted?

Can we test our containers to make sure? Even if we do the simplest of checks: is this software installed, might it run when it is deployed to an environment?

In fact, we can use a Python module from RedHat, named conu to write tests to know this!

I’ve created a Github repo showing this: rwilcox/testing_containers_with_conu, but this blog article annotates out all the pieces.

Behaviors of Container

It’s often important to know what behaviors we are testing before we write code or tests. In this case I want to provide examples on how to write the following tests:

1. As a tester I should be able to know if Ruby really is installed on this Ruby container
2. As a tester I should be able to know if a microservice will start and return something, anything. A health check! Anything!

I’m not talking about integration tests in Docker containers - although I’ll get close to that in this article. I’m talking simple tests: does this container really have the JRE? Does Ruby run, or segfault on launch because something else isn’t installed?

As a tester I should be able to know if Ruby really is installed on this Ruby container

We all have base image for our applications. One general place to do 90% of the software install all of our microsevices need. Our service Dockerfiles should not be doing something as low level as installing Ruby, you should use a base image for that.

But testing these base images is somewhat hard: all we can do is check for the presence of some binary: there’s really nothing to run, nothing to poke at.

Conu to the rescue

This is actually pretty easy in Conu. From my example on how to do this:

from conu import DockerBackend, DockerRunBuilder
from conu.helpers import get_container_output


def execute_container_testing_cmd():
    with DockerBackend() as backed:
    image = backed.ImageClass("nginix_rpw", "1.0.0")
    cmd = DockerRunBuilder(command = ["which", "nginx"])
    our_container = image.run_via_binary(cmd)

    assert our_container.exit_code() == 0, "command not found"
    print( "******* ngnix was installed in container *******************8" )

This isn’t a great test, but it is a test: we know that nginx is installed. Does it run? We could test that too, but for now knowing that things could work is way better than “build and wish”.

How do we integrate this in CI?

In your testing pipeline create a convention (“Docker tests go in docker/tests/test.py”) and call that python script from your CI pipeline. You could use a Python testing framework if you wanted to (I also layered on the new unit test module’s test discovery features), but Python’s assert statement will return a non-zero exit code, which is enough to trigger build step failure in most test runners.

As a tester I should be able to know if a microservice will start and return something, anything. A health check! Anything!

This is a little bit harder than the first solution. We can’t just check to see if some binary is installed, we want to know if it runs. But a microservice running is sometimes hard: often it requires a database, redis, some secret store, who knows.

So, how can a developer make sure these things exist for test? The answer: a Docker Compose file for Docker build testing!

Docker Compose as a solution for getting a valid “can it run?” test

Docker Compose lets you set containers that depend on other containers, so you can instruct Docker Compose to launch your database container before your microservice container.

Docker Compose is awesome because it does a number of things for us:

1. Gives our Docker container essentially a namespace. (This is based on the name of the containing folder. If you decale a ‘postgres’ container Docker Compose will translate that to my_service_postgres.)
2. Writes up the Docker network linking so that these microservices can talk to each other. They also get their separate Docker network, so they can/will only talk to services in the same compose file (by default).

Outside the fairy-tale of Docker Compose

In the simpliest world, those feature + the docker-compose.yml’s depends_on statement works. More likely your microservice startup looks like:

1. Launch database container for microservice
2. Wait for database to boot up, because it’s that kind of database. I’m looking at you, Cassandra. <--- you could do this with conu too, which would be an improvement over ‘just wait 10 seconds and hope’ I’ve seen everyone do
3. Perform database migration to get the newly launched / blank database with some database structure or seed data
4. Launch microservice container

From a CI/CD perspective, that’s a lot of garbage we can’t standardize.

Establish a convention

From a CI/CD perspective I want to run one thing to get the microservice up and running in Docker Compose, so I can run some simple health checks that may pass.

Let’s call that file docker/tests.sh.

$ ls docker/ docker-compose-tests.yml tests.sh tests.py

And an example contents of tests.py

#!/bin/sh

set -e 

replace_railsapps_image_statement_with $1
docker-compose -f docker-compose-tests.yml start db
sleep 10
docker-compose -f docker-compose-tests.yml run railsapp rake db:setup
docker-compose -f docker-compose-tests.yml start railsapp
python3 tests.py # or call everything above this docker/pre_test.sh

We want to test the container we’ve built, so from a CI/CD perspective we want to pass the tag/label of the image we built into this shell script. The shell script should do something clever with that, with sed or with i-don’t-care.

Now CI/CD just calls docker/tests.sh and is abstracted away from the mess of creating databases or whatever.

Conu for Docker Compose based, running, containers

It took me a couple days of background thought to realize how to get conu to test Docker Compose launched applications. I (eventually!) remembered that Docker Compose containers are just namespaced containers, at least from a docker ps perspective.

Given that we could write some clever conu code to find our container and run a health check against it.

from conu import DockerBackend, DockerRunBuilder
from conu.helpers import get_container_output


def iterate_containers(name):
    with DockerBackend() as backend:
        for current in backend.list_containers():

            # need the name of the container, not the name of the image here,
            # as we may be running containers whose image name is the same (ie on a CI server)
            # BUT Docker Compose namespaces _container_ names
            docker_name = current.get_metadata().name
            if docker_name.find(name) > -1:
                return current


def is_container_running(containerId, containerName):
    with DockerBackend() as backend:
        container = backend.ContainerClass(None, containerId, containerName)

        assert container.is_running(), ("Container found, but is not running (%s)" % containerName)

        with container.http_client(port=8081) as request:
            res = request.get("/health")  # HAHA, http-echo returns what we say EXCEPT for /health. That is special. Thanks(??) Hashicorp. WD-rpw 09-29-2018
            text = res.text.strip()
            assert text == """{"status":"ok"}""", ("Text was %s", text)


docker_compose_namespace = "test_docker_compose_service"
docker_compose_container_name = "%s_sit" % docker_compose_namespace
found = iterate_containers( docker_compose_container_name )

assert found != None, ("No container found for %s", docker_compose_container_name)
is_container_running( found.get_id(), found.get_image_name() )

print("****** TESTS DONE EVERYTHING IS FINE *********************")

Now it’s same as it ever was: let Python’s assert statements return non-zero error codes if something is borked.

Conclusions

We can now test Docker base images for their validity, and launch just enough of a microservice to test how that works. As a microservice may fail early if it can’t find some of it dependencies (connection to database), we want to make sure those are there too: and in fact we can, with Docker Compose!

Conu is pretty awesome software, and that with some conventions gives us a nice CI/CD pipeline to make sure that higher deployment environments get images that at least launch (QA testers get mad when they run into issues because the stupid Docker container didn’t launch!)

Test code, test containers!