February 18, 2018

Sending Heroku logs to CloudWatch Logs

Filed under: ResearchAndDevelopment — Ryan Wilcox @ 10:26 pm

For early stage experiments I like deploying to Heroku. Heroku’s free model will spin down servers that haven’t gotten traffic in a while, and none of my free experiments have enough continuous load to require anything more.

When I set up a Heroku app I usually set up Cloudfront as a CDN and S3 as a blob storage. Heroku is already just a layer over AWS EC2, so these choices are staying in the ecosystem, and easy to set up too

The other week I realized I could set up logging the same way: send to Cloudwatch Logs. I’ve used Cloudwatch Logs for other production applications, and it’s an OK common denominator tool. Cheap, and I can hook up subscription filters to send it to some other microservice for further processing.

So I wrote heroku-cloudwatch-sync: you deploy the app, set up a Heroku drain, and it goes to the specified Cloudwatch Log Group and Log Stream.

This was my first time with AWS Lambda functions, and current state of the world allows pretty rapid prototyping by just editing the file through a web based editor. For more complex things I ended up writing a complex Makefile to take care of packaging and deploying the script, and Cloudformation templates to set up the infrastructure.).

When I do further serverless work I’ll have to check to see if the serverless frameworks have any wins here: I don’t know if they make the deployment story any better or worse than what I figured out.

Anyway, my heroku cloudwatch sync provides an easy and cheap way to get logs from Heroku into an easy and cheap log store!