Alan Cruikshanks

Docker for development

2017-10-03T00:00:00+00:00

Using Docker for local development is different to actually building images that you expect to be deployed and run as containers elsewhere.

The aim for local development is to simplify creating the necessary environment for running your app.

So for example the main docker tutorial assumes I have Python and PIP installed locally so that I can then grab the project’s dependencies. The Dockerfile focuses on copying the project including dependencies into an image, and specifiying its startup behaviour.

For local development I’m more interested in starting a container that contains everything I need to build and run my app, whilst still being able to make changes to the project, and have that reflected in the running application (assuming you’re building in a dynamic rather than static language).

This post details some simple examples; one for Ruby development, and the other for Node.js.

Docker for Ruby

Here we demonstrate using Docker to provide an environment for developing a basic ruby Sinatra application.

Specifically we have an app with a simple root GET, which returns Hello world as we want to demonstrate a very basic app that

has a dependency and so needs a Gemfile
has something we can interact with from the host
is basic enough that it won’t complicate what we’re trying to show in Docker

The project (ruby)

First create the Gemfile

source 'http://rubygems.org'

# Sinatra is a DSL for quickly creating web applications in Ruby with minimal
# effort
gem 'sinatra'

# Restarts an app when the file system changes
gem 'rerun'

Then create a file called server.js

require 'sinatra'

set :port, 4567
set :bind, '0.0.0.0' # Required to bind to all interfaces

get '/' do
  'Hello world'
end

Finally create the Dockerfile

# Sets the base image
FROM ruby:2.4.1

# Use an env var to hold what will be the main directory within the container.
# As we need to refer to it a few times it can be easier to set an env var
# and then refer to it in subsequent commands
ENV APP_HOME /app

# This will execute any commands in a new layer on top of the current image and
# commit the results. In this case we create the /app directory
RUN mkdir $APP_HOME

# Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD
# instructions that follow it
WORKDIR $APP_HOME

# Copies new files or directories from <src> and adds them to the filesystem of
# the container at the path <dest>
COPY . $APP_HOME

# Here we tell docker to download our projects dependencies in another layer
# that then gets committed
RUN bundle install

# Informs Docker that the container should listen on the specified network port
# at runtime. EXPOSE does not make the ports of the container accessible to the
# host. To do that, you must use still use the `docker run` -p flag
EXPOSE 4567

# There can only be one CMD instruction in a Dockerfile. If you list more than
# one CMD then only the last CMD will take effect.

# Purpose of a CMD is to provide defaults for an executing container. In this
# case it sets the command and arguments to be executed when running the image
CMD ["rerun", "--background", "server.rb"]

If this is a bit too much hassle, feel free to make use of one I made earlier.

Build (ruby)

Now we’ve got our project, we need to create an image from it. Call

docker build -t docker-for-ruby-dev .

Don’t forget the . at the end there! You can call it something other than docker-for-ruby-dev. This is just what I have called it for this example. If we ran it on a machine clean of other Docker images, running docker images would display something like this

REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
docker-for-ruby-dev   latest              5ec4ed806769        3 minutes ago       713MB
ruby                  2.4.1               3630c02d3d1b        3 weeks ago         679MB

Run (ruby)

So we’ve built an image, now we need to create and run a container from it

docker run --rm -v "$(pwd)":/app -p 4567:4567 docker-for-ruby-dev

Breaking down this command we have

docker run first creates a writeable container layer over the specified image, and then starts it using the specified command
--rm automatically removes the container on exit
-v "$(pwd)":/app mounts the local folder to /app in the container
-p 4567:4567 bind port 4567 on the host to 4567 in the container
docker-for-ruby-dev the specified image to create and run a container from

Running in this way ensures we see any output from our app. Add the -d flag if you prefer the container to run as a daemon.

Develop (ruby)

Now we have a container we want to be able to make changes. As we have mounted the project’s root folder to /app in the container we can open it in our preferred editor, make changes, and see them reflected in the app.

To prove this first run curl -XGET http://localhost:4567 from the host. The response should be Hello world.

Edit server.rb and change line 7 to be 'Hello docker world'. Running the same curl command again will result in Hello docker world.

Docker for node.js

Now we’ll demonstrate using Docker to provide an environment for developing a basic Express app.

Again we just want an application with a simple root GET, which returns Hello world, and covers the following

has a dependency and so needs a package.json
has something we can interact with from the host
is basic enough that it won’t complicate what we’re trying to show in Docker

The project (node)

Create the package.json first

{
  "name": "docker-for-node-dev",
  "version": "0.1.0",
  "description": "Example site used to demo Docker for node development",
  "homepage": "https://github.com/Cruikshanks/docker-for-dev",
  "license": "MIT",
  "author": {
    "name": "Alan Cruikshanks",
    "url": "https://github.com/Cruikshanks"
  },
  "private": true,
  "scripts": {
    "start": "nodemon server.js"
  },
  "dependencies": {
    "express": "^4.15.4"
  },
  "devDependencies": {
    "nodemon": "^1.11.0"
  }
}

Next create the app in server.js

const express = require('express')
const app = express()

app.get('/', function (req, res) {
  res.send('Hello world')
})

app.listen(3000, function () {
  console.log('Example app listening on port 3000')
})

Finally we add our Dockerfile. Take note of the change we’ve had to make to allow for where npm wants to install the dependencies.

# Sets the base image
FROM node:8.4.0

# Use an env var to hold what will be the main directory within the container.
# As we need to refer to it a few times it can be easier to set an env var
# and then refer to it in subsequent commands
ENV APP_HOME /app

# This will execute any commands in a new layer on top of the current image and
# commit the results. In this case we create the /app directory
RUN mkdir $APP_HOME

# Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD
# instructions that follow it
WORKDIR $APP_HOME

# Copies new files or directories from <src> and adds them to the filesystem of
# the container at the path <dest>
COPY ./package.json $APP_HOME

# Here we tell docker to download our projects dependencies, then move that
# folder to the root (why explained below) in another layer that then gets
# committed
RUN npm install \
    && mv $APP_HOME/node_modules /node_modules

# ##############################################################################
# Npm unlike Bundler defaults to installing dependencies within the local
# project. That's fine when developing on the host, but as we want to install
# them as part of the image and have them present in the container when we start
# developing we face a problem.
#
# To use the container for development we mount the project folder to /app when
# calling `docker run`. If the container's app folder still held `node_modules`,
# the bind would cause it to become hidden as it doesn't exist on the host. We
# therefore need to move the `node_modules` somewhere else, in this case the
# root.
#
# In doing so we take advantage of
# https://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders
# and the way Node.js traverses the directory tree to locate dependencies.
# ##############################################################################

# Informs Docker that the container should listen on the specified network port
# at runtime. EXPOSE does not make the ports of the container accessible to the
# host. To do that, you must use still use the `docker run` -p flag
EXPOSE 3000

# There can only be one CMD instruction in a Dockerfile. If you list more than
# one CMD then only the last CMD will take effect.

# Purpose of a CMD is to provide defaults for an executing container. In this
# case it sets the command and arguments to be executed when running the image
CMD ["/node_modules/.bin/nodemon", "server.js"]

Or just use one I made earlier.

Build (node)

Next step; create the image.

docker build -t docker-dev-node .

This time if we run docker images we’ll see something like

REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
docker-for-node-dev   latest              c7c5150e2d55        2 minutes ago       677MB
node                  8.4.0               5e553613f1d8        17 hours ago        669MB

Run (node)

With the image built, we now need a container

docker run --rm -v "$(pwd)":/app -w /app -p 3000:3000 docker-for-node-dev

The breakdown is exactly the same as the ruby app, we’ve just changed the port and name we want to use.

Develop (node)

And development works in the same way as the ruby app. Open the code on the host in your preferred IDE, make changes, and see them reflected in the app.

You can test it by running curl -XGET http://localhost:3000 from the host. Edit server.js and change line 5 to be res.send('Hello docker world'). Running the same curl command again should result in Hello docker world.

All done

Using Docker for development will help you and your team avoid it works on my machine issues, and help get new members up and running quickly. The Dockerfile is also great at documenting what you actually need to run the app.

The examples here are very basic though, and its likely you will also depend on other services and tools. But pretty much every other example of using Docker for development immediately launches into using Compose, so I wanted to provide a stripped back example. I hope it helps those like me who just wanted to get to grips with what you need to conisder just getting ‘your’ app up and running.

Finally you will have to get used to calling Docker to start your apps, rather than your typical ruby server.js or npm start. And as you’ve seen the command can be complex so bear in mind how you can keep repeating it.

The Docker blocker

2017-07-28T00:00:00+00:00

When you get started with Docker, you’ll come across bumpff telling you that Docker containers are not virtual machines, and this will be emphasised by pictures showing containers not requiring a guest O/S.

Container		Virtual machine

Images taken from https://www.docker.com/what-container

Nice.

For those of us who don’t need to go deep on a subject before we have a play, you’ll accept this and move on. Next step is typically putting together your first Docker file. And the first statement in a Dockerfile is FROM foo.

The problem is foo, will actually be something like Alpine, Debian, or Ubuntu. So your image will be based on an O/S image.

Huh?

The blocker

I’m guessing others have either done the research first, or just accept this and move on. I can’t explain why, but for some reason this blocked me. I could see the benefits of Docker, but I couldn’t engage with it because of this conundrum.

And having spoken to others, I’m not the only one. It seems to be an incongruity that niggles enough at the back of your mind that you can’t focus on progressing.

So why when the docs talk about containers not needing an O/S, do so many examples include one.

So this post is the result of me learning just enough to answer this question for myself. I’ve put it out there in the hope it can help others.

Batman vs Superman

Or in our case containers versus virtual machines. The first thing I found helpful was thinking of Docker as “an abstraction at the application layer”, and virtual machines as “an abstraction at the hardware layer” (thanks Docker docs!)

A virtual machine will get dedicated resources and run isolated from everything else. The virtual manager and hypervisor will take this isolation down to the metal, and it’s why for all intents and purposes a virtual machine is separate and distinct from the host.

Containers aim to reuse the resources of the host, whilst still keeping things isolated at the app layer. The main way it does this is by sharing the underlying kernel.

A kernel of an idea

Though we have the Linux kernel and the Windows kernel, kernels are more a concept than a specific thing.

By Bobbo - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=4392180

The kernel is the bit of the operating system that connects the application software to the underlying hardware of the machine. Typically it gets loaded into protected memory when the system starts up, and this is known as kernel space. Everything it does happens here, which is taking system calls from processes for access to resources. Everything else like working with documents, applications, the file and window system happens in user space.

The linux family of operating systems use the same linux kernel; what makes them different what they do within the user space.

Back to the bottom

So we’ve gone a little deeper on differences between a container and a virtual machine, and we understand that an operating system is made up of 2 parts; the kernel and everything else!

So if Docker containers get a kernel, but no O/S what do we do for a file structure and system? How do we pull down the binaries our projects depend on when building the image? How do we get system dependencies we depend on?

That’s what FROM ubuntu is doing. These are container specific images built to provide just the user space stuff you need when deploying and running your project. It’s not a guest O/S in the same way as a virtual machine, its just filling in for the bits you probably need but which the kernel doesn’t cover.

Doing it from scratch

To prove a point, you really don’t need that O/S layer in your image. All Docker images derive from the base image called scratch.

You can build an image FROM scratch using a pre-compiled C binary

FROM scratch
ADD my_binary /
CMD ["/my_binary"]

You could theoretically do the same for a Ruby project, if you were willing to manually create the file system, and load all the binaries you depend on, and your gems depend on etc.

So yeah, never gonna happen!

Hence most projects will have a FROM statement that either directly uses an O/S image, or somewhere down the chain will have.

Move along please. Nothing to see here

Docker containers don’t have a guest O/S, if we define an O/S as everything plus the kernel. They also don’t have everything we need to work productively with Docker when creating an image, hence we use parent images to bring in the bits that make our lives easier.

And they’re not the same as virtual machines. So can we move along please?

Local gems for local projects

2017-06-16T00:00:00+00:00

When working if I find the same code is needed in other projects, I’ll shift it to a gem.

This is all well and good, but I’ll sometimes find I want to make changes to project foo, that depend on changes to gem bar. This leaves me with 2 options

Update the reference in the gem file to add the argument path
- Pro I can immediately get on with hacking the solution
- Con I’m liable to forget I made the change and so will push a change that breaks the build and other peoples environments
Make the changes to the gem, push them and then bundle install
- Pro the Gemfile is left untouched, removing the risk of pushing a broken version
- Con I have to commit the changes, and will find that when used they are not right or are incomplete

A third way

In the truest meaning of the “Third way”, Bundler does provide an option that means we don’t have to make changes to the gemfile, whilst still working against a local version of the gem.

Using the Bundler config command you can tell it when looking for gem bar, use my local version.

bundle config --local local.quke /Users/acruikshanks/projects/defra/quke

N.B. The --local flag scopes it to just this project.

Strike 1!

The gem might be referencing a repo rather than rubygems

gem "quke",
  git: "https://github.com/DEFRA/quke",
  branch: "master"

If you only ran the bundle config command you’d be likely to hit an error.

Local override for quke at /Users/acruikshanks/projects/defra/quke is using branch fix/make-the-tea but Gemfile specifies master

Basically the branch specified in the gem file doesn’t match what is currently checked out in your gem project (and why would it!?)

To resolve this run bundle config disable_local_branch_check true to silence the error message.

Strike 2!

So you do this, and think all is well. Then you start making changes to your gem code, but your project doesn’t see them.

You might even run bundle update my_gem to try and resolve things to no avail.

There is a gotcha with using this method. It only sees changes that have been committed. That’s the one benefit that using path has over this method. But if you remember this and make use of some WIP commits that you’ll clean up later, its easy to live with.

Strike 3, you’re outta here!

Not sold? Or simply want to switch back to using the proper source for final testing. You can reverse the change using

bundle config --delete local.pafs_core

Thank you

Thank you to Tony Headford for telling me the right combination of commands and gem file content to use (others had attempted this before but they still seemed to require messing with the gem file).

Also to Thilo Rusche for his post, which confirmed what I was doing and highlighted why my changes couldn’t be seen.

With just a few tags you too can have rich links

2017-05-23T00:00:00+00:00

A key collaboration tool at work is Slack, and for those times when left with idle hands (queuing, kettle boiling, hiding in the toilet from the kids) I’m checking Twitter. What they have in common is when you add a link to your message/tweet they present it in a nice box.

Example from Slack

Example from Twitter

Ignorantly (and ashamedly) I just assumed that each of these apps, and any others that do something like this automatically generate them from the link you provide. So when I came to start creating my own stuff there was nothing I needed to do.

Pity the fool!

Imagine my disappointment when proudly tweeting a link to my first post, I found the link just hung there in its vanilla, naked pale blue ugliness.

Plonker, of course I need to do something at my end!

So just in case you’re as ignorant as me, this post will cover how I added rich preview support to my site.

It’s meta

Props to Richard Oosterhof and his post How to optimize your site for Rich Previews for putting me on the right track.

You basically need to add a bunch of meta tags to the <head> section of your page. Richard’s post talks about the following tags

title
description
og:title
og:description
og:image

He also states your site should have a Favicon (so that was actually the first thing I fixed using RealFaviconGenerator).

So what exactly was the syntax? Well I’d seen people post enough Medium articles like Richard’s to know it worked for them, so I just pinched what I found using ‘view source’.

<meta name="title" content="How to optimize your site for Rich Previews – Richard Oosterhof – Medium" />
<meta name="description" content="You probably have seen them already, when you send a website link in chat apps…" />
<meta property="og:title" content="How to optimize your site for Rich Previews – Richard Oosterhof – Medium" />
<meta property="og:description" content="You probably have seen them already, when you send a website link in chat apps…" />
<meta property="og:image" content="https://cdn-images-1.medium.com/max/1200/1*VDDZ-GgOIHnd4Au4ncYlMg.png" />

All this stuff goes into the <head>, and Richard has even built a handy tool to check you’ve got it setup correctly at richpreview.com.

Thanks Richard!

But I’m different

That’s great if you’re hand cranking each static page, but no good for me, as I’m using Jekyll to generate my pages from Markdown documents.

However the implementation didn’t require major changes. The site uses a default layout for all pages, so the place to put the tags was it.

As my post pages don’t feature unique images I chose instead to go with a single default image I’d use for all pages. With that I could simply hard code the url to the image in the og:image tag.

title and description needed to come from each page though. Jekyll’s concept of front matter already provided title, so next I needed to go back through all my pages and add a new property for description.

So the final bit was to pull title and description from each page. As Jekyll uses the Liquid templating language this just meant inserting {{ page.title }} into the relevant tags. So the final implementation was

<meta name="title" content="{{ page.title }}" />
<meta name="description" content="{{ page.description }}" />
<meta property="og:title" content="{{ page.title }}" />
<meta property="og:description" content="{{ page.description }}" />
<meta property="og:image" content="http://cruikshanks.co.uk/assets/images/default_rich_preview.png" />

That it?

Not quite. I then got curious so Googled what an og: meta tag was and came across http://ogp.me/. Turns out adding these tags turns your web page into a graph object, and that enables it to become a rich object in a social graph.

I’ll leave you to read further on that, suffice to say what I’d added didn’t meet the minimum requirements for Open graph, plus there were some optional tags that might be relevant.

So I expanded the tags to include a few more.

<meta property="og:type" content="website" />
<meta property="og:url" content="http://cruikshanks.co.uk/{{ page.permalink }}" />
<meta property="og:site_name" content="cruikshanks.co.uk" />
<meta property="og:locale" content="en_GB" />

Again you can see how I used Liquid to grab a front matter property to build the url dynamically for each page.

Now are we done?

Almost! Seeing as generally I intended to use Twitter to post about my links I thought it worth while to also do some Googling for rich preview support in Twitter.

And lo, there be documents and a bit more to do. Twitter calls them Summary cards and on initial reading appears to rely on Twitter specific tags, for example <meta name="twitter:title" content="My title" />. However further research turned up Cards Markup Tag Reference.

This showed that where a Twitter meta tag did not exist, it would fall back on Open graph. I also found a handy card validator so you can check that your pages meet the requirements for Twitter to display a summary card.

So the final set of tags I added to my default layout were

<meta name="twitter:site" content="@alancruikshanks" />
<meta name="twitter:image:alt" content="Avatar for Alan Cruikshanks" />

twitter:site is required by Twitter, and twitter:image:alt allows me to provide alt text for my image, something I’ll always take advantage of.

Mic drop

That’s it for me and this site at least. It may well be if I was to post links in other apps I wouldn’t get a preview, but I’ll tackle those as I find them. For now, I’m Open graph compliant and Twitter likes me, so I shall leave it there.

Who me!? (Being a dev lead)

2017-05-15T00:00:00+00:00

A recent Scott Hanselman podcast entitled “road to lead developer” got me thinking about my own trips along that thoroughfare.

I’d like to say it’s always been a well planned journey, ending with some wise person anointing me as lead dev. In reality it’s been more a case of groping around in the dark until I eventually tripped and impaled myself on the anointing sword.

What I have learnt is there is no one sure path, and it will differ depending on your organisation and the people in it. Often you’ll find yourself somewhere such a role isn’t even recognised.

But I think it’s a required one in a successful delivery team.

A definition

Before we go further let’s be clear what I’m talking about. I’m referring to the person who takes point on any technical decisions, not the one who approves your holiday or who sets out your individual performance plan. That may be the same person in some places, but I’m interested in just the techie stuff.

It should also be differentiated from seniority. So being the most senior dev does not automatically make you this. Previously my assumption would have been the person tagged as lead dev (or dev lead, technical lead, grand poohba, whatever your preference is) was the most technically gifted and/or experienced developer in the room.

In practice it’s my experience they’re the person willing to do two things

make decisions
own the decisions they make

Everyone has opinions and many are happy to make them known. A lead dev however is the one setting a direction by being willing to make a decision.

Good ones make them having listened to the people around them. Poor ones either ignore them or just go with the loudest.

Yes, you!

If you are one, ideally you were selected. I can imagine that would help give you confidence in the decisions you make. However you may be like me, and find yourself often taking on the role because you sense the vacuum and it’s impact on your team.

Either way I think my two requirements apply. You must be willing to

make decisions
own the decisions you make

Clearly there’s more to it than just this, and I don’t claim to be a guru on the matter. But unless you have a hyper-performing, super self-organising team i.e. you’re the exception, I believe you need someone to lead, and that leader has to be willing to do those two things.

For reference, it’s probably an arrogance in my abilities to make decisions, or an ignorance of the consequences of them (or a bit of both) which explains why I’m happy to keep impaling myself. But I’m hopeful if I do it enough I will finally do a half decent job of it!

Boxed in dependencies

2017-05-05T00:00:00+00:00

First a clarification. I refer to my Vagrant images as boxes, but you should be aware box has a specific meaning in Vagrant land. I’ll highlight when I’m using the Vagrant version.

Sometime back I learnt of Vagrant and having had a play, immediately got stuck in and now use it daily. Initially I had grand plans to base all my development on Vagrant boxes, but in practise this never materialised.

Version and package managers seem to do all the heavy lifting, so I don’t feel the need to build a box for every project I’ve got going on. I’ve also found a reticence from other dev’s to using them, and like any part of an evolving project Vagrant files need to be fed and watered so there’s an overhead. Plus there’s the elephant in the room, otherwise known as Docker.

Custom boxes

So after listing a bunch of reasons for not using them, how do I use them daily? Well I use them for my general dependencies, databases being the primary example. Rather than install database x directly on my host machine, I instead stick it in a box.

You could create a box specific to each project, but I tend to share mine among them. So for example I’ll stick PostgreSQL in an box, and use it for all my projects that require it.

Big fish, little fish, Vagrant box!

So what steps are involved?

Installation

Obviously start with installing Vagrant. Next you’ll need to decide on your ‘provider’ and install it. This is basically the app used to create and run the VM’s you are going to install your dependencies on.

Vagrant uses VirtualBox as its default, and its what I use as its free and simple.

Initialise

With those two things installed, next create a new folder (named after the thing you intend to install), drop into it and run vagrant init from your command line.

This adds to the folder the files you’ll need to get started with Vagrant, the most important one being Vagrantfile.

As I want to make my boxes reusable, I also tend to run git init and add things like a .gitignore file, a README and a LICENSE, before pushing it to GitHub.

Vagrantfile

In here you’ll find the bare bones for configuring and setting up your box, which by default will be a VirtualBox. The first decision you need to make is what box (the Vagrant kind!) to base your box on. Basically what OS to use. There is a publicly accessible list to pick from, but if it helps I tend to use ubuntu/trusty64, and based on the download numbers so does everybody else.

I’m not going to get into the details, I encourage you to check out the documentation. But in essence the Vagrantfile is divided into two sections; configuration, and provisioning i.e. how to install your dependency.

Getting the install right

When getting started with a new box, I tend to begin with a standard bit of installation

# General provisioning of the box. Ensure time is set correctly and do an
# initial apt-get update, plus install packages commonly needed by all boxes
config.vm.provision "shell", name: "common", inline: <<-SHELL
  timedatectl set-timezone Europe/London
  apt-get update > /dev/null
  apt-get install -y git build-essential tcl8.5 wget curl make libqt4-dev
SHELL

I’ll then run vagrant up, which when complete will give me an empty box. I then access it using vagrant ssh and once on, begin following whatever install process I’ve managed to Google.

I’ll record all the commands I use and update the provisioning details when done. I’ll then exit the box and call

vagrant halt
vagrant destroy
vagrant up

This allows me to test that when I create this box in future I know my provisioning commands will work

Benefits?

I see plenty. I now have a reproducible way to get something like PostgreSQL on the machine I’m using.

Not only do I have a project that will automate the process, its also documents what’s involved.

I only need to run these services when using them. It’s easy to halt a box when its not being used.

I reduce bloat on my dev machine, and the risk of dependencies conflicting.

Examples

I have some existing boxes you can check out, copy from, or use.

Hey browser, these guys are with me

2017-04-27T00:00:00+00:00

This covers something I learnt when, having joined a project late I was asked to pick up some outstanding actions from their recent PEN test. Specifically the test had highlighted that the service was missing a Content Security Policy (CSP).

A what now?

A CSP is delivered via a HTTP response header, and defines approved sources of content that the browser may load. It’s seen as an effective countermeasure to Cross Site Scripting (XSS) attacks, is widely supported and usually easy to deploy.

When your browser loads a page it also loads a bunch of other assets along with it; images, fonts, styles, and JavaScript. It does this because we’ve told it to do so via things like our script tags.

The browser however doesn’t differentiate. So if a hacker, via some content they have contributed (a comment is the classic example), manages to inject a properly formed script tag into the page the browser will load it along with everything else.

So a CSP is designed to prevent this by essentially adopting the idea of a white list. The policy sets out what are the valid sources for the various types of assets your page may load, so the browser can differentiate.

Getting it in there

The project already had code for adding custom headers, specifically to manage caching. This was done within the application controller (oh, and its a Rails project by the way) using a before_action filter to call a method that did something like this.

response.headers["Cache-Control"] = "no-cache, no-store, max-age=0, must-revalidate, private"

So the initial implementation simply added something along the lines off

response
  .headers["Content-Security-Policy"] = "default-src 'self'; "\
                                        "script-src 'self' 'unsafe-inline'; "\
                                        "font-src 'self' data:; "\
                                        "report-uri https://mycustomname.report-uri.io/r/default/csp/enforce"

Specifically we are saying

scripts can only come from the service, however we permit unsafe-inline scripts
fonts can only come from the service, or via the data: scheme (a data: uri is a way to embed small files inline)
send any violation reports to a service we have setup for this CSP
for all other items their source must be the service (default-src)

All done?

Not quite! For reasons we spotted that this didn’t cover assets we were bringing in to support Google Analytics. Also one of those assets was an inline JS script, which our current use of unsafe-inline covered. But this was intended to only be temporary until an issue in a dependency was resolved and we could switch it off. So ideally we wanted to also be able to support nonces. Basically these are one time values generated on the server and added to the script tag, but also into the CSP. The browser can then cross check them, see they match and therefore permit the specified inline-script to run.

At this point the temptation is start rolling your own solution but a quick search found one already existed; enter the Secure Headers gem.

Not only did this come with support for nonces, but it also highlighted a bunch of other headers we really should be thinking of using.

To implement add a file to config/initializers (we called it secure_headers.rb). Then configure it using something like this

require `secure_headers`

SecureHeaders::Configuration.default do |config|
  config.csp = {
    default_src: %w('self'),
    font_src: %w('self' data:),
    img_src: %w('self' www.google-analytics.com),
    object_src: %w('self'),
    script_src: %w('self' 'unsafe-inline' www.googletagmanager.com www.google-analytics.com),
    style_src: %w('self'),
    report_uri: %w(https://mycustomname.report-uri.io/r/default/csp/enforce)
  }
end

So now we’re saying

All external scripts must be sourced from either the service, googletagmanager.com, or google-analytics.com, plus inline scripts are permitted to run
All images must be sourced from either the service or google-analytics.com (GA because it sends the data to the Analytics server via a list of parameters attached to a single-pixel image request)
All fonts must be sourced either from the service or via a data: scheme
Browsers should report any violations of this policy to the service we have setup for this CSP
For everything else default to requiring the source to be the service itself

(By the way, big thank you to Foundeo Inc for putting up an faq post on CSP that covered Google Analytics).

Jekyll and GitHub pages oh my!

2017-04-15T00:00:00+00:00

So it was always on the to-do list to sort out my site, and recently I embarked on a venture (more about that in future posts) which involved me sending someone a link and them ending up on said site.

Only it wasn’t up, and to be honest if it was there wouldn’t have been anything to see. I’d already been pondering posting my lessons learnt rather than just squirreling them away in Evernote, so this felt like the time to pull my finger out and fix it.

I’d played with Jekyll as part of some work stuff, and knew about GitHub Pages so figured this would be a good opportunity to get to grips with both. Plus it seemed a low friction way to maintain my personal web site in the future.

It’s for newbs, right?

Or certainly that’s what the GitHub Pages intro page made me believe. I started by following the steps provided and yep, I had my hello world version up and running in no time.

But I knew I wanted the ability to create and run stuff locally, and like, presumed Jekyll had to feature somewhere so moved onto the next step of setting up Jekyll (this is linked to from the GitHub Pages site).

So got that all running, pushed it up and now I had an out of the box Jekyll site accessible from http://cruikshanks.github.io.

Themes

Well that’s nice, but the first thing I wanted to do was select a different theme. And from what I was reading that just seemed to require a tweak to the _config.yml file.

But whenever I did this it just caused a mass of errors. Plus the docs weren’t quite lining up with how I’d created my site.

Did I need to do this in my GitHub settings as well? Or was I supposed to be adding the theme as a gem to my Gemfile?

Whatever I was trying my stuff simply broke the moment I touched it. Clearly I was missing something, but I couldn’t tell if it was my ignorance or existing knowledge that was getting in the way.

Some googling later

After some googling and experimentation I realised the following

If you want the simplest, out of the box GitHub Pages solution, then create your my-org-name.github.io project, go to settings, select your theme, and start adding stuff via the GitHub web UI
If you want a supported Jekyll solution, so you can run the site locally to view changes before pushing them then find the theme you’re interested in and fork the project
The latest version of Jekyll themes are managed via the _config.yml and Gemfile using gems, but this is not supported in GitHub Pages
There are some nice themes out there (I started looking into Minimal Mistakes) but if you have no clue what you are doing, changing anything just seems to lead to lots of errors.

Ta-da!

Then I came across the GitHub Pages page in the Jekyll docs and it led me to Creating and Hosting a Personal Site on GitHub, a marvellous guide in Jekyll’s own words.

Rather than generating a project with Jekyll or forking an existing one from GitHub, the guide has you create a project from scratch.

This helped cement how layouts work and how they link with the actual content, what front matter does and essentially how a Jekyll site is structured. I could then combine this with lessons learnt during my experimentation to get a desired result. A GitHub pages compatible Jekyll based web site that I could deploy and run locally, which I understand.

After feeling like a dunce trying to get this up and running that last point was critical.

Small steps

So what I’m left with is a very basic web site which does very little at the moment, and which has a tonne of ways it could be improved. But its simple, easy to maintain and something I can build on going forward.

I can now get on with recording what I’ve learnt as posts, if only to prove to anyone watching that I do care about my work. If they happen to be of help; all the better!