Short Docker Stack tutorial

You’ve probably heard of Docker; since its 2013 release it has gained a lot of traction in the tech community. In this blog Dan Lyons – software developer @ Clicksco – will be explaining some of its primary functionality and applications.

 

"

 

What is Docker?

To begin with, Docker allows applications to run in isolation inside light-weight containers, separate from the host operating system.  This is achieved through configuring an ‘image’ with the files and settings required to run your application. Once an image is configured, it can be pushed to a Docker image registry allowing it to be pulled down to any machine. Images are the environment run within a container. Therefore once an image exists in a remote repository, a container can be launched anywhere, on any machine with Docker installed, and your application will run in the same way. This completely removes situations where your app runs on your machine – but not in deployment.

Sound good? Keep reading.

How does Docker Stack help?

Docker Stack sits at a higher level than Docker containers and helps to manage the orchestration of multiple containers across several machines. Docker Stack is run across a Docker Swarm, which is essentially a group of machines running the Docker daemon, which are grouped together, essentially pooling resources.

Stacks allow for multiple services, which are containers distributed across a swarm, to be deployed and grouped logically. The services run in a Stack can be configured to run several replicas, which are clones of the underlying container. This number will be maintained as long as the service is running. Adding or removing replicas of a service is simple. This makes Docker Services ideal for running stateless processing applications.

Docker Stack in production

Docker Services are easily scalable and ideal for stateless applications. We have implemented them for event streaming and message based process intensive applications, handling millions of messages daily. And since Docker Services can communicate easily and live on the same ‘network’, they lend themselves particularly well to a micro-service architecture.

The deployment of many inter-communicating microservices is where Docker Stack comes in. The stack is configured using a docker-compose file. This is a YAML file written in a domain-specific language to specify Docker services, containers and networks. Given a compose file, it is as simple as one command to deploy the stack across an entire swarm of Docker nodes.

Our case for using Docker stack is the processing of millions of user page visit events containing information on the page a user has visited. This information is processed by a series of microservices, each feeding data to the next to attempt to find a good categorisation within our taxonomy for the page visit. Eventually a categorisation within our taxonomy is decided upon and this is written to the profile of the user who landed on the page. From then on whenever this user is seen within our network we can be sure that this person has an interest in the taxonomy node assigned by this process.

Setting Up Your Stack

As I said earlier to get a stack up and running just requires a docker-compose file. The following is run in a docker-quickstart-terminal, an easy way to get started with docker on windows. Docker toolbox for windows can be downloaded here.

  • To deploy a stack you need to be running on a swarm master node, so initialize a swarm:
docker swarm init
  • Create a file named “docker-compose.yaml” containing the following:

 

This specifies 3 services, each running a busy box image, which will be pulled from docker hub, and each will just sleep for 5 days on startup.

  • Use the compose file to deploy your stack of 3 services
docker stack deploy --compose-file {path-to-compose-file} my-new-stack
  • Check that your services are deployed
docker service ls

You should see something like this:

 

 

We now have 3 services running on a docker swarm.

Scaling your stack

When our services are running on a swarm it is trivial to scale the number of instances.

  • Now we have our services running we can scale them:
docker service scale my-stack_machine1=5
  • Let’s check our replica counts now:
docker service ls

Should show:

 

 

And the underlying containers can be seen by running:

docker ps

Showing these containers:

my-stack_machine1.5.r4vi6qme6d0qyo5lt4nfcnvyb
my-stack_machine1.4.7ts1e06bio6ktj1kcus6p3n17
my-stack_machine1.3.c0vr05p6ki3clhalgbnabd8wn
my-stack_machine1.2.xbilowtxjh30bk2q2c2s6zo1k
my-stack_machine2.1.riq2krbk93z0if4flwh5re5u1
my-stack_machine1.1.j3ut8y99v0colo6yk3tbwpwdb
my-stack_machine3.1.nafwoz48jyvlfsnxs45c5p48v

We now have 5 instances of the ‘my-stack_machine1’ service and one of machine 2 and 3, but 2 and 3 can be scaled in the same way.

Adding swarm workers

If you have docker-machine installed it is easy to demonstrate how we add workers to a swarm. A worker is another machine which can host containers, or service replicas. This section will show you how to add more machines to your swarm allowing their resources to be used to run your containers.

  • Create a new docker-machine:
docker-machine create worker-node
  • Get a worker join token from your master swarm node:
docker swarm join-token worker

“join-token” should output a command to run on a worker node like the following:

docker swarm join --token {TOKEN} {MASTER-IP}:2377
  • Connect to your new machine:
docker-machine ssh worker-node
  • Paste the join command you received earlier

If everything worked well you should see the following:

This node joined a swarm as a worker.
  • Now if you scale a service on your master node it should be distributed across the two nodes

Leave your worker docker-machine and on your master run:

docker service scale my-stack_machine2=5
  • Check the containers running on your master:
docker ps

You should not be able to see all 5 replicas of machine 2 on your master node

  • Connect to your worker machine again
docker-machine ssh worker-node
  • Check on your containers here
docker ps

You should be able to see the rest of your instances on this worker machine

You now have multiple services capable of using the pooled resources of a swarm. It is common practice for swarms to consist of multiple cloud computing machines and the services will be distributed between all of them.

Conclusions

To summarize, in the words of Docker themselves “Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications”, and Docker Stack sits a level above this to orchestrate multiple associated applications. In the above tutorial we created multiple Docker services and distributed them across multiple machines, showing how easy it is to work with a Docker Stack.

If you wish to find out more about Docker there are plenty of resources on their website https://www.docker.com/. There are also plenty of useful tutorials ranging in complexity at https://training.play-with-docker.com/alacart/.

 

Author: Dan Lyons, Software Developer @ Clicksco