Zocker: DIY Docker on FreeBSD

This is about Zocker, a project where I learnt container technologies by doing. The approach was to recreate a simple Docker clone using FreeBSD jail mechanism and ZFS. The essential concepts were quite easy to make and the result is 1.5 kloc of sh script. I believe that we should put all the effort in Docker project and I wish that it will have FreeBSD support one day. However, for me and my limited free time it was easier to hack with an independent project.

Introduction

Operating-system-level virtualization technologies have a long history, but the Docker phenomenon has made container technologies increasingly popular today. There must be many reasons to Docker popularity, but for me the most important is standardization. Docker concepts will drive application deployment in one common direction.

To me following are the key concepts in Docker. The first is an isolated process. A container runs a single process and containers are isolated using operating-system-level virtualizatiin. A container contains also the environment of the process, that is files and operating system reources.

However, to make containers portable they are decoupled from the host operating system. All shared resources: ports and shared volumes for example, must be explicitly specified on boot.

To help this, containers contain metadata. Parameters like incoming ports, shared volumes and environment variables are defined in container's metadata. In fact the command that is run inside a container is also one of the metadata parameters. We can say that a running container is combination of a process, a filesystem and metadata.

The filesystem and metadata can be saved into an image. It is a series of incremental changes in the filesystem or metadata. The image concept is essential as new containers can be created only from images.

To simplify image creation there is a Dockerfile. It is a list of parameters that will define an image and the image can be built easily by providing the file.

To share images easily there is a repository. There is a one common Docker repository, but you can also create another one for private use.

When a container is destroyed container's state is lost if the state is not saved into an image or if a volume is not used. A volume is a shared filesystem that can be shared with the host operating system or with an anohter container. Volume mount points are also defined in container's metadata.

Containers are isolated on network level as well. Operator needs to expose ports defined in metadata in boot or he can link the containers. Linking is a way to link multiple containers together in network level and send connection information from one to another.

Simply put, a container is a process, its environment and metadata in one package. This helps in deploying application in scale. To me this concept in container technologies is more important than fast boot times.

I wanted to deploy my application using the concepts mentioned above on FreeBSD and I wrote Zocker. Zocker covers the concept of an isolated process, decoupling, metadata, an image, a Dockerfile, a repository and a volume. However, networking is different. Zocker doesn't use network stack virtualization, it simply uses an IP address per container. I also like the simple approach as now I don't need to worry about port mappings when deploying containers. The linking concept is not implemented because of the simplified networking. There are also dozens of minor differences, for example volumes can be mounted only from the host system and Zocker doesn't implement container daemonizing. However, I want to emphasize that Zocker is not going to be a feature complete Docker clone, but it is a proof of concept tool to implement similar features.

The End Result

Zocker can be found here. Zocker has a CLI interface that tries to mimic Docker interface closely. Following tutorial goes through most of the commands. Zocker should be run as root and the examples are for tcsh shell.

Preparations

To start out we need a FreeBSD 10 (Zocker uses jail.conf) installation with /usr/src to compile a base jail.

Following creates a ZFS filesystem for testing purposes, does a git checkout and configures Zocker (replace mypool and em0 with proper values)

~# zfs create mypool/zocker
~# zfs create mypool/zocker/zocker
~# cd /mypool/zocker/zocker
/mypool/zocker/zocker# git clone https://github.com/toddnni/zocker.git .
/mypool/zocker/zocker# echo 'ZFS_FS=mypool/zocker' > config
/mypool/zocker/zocker# echo 'HOST_INTERFACE=em0' >> config

We need to add zocker to our PATH somehow. To get going let's set

/mypool/zocker/zocker# setenv PATH "${PATH}:/mypool/zocker/zocker"

Create base image from sources using example script

/mypool/zocker/zocker# sh examples/src.sh

This is the longest part in the example, because the compilation will take time.

srcbuild
--- buildworld ---
--- buildworld_prologue ---
--------------------------------------------------------------
>>> World build started on Mon May  4 20:12:12 EEST 2015
>>> --------------------------------------------------------------
>>>
...
943a43e0-f4e6-11e4-972e-27ff10bf3cd7

zocker images

Now we have an image. Run

/mypool/zocker/zocker # zocker images

to see it

TAG              IMAGEID                              DATE                    USAGE         PARENT
10.1-RELEASE-p9  943a43e0-f4e6-11e4-972e-27ff10bf3cd7 Thu May  7 21:26 2015   205M (205M)   cd43b57f-f29a-11e4-972e-27ff10bf3cd7
scratch          cd43b57f-f29a-11e4-972e-27ff10bf3cd7 Mon May  4 23:19 2015   19K (19K)     -

zocker tag

We need to tag it with tag 'base' so that rest of the examples will work.

/mypool/zocker/zocker # zocker tag 10.1-RELEASE-p9 base

zocker run

To try out the image let's fetch URL http://www.google.com. Zocker's simple networking doesn't assign network addresses automatically and we need to assign them manually to containers. Zocker configures jail tool to resolve an address using container's hostname, and to make commands zocker build or zocker run to work we need to assign resolvable hostnames. So let's have hostnames build and test in hosts file

/mypool/zocker/zocker # echo '192.168.0.240 build' >> /etc/hosts
/mypool/zocker/zocker # echo '192.168.0.241 test' >> /etc/hosts

Now this should work

/mypool/zocker/zocker # getent hosts test

and we can run a container using the hostname

/mypool/zocker/zocker # zocker run -n test base 'printf "GET / HTTP/1.0\r\n\r\n" | nc www.google.com 80'
test: created
HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
...
test: removed

Zocker will copy /etc/resolv.conf from the host to a container on boot.

zocker ps

Let's examine our container. We need to use -a because the container is down.

/mypool/zocker/zocker # zocker ps -a
JAIL             IMAGEID                              DATE                  STAT IP              USAGE         CMD
test             943a43e0-f4e6-11e4-972e-27ff10bf3cd7 Thu May  7 21:37 2015 down -               205M (110K)   'printf "GET / HTTP/1.0\r\n\r\n" | nc www.google.com 80'

zocker rm

Containers will not be removed automatically. If we don't want to store it for later use, we need to be remove it

/mypool/zocker/zocker # zocker rm test

zocker build

Now we can build an apache image using an example Zockerfile

/mypool/zocker/zocker # zocker build -t apache examples/apache
## Using container 'build', it must be manually removed in some errors
## FROM base
# - 943a43e0-f4e6-11e4-972e-27ff10bf3cd7
## NET build
# - 943a43e0-f4e6-11e4-972e-27ff10bf3cd7
## RUN env ASSUME_ALWAYS_YES=YES pkg bootstrap
build: created
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[build] Installing pkg-1.5.1...
[build] Extracting pkg-1.5.1: 100%
Message for pkg-1.5.1:
If you are upgrading from the old package format, first run:

  # pkg2ng
build: removed
# - 829e6f61-f4e8-11e4-972e-27ff10bf3cd7
## RUN mkdir /apachelogs
build: created
build: removed
# - 8410f58e-f4e8-11e4-972e-27ff10bf3cd7
## RUN pkg install -y apache24
build: created
Updating FreeBSD repository catalogue...
[build] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
[build] Fetching packagesite.txz: 100%    5 MiB   2.7MB/s    00:02 

Processing entries: 100%
FreeBSD repository update completed. 23912 packages processed.
Updating database digests format: 100%
The following 9 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        apache24: 2.4.12
        expat: 2.1.0_2
        perl5: 5.18.4_14
        pcre: 8.35_2
        apr: 1.5.1.1.5.4
        gdbm: 1.11_2
        indexinfo: 0.2.3
        gettext-runtime: 0.19.4
        db5: 5.3.28_2

The process will require 130 MiB more space.
31 MiB to be downloaded.
[build] Fetching apache24-2.4.12.txz: 100%    4 MiB   1.9MB/s    00:02
[build] Fetching expat-2.1.0_2.txz: 100%   98 KiB 100.1kB/s    00:01
[build] Fetching perl5-5.18.4_14.txz: 100%   13 MiB   3.5MB/s    00:04
[build] Fetching pcre-8.35_2.txz: 100%    1 MiB   1.1MB/s    00:01 
[build] Fetching apr-1.5.1.1.5.4.txz: 100%  402 KiB 411.3kB/s    00:01
[build] Fetching gdbm-1.11_2.txz: 100%  142 KiB 145.6kB/s    00:01 
[build] Fetching indexinfo-0.2.3.txz: 100%    5 KiB   5.0kB/s    00:01
[build] Fetching gettext-runtime-0.19.4.txz: 100%  146 KiB 149.3kB/s    00:01
[build] Fetching db5-5.3.28_2.txz: 100%   12 MiB   1.4MB/s    00:09
Checking integrity... done (0 conflicting)
[build] [1/9] Installing indexinfo-0.2.3...
[build] [1/9] Extracting indexinfo-0.2.3: 100%
[build] [2/9] Installing gettext-runtime-0.19.4...
[build] [2/9] Extracting gettext-runtime-0.19.4: 100%
[build] [3/9] Installing expat-2.1.0_2...
[build] [3/9] Extracting expat-2.1.0_2: 100%
[build] [4/9] Installing gdbm-1.11_2...
[build] [4/9] Extracting gdbm-1.11_2: 100%
[build] [5/9] Installing db5-5.3.28_2...
[build] [5/9] Extracting db5-5.3.28_2: 100%
[build] [6/9] Installing perl5-5.18.4_14...
[build] [6/9] Extracting perl5-5.18.4_14: 100%
[build] [7/9] Installing pcre-8.35_2...
[build] [7/9] Extracting pcre-8.35_2: 100%
[build] [8/9] Installing apr-1.5.1.1.5.4...
[build] [8/9] Extracting apr-1.5.1.1.5.4: 100%
[build] [9/9] Installing apache24-2.4.12...
===> Creating users and/or groups.
Using existing group 'www'.
Using existing user 'www'.
[build] [9/9] Extracting apache24-2.4.12: 100%
Message for apache24-2.4.12:
To run apache www server from startup, add apache24_enable="yes"
in your /etc/rc.conf. Extra options can be found in startup script.

Your hostname must be resolvable using at least 1 mechanism in
/etc/nsswitch.conf typically DNS or /etc/hosts or apache might
have issues starting depending on the modules you are using.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

- apache24 default build changed from static MPM to modular MPM
- more modules are now enabled per default in the port
- icons and error pages moved from WWWDIR to DATADIR

   If build with modular MPM and no MPM is activated in
   httpd.conf, then mpm_prefork will be activated as default
   MPM in etc/apache24/modules.d to keep compatibility with
   existing php/perl/python modules!

Please compare the existing httpd.conf with httpd.conf.sample
and merge missing modules/instructions into httpd.conf!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
build: removed
# - ade40bb2-f4e8-11e4-972e-27ff10bf3cd7
## COPY httpd.conf /usr/local/etc/apache24
build
# - cd3b2e21-f4e8-11e4-972e-27ff10bf3cd7
## VOLUME /var/empty:/apachelogs:rw
build
# - ce0d457a-f4e8-11e4-972e-27ff10bf3cd7
## CMD apachectl -D FOREGROUND
build
# - cf1bd858-f4e8-11e4-972e-27ff10bf3cd7
# Tagged 'apache'

The build will use the hostname build that was defined above. When the build is finished we have

/mypool/zocker/zocker # zocker images
TAG              IMAGEID                              DATE                    USAGE         PARENT
apache           cf1bd858-f4e8-11e4-972e-27ff10bf3cd7 Thu May  7 21:42 2015   349M (36K)    ce0d457a-f4e8-11e4-972e-27ff10bf3cd7
10.1-RELEASE-p9  943a43e0-f4e6-11e4-972e-27ff10bf3cd7 Thu May  7 21:26 2015   205M (205M)   cd43b57f-f29a-11e4-972e-27ff10bf3cd7
base             943a43e0-f4e6-11e4-972e-27ff10bf3cd7 Thu May  7 21:26 2015   205M (205M)   cd43b57f-f29a-11e4-972e-27ff10bf3cd7
scratch          cd43b57f-f29a-11e4-972e-27ff10bf3cd7 Mon May  4 23:19 2015   19K (19K)     -

Let's run the apache using the test IP address. The following code creates a directory for log files and then starts an apache container in foreground

/mypool/zocker/zocker # mkdir /tmp/apachelogs
/mypool/zocker/zocker # zocker run -n test -v /tmp/apachelogs:/apachelogs:rw apache
test: created
[Thu May 07 21:45:18.320406 2015] [core:warn] [pid 83097] (2)No such file or directory: AH00075: Failed to enable the 'httpready' Accept Filter

Test apache in another terminal session

~# curl http://test
<html><body><h1>It works!</h1></body></html>

and see that a log line was generated to the shared volume

~# cat /tmp/apachelogs/httpd-access.log
192.168.0.241 - - [07/May/2015:21:45:35 +0300] "GET / HTTP/1.1" 200 45 "-" "curl/7.42.1"

Now we can stop the apache container in the first terminal session using CTRL-C and remove the container

/mypool/zocker/zocker # zocker stop test
test: removed
/mypool/zocker/zocker # zocker rm test

zocker push

Images are transferred to a Zocker repository via SSH. We need to configure SSH keys for a user in server side and configure REPOSITORY variable in client side config file. After the configuration, a push will transfer all the intermediate images to a server

/mypool/zocker/zocker # zocker push apache

When there are images in a repository zocker search and zocker pull will work. There is also zocker del to delete images from a repository.

Discussion

There is a lot going on around the Docker phenomenon and I wanted to share my research. I'm impressed how easily the container concept can be implemented using ZFS and jails. Zocker is in fact a thin wrapper that doesn't add that much on top of existing technologies.

Take a look at Zocker repository and try it out. To learn more about use cases find articles related to Docker. The Docker tutorials and command line reference can be helpful too.

Before starting the project I also found out that Kato Kazuyoshi has been working with Docker FreeBSD port, but it is progressing slowly. There is also Rocket and App Container Specification which sound promising according to Maciej Pasternacki's research.

I'm open for suggestions and improvements, but please keep in mind that Zocker aims to be a simple, but functional tool to hack and learn the container concepts. The quality of code is currently not up to par and there are lot of things to fix. However, it works and I use it to manage my private systems.