Docker cache on Travis and Docker 1.12

I blogged before about building Docker images on Travis and suggested uploading images after successful test runs to Docker Hub and use them as Cache after downloading them in next Travis runs.

Travis upgraded recently to Docker version 1.12 (from 1.9) and since version 1.10 Docker features Content Addressability for layers. This change breaks caching and we need to implement a workaround using Travis cache.

Changes need to be made in .travis.yml:

Enable cache

cache:
  directories:
    - /home/travis/docker/

Start by requesting Travis to cache /home/travis/docker directory.

Load Docker images previously cached

before_install:
  - if ...

in software · Thu 01 September 2016

User friendly website analytics with Sandstorm Oasis and Piwik

Piwik is a great FLOSS website analytics platform. I've been self hosting it for different small websites I've managed through the years. Although it's fairly easy to setup and maintain at this level of use, I want to avoid having another service in my maintenance list.

While looking for user and web respecting alternatives -read looking for something else than Google Analytics- I realized that Sandstorm Oasis does support Piwik.

I logged-in and setup my Piwik instance, or Grain as Sandstorm calls instances, in less than 30 of seconds. The tricky part is to copy the code ...

in software · Tue 26 July 2016

Build and Test against Docker Images in Travis

The road towards the absolute CI/CD pipeline goes through building Docker images and deploying them to production. The code included in the images gets unit tested, both locally during development and after merging in master branch using Travis.

But Travis builds its own environment to run the tests on which could be different from the environment of the docker image. For example Travis may be running tests in a Debian based VM with libjpeg version X and our to-be-deployed docker image runs code on-top of Alpine with libjpeg version Y.

To ensure that the image to be deployed to ...

in software · Fri 17 June 2016

Remove users from git-crypt enabled repository

Git crypt is a neat git extension to encrypt some files - if not all - in a git repository. Integrates nicely with git using filters and it's use is completely transparent once you have unlocked a repository.

Using git-crypt you can still share a repository in public and maintain a set of files with secrets that are accessible to a limited number of users. Especially useful for open source projects.

At some point maybe you'll need to remove one of the users who have access to the encrypted files. Git-crypt does not provide a command to remove users (yet ...

in software · Tue 24 May 2016

Takis - A util that blocks until a port is open.

Over at Mozilla's Engagement Engineering we use Docker to ship our websites. We build the docker images in CI and then we run tests against them. Our tests usually need a database or a cache server which you can get it running simply with a single command:

docker run -d mariadb

The problem is that this container will take some time to initialize and become available to accept connections. Depending on what your test and how you run your tests this delay can cause a test failure to due database connection timeouts.

We used to wait on executing our ...

in software · Mon 16 May 2016

Static site hosting on Dokku and Deis

Deis and Dokku, the open source Heroku-like PaaS, can be used for hosting static sites too. Since they both support Dockerfile based deployments all we need is an Docker Image with Nginx.

I created giorgos/dokku-static-site which uses the ONBUILD instruction. To use it create a Dockerfile at the root directory of your static site with only one line:

FROM giorgos/dokku-static-site

and then place all your files under the html directory. If moving your website files to another directory isn't in your plans, you can alternatively create a symbolic link

ln -s . html

Then push to your Deis ...

in software · Mon 09 May 2016

Host your own music server with supysonic

I like to self-host services to make things fit my needs instead of the other way around and to satisfy my need to control my data. Also remember that self hosting is really important for the health of the open web and internet.

There are a couple of options for self-hosted media servers. I always go for FLOSS solutions, so Plex and friends are not an option. A couple of years ago I tried Subsonic, including the MadSonic and Musicabinet forks. Subsonic looked really outdated at that time, so that didn't last long. Musicabinet was second in my evaluation ...

in software · Wed 27 April 2016

Snippet testing with Firefox on Docker

Mozilla's Snippets Service feeds Firefox for Desktop and Android with small HTML or JSON blobs with content to display about:home page. That's the page users get when they open up Firefox.

Desktop snippets are usually a line of text and an icon but the service supports much more complicated snippets. Multiple choice forms for our end of year fund raising campaigns, videos about our mission, interactions with the browser Chrome and others. We even had a HTML5 game going once!

As the snippets increase more and more in complexity we need better ways to test them. Unfortunately ...

in software, · Thu 12 November 2015

Programming Pro Trinket with an Arduino

Pro Trinket is a ATmega328 based board by Adafruit. It's compatible with Arduino IDE and most of the code written for Arduino. It's small factor, cheap and comes with neat accessories like this LiPo backpack addon.

Pro Trinket is cheaper, smaller and consumes less power than a typical Arduino board because it lacks things like the FTDI chip used to communicate between the programmer and the board.

To program the Pro Trinket you need to have an external FTDI board or figure out how to make programming over USB work with Linux. I opted for the first option ...

in hardware, · Sat 31 October 2015

Downloading HDS Flash (f4m) video streams.

Some WebTV or other video streaming websites use a special Adobe (surprise, surprise) inspired protocol to stream their content. The special thing about this protocol is that the file is split into chunks which I guess is good if you want to jump on different points in the video or you have an unstable internet connection but really annoying if you just want to download the video and view it in a different player.

This protocol is called Abode Dynamic Streaming for Flash and usually the sites link to f4m manifest files. From the SWF Wikipedia Page we read get ...

in software · Wed 21 October 2015

Fleet job to remove unused docker images

Engagement Engineering, the team that I'm part of at Mozilla, runs two Deis clusters on AWS to host important websites including www.mozilla.org.

Deis is a Heroku-inspired PaaS which utilizes CoreOS and Docker. It's a great open-source project, developed in the public, with a great Community and commercially backed by Engine Yard.

Apps on Deis run within Docker containers which run on CoreOS machines that form the Deis cluster. Each new release of your code, i.e. each new deis pull or git push, creates a new Docker image that is stored in the internal Deis Docker ...

in software · Tue 13 October 2015

List the extensions of files under a directory

I want to list the extensions of filenames in a directory, sorted and unique.

First list all the files, leaving out directories using:

find -type f .

Then use awk to get the last three characters of each line, i.e. filename.

find . -type f | awk '{ print substr( $0, length($0) - 3, 4) }'

AWK's substr function extracts four characters (last argument) of string $0 (first argument) starting at full lenth of the string minus three characters (second argument).

This will list the extensions from all files in the current directory and all directories below.

Now sort and uniq the output ...

in software · Mon 15 June 2015

Don't hesitate to get a VPS to do your work.

This is not going to be one of these posts about how I got a super slim, slick machine or tablet and I moved all my work in the cloud. Fortunately this is old news and nobody talks about it. That being said I have been working on VPS machines for the last 6 months for some tasks.

When your bandwidth is not enough for what you've to do renting a VPS is a great option. I've seen speeds up to 200MB/s on the cheap DigitalOcean or Vultr VPS, with the later being faster most of the ...

in software, · Tue 05 May 2015

Duplicity scp backend weirdness

I'm a big fan of Duplicity and I use to backup my laptop to my office based server over scp which in its turn encrypts everything with EncFS and syncs everything up to my SpiderOak account. I was going under a typical maintenance today and I run duplicity collection-status to see how many backup sets I have.

Surprisingly duplicity returned only one full backup set more than a year old and no other backup sets. After some digging I found that the returned backup set was indeed the only full set I had but besides that I had many ...

in software · Tue 28 April 2015