Continuous Integration Systems I: CircleCI, TravisCI, Codeship, Jenkins

Continuous Integration and Unit Testing are two critical processes in the development life cycle of every software project. Therefore, the question is not if it should be done, but how it should be done. Luckily there are plenty of options available for free (for open source projects). This post was written to help you decide which of the four Continuous Integration systems (CircleCI, TravisCI, Codeship and Jenkins) to use depending on your priorities, using a scaffold application and created with a generator. The application uses nodejs 4.4.6, npm, gulp and connects to mongod. However, this post will not cover the delivery process.

In this first post in my two-part series, I will cover CircleCI and TravisCI. In my next post, I will cover Codeship and Jenkins and give my final evaluation. This comparison shows how easy/hard it is to make each option work and what considerations you should have when you deploy a similar application.

Evaluation Method

The following aspects were evaluated on each system:

Initial Setup: Complexity for creating the project and running basic tests
Services: Complexity for connecting services (mongod in this case)
Performance: Time it takes to run the operations
UI/UX: Satisfaction in the opinion of the author
Pricing (For evaluation purposes, I used free offerings only)

Each system is configured so that builds are triggered when changes are committed. Then, performance and UI/UX is evaluated. Ten iterations were considered to average each system’s performance without changing the code.

Test Program

These operations are considered as standard to build a MEAN application:

Preparation

rm -rf node_modules
npm install –no-optional

Unit Testing

npm test

Build

gulp build

Some systems require additional operations like installing underlying software, that time is considered as part of their performance evaluation.

We will begin here with CircleCI:

—

CircleCI

Initial Setup

Sign up requires only an authorization of CircleCI with Github. You will then see a list of repositories, so you can create the project with one click. The system looks for a file called circle.yml to determine what steps conform the build. The documentation is very helpful, providing many examples for the most common languages and features. Setting the node version only requires a statement.

Rating: 10

Services

CircleCI provides many services out of the box and it’s bounded to localhost. The list includes: Postgres, MySQL, MongoDB, Cassandra, redis, elasticsearch, rabbit-mq, etc, so for this application nothing is required.

Rating: 10

Performance

Main WindowUI/UX

The main page is informative, showing a list of the latest builds with the commit message, time ago and time it took.

This page is responsive, but, in my opinion, it doesn’t adapt well enough when the width is reduced. This is because the repository list is kept at the same width occupying empty space below the repo name.

Build Window

Each command is captured in a panel that is collapsed once the command has completed. Scrolling to the bottom of the page will keep it autoscrolling.

ANSI Support

CircleCI correctly supports ANSI codes for different colors.

Mobile

The same responsive view is shown on mobile devices, which doesn’t use space appropriately.

UI/UX Rating: 8

Pricing

Free for 1 job, 1 container
$150/m for 4 containers
$350/m for 8 containers

—

Travis CI

Initial Setup

The first thing to learn about TravisCI is that there are two sites: Travis-CI.com for enterprise clients and Travis-CI.org for open source projects. Immediately, I discovered a big delay, as I found out that Travis comes with a really old C++ compiler that doesn’t support C++11. This is important because nodejs and many other languages compile libraries/gems using C++, so you will get strange errors that don’t seem to relate to your code. Finally, you must add a file called .travis.yml to your repo and Travis will start triggering builds when code is committed to Github. TravisCI does have good documentation and community base.

Rating: 7

Services

Services in Travis work similar to CircleCI, with the exception that in Travis these are not started by default. So, you need to specify which services you want load from this list: MySQL, Postgres, MariaDB, MongoDB, CouchDB, RabbitMQ, memcached, Redis, Cassandra, Neo4j, ElasticSearch, RethinkDB, etc. For each of these Travis supports a number of versions.

Rating: 10

Performance

The time shown here includes installing a new compiler as noted above.

Main PageUI/UX

The main page shows the latest build which is very useful. As in CircleCI, it shows time ago, commit message and commit id, but also a link to compare the last changes, which is also very useful too.