Continuous Integration Systems: CircleCI vs TravisCI vs Codeship vs Jenkins Part 2

In part 1 of this series on Continuous Integration Systems, I outlined features of CircleCI and TravisCI. In this post, I will review Codeship and Jenkins, show you a results table comparing all four systems, interpret the results table and declare the overall winner.

Codeship

Initial Setup

To start a project you need to authorize Codeship against Github and then manually write the repository URI, which is very disappointing. Also, you have to write the commands required to build your project in their interface. This is a huge disadvantage because your code will be detached with the build steps. So, if you need to change the NodeJS version, or add one step, you have to do it on Codeship’s interface and not your repository’s code.

Rating: 7

Services

Codeship does provide Cassandra, MongoDB, MySQL, Postgres, RethinkDB, ElasticSearch, Memcached, RabbitMQ, Redis. Some of this requires setup, but others, like Mongo, don’t require any configuration.

Rating: 8

Performance

Main WindowUI/UX

The project has to be selected from the top of the page and then you see a list of the latest builds. It shows time ago, commit id and commit message.

Build Window

Codeship collapses the output of the build by commands similar to how CircleCI does it.

UI/UX Rating: 7.5

Pricing

1. Free for 1 concurrent build
2. $49/m for 1 concurrent build, 2 test pipelines
3. $99/m for 2 concurrent builds, 2 test pipelines
4. $199/m for 3 concurrent builds, 3 test pipelines

—

Jenkins

Initial Setup

Starting from scratch with Jenkins may be the worst option for a development group without sysadmin experience. However, at the same time, it can be a totally viable and desirable option for a group with sysadmin experience. This is due to the ability to customize the service, integrate it with slack or other services, and get better performance over the hosted offerings. It is worth to mention that Jenkins 2.x comes with a wizard that facilitates the setup.

Besides installing Jenkins 2.x, in this case, I also had to install nvm, development tools and libraries (depending on your OS) and a few Jenkins plugins. Once the setup is ready, you have to add a Jenkinsfile to your repository that describes the process to test/build the application.

Rating: 7

Services

You have to manually install services you want to use. This has to be considered when setting the specs of your server, because Jenkins requires memory and disk depending on your use, and that adds up with the memory, disk and cpu required by the different services.

Rating: 7

Performance

Main WindowUI/UX

Jenkins provides good navigation between builds, but it doesn’t show details like commit message or id in its main window, which I find unfavorable. There are useful things from this window, like showing the trend above the build history listing, or the overview of changes in your repo with the corresponding build.

Trend Window

Recent Changes

ANSI

Jenkins does not support ANSI codes by default, which is very disappointing. There is a plugin you can configure, but this is a feature that should already be part of the base system.

Jenkins’s advantage is that you can customize it to fit your needs, but the default configurations are being evaluated in this case.

UI/UX Rating: 7

Pricing

For this exercise, a t2.medium instance in AWS was used. t2.micro could not support Jenkins and Mongo running on the same host and t2.medium performed better than the hosted service,s since those offer containers with shared resources.

1. $37/m for t2.medium instance in AWS

Results Table

• Gitlab was initially considered for this comparison, but it was removed due to the complexity to configure additional services like DB.
• AWS CodePipeline is not included in this evaluation because it is not a CI tool, but more a CD framework.

Interpretation of Results

There is clear winner in terms of performance due to the fact that all hosted solutions offer containers. The difference is significant considering the price is much lower than the hosted solutions. This is at the cost, of course, of some dedication to configure the service and maintain it internally.

CircleCI is a very good option if you are not concerned about performance, but it is clear that it taxes their servers with more containers per CPU. Besides this, it offers great UI and availability of services.

My personal best (excluding Jenkins) is TravisCI, because it provides a very useful UI and availability of services, but it is also very well documented and used. Also, you can have any number of tests configured and only pay for 2 concurrent.

Conclusions

If you have internal resources that can setup and maintain it, go with Jenkins. If not, check TravisCI first and then CircleCI second.

—

If you have any questions or would like to see more about continuous integration, please comment below. You can also read this post on GitHub here. Gorilla Logic hires only the best software engineers. Think you have what it takes to become a Gorilla? Check out or careers section by clicking below.