Karate vs REST-Assured: Comparing Powerful API Testing Frameworks

Modern development stacks are becoming increasingly reliant on splitting back-end development from front-end development. This paradigm brings several benefits to development teams:

• Faster speed to market

• Isolated points of failure

• More rapid, agile, and resilient delivery of new functionality through the separation of back-end and front-end projects, with some teams even able to deliver a single microservice at a time

• Greater ability to deliver products to a wider range of users (smartphones, monitors, computers, IoT devices, and more) by creating multiple front-end applications that target different platforms and leverage ubiquitous data

These changes require new ways to more efficiently manage aspects of the development cycle such as product creation, API testing, security, deployments, and more. In this article, I will focus on API testing. I will review two of the most powerful API testing frameworks in the market, and compare the pros and cons of each.

Why We Need API Testing

Back-end teams create API’s at a blazing pace today. QA engineers rely on tools such as Postman to verify complete product functionalities using API endpoints. This approach provides:

• Faster results, because teams can get feedback from the API in a matter of minutes.

• Improved scalability, because faster feedback lets teams deploy faster.

• Simplified test creation, because the visual GUI (Graphical User Interface) makes the process easier.

• More stable testing than with UI tests and more accurate testing than with unit tests.

Despite these benefits, this approach creates a challenge—QA engineers become solely responsible for test creation. This means that developers lose knowledge of how to create the API endpoint test, which can be a loss to development teams.  

There are, however, ways to give some power back to developers so they can test their own endpoints, including the use of two of the most interesting API testing tools in the market: Karate and REST-Assured.

API Testing Tools: REST-Assured and Karate

REST-Assured is one of the most established API testing tools in the market. It has a powerful Java-based API that leverages Hamcrest as the matcher tool. It also has a simple set of methods that allows method chaining and Pseudo-Gherkin Syntax.

Karate, on the other hand, is a Gherkin-based API testing tool. It’s ideal for quick prototyping as well as simpler testing that can be as in-depth as you want. Karate includes a simple quick start mode, integrated dashboards, and other goodies that are gaining attention from the testing community.

Let’s Get Our Hands Dirty

We will create two projects/folders, one for each tool we will compare. The first project will use Karate; the second one will use REST-Assured. Within each project, we will install the tool, configure it, and create tests with it. 

For a fair comparison, we will create the same tests in both projects. Both projects are based on Java and the Java Virtual Machine (JVM), so you will need to have the JVM and Maven installed.

Karate

To install Karate, simply download the Karate-x.x.x.jar.

REST-Assured

REST-Assured works well with Maven, so you can start a new project using any Maven archetype and then add the following dependencies:

REST-Assured dependencies

A full pom.xml file can be found here.

Execute “mvn compile” and all of the dependencies will be downloaded.

The API Tests

We will have both frameworks verify the same application and use the same tests.

Our application under test will be JSONPlaceHolder, which offers a simple REST API for testing.

• The API returns a hard-coded list of TODO’s

• We will verify the first TODO and the most common operations available:

             • Equality

             • Contains and Not Contains

Karate – Equality
Feature: TODO API Verificator

Background:
    * url 'https://jsonplaceholder.typicode.com/todos/1'

Scenario: Verify First Todo is equals to "delectus aut autem"
    Given request
    When method get
    Then status 200   
    And match response == { userId: 1, id: 1, title: "delectus aut autem", completed: false } # Karate offers full JSON Comparison out of the box

 Scenario: Verify First Todo is equals to the sum of its parts
    Given request
    When method get
    Then status 200    
    And match response.userId == 1
    And match response.id == 1
    And match response.title == "delectus aut autem"
    And match response.completed == false # Also offers the standard comparison by element
REST-Assured – Equality
private String url = "https://jsonplaceholder.typicode.com/todos/1";
    @Test
    public void verifyFirstTodoIsEqualToTheSumOfItsParts() {
        given()
        .when()
            .get(url). // Yay, Gherkin
        then()
            .assertThat().body("userId", equalTo(1))
            .and().statusCode(200)
            .and().body("id", equalTo(1))
            .and().body("title", equalTo("delectus aut autem"))
            .and().body("completed", equalTo(false))
            ;
    }

 

Unfortunately, there is no built-in way to compare a whole JSON in Rest-Assured, however, the Gherkin Syntax included allows for powerful matching logic.

Karate – Contains
Feature: TODO API Verificator

Background:
    * url 'https://jsonplaceholder.typicode.com/todos/1'

Scenario: Verify First Todo title contains "delectus"
    Given request
    When method get
    Then status 200    
    And match response contains { title: "#regex ^(\\w+[delectus]).+"} # Supports regex out of the box
REST-Assured – Contains
private String url = "https://jsonplaceholder.typicode.com/todos/1";
    @Test
    public void verifyTitleContainsDelectus() {
        given()
        .when()
            .get(url).
        then()
            .assertThat().statusCode(200)
            .and().body("title", containsString("delectus")) // Hamcrest Matchers, no need for regex
            ;
    }
Karate – Not contains value
Feature: TODO API Verificator

Background:
    * url 'https://jsonplaceholder.typicode.com/todos/1'

Scenario: Verify First Todo title field does not contain lorem
    Given request
    When method get
    Then status 200    
    And match response contains { title: "#regex ^((?!lorem).)*$"}  # Regex to verify value is not contained
REST-Assured – Not contains value
private String url = "https://jsonplaceholder.typicode.com/todos/1";
    @Test
    public void verifyTitleNotContainsLoremIpsum() {
        given()
        .when()
            .get(url).
        then()
            .assertThat().statusCode(200)
            .and().body("title", not(containsString("lorem ipsum"))) // Use the compound matcher not()
            ;
    }

 

What if we want to verify that an element is not a part of the response?

Karate – Not contains ELEMENT
Feature: TODO API Verificator

Background:
    * url 'https://jsonplaceholder.typicode.com/todos/1'

Scenario: Verify First Todo does not contains author field
    Given request
    When method get
    Then status 200    
    And match response.author == '#notpresent' 
    
    # Built-In Mechanism to verify for element existente
REST-Assured – Not contains ELEMENT
private String url = "https://jsonplaceholder.typicode.com/todos/1";
    @Test
    public void verifyResponseDoesNotContainAuthor() {
		given()
        .when()
            .get(url).
        then()
            .assertThat().statusCode(200)
            .and().body( "any { it.key == 'author'}", is(false)) // Use JSONPath to verify element existence
            ;
    }

The Results

To execute Karate’s feature files:

• Place all feature files in a directory called “features”

              • For example: features\TODO-verifyNotContains.feature

• Execute: java -jar Karate-x.x.x.jar features\TODO-verifyNotContains.feature

Karate’s built-in reporting tool, which is powered by Cucumber, gives you the test results. 

Test results from Karate's built-in reporting tool

 

 

 

 

 

 

 

 

REST-Assured leverages JUnit or TestNG as test runners. To execute REST-Assured tests:

• Place all the Test files into the src\test\java folder.

• Open a terminal in the root folder and execute mvn clean compile test.

Note that, by default, REST-Assured doesn’t include a nice reporting tool, yet…

Rest-Assured results

 

 

 

 

 

 

 

 

 

 

We can use the powerful Extent Framework to create beautiful reports, something I will talk about more in my next article.

Which Is the Best API Testing Tool for You? It Depends.

Both API testing tools have strengths and weaknesses. Each can easily empower your team. Consider the following recommendations when selecting your tool. 

I recommend using Karate when: 

• You want a quick start to do API tests, without a major hassle.

• Your team lacks a Java background, or wants to work with Gherkin.

• Your product owners are empowered in code and can write some tests in Gherkin.

• You’re comfortable enough with Karate’s standard reports and don’t want to spend the time creating enhanced reporting.

I recommend using REST-Assured when:

• Your team has Java expertise.

• You want to give more responsibility for test creation to developers (improving the quality assurance methodology).

• You need or want to use the improved matchers offered by Hamcrest.

• You’re working with an API that requires intricate logic that is easier to write in Java.

The Repo

All of the tests and code examples from this article can be found in this repo: Karate vs REST-Assured.

The Karate Folder contains all of the feature files used for the example, as well as the Karate.jar binary file for quickstart.

The REST-Assured folder contains a quick Maven project to start testing, as well as all of the examples from this article.

 

Subscribe to our Blog

Frederik Rodriguez
Frederik Rodriguez
Frederik is a QA Automation Engineer based out of our office in Medellín, Colombia. He has been working in software development for over ten years and has become an avid fan of the Agile process. In his free time, he enjoys playing soccer. When asked to describe himself he says he is an "Agile lover" and a "puppy hugger."

Deliver off-the-chart results.

WordPress Video Lightbox Plugin