Loadmill
Learn moreSchedule a demo
  • Introduction
    • Loadmill - AI - Powered Solution
    • Deviceless mobile testing
      • Capturing traffic with Loadmill MITM Proxy
      • Loadmill desktop recorder
        • Generating test flows
      • Installing certificate on mobile devices
        • iOS certificate installation
        • Android certificate installation
      • Configuring proxy on mobile devices
        • iOS Wi-Fi settings
        • Android Wi-Fi settings
      • Troubleshooting
    • What is an API
      • API - Data Fetching
      • Quick examples of API requests
      • What is an API endpoint?
    • API Server Testing
      • Contract testing
      • Regression Testing
  • Quick Guide
    • Create Account
    • Download Test Composer
    • Register your first API flow
    • Running Your API Test
  • Loadmill Test Composer
    • Quickstart
    • Composer Layout
    • Filter Settings
  • Test Editor
    • Layout
    • Flows
      • Generated Flow Code
      • Test Flow editor
      • Flow Controls
      • Add CSV to Flow
      • Flow Execution
    • Steps
      • Request step
      • Code step
      • Extraction & Assertion step
      • Web Socket step
    • Extractions - Set Parameters
    • Assertions - Verify Response
    • Parameters
      • Parameter Execution Order
      • Test Suite Parameters
      • Parameters Sets
    • ⨍(⨯) FUNCTIONS
    • Postscript
      • Running Postscript
      • Accessing w/ Postscript
      • Validating Postscript
    • Login/Authentication Flow
    • Before & After Hooks
  • Load Testing
    • Load Test Editor
    • Load Testing Guide
    • Analyzing Load Test Results
    • Parameterized Load Test
    • Domain Verification
    • Configuration Files
    • Load Testing FAQs
    • Load Testing Troubleshooting
  • User Behavior Testing
    • Overview
    • Setup
    • Recording troubleshooting
    • Additional recording methods
    • Recording settings
    • How to work with Recordings
  • Auth
    • Okta SSO integration
    • API Tokens
    • Testing with CORS
    • REST API
  • Integrations
    • Loadmill Agent
    • CI integration
    • GitHub
      • CI integration
      • Data sync
    • GitLab
    • Bitbucket
    • Jira
    • New Relic
    • Slack integration
    • TestRail integration
    • Database Testing
    • Kafka Testing
    • Datadog Integration
    • ✉️Email Testing
    • Webhook Testing
    • Integrations FAQs
    • XRay
    • TestRail
    • gRPC Support
  • Collaboration
    • Collaboration
    • Teams
    • Groups & Reports
    • Test Suite Collaboration
    • Reviews
    • Shared Flows
    • Labels
  • Reporting
    • API Catalog & Coverage
      • API Catalog
        • Unique Entity ID's Mapping
        • Domain Mapping and grouping
        • Endpoints grouping
        • OpenAPI upload
      • Test Coverage
        • Generating API test coverage report
  • General
    • Billing
      • Usage report
    • Settings
      • 📈Analytics
        • Flow Run History
      • 🧳Import & Export
    • General FAQs
    • General troubleshooting
    • Comparisons
      • Loadmill vs. SoapUI
      • Loadmill vs. JMeter
      • Loadmill vs. Blazemeter
      • Loadmill vs. WebdriverIO
      • Loadmill vs. Potato
    • Miscellaneous
      • Running a Test Suite
      • Test Plan
      • API Testing troubleshooting
      • API Testing FAQs
      • Test Editor
        • API Tests - Data from CSV files
Powered by GitBook
On this page
  • Assertions editor
  • Assertions Creation from Response
  • Assertion Types
  • Suggestions
  • Caveats
  1. Test Editor

Assertions - Verify Response

PreviousExtractions - Set ParametersNextParameters

Last updated 1 year ago

When testing an API, it is usually not enough to verify that all our HTTP requests completed successfully. Often, it is necessary to make sure that we got the correct response from the server by examining the response status, body or headers.

With , this is made easy by using Assertions. Assertions are used in conjunction with to do just that: examine the server's response and assert its correctness.

You may have an arbitrary number of assertions executed after each successful request. If an assertion fails, the next request will not be executed and the test scenario will be marked as failed, but the subsequent assertions for the current request will be executed nonetheless.

Assertions editor

Assertions editor can be found in loadmill under each request cubical on it's expanded form.

Assertions Creation from Response

It is also possible to create assertions directly from the response panel from the 'JSONPATH' window (only when the response is a JSON). After executing the test, click on one of the responses to expand it, which will open the response panel, as follows:

Now, simply click on the plus sign located in the top right corner, set the extraction, and add the desired assertion right from there!

You can then see both the extraction and the assertion in the request editor panel:

Assertion Types

The target of an assertion is always a parameter value. You may use built-in parameters, default parameters or any parameter extracted from the current or previous requests in the current scenario as the target.

There are several types of assertions:

Exists - Validates that a parameter exists and not null.

Doesn't exist - Validates that a parameter doesn't exists or null.

Equals - Validates that a parameter is equal to the given expression. The equality check is case sensitive.

Doesn't equal - Validates that a parameter is NOT equal to the given expression. The equality check is case sensitive.

Contains - Validates that a parameter contains a sub-string. The containment check is case sensitive.

Doesn't contain - Validates that a parameter DOESN'T contain the given expression. The containment check is case sensitive.

Matches - Validates that a parameter matches the given regular expression.

Greater than - Validates that a parameter is greater than the given expression.

Less than - Validates that a parameter is less than the given expression.

JSON Contains - Validates that a json contains a subset json, in such way dynamic fields can be omitted to avoid false negatives.

Accepting any value for individual field will accept "*" notation Note, the subset json object should preserve the same hierarchal structure as the containing object while fields can be omitted.

XML Contains - Validates that an xml contains a subset xml, in such way dynamic fields can be omitted to avoid false negatives.

Accepting any value for individual field will accept "*" notation Note, the subset xml object should preserve the same hierarchal structure as the containing object while fields can be omitted.

You may embed parameters in any assertion expression. These parameters will be evaluated right before the assertion is executed.

  1. Default assertions - for any extraction we find in the recorded test, we add a default assertion to it (example: "id" exists).

  2. Specific assertions - for any parameter with given user key (for example, "success") we extract it and assign to it an assertion (example: "success" equals "true").

Suggestions

Then, each user can add the assertions from the repository by clicking on + SUGGESTIONS.

By default, there are a few common extraction and assertion examples in the repository. Team admins can also navigate to the Suggestions Settings directly from within the suggestion dialog window.

Caveats

Keep in mind that all parameter values are textual, i.e. a parameter has no type such as Number or Array that we know from common programming languages.

This is important in order to avoid confusion when using parameter extractors such as JSONPath. For example, consider the following scenario:

  1. Extract the value for books via the JSONPath query student.books on

     {
         "student": {
             "books": []
         }
     }
  2. Assert books is Not Empty.

You may expect this assertion to fail but, in fact, it will succeed. This is because the parameter books is evaluated to [] and therefore is considered as a non-empty string. One possible way to correct this is to use a RegExp assertion on the books parameter instead: \[[^\s]+\].

Furthermore, there is an option to disable an assertion by clicking the three dots () in the assertion line. A disabled assertion will still be evaluated, but it won't cause the test to fail.

Click on , then navigate to the desired parameter. For example, let's find the 'content-length':

When using assertions, it will usually require to use first to set the parameter with proper data and only then to attempt to validate it's form.

Example 1: Assigning empty term to param1 will result in assertion failure

Example 2: Assigning any value to param1 will result in passed assertion

Example 1: Assigning any value to param1 will result in assertion failure

Example 2: Not setting param1 entirely will result in passed assetion

Example 1: The assigned param1 doesn't equals to loadmill hence will fail in assertion

Example 2: Extracting $.args.app json path field to param1(from the result that is not displayed) should be equal to "Demo" and will result the assertion to pass Assigning numeric value to param2 of 11.0 in postscript should be equal to "11", and the assertion will pass. Note that all values are implicitly compared as strings.

Example 1: param1 shouldn't be equals to 'demo' but it is, hence the assertion will fail

Example 2: param1 doesn't equals to 'loadmill' hence the assertion will pass

Example 1: When param1 set with the value 'loadmill-demo' and doesn't contains the string 'example' the assertion will fail.

Example 2: When param1 set with the value 'loadmill-demo' and contains the string 'demo', the assertion will pass.

Example 1: When param1 set with the value 'loadmill-demo' and contains the string 'demo', the assertion will fail.

Example 2: When param1 set with the value 'loadmill-demo' and doesn't contain the string 'example', the assertion will pass.

Example 1: The assigned epoch representation of the current time to param1 will contain 13 digits would not match the regex (^\d{10}$), hence the assertion will fail

Example 2: The assigned epoch representation of the current time to param1 will contain 13 digits and would match the regex (^\d{13}$), hence the assertion will pass

Example 1: param1 is assigned with the value 31 that is not greater than 40 hence the assertion will fail.

Example 2: param1 is assigned with the value 31 which is greater than 30 will make the assertion to pass

Example 1: param1 is assigned with the value 30 that is not less than 31 hence the assertion will fail.

Example 2: param1 is assigned with the value 31 which is less than 40 will make the assertion to pass

JSON Schema - Validate that a parameter comply with given .

Example 1: Given param1 was set with a json as below, opposed to the expected schema the field "name" is not numeric hence the assertion will fail. Note, the assertion will suggests to exclude the failing field from the validation schema.

Example 2: Given param1 was set with a json as below and the expected schema describes it correctly, the assertion will pass.

To prepare initial JSON schema out of a JSON you can search for 'from json to schema' in google or use a .

Example 1: Given param1 was set with a json as below, any mismatch in a key or value on same hierarchy will result in assertion error: Note to the "Exclude ..." notations, clicking on them will adjust the value of the assertion to exclude the failing field.

Example 2: Given param1 was set with a json as below, any partial subset in json-contains validation value with same structure will result pass in assertion: - Using star notation to accept any value for latitude and country fields. Note, the "*" notation works only on simple values (strings/numerics/booleans)

Example 1: Given param1 was set with an xml as below, any mismatch in a key or value on same hierarchy will result in assertion error:

Example 2: Given param1 was set with an xml as below, any partial subset in xml-contains validation value with same structure will result pass in assertion: - Using star notation to accept any value for latitude and country fields. Note, the "*" notation works only on simple values (strings/numerics/booleans)

See an example of the assertion that validates the generated ID is in below:

When creating tests via our , Loadmill will create two automatic assertion types for you:

Optional automatic JSON Schema assertion: you can enable within Settings - Recordings.

In many cases Loadmill users use the same or similar extractions and assertions. We've implemented the Suggestions feature that allows team admins to configure a repository of and assertions within Settings - Suggestions.

extractions
JSON Schema
tool like this
the format of UUID
🧙‍♂️
Chrome recorder extension
this assertion
extractions
Loadmill
parameters
Assertion section in request editor
The request Assertions - Verify Response section