Assertions - Verify Response
When testing an API, it is usually not enough to verify that all our HTTP requests completed successfully. Often, it is necessary to make sure that we got the correct response from the server by examining the response status, body or headers.
With Loadmill, this is made easy by using Assertions. Assertions are used in conjunction with parameters to do just that: examine the server's response and assert its correctness.
You may have an arbitrary number of assertions executed after each successful request. If an assertion fails, the next request will not be executed and the test scenario will be marked as failed, but the subsequent assertions for the current request will be executed nonetheless.
Assertions editor
Assertions editor can be found in loadmill under each request cubical on it's expanded form.

Furthermore, there is an option to disable an assertion by clicking the three dots () in the assertion line. A disabled assertion will still be evaluated, but it won't cause the test to fail.


Assertions Creation from Response
It is also possible to create assertions directly from the response panel from the 'JSONPATH' window (only when the response is a JSON). After executing the test, click on one of the responses to expand it, which will open the response panel, as follows:

Click on , then navigate to the desired parameter. For example, let's find the 'content-length':

Now, simply click on the plus sign located in the top right corner, set the extraction, and add the desired assertion right from there!

You can then see both the extraction and the assertion in the request editor panel:

Assertion Types
The target of an assertion is always a parameter value. You may use built-in parameters, default parameters or any parameter extracted from the current or previous requests in the current scenario as the target.

When using assertions, it will usually require to use extractions first to set the parameter with proper data and only then to attempt to validate it's form.
There are several types of assertions:
Exists - Validates that a parameter exists and not null.
Example 1: Assigning empty term to
param1
will result in assertion failureExample 2: Assigning any value to
param1
will result in passed assertion
Doesn't exist - Validates that a parameter doesn't exists or null.
Example 1: Assigning any value to
param1
will result in assertion failureExample 2: Not setting
param1
entirely will result in passed assetion
Equals - Validates that a parameter is equal to the given expression. The equality check is case sensitive.
Example 1: The assigned
param1
doesn't equals to loadmill hence will fail in assertionExample 2: Extracting
$.args.app
json path field toparam1
(from the result that is not displayed) should be equal to "Demo" and will result the assertion to pass Assigning numeric value toparam2
of 11.0 in postscript should be equal to "11", and the assertion will pass.Note that all values are implicitly compared as strings.
Doesn't equal - Validates that a parameter is NOT equal to the given expression. The equality check is case sensitive.
Example 1:
param1
shouldn't be equals to 'demo' but it is, hence the assertion will failExample 2:
param1
doesn't equals to 'loadmill' hence the assertion will pass
Contains - Validates that a parameter contains a sub-string. The containment check is case sensitive.
Example 1: When
param1
set with the value 'loadmill-demo' and doesn't contains the string 'example' the assertion will fail.Example 2: When
param1
set with the value 'loadmill-demo' and contains the string 'demo', the assertion will pass.
Doesn't contain - Validates that a parameter DOESN'T contain the given expression. The containment check is case sensitive.
Example 1: When
param1
set with the value 'loadmill-demo' and contains the string 'demo', the assertion will fail.Example 2: When
param1
set with the value 'loadmill-demo' and doesn't contain the string 'example', the assertion will pass.
Matches - Validates that a parameter matches the given regular expression.
Example 1: The assigned epoch representation of the current time to
param1
will contain 13 digits would not match the regex (^\d{10}$
), hence the assertion will failExample 2: The assigned epoch representation of the current time to
param1
will contain 13 digits and would match the regex (^\d{13}$
), hence the assertion will pass
Greater than - Validates that a parameter is greater than the given expression.
Example 1:
param1
is assigned with the value 31 that is not greater than 40 hence the assertion will fail.Example 2:
param1
is assigned with the value 31 which is greater than 30 will make the assertion to pass
Less than - Validates that a parameter is less than the given expression.
Example 1:
param1
is assigned with the value 30 that is not less than 31 hence the assertion will fail.Example 2:
param1
is assigned with the value 31 which is less than 40 will make the assertion to pass
JSON Schema - Validate that a parameter comply with given JSON Schema.
Example 1: Given
param1
was set with a json as below, opposed to the expected schema the field "name" is not numeric hence the assertion will fail.Note, the assertion will suggests to exclude the failing field from the validation schema.
Example 2: Given
param1
was set with a json as below and the expected schema describes it correctly, the assertion will pass.
JSON Contains - Validates that a json contains a subset json, in such way dynamic fields can be omitted to avoid false negatives.
Accepting any value for individual field will accept "*" notation Note, the subset json object should preserve the same hierarchal structure as the containing object while fields can be omitted.
Example 1: Given
param1
was set with a json as below, any mismatch in a key or value on same hierarchy will result in assertion error:Note to the "Exclude ..." notations, clicking on them will adjust the value of the assertion to exclude the failing field.
Example 2: Given
param1
was set with a json as below, any partial subset in json-contains validation value with same structure will result pass in assertion:- Using star notation to accept any value for
latitude
andcountry
fields.Note, the "*" notation works only on simple values (strings/numerics/booleans)
XML Contains - Validates that an xml contains a subset xml, in such way dynamic fields can be omitted to avoid false negatives.
Accepting any value for individual field will accept "*" notation Note, the subset xml object should preserve the same hierarchal structure as the containing object while fields can be omitted.
Example 1: Given
param1
was set with an xml as below, any mismatch in a key or value on same hierarchy will result in assertion error:Example 2: Given
param1
was set with an xml as below, any partial subset in xml-contains validation value with same structure will result pass in assertion:- Using star notation to accept any value for
latitude
andcountry
fields.Note, the "*" notation works only on simple values (strings/numerics/booleans)
You may embed parameters in any assertion expression. These parameters will be evaluated right before the assertion is executed.
See an example of the assertion that validates the generated ID is in the format of UUID below:

Suggestions
In many cases Loadmill users use the same or similar extractions and assertions. We've implemented the Suggestions feature that allows team admins to configure a repository of extractions and assertions within Settings - Suggestions.
Then, each user can add the assertions from the repository by clicking on + SUGGESTIONS.

By default, there are a few common extraction and assertion examples in the repository. Team admins can also navigate to the Suggestions Settings directly from within the suggestion dialog window.

Caveats
Keep in mind that all parameter values are textual, i.e. a parameter has no type such as Number
or Array
that we know from common programming languages.
This is important in order to avoid confusion when using parameter extractors such as JSONPath. For example, consider the following scenario:
Extract the value for
books
via the JSONPath querystudent.books
on{ "student": { "books": [] } }
Assert
books
is Not Empty.
You may expect this assertion to fail but, in fact, it will succeed. This is because the parameter books
is evaluated to []
and therefore is considered as a non-empty string. One possible way to correct this is to use a RegExp assertion on the books
parameter instead: \[[^\s]+\]
.
Last updated