Assertions - Verify Response
Last updated
Last updated
When testing an API, it is usually not enough to verify that all our HTTP requests completed successfully. Often, it is necessary to make sure that we got the correct response from the server by examining the response status, body or headers.
With Loadmill, this is made easy by using Assertions. Assertions are used in conjunction with parameters to do just that: examine the server's response and assert its correctness.
You may have an arbitrary number of assertions executed after each successful request. If an assertion fails, the next request will not be executed and the test scenario will be marked as failed, but the subsequent assertions for the current request will be executed nonetheless.
Assertions editor can be found in loadmill under each request cubical on it's expanded form.
It is also possible to create assertions directly from the response panel from the 'JSONPATH' window (only when the response is a JSON). After executing the test, click on one of the responses to expand it, which will open the response panel, as follows:
Now, simply click on the plus sign located in the top right corner, set the extraction, and add the desired assertion right from there!
You can then see both the extraction and the assertion in the request editor panel:
The target of an assertion is always a parameter value. You may use built-in parameters, default parameters or any parameter extracted from the current or previous requests in the current scenario as the target.
When using assertions, it will usually require to use extractions first to set the parameter with proper data and only then to attempt to validate it's form.
There are several types of assertions:
To prepare initial JSON schema out of a JSON you can search for 'from json to schema' in google or use a tool like this.
Accepting any value for individual field will accept "*" notation Note, the subset json object should preserve the same hierarchal structure as the containing object while fields can be omitted.
Accepting any value for individual field will accept "*" notation Note, the subset xml object should preserve the same hierarchal structure as the containing object while fields can be omitted.
You may embed parameters in any assertion expression. These parameters will be evaluated right before the assertion is executed.
See an example of the assertion that validates the generated ID is in the format of UUID below:
Default assertions - for any extraction we find in the recorded test, we add a default assertion to it (example: "id" exists).
Specific assertions - for any parameter with given user key (for example, "success") we extract it and assign to it an assertion (example: "success" equals "true").
Optional automatic JSON Schema assertion: you can enable this assertion within Settings - Recordings.
In many cases Loadmill users use the same or similar extractions and assertions. We've implemented the Suggestions feature that allows team admins to configure a repository of extractions and assertions within Settings - Suggestions.
Then, each user can add the assertions from the repository by clicking on + SUGGESTIONS.
By default, there are a few common extraction and assertion examples in the repository. Team admins can also navigate to the Suggestions Settings directly from within the suggestion dialog window.
Keep in mind that all parameter values are textual, i.e. a parameter has no type such as Number or Array that we know from common programming languages.
This is important in order to avoid confusion when using parameter extractors such as JSONPath. For example, consider the following scenario:
Extract the value for books via the JSONPath query student.books on
Assert books is Not Empty.
You may expect this assertion to fail but, in fact, it will succeed. This is because the parameter books is evaluated to [] and therefore is considered as a non-empty string. One possible way to correct this is to use a RegExp assertion on the books parameter instead: \[[^\s]+\].
Furthermore, there is an option to disable an assertion by clicking the three dots () in the assertion line. A disabled assertion will still be evaluated, but it won't cause the test to fail.
Click on , then navigate to the desired parameter. For example, let's find the 'content-length':
Example 1:
Assigning empty term to param1 will result in assertion failure
Example 2:
Assigning any value to param1 will result in passed assertion
Example 1:
Assigning any value to param1 will result in assertion failure
Example 2:
Not setting param1 entirely will result in passed assetion
Example 1:
The assigned param1 doesn't equals to loadmill hence will fail in assertion
Example 2:
Extracting $.args.app json path field to param1(from the result that is not displayed) should be equal to "Demo" and will result the assertion to pass
Assigning numeric value to param2 of 11.0 in postscript should be equal to "11", and the assertion will pass.
Note that all values are implicitly compared as strings.
Example 1:
param1 shouldn't be equals to 'demo' but it is, hence the assertion will fail
Example 2:
param1 doesn't equals to 'loadmill' hence the assertion will pass
Example 1:
When param1 set with the value 'loadmill-demo' and doesn't contains the string 'example' the assertion will fail.
Example 2:
When param1 set with the value 'loadmill-demo' and contains the string 'demo', the assertion will pass.
Example 1:
When param1 set with the value 'loadmill-demo' and contains the string 'demo', the assertion will fail.
Example 2:
When param1 set with the value 'loadmill-demo' and doesn't contain the string 'example', the assertion will pass.
Example 1:
The assigned epoch representation of the current time to param1 will contain 13 digits would not match the regex (^\d{10}$), hence the assertion will fail
Example 2:
The assigned epoch representation of the current time to param1 will contain 13 digits and would match the regex (^\d{13}$), hence the assertion will pass
Example 1:
param1 is assigned with the value 31 that is not greater than 40 hence the assertion will fail.
Example 2:
param1 is assigned with the value 31 which is greater than 30 will make the assertion to pass
Example 1:
param1 is assigned with the value 30 that is not less than 31 hence the assertion will fail.
Example 2:
param1 is assigned with the value 31 which is less than 40 will make the assertion to pass
Example 1:
Given param1 was set with a json as below, opposed to the expected schema the field "name" is not numeric hence the assertion will fail.
Note, the assertion will suggests to exclude the failing field from the validation schema.
Example 2:
Given param1 was set with a json as below and the expected schema describes it correctly, the assertion will pass.
Example 1:
Given param1 was set with a json as below, any mismatch
in a key or value on same hierarchy will result in assertion error:
Note to the "Exclude ..." notations, clicking on them will adjust the value of the assertion to exclude the failing field.
Example 2:
Given param1 was set with a json as below, any partial subset in json-contains validation value with same structure will result pass in assertion:
- Using star notation to accept any value for latitude and country fields.
Note, the "*" notation works only on simple values (strings/numerics/booleans)
Example 1:
Given param1 was set with an xml as below, any mismatch
in a key or value on same hierarchy will result in assertion error:
Example 2:
Given param1 was set with an xml as below, any partial subset in xml-contains validation value with same structure will result pass in assertion:
- Using star notation to accept any value for latitude and country fields.
Note, the "*" notation works only on simple values (strings/numerics/booleans)
When creating tests via our Chrome recorder extension, Loadmill will create two automatic assertion types for you:
