Functional Integration Testing with Postman

I previously wrote about my initial experiences working with Azure Functions during a summer project earlier this year.

See Azure Functions - first encounter

As great of an experience serverless and Azure Functions might provide, I still had some major obstacles I found hard to overcome. One being able to writing good tests, verifying the behavior of the functions as a whole. Cloud storage abstractions are injected as function parameters and all methods are static. This makes the task of writing maintainable and testable code - different, for lack of a better phrase.

There are many things one can do, but not yet a common "best practice", and working with a time limited project, writing tests that do not depend on hosting type or coding style was the best option. That can be done using postman's built in testing framework to verify functionality by actually issuing HTTP requests to the function app. Having a limited set of functional tests like this is a great way of verifying behavior independent of implementation and infrastructure. The test suit should emulate the behavior of an actual client, verifying that the current running version satisfies the current version of the client code. Being independent of the rest of the code is valuable, but also the greatest weakness of these kinds of tests, as they quickly become hard to maintain. Essentially one has to remember to update the tests when updating the client behavior, else the tests might pass while the client breaks.

In the mentioned project the client/server interaction was quite stable, changes consisting mainly of added functionality, making maintainability of the automatic tests less of a problem. This is typical for e.g. mobile applications as you most of the time cannot expect the users to update the client after installation, thus having to limit breaking changes in the back-end.

Application Behavior

The application under consideration is a mobile application which includes a loyalty program where users register purchases and eventually are allowed to redeem a free item.

The function app has the following functions in need of testing:

  • RegisterPurchase(GUID, UserId)
  • FetchUserPurchases(UserId)
  • RedeemBonus(UserId)
  • RegisterEmployee(Name)

The user will fetch purchases which will be visualized in the app, and purchases are registered through scanning a QR-code. Consecutive calls to the register function for the same userId should be rejected, and redeeming bonuses should only be allowed if enough purchases are registered. Each purchase is connected to an employee. New employees can be added through the RegisterEmployee-function, which is necessary as we don't want our tests to be dependent on existing data to run.

The standard interaction which we would like to define with tests, consists of the following steps:

  1. Register a new employee and keep the id as reference
  2. Register purchase with the previously created employee id
  3. Register another purchase without waiting - this should be rejected
  4. Wait for a configurable interval, then register another purchase
  5. Fetch purchases for the user, verify the number of active purchases
  6. Try to redeem a bonus- should be rejected
  7. Register 5 more purchases, waiting between each operation
  8. Redeem bonus
  9. Fetch active purchases again - should be back to zero

These steps showcase the whole data and business logic of the application, and having tests to verify and document it is very useful. The flow involves several functions and storage tables, which might have made it hard to define in other types of tests.

Functional Tests

Functional tests are simply tests that verify functionality. Where other types of tests verify units of code, integrations, or behavior of internal components, these tests run with a higher level of abstraction. In this example the tests issue HTTP requests chained together via the response payloads, without any knowledge about framework or storage. This is valuable both as documentation and to increase the chance of catching mistakes early. When relying solely on tests defined in the same solution/code repository as the business logic, it is easy to break the existing logic simply through refactoring. Refactoring tools can help you make such mistakes e.g. by automatic name changes or method signature updates, which will update your tests simultaneously, possibly breaking some contract or some setting not directly covered by tests.

The con with tests residing and running elsewhere than the rest of the code is that maintainability will quickly be a concern. Also when creating tests using Postman, which will be showcased shortly, the definition is in Json and requires the postman application itself to be edited. This makes it expensive and cumbersome to update and maintain. It is thus recommended to limit such tests to cover only the most important logic that rarely change. In the example presented it is quite unlikely that any of the steps will change, at least not intentionally.

Postman Test Chains

Postman is a widely used tool for manually issuing HTTP requests during api development and testing. Requests can be stored in collections and parameterized to quickly change between "environments". Postman also has some built-in parameters like $guid which can be used in the requests, together with environment variables.

Screen-Shot-2018-09-13-at-21.32.18

Additionally you can write javascript code to make assertions on the responses which will alert you when failing. After verifying the response, you can store parts of the response in variables which will be available to other tests in the same collection. In the following example, we check that the response code is 201 and that a given property is of correct length, then create a new variable based on data from the response.

pm.test("Status code is 201", function () {
    pm.response.to.have.status(201);
});

var responseBodyJson = pm.response.json();

pm.test("Expect new row key", function () {
    pm.expect(responseBodyJson.RowKey.length).to.be.above(10);
});

pm.environment.set("newQrGuid", responseBodyJson.QrCode);

Now we can use this variable in another request in the same collection. The next request uses the variables set previously to issue a new request and tests verify the response again.

Screen-Shot-2018-09-13-at-21.35.46

Chaining requests with tests attached, we can define complete application user scenarios testing the manipulated data etc. Tests in the same "collection" can be run together using the collection runner.

Screen-Shot-2018-09-13-at-22.03.43

The tests can also be exported and run from the command line using the npm package newman. Exporting the collection and the environment from the postman client results in a json file that can be checked in and used in build scripts. One way of managing this and the newman dependency is to create a directory containing a package.json with the following content together with the collection file from postman.

{
  "name": "api-tests",
  "version": "1.0.0",
  "description": "api-tests",
  "main": "index.js",
  "scripts": {
    "testapi": "newman run integration-tests.postman_collection.json -e test.postman_environment.json",
  },
  "devDependencies": {
    "newman": "^3.9.3"
  },
  "author": ""
}

Running npm install && npm run testapi will run the tests and output the results.

Integration Testing

→ RegisterEmployee - succsessfully create new employee
  POST https://my-functions-uri.net/api/RegisterEmployee [201 Created, 480B, 5.5s]
  ✓  Status code is 201
  ✓  Expect new row key

→ FetchUserPurchases  - Verify empty on first load
  GET https://my-functions-uri.net/api/FetchUserPurchases/d117603a-fd8e-4005-82cc-767adf94e97e [200 OK, 344B, 202ms]
  ✓  Status code is 200
  ✓  No purchases registered on initial load

→ RegisterPurchase - successfully register first purchase
  POST https://my-functions-uri.net/api/RegisterPurchase [201 Created, 673B, 922ms]
  ✓  Purchase to be successfully registered

→ RegisterPurchase - consecutive purchase fails
  POST https://my-functions-uri.net/api/RegisterPurchase [400 Bad Request, 206B, 79ms]
  ✓  Purchase to be rejected
  ✓  Response to have successfull text

→ RegisterPurchase - successfully registration when waiting
  POST https://my-functions-uri.net/api/RegisterPurchase [201 Created, 673B, 75ms]
  ✓  Purchase to be successfully registered

→ RegisterPurchase - 8. purchase
  POST https://my-functions-uri.net/api/RegisterPurchase [201 Created, 673B, 139ms]
  ✓  Purchase to be successfully registered

→ ClaimBonus - succeedes
  PUT https://my-functions-uri.net/api/ClaimBonus/d117603a-fd8e-4005-82cc-767adf94e97e [200 OK, 605B, 100ms]
  ✓  Should reject bonus request

→ FetchUnredeemedPurchases - 1 unreedeemed purchase left
  GET https://my-functions-uri.net/api/FetchUnredeemedPurchases/d117603a-fd8e-4005-82cc-767adf94e97e [200 OK, 747B, 121ms]
  ✓  Status code is 200
  ✓  Should have one purchase registered

┌─────────────────────────┬──────────┬──────────┐
│                         │ executed │   failed │
├─────────────────────────┼──────────┼──────────┤
│              iterations │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│                requests │       15 │        0 │
├─────────────────────────┼──────────┼──────────┤
│            test-scripts │       15 │        0 │
├─────────────────────────┼──────────┼──────────┤
│      prerequest-scripts │        0 │        0 │
├─────────────────────────┼──────────┼──────────┤
│              assertions │       20 │        0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 1m 58.4s                  │
├───────────────────────────────────────────────┤
│ total data received: 5.75KB (approx)          │
├───────────────────────────────────────────────┤
│ average response time: 523ms

Wrap Up

Using functional tests in this fashion is useful during development, but also as tests run as part of a CI/CD pipeline. Especially to be able to refactor how functions interact or change storage, it is important to have tests that verify the flow and mutation of data end-to-end. Keep the postman test suite limited to the most important functionality might be a good idea, as maintaining the test can be a bit inconvenient.

Read more about using postman for testing of APIs in this blogg post at the postman websites.