WSO2 API Manager 4.0 | How to perform A/B Testing

In this post we are going to implement a use case where an organisation wants to A/B test an API to see which API backend get more traction compared to the other.

There are many ways to achieve above use case using WSO2 API Manager but in this post we are going to use an API with dynamic backend endpoint with a custom mediation to route the request to a particular backend based on a request header to achieve this scenario. Then we can utilise API Analytics to determine which backend API gets more requests.

Let’s start with configuring the API.

1. Start WSO2 API Manager.
2. Create an API using provided[1] open api definition(openapi-train-operations.yaml) using Import Open API option. Here leave the endpoint empty.

Create API

3. Click the created API and go to the Endpoints section and select Dynamic Endpoints and click Save.

Endpoints Section

4. Go to the Runtime section and import the ABPolicy.xml[1] as a Request mediation policy.

Custom Mediation Policy

5. Select proper Business Plan.

Select Business Plans

6. Deploy the API.

Deployments

7. Publish the API.

Publish API

8. Login to the developer console.

9. subscribe to an Application and generate access token.

Subscribe to Application

8. Invoke the API using postman without header ABPolicyHeader:A

Without Header

9. Invoke the API using postman with header ABPolicyHeader:A

With Header

As you can see, the response is different(additional attribute: runBy) in two occasions since the request is now redirecting to different backends.

Sample Policy to handle above case is as follows.

ABPolicy.xml(Mediation Policy)

Now when checking the API Analytics the users can determine which backend API has been prefered.

All the artefacts related to trying out above use can be found below link. You can deploy the backend(trainop.war) in a local tomcat server.

[1] https://drive.google.com/drive/folders/1QBzj44YvZlEkyy7EaLG3Gd5saAIObfCt?usp=sharing