Approval Testing your Open API/Swagger Documents

The team I work in at Just Eat have a number of HTTP based APIs which are consumed by other components, some in internally others externally. Like a lot of people building HTTP based APIs we use Swagger for both developer experimentation and documentation of the endpoints exposed. This is done via the Swagger UI, which uses an Open API document (formally Swagger docs) that describes the endpoints and resources in JSON form.

As our APIs evolve it’s imperative that we don’t unintentionally break any of these public contracts as this would cause headaches for consumers of our service.

What we need is visibility of intentional or unintentional changes to this contract, this is where approval testing has helped.

What is approval testing?

Approval testing is a method of testing that not many are familiar with. If I were to hypothesise why this is, I’d say it’s because of it narrow application, especially when contrasted to other more common forms of forms of testing.

Assertion testing

We’re all familiar with assertion based testing - you order your test in an arrange, act, assert flow where you first ‘arrange’ the test, then execute the action (act) then ‘assert’ the output.

For instance:

// Arrange
var child1 = 13;
var child2 = 22;

// Act
var age = calculateAge(13, 22);

// Assert
age.ShouldBe(35);

Approval testing

Approval testing follows the same pattern but with one difference; instead of asserting the expected return value given your set of input parameters, you ‘approve’ the output instead.

This slight shift changes your perspective on what a failing test means. In other words, with Approval Testing the failed test doesn’t prove something has broken but instead flags that the given output differs from the previous approved output and needs approving.

Actions speak louder than words so let’s take a look at how we can apply this method of testing to our Open API documents in order to gain visibility of changes to the contract.

Test Setup

In this example I’ll use a simple .NET Core based API that has Swagger setup with Swagger UI. The test will use the Microsoft.AspNetCore.Mvc.Testing package for running our API in-memory. If you’re not familiar with testing this way then be sure to check out the docs if you wanted to try this yourself.

First let’s take a look at our application:

Our ASP.NET Core Application

// ProductController.cs 

[Route("api/[controller]")]
[ApiController]
public class ProductController : ControllerBase
{
    [HttpGet]
    public ActionResult<IEnumerable<ProductViewModel>> Get()
    {
        return new List<ProductViewModel>
        {
            new ProductViewModel {Id = 1, Name = "Product 1"},
            new ProductViewModel {Id = 2, Name = "Product 1"},
            new ProductViewModel {Id = 3, Name = "Product 1"}
        };
    }
}

// Program.cs

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }
    
    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
        .UseStartup<Startup>();
}

// ProductViewModel.cs

public class ProductViewModel
{
    public string Name { get; set; }

    public int Id { get; set; }

    public string Description { get; set; }
}

// Startup.cs

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        ...

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "My API", Version = "v1" });
        });
        
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        ...
        
        app.UseSwagger();
        app.UseSwaggerUI(c =>
        {
            c.SwaggerEndpoint("/swagger/v1/swagger.json", "My API V1");
        });
    }
}

If we were to launch our API and go to /swagger/v1/swagger.json we should see a json document that includes information about the resources our API exposes.

Now we have a working API, let’s look at our test setup.

Our Test Setup

As mentioned above, our test will use the Microsoft.AspNetCore.Mvc.Testing package for running our API in-memory. Why do we need to run it in-memory? Hold that thought and read on.

We’ll use a simple xunit fixture class that uses WebApplicationFactory<T> to run our API in memory.

// ApiFixture.cs

public class ApiFixture
{
    private readonly WebApplicationFactory<Startup> _fixture;

    public ApiFixture()
    {
        _fixture = new WebApplicationFactory<Startup>();
    }

    public HttpClient Create()
    {
        return _fixture.CreateClient();
    }
}

Now we’ve got our API and infrastructure setup, this is where the magic happens. Below is a simple test that will perform the following:

1. Run our ASP.NET Core API in memory and allow the tests to call it via an HttpClient.
2. Make a GET request to the Open API document (Swagger docs) and parse the content into a string.
3. Call .ShouldMatchApproved() on the string content.

What is ShouldMatchApproved()? Read on…

// OpenApiDocumentTests.cs

public class OpenApiDocumentTests : IClassFixture<ApiFixture>
{
    private readonly ApiFixture _apiFixture;

    public OpenApiDocumentTests(ApiFixture apiFixture)
    {
        _apiFixture = apiFixture;
    }
    
    [Fact]
    public async Task DocumentHasChanged()
    {
        // Arrange
        var client = _apiFixture.Create();

        // Act
        var request = await  client.GetAsync("/swagger/v1/swagger.json");
        var content = await request.Content.ReadAsStringAsync();
        
        // Assert
        content.ShouldMatchApproved();
    }
}

Shouldly and .ShouldMatchApproved()

Shouldly is an open-source assertion testing framework that I frequently use (and as a result, contribute to) that simplifies test assertions.

One assertion method Shouldly provides that sets it apart from others is the .ShouldMatchApproved() extension which hangs off of .NET’s string type. This extension method enables us to easily apply approval based testing to any string. There are other libraries such as ApprovalTests.Net that support more complex use cases, but for this example Shouldly will suffice.

How does .ShouldMatchApproved() work?

Upon executing your test .ShouldMatchApproved() performs the following steps:

1. First it checks to see if an approved text file lives in the expected location on disk.

2. If the aforementioned file exists Shouldly will diff the contents of the file (containing the expected) against the input (the actual) of the test.

3. If ShouldMatchApproved detects a difference between the expected and actual values then it will scan common directories for a supported diff tool (such as Beyond Compare, Win Merge etc, or even VS Code).

4. If it finds a supported difftool it will automatically launch it, prompting you to approve the diffs and save the approved copy to the location described in Step 1.

5. Once approved and saved you can rerun the test and it will pass.

Any changes to your API from this point on will result in a test pass, providing no difference is detected between your test case and the approval file stored locally. If a change is detected then the above process starts again.

One of the powerful aspects of this method of testing is that the approval file is committed to source control, meaning those diffs to the contract are visible to anyone reviewing your changes, whilst also keeping a history of changes to the public contract.

Demo

Along side the source code to the demonstration in this post and the below video, I’ve published quick demonstration on YouTube which you can take a look at. The video starts by first creating our approved document, I then go on to make a change to a model exposed via the Open API document, approve that change then rerun the test again.