Test Driven Infrastructure with varnishtest

Infrastructure as code and continuous integration/continuous delivery (CI/CD) pipelines are becoming the norm for delivering software applications. By codifying infrastructure and leveraging automation, organizations can release changes faster and with more reliability.

A key practice in modern software delivery is test driven development (TDD), where tests are written before code. This ensures that the code is testable and aligns with requirements. The same TDD principles can be applied when managing infrastructure configurations.

In this article, we will look at how to build a CI/CD pipeline with test driven infrastructure configurations, using Varnish Cache and the varnishtest tool as a practical example.

What to expect from this post

This article aims to provide a guide on leveraging test driven development principles for infrastructure code and configurations, using examples of Varnish Cache's varnishtest. Specifically:

• The benefits of test driven infrastructure configurations and CI/CD pipelines
• An overview of Varnish Cache and how it is configured through VCL
• Introduction to the varnishtest tool for testing Varnish logic
• Best practices for writing robust varnishtest test cases
• How to effectively incorporate varnishtest in a CI pipeline
• Conclusion on how Varnish Cache and varnishtest enable test driven infrastructure.

Overview

Varnish Cache is a popular open source web application accelerator. It can speed up websites by caching content close to visitors. The Varnish configuration determines its exact behavior and is described in VCL (Varnish Configuration Language).

To validate that configuration changes in Varnish VCL work as intended, the varnishtest tool can be used. It is part of Varnish Cache and allows writing integration tests that interact with a Varnish instance to test scenarios and expectations.

By incorporating varnishtest in an automated CI/CD pipeline, infrastructure changes can be validated before they are deployed to production environments. The workflow would look like this:

1. A developer makes a change to the Varnish VCL configuration file
2. They write an integration test with varnishtest that validates the expected behavior
3. A pull request is opened with the VCL change and test
4. In the CI pipeline, the varnishtest is executed against the new VCL
5. If the test passes, the change can be reviewed and merged
6. In the CD pipeline, the VCL change is deployed after passing all tests

This ensures infrastructure changes are thoroughly tested before they are deployed. The varnishtest tool essentially allows test driven development for infrastructure configurations.

In the rest of this article, we will explore varnishtest and Varnish in more detail.

Introduction to Varnish Cache

Varnish Cache is an HTTP accelerator designed for speeding up websites and APIs. It can store cacheable content in memory, closer to your visitors. This reduces latency and takes load off your web servers.

Some typical use cases for Varnish include:

• Caching static assets like images, CSS and Javascript files
• Caching API responses
• Offloading traffic from application servers
• Implementing edge logic like routing and access control lists

Varnish is often deployed as a reverse proxy. Visitors send requests to Varnish, which forwards cache misses to application servers in the backend.

To determine the exact behavior of Varnish, you configure it with VCL (Varnish Configuration Language). It describes how Varnish should handle requests and responses.

Some examples of VCL functionality:

• Defining when content can be cached
• Selecting which backend server to route requests to
• Transforming requests and responses
• Implementing access control lists

Changes in VCL can introduce bugs or incorrect behavior. That's why testing the VCL is important.

Introducing varnishtest

The varnishtest tool is part of Varnish Cache and makes it easy to test VCL programmatically. It allows writing integration tests for your Varnish configuration.

The varnishtest tool performs the following:

• It starts a Varnish test instance with the VCL configuration under test
• HTTP clients can send requests to the test Varnish and receive responses
• Conditions and expectations can be asserted

This validates that the VCL logic results in the correct Varnish behavior.

A test case typically interacts with the following components:

• Varnish instance
• Backend server
• HTTP client

These are declared in VTC files. VTC stands for Varnish Test Case and uses a domain specific language.

Here is an example of a Varnish Test Case:

varnish v1 -vcl+backend {
  # VCL configuration
} -start

server s1 {
  # Handles requests
} -start  

client c1 {

  # Send request
  txreq

  # Receive response
  rxresp

  # Validate response
  expect resp.status == 200

} -run

This starts a Varnish instance v1 with the provided VCL, as well as a backend server s1. The client c1 sends a request and validates that the response status code is 200 OK.

Multiple requests can be executed and expectations can be validated. This allows testing a variety of scenarios.

Running varnishtest will execute the test case and report if any expectations fail:

$ varnishtest example.vtc
Varnish test case example.vtc passed!

varnishtest allows test driven development of infrastructure configurations. By writing the test case upfront, you can validate that intended behavior is implemented correctly.

Now let's look at how to leverage varnishtest in a CI/CD pipeline.

Building a CI/CD pipeline

To prevent defects in production, it's important to test infrastructure configuration changes before deploying them. By incorporating automated testing in a CI/CD pipeline, we can enable test driven infrastructure.

To setup a CI/CD tool for testing Varnish config changes with varnishtest the basic workflow will look like this:

When a pull request is opened with changes to VCL and varnishtest files, the workflow will:

1. Check out the code
2. Run varnishtest to validate the VCL logic
3. Publish the test results
4. Notify about the test outcome

This gives fast feedback on whether the infrastructure change works as expected.

Let's walk through the workflow step-by-step.

1. Check out code

The first job in the workflow checks out the code from the pull request. This makes the changed files available where the next steps will run.

2. Run varnishtest

Next up, varnishtest is executed to validate the VCL logic:

varnishtest -q varnishtest.vtc 

The Varnish package is installed and then varnishtest is called to run the integration test suite in quiet mode.

If any expectations fail, varnishtest will return a non-zero exit code that fails the step.

3. Publish test results

To get visibility into the test execution, the results are published.

4. Notify about test results

Finally, a comment is posted on the pull request to notify about the test outcome:

This will post a message on Slack whenever a pull request is made. The if: always() ensures it runs even when the tests fail, so developers are notified.

That concludes the CI workflow! It runs varnishtest to validate Varnish config changes and provides automated feedback through published test results and Slack notifications.

Now let's look at implementing test cases with varnishtest in more detail.

Writing varnishtest cases

To get the most out of varnishtest for test driven infrastructure, it's important to understand how to write robust test cases.

Some best practices for writing good varnishtest tests:

Test a single use case

Each test case should focus on one specific use case to make them short and maintainable.

Validate expected behavior

Use assertions to validate that Varnish responds as expected when hitting a test endpoint.

Isolate test data

Unique data should be used in each test case. This prevents cascading failures when a test pollutes Varnish state.

Simulate edge conditions

Write test cases for edge scenarios like cache expiration, failures, irregular input etc.

Start simple

Begin with a simple straight forward test and build from there. Don't make complex test cases from the start.

Reuse test logic

Create reusable test building blocks like helper functions for common test logic.

Let's look at an example test case that implements these best practices.

Cache control test

Here is a varnishtest file that tests cache control behavior:

varnish v1 -vcl+backend {
  # VCL configuration
} -start

varnish v2 -vcl+backend {
  # VCL configuration
} -start

backend b1 {
  .host = "test-backend";
} 

backend b2 {
  .host = "test-backend";
}

server s1 {
  rxreq {
    expect req.url == "/test1"
  }
  
  txresp {
    set resp.http.Cache-Control = "public, max-age=10";
  }
} -start

vtc_begin_test {
  txreq -url "/test1"
  rxresp
  expect resp.http.Cache-Control == "public, max-age=10"
} -run

vtc_begin_test {
  # Wait for cache to expire
  sleep(11)
  
  txreq -url "/test1" 
  rxresp
  expect resp.http.Cache-Control == "public, max-age=10"
} -run

It tests the following scenarios:

Cacheable response

The first test validates that the Cache-Control header is set to public with 10 second max-age.

Cache expiration

The second test waits 11 seconds so the cache expires and sends a new request to verify the Cache-Control header is still set.

This is a focused test case that validates the cache control behavior with a simulated edge condition. The tests are isolated by the generated URL. And it uses the reusable vtc_begin_test {} helper.

By following varnishtest best practices, robust test cases can be written to test drive infrastructure configuration delivery.

Executing varnishtest in CI

Now let's look at how to incorporate varnishtest in a CI/CD pipeline for automated testing. Here are some best practices when running varnishtest in CI:

Install Varnish

Use the same method of installing Varnish as is used to deploy Varnish to produciton. Also ensure that the same version is deployed as will be deployed in the same delivery cycle to production.

Quiet mode

Use the -q option for quieter test output without color coding and progress indicators. This eliminates spurious CI/CD job output that makes it hard for engineers to read when things go belly up.

Fail fast

Make sure to set the CI job fails if varnishtest returns a non-zero exit code.

Parallelization

The -j option allows running test cases in parallel to speed up execution. Be careful that your tests can be run in parallel from a correctness perspective. You might need to separate tests that can be run in parellel to run in one job and the sequential tests in another. These different jobs can then be run in paraellel as separate jobs.

Artifacts

Persist the varnishtest logs as an artifact to debug any failed tests.

Smoke tests

Have a fast running smoke test suite to quickly validate across PRs and commits. And a full regression suite for more thorough validation before merging PRs.

Conclusion

In this article, we looked at how to implement test driven infrastructure configurations with Varnish Cache and varnishtest:

• Varnish Cache accelerates websites by caching content at the edge. Its exact behavior is determined by VCL configuration.
• varnishtest allows writing integration test cases that validate Varnish functionality and VCL logic.
• A CI/CD workflow with varnishtest testing provides automated feedback on infrastructure changes before deploying them.
• Following best practices for writing varnishtest cases ensures robust test coverage.

Adopting test driven infrastructure brings many benefits:

• Reliable and rapid releases by testing infrastructure code upfront
• Prevent production defects by validating configurations
• Improved collaboration between developers and operations
• Documentation of intended behavior through test cases

As infrastructure as code practices grow, having the testing tools to validate these programmable configurations is key. Varnish Cache and varnishtest are great examples of how test driven infrastructure can be implemented for fast and robust delivery of infrastructure changes.

References

• Varnish Cache
• varnishtest
• Varnish Configuration Language