Assertion Roulette doesn't mean that multiple assertions are bad.
When I coach teams or individual developers in test-driven development (TDD) or unit testing, I frequently encounter a particular notion: Multiple assertions are bad. A test must have only one assertion.
That idea is rarely helpful.
Let's examine a realistic code example and subsequently try to understand the origins of the notion.
Outside-in TDD
Consider a REST API that enables you to make and cancel restaurant reservations. First, an HTTP POST request makes a reservation:
POST /restaurants/1/reservations?sig=epi301tdlc57d0HwLCz[...] HTTP/1.1
Content-Type: application/json
{
"at": "2023-09-22 18:47",
"name": "Teri Bell",
"email": "terrible@example.org",
"quantity": 1
}
HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Location: /restaurants/1/reservations/971167d4c79441b78fe70cc702[...]
{
"id": "971167d4c79441b78fe70cc702d3e1f6",
"at": "2023-09-22T18:47:00.0000000",
"email": "terrible@example.org",
"name": "Teri Bell",
"quantity": 1
}
Notice that in proper REST fashion, the response returns the location of the created reservation in the Location header.
If you change your mind, you can cancel the reservation with a DELETE request:
DELETE /restaurants/1/reservations/971167d4c79441b78fe70cc702[...] HTTP/1.1
HTTP/1.1 200 OK
Imagine that this is the desired interaction. Using outside-in TDD you write the following test:
[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservation(
int days, int hours, int minutes,
string email, string name, int quantity)
{
using var api = new LegacyApi();
var at = DateTime.Today.AddDays(days).At(hours, minutes)
.ToIso8601DateTimeString();
var dto = Create.ReservationDto(at, email, name, quantity);
var postResp = await api.PostReservation(dto);
Uri address = FindReservationAddress(postResp);
var deleteResp = await api.CreateClient().DeleteAsync(address);
Assert.True(
deleteResp.IsSuccessStatusCode,
$"Actual status code: {deleteResp.StatusCode}.");
}
This example is in C# using xUnit.net because we need some language and framework to show realistic code. The point of the article, however, applies across languages and frameworks. The code examples in this article are based on the sample code base that accompanies my book Code That Fits in Your Head.
In order to pass this test, you can implement the server-side code like this:
[HttpDelete("restaurants/{restaurantId}/reservations/{id}")]
public void Delete(int restaurantId, string id)
{
}
While clearly a no-op, this implementation passes all tests. The newly-written test asserts that the HTTP response returns a status code in the 200 (success) range. This is part of the API's REST protocol, so this response is important. You want to keep this assertion around as a regression test. If the API ever begins to return a status code in the 400 or 500 range, it would be a breaking change.
So far, so good. TDD is an incremental process. One test doesn't drive a full feature.
Since all tests are passing, you can commit the changes to source control and proceed to the next iteration.
Strengthening the postconditions
You should be able to check that the resource is truly gone by making a GET request:
GET /restaurants/1/reservations/971167d4c79441b78fe70cc702[...] HTTP/1.1
HTTP/1.1 404 Not FoundThis, however, is not the behavior of the current implementation of Delete, which does nothing. It seems that you're going to need another test.
Or do you?
One option is to copy the existing test and change the assertion phase to perform the above GET request to check that the response status is 404:
[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservationActuallyDeletes(
int days, int hours, int minutes,
string email, string name, int quantity)
{
using var api = new LegacyApi();
var at = DateTime.Today.AddDays(days).At(hours, minutes)
.ToIso8601DateTimeString();
var dto = Create.ReservationDto(at, email, name, quantity);
var postResp = await api.PostReservation(dto);
Uri address = FindReservationAddress(postResp);
var deleteResp = await api.CreateClient().DeleteAsync(address);
var getResp = await api.CreateClient().GetAsync(address);
Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode);
}
This does, indeed, prompt you to properly implement the server-side Delete method.
Is this, however, a good idea? Is the test code easy to maintain?
Test code is code too, and you have to maintain it. Copy and paste is problematic in test code for the same reasons that it can be a problem in production code. If you later have to change something, you have to identify all the places that you have to edit. It's easy to miss one, which can lead to bugs. This is true for test code as well.
One action, more assertions
Instead of copy-and-pasting the first test, why not instead strengthen the postconditions of the first test case?
Just add the new assertion after the first assertion:
[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservation(
int days, int hours, int minutes,
string email, string name, int quantity)
{
using var api = new LegacyApi();
var at = DateTime.Today.AddDays(days).At(hours, minutes)
.ToIso8601DateTimeString();
var dto = Create.ReservationDto(at, email, name, quantity);
var postResp = await api.PostReservation(dto);
Uri address = FindReservationAddress(postResp);
var deleteResp = await api.CreateClient().DeleteAsync(address);
Assert.True(
deleteResp.IsSuccessStatusCode,
$"Actual status code: {deleteResp.StatusCode}.");
var getResp = await api.CreateClient().GetAsync(address);
Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode);
}
This means that you only have a single test method to maintain instead of two duplicated methods that are almost identical.
But, some of the people I've coached might say, this test has two assertions!
Indeed. So what? It's one single test case: Cancelling a reservation.
While cancelling a reservation is a single action, we care about multiple outcomes:
- The status code after a successful DELETE request should be in the 200 range.
- The reservation resource should be gone.
Developing the system further, we might add more behaviors that we care about. Perhaps the system should also send an email about the cancellation. We should assert that as well. It's still the same test case, though: Successfully cancelling a reservation.
There's nothing wrong with multiple assertions in a single test. The above example illustrates the benefits. A single test case can have multiple outcomes that should all be verified.
Origins of the single assertion notion
Where does the only one assertion per test notion come from? I don't know, but I can guess.
The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure.
It looks to me as though the only one assertion per test 'rule' stems from a misreading of the Assertion Roulette description. (I may even have contributed to that myself. I don't remember that I have, but to be honest I've produced so much content about unit testing over the decades that I don't want to assume myself free of guilt.)
xUnit Test Patterns describes two causes of Assertion Roulette:
- Eager Test: A single test verifies too much functionality.
- Missing Assertion Message
You have an Eager Test when you're trying to exercise more than one test case. You may be trying to simulate a 'session' where a client performs many steps in order to achieve a goal. As Gerard Meszaros writes regarding the test smell, this is appropriate for manual tests, but rarely for automated tests. It's not the number of assertions that cause problems, but that the test does too much.
The other cause occurs when the assertions are sufficiently similar that you can't tell which one failed, and they have no assertion messages.
That's not the case with the above example. If the Assert.True assertion fails, the assertion message will tell you:
Actual status code: NotFound.
Expected: True
Actual: False
Likewise, if the Assert.Equal assertion fails, that too will be clear:
Assert.Equal() Failure
Expected: NotFound
Actual: OK
There's no ambiguity.
One assertion per test
Now that you understand that multiple assertions per test are fine, you may be inclined to have a ball adding assertions like there's no tomorrow.
Usually, however, there's a germ of truth in a persistent notion like the one test, one assertion 'rule'. Use good judgement.
If you consider what an automated test is, it's basically a predicate. It's a statement that we expect a particular outcome. We then compare the actual outcome to the expected outcome to see if they are equal. Thus, in essence, the ideal assertion is this:
Assert.Equal(expected, actual);I can't always attain that ideal, but whenever I can, I feel deep satisfaction. Sometimes, expected and actual are primitive values like integers or strings, but they might also be complex values that represent the subset of program state that the test cares about. As long as the objects have structural equality, such an assertion is meaningful.
At other times I can't quite find a way to express the verification step as succinctly as that. If I have to add another assertion or two, I'll do that.
Conclusion
There's this notion that you're only allowed to write one assertion per unit test. It probably originates from real concerns about badly-factored test code, but over the years the nuanced test smell Assertion Roulette has become garbled into a simpler, but less helpful 'rule'.
That 'rule' often gets in the way of maintainable test code. Programmers following the 'rule' resort to gratuitous copying and pasting instead of adding another assertion to an existing test.
If adding a relevant assertion to an existing test is the best way forward, don't let a misunderstood 'rule' stop you.