unit tests Archives - CodeJourney.net

What is Mutation Testing and Why Should You Use It?

Dawid Sibiński — Tue, 01 Oct 2024 18:22:58 +0000

Do you know what is a better testing metric than code coverage? Does 100% code coverage mean your code is well-tested? How to generate unit test cases for free, without using AI?

The answer to all these questions is mutation testing. Interested? Read on

Code Coverage – Our Old Friend

You probably know what code coverage is. Traditionally, many teams tend to measure how good their tests’ suite is by percentage line / statements / functions coverage. Some even require this metric to be 100%. Let’s start with this well-known approach first.

We are working with TicketsPriceCalculator class today. It contains some simple business logic:

	export class TicketsPriceCalculator {
	calculateTotalTicketsPrice(tickets: Ticket[]): number {
	if (tickets.length === 0) {
	return 0;
	}

	let totalPrice = tickets.reduce((sum, ticket) => sum + ticket.price, 0);
	const businessClassTickets = tickets.filter(
	(ticket) => ticket.isBusinessClass
	).length;

	let totalDiscountToApply = 0;

	// Apply discounts based on total price
	if (totalPrice > 2000) {
	totalDiscountToApply += 0.25;
	} else if (totalPrice > 1000 && totalPrice <= 2000) {
	totalDiscountToApply += 0.2;
	} else if (totalPrice > 500 && totalPrice <= 1000) {
	totalDiscountToApply += 0.15;
	} else if (totalPrice >= 200 && totalPrice <= 500) {
	totalDiscountToApply += 0.1;
	}

	// Apply additional discount based on number of tickets
	if (tickets.length >= 10) {
	totalDiscountToApply += 0.05;
	}

	// Apply business class discounts
	if (businessClassTickets >= 10) {
	totalDiscountToApply += 0.2;
	} else if (businessClassTickets >= 5) {
	totalDiscountToApply += 0.1;
	}

	return totalPrice * (1 – totalDiscountToApply);
	}
	}

view raw ticketsPriceCalculator.ts hosted with ❤ by GitHub

As you can see, the class has one function whith returns the total price to be paid for a set of tickets. On the way, it applies various discounts based on some business rules. Fair enough.

It’s a critical piece of business logic for the app we’re working on, but it has never had any tests. We were asked to write unit tests for this class. Our manager strictly requires the code coverage to be 100%. Let’s do it then. We will go line by line, analyze the business logic and add the tests. Below, you can see the results of running those tests (you can find the tests code here):

Wow, that’s awesome! We did a great job. We have 8 unit tests which provide 100% statements, branch, functions and lines coverage!

So now we are really safe, aren’t we?

Yeah… until you know what mutation testing is

What Is Mutation Testing?

The idea of mutation testing is to mutate (change) our code. Mutation testing tools introduce changes into our production code. These changes can be really anything – from reverting a condition of an if statement, to changing some operators or changing a string value. For instance, here are some logical mutations Stryker Mutator is able to introduce (you can find the list of all mutations supported by Stryker here):

In effect, they generate modified versions of our code, which are called mutants. Makes sense

Each mutant contains a single modification in our source code. What happens next is crucial. Mutation testing tool takes this mutant (it is used as the code under tests now) and executes all of our unit tests against it. The result may be twofold:

all tests have passed. In other words, the mutant has survived. This is a bad situation. It implies that the modification of our source code was not detected by our tests
at least one test has failed. In other words, the mutant has been killed. This is a good scenario. It means that our tests detect changes in the production code and efficiently cover this particular scenario.

The flow is more or less like this schema from Peter Evans’ blog presents:

But why would anyone change the production code on purpose by introducing some stupid changes? It may seem counterintuitive for now, but let’s see how we can use it to our leverage.

Mutation Testing with Stryker Mutator

Let’s clearly state what we are starting with:

8 unit tests, 100% code coverage

Let’s now run mutation testing on our code. We are using TypeScript here, so let’s go with Stryker Mutator.

Here’s the result of running Stryker with npx stryker run on our code:

As you can see, 53 mutants were killed, but 15 of them survived. This gives us a mutation score of 78%. Let’s call it mutation coverage from now on. So to make it clear, this is our starting situation:

8 unit tests, 100% code coverage, 78% mutation coverage

Let’s now see how to kill those remaining mutants.

Killing First Mutants

In order to kill the mutants and improve our mutation score, we need to write more unit tests. Sounds logical, doesn’t it?

If still not, let’s take a deeper look at mutation tests results in VS Code. Stryker lists details of every survived mutant, for example this one:

Stryker changed the greater or equal (>=) operator to greater operator (>). This is a very common and very useful mutation. It usually means that our tests don’t cover the edge case associated with this particular condition.

In this case, the edge case value seems to be 5. More precisely, what happens if the number of business class tickets is exactly 5? We don’t know, because there is no test for this particular edge case!

Let’s add it:

	test("should apply 10% discount if there are exactly 5 business class tickets", () => {
	const tickets = Array.from({ length: 5 }, () => ({
	price: 10,
	isBusinessClass: true,
	}));

	const totalPrice = calculator.calculateTotalTicketsPrice(tickets);
	expect(totalPrice).toBe(45); // 10% discount
	});

view raw ticketsPriceCalculator_edgeCase.test.ts hosted with ❤ by GitHub

The test passes. Let’s run mutation tests again:

That’s it! We have a new unit test and 1 more mutant killed Current state:

9 (+1) unit tests, 100% code coverage, 79% mutation coverage

I think you’re now getting what this is all about. We should be adding unit tests to cover the edge cases detected by mutations until all (or most) mutants are killed.

Mutation Testing – Free Unit Tests Generator

After adding 1 more test, the results look as follows:

10 (+2) unit tests, 100% code coverage, 86% mutation coverage

Finally, this is what we managed to get after a few minutes of work:

And we now have 17 unit tests in our suite! The final score looks like that:

17 (+9) unit tests, 100% code coverage, 96% mutation coverage

Notice what happened here. We started with 8 unit tests and finished with 17! This is awesome! That’s how mutation testing is a free unit tests generator. And you don’t even need AI for that

What’s more, the generated mutations even forced me to make a little tweak to the production code. It exposed some things that didn’t make sense. I could make this change to the production code, because my tests’ suite was already quite substantial.

Note that I didn’t push to gain 100% mutation coverage. The 3 mutants that survived don’t make much sense in our case. I encourage you to get the code and execute npx stryker run in the application folder to see for yourself

The percentage value doesn’t matter in that case. What matters is that we got 9 new unit tests and our tests’ suite is more robust. Now we can go and tell our manager that the business-critical code is well tested

So What Now?

I hope you see how valuable mutation testing is. It’s not a perfect tool that suits all cases. For example, it’s not easy to use mutation testing for complex legacy code. However, if your logic is nicely isolated, it could be a great tool. Especially if your production code heavily depends on calculations and many conditions. You may be surprised how many edge cases mutation testing can find.

I also love using mutation testing with TDD. These two techniques nicely complement each other.

I encourage you to go and try mutation testing. There are different mutation tools for various programming languages:

C#, JavaScript, TypeScript, Scala: Stryker Mutator
Ruby: mutant
Java: pitest
PHP: Infection
C++: Mutate++

…and probably many more. Just do some research for the language you work with and give it a try Because testing is cool, isn’t it?

You can find the full source code used in this article here.

The post What is Mutation Testing and Why Should You Use It? appeared first on CodeJourney.net.

Write Test Progress To The Console With NUnit

Dawid Sibiński — Sun, 19 Jun 2022 13:07:28 +0000

I recently needed to write test progress to the console with NUnit. The task we want to solve here is basically the TODO part of this snippet:

	[TestFixture]
	public class MyTests
	{
	[Test]
	public void SampleTest()
	{
	// some operations here, like starting a server for tests in-memory…

	// TODO: inform the 'user' (the one who runs the test) that the server is already running
	while (true)
	{
	// keep it running on purpose (e.g. for E2E tests)
	}

	}
	}

view raw NUnit_test_progress_todo.cs hosted with ❤ by GitHub

The title picture of this article shows the end result. If you want to know the solution, keep reading

Context

The context of this need is very simple. I have a unit test in which I want to perform some operations and keep the test running indefinitely afterwards. The use case might be instantiation of an in-memory test server. After that, I want to inform the user (programmer or the one running the test) that the server is up and running. I want to do that before the test finishes. In other words: I want to write the test’s output while it’s still running.

In my case, I was running two .NET Core apps in-memory. The backend server app and the ASP.NET Core web application. In the process, the web app got its IP address assigned dynamically. Meaning that the IP was different with each test run. This particular test was used to run the app in-memory and let the user manually use the application. It means the user must know the IP/URL of the in-memory web app. That’s why I needed to output the dynamic IP address into the NUnit test output

Solution

The solution to write test progress to the console with NUnit is very simple. To do that, use TestContext.Progress.WriteLine(string) method from NUnit.Framework namespace:

	[TestFixture]
	public class MyTests
	{
	[Test]
	public void SampleTest()
	{
	// some operations here, like starting a server for tests in-memory…

	TestContext.Progress.WriteLine("The server is running now! You can reach it on https://localhost:8067/");
	while (true)
	{
	// keep it running on purpose (e.g. for E2E tests)
	}

	}
	}

view raw NUnit_test_progress.cs hosted with ❤ by GitHub

And that’s it! I’m publishing this simple solution, because it took me some time to find

I previously tried with TestContext.Out.WriteLine, System.Diagnostics.Trace.WriteLine and System.Console.WriteLine, but all of them output the text after the test finishes.

Finally, you can also run the test from cmd using dotnet test:

Hope it’s useful!

The post Write Test Progress To The Console With NUnit appeared first on CodeJourney.net.

Improve your tests with Assert Object Pattern

Dawid Sibiński — Mon, 09 Nov 2020 20:42:34 +0000

Today I’d like to share with you a very interesting concept in software testing – Assert Object pattern. It makes the Assert part of a test much simpler and more readable. Let’s dive right into it

The Asserts Hell

Let’s consider the following unit test:

What’s wrong here? Given and When sections are great single-liners. We know straightaway what’s the input and the action executed. However, Then block is too complex. It’s hard to figure out, just passing quickly through this test, what is expected. This case is even not that bad thanks to the usage of FluentAssertions.

I spend a lot of time writing tests (unit or integration), but I spend even more time reading them. I always hope that I will find these idyllic tests that act as code documentation…

However, let’s go step by step. I recently took place in a software testing course. That’s where I discovered the solution to The Asserts Hell.

Assert Object for Better Asserts

Assert Object pattern is what solves our problem. The idea is to create an Assert class which wraps the original object being tested. In our case the tested object is of Product class, so our assert class will be called ProductAssert.

Let’s see the implementation:

As you can see, we simply moved the asserts from our test’s Then section into ProductAssert class’s methods. Additionally, we always return this from each asserting method, which allows chaining functions calls.

We can now use such assert object in our unit test:

You must agree it’s much more readable now

We can of course improve it even more, for example by wrapping these 3 assertion methods (BeAvailable(), HaveDiscount() and Cost()) into a single one. We can also easily create an extension method for the Product class itself, so the assert object doesn’t have to be instantiated in our unit test. I could even add an extension to FluentAssertions library.

You can find complete source code used as examples in this article here.

Assert Object – summary

I personally find Assert Object a very useful tests code refactoring method. As a programmer, you spend most of your time not on writing, but on reading the source code. Clear and simple tests can make your (and your colleagues’) life much better

Of course, the Assert Object pattern should not be your default way of asserting. If you can complete your test with one or two assertions, then it’s probably better to keep it without wrapping into any additional objects. Having too many assert statements might also mean that the objects being tested are poorly designed. However, as we often work with legacy code which we can’t easily change or refactor, it might be a good solution.

What patterns for writing better tests do you use? Share your tips in the comments!

The post Improve your tests with Assert Object Pattern appeared first on CodeJourney.net.