Testing, Tools, and Frameworks

First, I’d like to get back a little bit of this framework’s architecture overview. Test Execution Progress As the above image and the previous story mentioned, the test suites/cases/steps are formed in a JSON file, and the framework will load and map these JSON files to the list of test suites/cases objects. Then, action steps are executed based on the class and method (keyword) that are specified in each test step (use the reflection technique to invoke the keyword — method). Our responsibility is to write the autotest script inside each keyword. I coded the core class and keywords for REST API Testing named RestAPIAction. What you need to use this framework is to define test suites/cases and test steps in the JSON format file. The source code is located on GitHub here and here. Download or clone it to research more. The automation libraries used in this framework are as follows: TestNG: A testing framework REST Assured: A library you can use to test HTTP-based REST services Extent Report 4.0: A framework for creating a test report Next, I’d like to show you the test parameters of the RestAPIAction keyword that will be defined in each REST API test step (please refer to the previous article linked in the introduction to get more detail on the test suite/test case format). 1. GET Method { "name": "Search repository", "class": "RestAPIAction", "method": "GET", "parameters": { "request": { "url": "https://api.github.com/search/repositories", "headers": { "Authorization": "@var->authorization@" }, "queryParams": { "q": "automationtestingframework", "sort": "stars", "order": "desc" } }, "response": { "statusCode": 200, "body": { "total_count": { "greaterThan": 1 }, "items.any{it.full_name = 'automationtester304/automationtestingframework'}": { "is": true } } } } } 2. POST Method { "name": "Send POST request to create Hello-World repository", "class": "RestAPIAction", "method": "POST", "parameters": { "request": { "url": "https://api.github.com/user/repos", "headers": { "Authorization": "@var->authorization@" }, "body": { "name": "Hello-World", "description": "This is your first repository", "homepage": "https://github.com", "private": false, "has_issues": true, "has_projects": true, "has_wiki": true } }, "response": { "statusCode": 201, "body": { "full_name": { "is": "automationtester304/Hello-World" } } } } } 3. PUT/PATCH and DELETE methods Please refer to CRUDRepository.json. As you can see, there are two “Request and Response” sections in each REST Action method. Request Section This is where you define the parameters’ values for sending a REST Request. url: Define the URL you want to send for a request headers: Where you define the header parameters such as Authorization, ContentType, etc. queryParams: Define the query parameters for the request body: Define the request body for POST/ PATCH methods Response Section This is where you define the expected value in the response. statusCode: Define the expected value of the response status code as 200,201,204,400, or 501 schemaPath: Define the path file that contains the JSON schema format of the response body: Define the expected value of the fields, and parameters in the response; it contains the JSON objects where we can define the field name and the query of identifying a specified value from the response Example: "body": { "total_count": { "greaterThan": 1 }, "items.any{it.name = 'automationtestingframework'}": { "is": true } } Inside the field name or the query, some Hamcrest matchers are defined to perform its assertion since REST-assured takes advantage of the power of this one. Please refer to the Hamcrest Tutorial for more information. 4. StoreResponseValue Method This keyword method is located after the REST method step to store the specified value from the response of the REST request method step right before. The next step can use the value stored from this method by the string @var->[var name]@ to verify something. Example: { "name": "Store owner and repoName values of the above response", "class": "RestAPIAction", "method": "storeResponseValue", "parameters": { "variables": [ { "varName": "owner", "key": "items.find{it.name = 'AutomationTesting'}.owner.login" }, { "varName": "repoName", "key": "items.find{it.name = 'AutomationTesting'}.name" } ] } } { "name": "Send GET request to get branches", "class": "RestAPIAction", "method": "GET", "parameters": { "request": { "url": "@var->githubapi@/repos/@var->owner@/@var->repoName@/branches", "headers": { "Authorization": "@var->authorization@" } }, "response": { "statusCode": 200, "body": { "any{it.name == 'master'}": { "is": true } } } } } 5. ValidateReponse Method As the method name suggests, you can use this method to validate the response of the REST request method. The situation of using this method is when you want to add some steps to handle the value from the response of the previous REST Request step, and then you’ll use this method to verify the responses’ values. So the parameters of this method are defined as well as the response section in the REST request step. Example: { "name": "Validate Response", "class": "RestAPIAction", "method": "validateResponse", "parameters": { "statusCode": 404, "body": { "message": { "is": "Not Found" } } } } Next Vision I’m going to tell you how I apply the combination without repetition of the algorithm to generate a suite of test cases in order to increase the test coverage for REST API testing. I’m developing the test tool UI to manage the visually generated test suite/test case JSON file.

A/B testing has long been the cornerstone of experimentation in the software and machine learning domains. By comparing two versions of a webpage, application, feature, or algorithm, businesses can determine which version performs better based on predefined metrics of interest. However, as the complexity of business problems or experimentation grows, A/B testing can be a constraint in empirically evaluating successful development. Multi-armed bandits (MAB) is a powerful alternative that can scale complex experimentation in enterprises by dynamically balancing exploration and exploitation. The Limitations of A/B Testing While A/B testing is effective for simple experiments, it has several limitations: Static allocation: A/B tests allocate traffic equally or according to a fixed ratio, potentially wasting resources on underperforming variations. Exploration vs. exploitation: A/B testing focuses heavily on exploration, often ignoring the potential gains from exploiting known good options. Time inefficiency: A/B tests can be time-consuming, requiring sufficient data collection periods before drawing conclusions. Scalability: Managing multiple simultaneous A/B tests for complex systems can be cumbersome and resource-intensive. Multi-Armed Bandits The multi-armed bandit problem is a classic Reinforcement Learning problem where an agent must choose between multiple options (arms) to maximize the total reward over time. Each arm provides a random reward from a probability distribution unique to that arm. The agent must balance exploring new arms (to gather more information) and exploiting the best-known arms (to maximize reward). In the context of experimentation, MAB algorithms dynamically adjust the allocation of traffic to different variations based on their performance, leading to more efficient and adaptive experimentation. The terms "exploration" and "exploitation" refer to the fundamental trade-off that an agent must balance to maximize cumulative rewards over time. This trade-off is central to the decision-making process in MAB algorithms. Exploration Exploration is the process of trying out different options (or "arms") to gather more information about their potential rewards. The goal of exploration is to reduce uncertainty and discover which arms yield the highest rewards. Purpose To gather sufficient data about each arm to make informed decisions in the future. Example In an online advertising scenario, exploration might involve displaying various different ads to users to determine which ad generates the most clicks or conversions. Even though some ads perform poorly initially, they are still shown to collect enough data to understand their true performance. Exploitation Exploitation, on the other hand, is the process of selecting the option (or "arm") that currently appears to offer the highest reward based on the information gathered so far. The main purpose of exploitation is to maximize immediate rewards by leveraging known information. Purpose To maximize the immediate benefit by choosing the arm that has provided the best results so far. Example In the same online advertising case, exploitation would involve predominantly showing the advertisement that has already shown the highest click-through rate, thereby maximizing the expected number of clicks. Types of Multi-Armed Bandit Algorithms Epsilon-Greedy: With probability ε, the algorithm explores a random arm, and with probability 1-ε, it exploits the best-known arm. UCB (Upper Confidence Bound): This algorithm selects arms based on their average reward and the uncertainty or variance in their rewards, favoring less-tested arms to a calculated degree. Thompson Sampling: This Bayesian approach samples from the posterior distribution of each arm's reward, balancing exploration and exploitation according to the likelihood of each arm being optimal. Implementing Multi-Armed Bandits in Enterprise Experimentation Step-By-Step Guide Define objectives and metrics: Clearly outline the goals of your experimentation and the key metrics for evaluation. Select an MAB algorithm: Choose an algorithm that aligns with your experimentation needs. For instance, UCB is suitable for scenarios requiring a balance between exploration and exploitation, while Thompson Sampling is beneficial for more complex and uncertain environments. Set up infrastructure: Ensure your experimentation platform supports dynamic allocation and real-time data processing (e.g. Apache Flink or Apache Kafka can help manage the data streams effectively). Deploy and monitor: Launch the MAB experiment and continuously monitor the performance of each arm. Adjust parameters like ε in epsilon-greedy or prior distributions in Thompson Sampling as needed. Analyze and iterate: Regularly analyze the results and iterate on your experimentation strategy. Use the insights gained to refine your models and improve future experiments. Top Python Libraries for Multi-Armed Bandits MABWiser Overview: MABWiser is a user-friendly library specifically designed for multi-armed bandit algorithms. It supports various MAB strategies like epsilon-greedy, UCB, and Thompson Sampling. Capabilities: Easy-to-use API, support for context-free and contextual bandits, online and offline learning. Vowpal Wabbit (VW) Overview: Vowpal Wabbit is a fast and efficient machine learning system that supports contextual bandits, among other learning tasks. Capabilities: High-performance, scalable, supports contextual bandits with rich feature representations. Contextual Overview: Contextual is a comprehensive library for both context-free and contextual bandits, providing a flexible framework for various MAB algorithms. Capabilities: Extensive documentation, support for numerous bandit strategies, and easy integration with real-world data. Keras-RL Overview: Keras-RL is a library for reinforcement learning that includes implementations of bandit algorithms. It is built on top of Keras, making it easy to use with deep learning models. Capabilities: Integration with neural networks, support for complex environments, easy-to-use API. Example using MABWiser. Python # Import MABWiser Library from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy # Data arms = ['Arm1', 'Arm2'] decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1'] rewards = [20, 17, 25, 9] # Model mab = MAB(arms, LearningPolicy.UCB1(alpha=1.25)) # Train mab.fit(decisions, rewards) # Test mab.predict() Example from MABWiser of Context Free MAB setup. Python # 1. Problem: A/B Testing for Website Layout Design. # 2. An e-commerce website experiments with 2 different layouts options # for their homepage. # 3. Each layouts decision leads to generating different revenues # 4. What should the choice of layouts be based on historical data? from mabwiser.mab import MAB, LearningPolicy # Arms options = [1, 2] # Historical data of layouts decisions and corresponding rewards layouts = [1, 1, 1, 2, 1, 2, 2, 1, 2, 1, 2, 2, 1, 2, 1] revenues = [10, 17, 22, 9, 4, 0, 7, 8, 20, 9, 50, 5, 7, 12, 10] arm_to_features = {1: [0, 0, 1], 2: [1, 1, 0], 3: [1, 1, 0]} # Epsilon Greedy Learning Policy # random exploration set to 15% greedy = MAB(arms=options, learning_policy=LearningPolicy.EpsilonGreedy(epsilon=0.15), seed=123456) # Learn from past and predict the next best layout greedy.fit(decisions=layouts, rewards=revenues) prediction = greedy.predict() # Expected revenues from historical data and results expectations = greedy.predict_expectations() print("Epsilon Greedy: ", prediction, " ", expectations) assert(prediction == 2) # more data from online learning additional_layouts = [1, 2, 1, 2] additional_revenues = [0, 12, 7, 19] # model update and new layout greedy.partial_fit(additional_layouts, additional_revenues) greedy.add_arm(3) # Warm starting a new arm greedy.warm_start(arm_to_features, distance_quantile=0.5) Conclusion Multi-armed bandits offer a sophisticated and scalable alternative to traditional A/B testing, particularly suited for complex experimentation in enterprise settings. By dynamically balancing exploration and exploitation, MABs enhance resource efficiency, provide faster insights, and improve overall performance. For software and machine learning engineers looking to push the boundaries of experimentation, incorporating MABs into your toolkit can lead to significant advancements in optimizing and scaling your experiments. Above we have touched upon just the tip of the iceberg in the rich and actively researched literature in the field of Reinforcement Learning to get started.

While comprehensive chaos testing tools offer a wide range of features, sometimes you just need a quick and easy solution for a specific scenario. This article focuses on a targeted approach: simulating network issues between Redis client and Redis Cluster in simple steps. These methods are ideal when you don't require a complex setup and want to focus on testing a particular aspect of your Redis cluster's behavior under simulated network issues. Set-Up This article assumes that you already have a Redis cluster and the client code for sending traffic to the cluster is set up and ready to use. If not, you can refer to the following steps: Install a Redis cluster: You can follow this article to set up a Redis cluster locally before taking it to production. There are several Redis clients available for different languages, you can choose what’s most suitable for your use case. Jedis documentation Lettuce documentation Redisson documentation redigo documentation go-redis/redis documentation redis-py documentation hiredis documentation Let’s explore a few ways to simulate network issues between Redis clients and the Redis Cluster. Simulate Slow Redis Server Response DEBUG SLEEP Shell DEBUG SLEEP <seconds> The DEBUG SLEEP command will suspend all operations, including command processing, network handling, and replication, on the specified Redis node for a given time duration effectively making the Redis node unresponsive for the specified duration. Once this command is initiated the response is not sent until the specified duration is elapsed. In the above screenshot, the response (OK) is received after 5 seconds. Use Case This command can be used to simulate a slow server response, server hang-ups, and heavy load conditions, and observe the system’s reaction to an unresponsive Redis instance. Simulate Connection Pause for Clients CLIENT PAUSE Shell CLIENT PAUSE <milliseconds> This command temporarily pauses all the clients and the commands will be delayed for a specified duration however interactions with replica will continue normally. Modes: CLIENT PAUSE supports two modes: ALL (default): Pauses all client commands (write and read). WRITE: Only blocks write commands (reads still work). It gives finer control if you want to simulate connection pause only for writes or all client commands. Once this command is initiated it responds back with “OK” immediately (unlike debug sleep) Use Case Useful for scenarios like controlled failover testing, control client behavior, or maintenance tasks where you want to ensure that no new commands are processed temporarily. Simulate Network Issues Using Custom Interceptors/Listeners Interceptors or listeners can be valuable tools for injecting high latency or other network issues into the communication between a Redis client and server, facilitating effective testing of how the Redis deployment behaves under adverse network conditions. Inject High Latency Using a Listener Interceptors or Listeners act as a middleman, listening for commands sent to the Redis server. When a command is detected, we can introduce a configurable delay before forwarding it by overriding the methods of the listener. This way you can simulate high latency and it allows you to observe how your client behaves under slow network conditions. The following example shows how to create a basic latency injector by implementing the CommandListener class in the Lettuce Java Redis client. Java package com.rc; import io.lettuce.core.event.command.CommandFailedEvent; import io.lettuce.core.event.command.CommandListener; import io.lettuce.core.event.command.CommandStartedEvent; import io.lettuce.core.event.command.CommandSucceededEvent; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.concurrent.TimeUnit; public class LatencyInjectorListener implements CommandListener { private static final Logger logger = LoggerFactory.getLogger(LatencyInjectorListener.class); private final long delayInMillis; private final boolean enabled; public LatencyInjectorListener(long delayInMillis, boolean enabled) { this.delayInMillis = delayInMillis; this.enabled = enabled; } @Override public void commandStarted(CommandStartedEvent event) { if (enabled) { try { // Introduce latency Thread.sleep(delayInMillis); } catch (InterruptedException e) { // Handle interruption gracefully, logger.error("Exception while invoking sleep method"); } } } @Override public void commandSucceeded(CommandSucceededEvent event) { } @Override public void commandFailed(CommandFailedEvent event) { } } In the above example, we have added a class that implements CommandListener interface provided by the Lettuce Java Redis client. And, in commandStarted method, we have invoked Thead.sleep() that will cause the flow to halt for a specific duration, thereby adding latency to each command that will be executed. You can add latency in other methods also such as commandSucceeded and commandFailed, depending upon the specific behavior you want to test. Simulate Intermittent Connection Errors You can even extend this concept to throw exceptions within the listener, mimicking connection errors or timeouts. This proactive approach using listeners helps you identify and address potential network-related issues in your Redis client before they impact real-world deployments The following example shows the extension of the commandStarted method implemented in the above section to throw connection exceptions to create intermittent connection failures/errors implementing CommandListener class in Lettuce Java Redis client. Java @Override public void commandStarted(CommandStartedEvent event) { if (enabled && shouldThrowConnectionError()) { // introduce connection errors throw new RedisConnectionException("Simulated connection error"); } else if (enabled) { try { // Introduce latency Thread.sleep(delayInMillis); } catch (InterruptedException e) { // Handle interruption gracefully, logger.error("Exception while invoking sleep method"); } } } private boolean shouldThrowConnectionError() { // adjust or change the logic as needed - this is just for reference. return random.nextInt(10) < 3; // 30% chance to throw an error } Similarly, Redis clients in other languages also provide hooks/interceptors to extend and simulate network issues such as high latency or connection errors. Conclusion We explored several techniques to simulate network issues for chaos testing specific to network-related scenarios in a Redis Cluster. However, exercise caution and ensure these methods are enabled with a flag and used only in strictly controlled testing environments. Proper safeguards are essential to avoid unintended disruptions. By carefully implementing these strategies, you can gain valuable insights into the resilience and robustness of your Redis infrastructure under adverse network conditions. References Create and Configure a Local Redis Cluster Redis Docs Lettuce Other Related Articles If you enjoyed this article, you might also find these related articles interesting. Techniques for Chaos Testing Your Redis Cluster Manage Redis Cluster Topology With Command Line

In this article, I would like to describe an approach to writing tests with a clear division into separate stages, each performing its specific role. This facilitates the creation of tests that are easier to read, understand, and maintain. The discussion will focus on using the Arrange-Act-Assert methodology for integration testing in the Spring Framework with mocking of HTTP requests to external resources encountered during the execution of the tested code within the system behavior. The tests under consideration are written using the Spock Framework in the Groovy language. MockRestServiceServer will be used as the mocking mechanism. There will also be a few words about WireMock. Problem Description When studying how to write integration tests for Spring, I often referred to materials on the topic. Examples for MockRestServiceServer mostly described an approach with the declaration of expectations as follows: Expected URI Number of requests to the expected URI Expectations for the structure and content of the request body Response to the request The code looked something like this: Java @Test public void testWeatherRequest() { mockServer.expect(once(), requestTo("https://external-weather-api.com/forecast")) .andExpect(method(HttpMethod.POST)) .andExpect(jsonPath("$.field1", equalTo("value1"))) .andExpect(jsonPath("$.field2", equalTo("value2"))) .andExpect(jsonPath("$.field3", equalTo("value3"))) .andRespond(withSuccess('{"result": "42"}', MediaType.APPLICATION_JSON)); weatherService.getForecast("London") mockServer.verify() assert .. assert .. } When applying this approach, I encountered a number of difficulties: Ambiguity in determining the reasons for AssertionErrorby the log text - the log text is the same for different scenarios: The HTTP call code is missing/not executed according to business logic. The HTTP call code is executed with an error. The HTTP call code is executed correctly, but there is an error in the mock description. Difficulty in determining the scope of the tested states due to their dispersion throughout the test code. Formally, the result verification is carried out at the end of the test (mockServer.verify()), but the verification assertions regarding the composition and structure of the request are described at the beginning of the test (as part of creating the mock). At the same time, verification assertions not related to the mock were presented at the end of the test. Important clarification: Using RequestMatcher for the purpose of isolating mocks within many requests seems like the right solution. Proposed Solution Clear division of test code into separate stages, according to the Arrange-Act-Assert pattern. Arrange-Act-Assert Arrange-Act-Assert is a widely used pattern in writing tests, especially in unit testing. Let's take a closer look at each of these steps: Arrange (Preparation) At this stage, you set up the test environment. This includes initializing objects, creating mocks, setting up necessary data, etc. The goal of this step is to prepare everything needed for the execution of the action being tested. Act (Execution) Here you perform the action you want to test. This could be a method call or a series of actions leading to a certain state or result to be tested. Assert (Result Verification) At the final stage, you check the results of the action. This includes assertions about the state of objects, returned values, changes in the database, messages sent, etc. The goal of this step is to ensure that the tested action has produced the expected result. Demonstration Scenarios The business logic of the service for which the tests will be provided can be described as follows: Gherkin given: The weather service provides information that the weather in city A equals B when: We request weather data from the service for city A then: We receive B Sequence Diagram Example Implementation for MockRestServiceServer Before Proposed Changes Tests for the above scenario will be described using MockRestServiceServer. Difficulty in Determining the Scope of Tested States Due to Their Dispersion Throughout the Test Code Groovy def "Forecast for provided city London is 42"() { setup: // (1) mockServer.expect(once(), requestTo("https://external-weather-api.com/forecast")) // (2) .andExpect(method(HttpMethod.POST)) .andExpect(jsonPath('$.city', Matchers.equalTo("London"))) // (3) .andRespond(withSuccess('{"result": "42"}', MediaType.APPLICATION_JSON)); // (4) when: // (5) def forecast = weatherService.getForecast("London") then: // (6) forecast == "42" // (7) mockServer.verify() // (8) } Setup stage: describing the mock Indicating that exactly one call is expected to https://external-weather-api.com Specifying expected request parameters Describing the response to return Execution stage, where the main call to get the weather for the specified city occurs Verification stage: Here, mockServer.verify() is also called to check the request (see item 3). Verification assertion regarding the returned value Calling to verify the mock's state Here we can observe the problem described earlier as "difficulty in determining the scope of tested states due to their dispersion throughout the test code" - some of the verification assertions are in the then block, some in the setup block. Ambiguity in Determining the Causes of AssertionError To demonstrate the problem, let's model different error scenarios in the code. Below are the situations and corresponding error logs. Scenario 1 - Passed an unknown city name: def forecast = weatherService.getForecast("Unknown") Java java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com 0 request(s) executed. at org.springframework.test.web.client.AbstractRequestExpectationManager.createUnexpectedRequestError(AbstractRequestExpectationManager.java:193) Scenario 2: Incorrect URI declaration for the mock; for example, mockServer.expect(once(), requestTo("https://foo.com")) Java java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com 0 request(s) executed. at org.springframework.test.web.client.AbstractRequestExpectationManager.createUnexpectedRequestError(AbstractRequestExpectationManager.java:193) Scenario 3: No HTTP calls in the code Java java.lang.AssertionError: Further request(s) expected leaving 1 unsatisfied expectation(s). 0 request(s) executed. The main observation: All errors are similar, and the stack trace is more or less the same. Example Implementation for MockRestServiceServer With Proposed Changes Ease of Determining the Scope of Tested States Due to Their Dispersion Throughout the Test Code Groovy def "Forecast for provided city London is 42"() { setup: // (1) def requestCaptor = new RequestCaptor() mockServer.expect(manyTimes(), requestTo("https://external-weather-api.com")) // (2) .andExpect(method(HttpMethod.POST)) .andExpect(requestCaptor) // (3) .andRespond(withSuccess('{"result": "42"}', MediaType.APPLICATION_JSON)); // (4) when: // (5) def forecast = weatherService.getForecast("London") then: // (6) forecast == "42" requestCaptor.times == 1 // (7) requestCaptor.entity.city == "London" // (8) requestCaptor.headers.get("Content-Type") == ["application/json"] } #3: Data capture object #7: Verification assertion regarding the number of calls to the URI #8: Verification assertion regarding the composition of the request to the URI In this implementation, we can see that all the verification assertions are in the then block. Unambiguity in Identifying the Causes of AssertionError To demonstrate the problem, let's attempt to model different error scenarios in the code. Below are the situations and corresponding error logs. Scenario 1: An unknown city name was provided def forecast = weatherService.getForecast("Unknown") Groovy requestCaptor.entity.city == "London" | | | | | | | false | | | 5 differences (28% similarity) | | | (Unk)n(-)o(w)n | | | (Lo-)n(d)o(-)n | | Unknown | [city:Unknown] <pw.avvero.spring.sandbox.weather.RequestCaptor@6f77917c times=1 bodyString={"city":"Unknown"} entity=[city:Unknown] headers=[Accept:[application/json, application/*+json], Content-Type:[application/json], Content-Length:[18]]> Scenario 2: Incorrect URI declaration for the mock; for example, mockServer.expect(once(), requestTo("https://foo.com")) Groovy java.lang.AssertionError: No further requests expected: HTTP POST https://external-weather-api.com 0 request(s) executed. Scenario 3: No HTTP calls in the code Groovy Condition not satisfied: requestCaptor.times == 1 | | | | 0 false <pw.avvero.spring.sandbox.weather.RequestCaptor@474a63d9 times=0 bodyString=null entity=null headers=[:]> Using WireMock WireMock provides the ability to describe verifiable expressions in the Assert block. Groovy def "Forecast for provided city London is 42"() { setup: // (1) wireMockServer.stubFor(post(urlEqualTo("/forecast")) // (2) .willReturn(aResponse() // (4) .withBody('{"result": "42"}') .withStatus(200) .withHeader("Content-Type", "application/json"))) when: // (5) def forecast = weatherService.getForecast("London") then: // (6) forecast == "42" wireMockServer.verify(postRequestedFor(urlEqualTo("/forecast")) .withRequestBody(matchingJsonPath('$.city', equalTo("London")))) // (7) } The above approach can also be used here, by describing the WiredRequestCaptor class. Groovy def "Forecast for provided city London is 42"() { setup: StubMapping forecastMapping = wireMockServer.stubFor(post(urlEqualTo("/forecast")) .willReturn(aResponse() .withBody('{"result": "42"}') .withStatus(200) .withHeader("Content-Type", "application/json"))) def requestCaptor = new WiredRequestCaptor(wireMockServer, forecastMapping) when: def forecast = weatherService.getForecast("London") then: forecast == "42" requestCaptor.times == 1 requestCaptor.body.city == "London" } This allows us to simplify expressions and enhance the idiomaticity of the code, making the tests more readable and easier to maintain. Conclusion Throughout this article, I have dissected the stages of testing HTTP requests in Spring, using the Arrange-Act-Assert methodology and mocking tools such as MockRestServiceServer and WireMock. The primary goal was to demonstrate how clearly dividing the test into separate stages significantly enhances readability, understanding, and maintainability. I highlighted the problems associated with the ambiguity of error determination and the difficulty of defining the scope of tested states and presented ways to solve them through a more structured approach to test writing. This approach is particularly important in complex integration tests, where every aspect is critical to ensuring the accuracy and reliability of the system. Furthermore, I showed how the use of tools like RequestCaptor and WiredRequestCaptor simplifies the test-writing process and improves their idiomaticity and readability, thereby facilitating easier support and modification. In conclusion, I want to emphasize that the choice of testing approach and corresponding tools should be based on specific tasks and the context of development. The approach to testing HTTP requests in Spring presented in this article is intended to assist developers facing similar challenges. The link to the project repository with demonstration tests can be found here. Thank you for your attention to the article, and good luck in your pursuit of writing effective and reliable tests!

This article is a high-level overview introducing some often overlooked concepts and the need for productive functional and or integration tests. Modern development should be founded on the following: Iterating with fast feedback Eliminating waste Amplifying value Based on Lean_software_development. Allen Holub tweeted: "Agile: Work small Talk to each other Make people’s lives better That’s it. All that other junk is a distraction." Here’s a link to the original tweet. Here’s more information. What’s the Problem? How can we effectively fulfill the above, if we don’t test productively? We should be focusing on delivering value, however, most companies and projects don’t focus on this. In the ideal world, your whole company would be aligned with Agile to facilitate fast feedback and quick iterations. This very concept is hard for some to imagine, maybe you think that you are already doing this? To paraphrase Allen Holub — we have world views; one person’s world view could be an agile one, as they have worked in a startup. Another, if they have worked in a non-tech savvy organization, couldn’t imagine what agile looks like. When it comes to testing, many people seem to have the world view that hard-to-maintain tests are the norm and acceptable. In my experience, the major culprits are BDD frameworks that are based on text feature files. This is amplifying waste. The extra feature file layer in theory allows; The user to swap out the language at a later date Allows a business person to write user stories and or acceptance criteria Allows a business person to read the user stories and or acceptance criteria Collaboration Etc… You have actually added more complexity than you think, for little benefit. I am explicitly critiquing the approach of writing the extra feature file layer first, not the benefits of BDD as a concept. You test more efficiently, with better results not writing the feature file layer, such as with Smart BDD, where it’s generated by code. Here I compare the complexities and differences between Cucumber and Smart BDD. In summary and reality the feature file layer: Doesn’t help with swapping out the language. Both language implementations and maintenance would have diminishing returns. Doesn’t help with a business person writing user stories and or acceptance criteria, because as the article above shows in more detail, the language you use is actually determined by the framework and limitations thereof. Therefore, it would need to be re-worded to some degree, hence a feature file isn’t helping. Doesn’t help with a business person reading the user stories and or acceptance criteria. Usually, feature files are badly written due to the complexity of the BDD framework, therefore a business person in reality is more likely to ask a developer how the system behaves. Smart BDD generates more concise and consistent documentation. Does help with collaboration, which is a major benefit What’s the Best Tool for the Job? If you want to be productive, you might seek the best tool for the job, but what is “the best tool for the job”? It’s retrospective; you don’t know upfront. Using an extra framework alongside your existing unit test framework could be wasteful. It’s better to learn one testing framework well. For example, Cucumber is evidently hard to master leading to poor quality tests that take longer and longer to maintain. You can easily underestimate the future complexities, especially with frameworks that make you write feature files first. You don’t plan to write poor-quality tests, it’s something that could happen over time. You don’t want to spend extra time battling over a framework. You could have the worst of all worlds and use a feature file-driven framework with language you’re not proficient in. If you’re a Java developer using Ruby or Typescript etc. for tests, this could also lead to poor quality and less productivity. I’m suggesting, that if you’re a Java developer, Smart BDD would be the closest to your main skill-set (the least friction) and it tries its hardest to promote good practices. You do less and get more out! Test Based on Your Project's Needs If you’ve heard of the testing pyramid, you can use it as a reference, but do what works for you. You don’t have to do more unit tests and less functional testing because you think that’s the shape of a triangle. You need to align your culture by thinking what value does something provide? Aim for what works for your team, not something designed by a committee. The number of unit tests or the coverage is an output, not an outcome. Aiming for some % test coverage is an amplifying waste. With unit tests, TDD is about code quality, it drives the architecture, and coverage is a side effect. Higher quality code is better for agility. Where does Smart BDD and or another productive testing framework fit in? If you are going to test, it’s best to test first, as it’s more work to test later, and you’ll miss many of the benefits of testing first. With any new feature and or requirement you should generally start outside in. If you start with the functional tests, you start to understand better the requirements and the features that you are delivering. A lot of software development is about learning and communication. Once you have validated that you are working on the right thing, and you’ve increased your understanding of the problem you’re solving, you can work on the code, ideally using TDD. Next is to get the feature in the hands of your client and or the next best available person for feedback. Obviously, feedback is feedback and not future requirements. Feedback from a client is not a silver bullet. It could be used with metrics, for example, was the new feature used as expected? Think of this process as learning, or even better, validated learning if you can prove something. You should strive to solve the problem; how can you get meaningful feedback, as soon as possible? It’s a red flag when you spend too long writing and maintaining functional tests. For example, if you use well-designed defaults and builders you can really reduce the amount of work required in creating and maintaining functional tests. You also want the test framework to be smart, for example; Create interaction diagrams Test boundary cases Produce metrics like timers Give insights And many more At the heart of this is specifying intent and not implementing it many times over. I think the industry in general is moving in the direction of declaring intent over implementing details. By using best practices you’ll get better at testing/documenting behavior at an appropriate level and not making the tests/documentation obfuscated with irrelevant data. Conclusion Culture is hugely important, I’m sure we and our bosses and senior leaders would all ultimately agree with the following: For more value, you need more feedback and less waste For more feedback, you need more value and less waste For less waste, you need more value and more feedback However, most work culture is not aligned with this. Agreeing with something and having the culture are very different, it’s the difference between agreeing that eating healthily and exercising is good for you and actually doing something about it. The next level in the healthy analogy is, having friends or a partner that are similarly minded. Is your company metaphorically encouraging a healthy lifestyle or just agreeing that being healthy makes sense? Culture drives your mindset, behavior, and processes. This has been a very brief introduction, hopefully, you’ll think about amplifying value, thanks for reading please do have the courage to do more or less of something and introduce change where needed.

Unit testing is a software testing methodology where individual units or components of software are tested in isolation to check whether it is functioning up to the expectation or not. In Java, it is an essential practice with the help of which an attempt to verify code correctness is made, and an attempt to improve code quality is made. It will basically ensure that the code works fine and the changes are not the point of breakage of existing functionality. Test-Driven Development (TDD) is a test-first approach to software development in short iterations. It is a kind of practice where a test is written before the real source code is written. It pursues the aim of writing code that passes predefined tests and, hence, well-designed, clean, and free from bugs. Key Concepts of Unit Testing Test automation: Use tools for automatic test running, such as JUnit. Asserts: Statements that confirm an expected result within a test. Test coverage: It is the code execution percentage defined by the tests. Test suites: Collection of test cases. Mocks and stubs: Dummy objects that simulate real dependencies. Unit Testing Frameworks in Java: JUnit JUnit is an open-source, simple, and widely used unit testing. JUnit is one of the most popular Java frameworks for unit testing. In other words, it comes with annotations, assertions, and tools required to write and run tests. Core Components of JUnit 1. Annotations JUnit uses annotations to define tests and lifecycle methods. These are some of the key annotations: @Test: Marks a method as a test method. @BeforeEach: Denotes that the annotated method should be executed before each @Test method in the current class. @AfterEach: Denotes that the annotated method should be executed after each @Test method in the current class. @BeforeAll: Denotes that the annotated method should be executed once before any of the @Test methods in the current class. @AfterAll: Denotes that the annotated method should be executed once after all of the @Test methods in the current class. @Disabled: Used to disable a test method or class temporarily. 2. Assertions Assertions are used to test the expected outcomes: assertEquals(expected, actual): Asserts that two values are equal. If they are not, an AssertionError is thrown. assertTrue(boolean condition): Asserts that a condition is true. assertFalse(boolean condition): Asserts that a condition is false. assertNotNull(Object obj): Asserts that an object is not null. assertThrows(Class<T> expectedType, Executable executable): Asserts that the execution of the executable throws an exception of the specified type. 3. Assumptions Assumptions are similar to assertions but used in a different context: assumeTrue(boolean condition): If the condition is false, the test is terminated and considered successful. assumeFalse(boolean condition): The inverse of assumeTrue. 4. Test Lifecycle The lifecycle of a JUnit test runs from initialization to cleanup: @BeforeAll → @BeforeEach → @Test → @AfterEach → @AfterAll This allows for proper setup and teardown operations, ensuring that tests run in a clean state. Example of a Basic JUnit Test Here’s a simple example of a JUnit test class testing a basic calculator: Java import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.AfterEach; import static org.junit.jupiter.api.Assertions.*; class CalculatorTest { private Calculator calculator; @BeforeEach void setUp() { calculator = new Calculator(); } @Test void testAddition() { assertEquals(5, calculator.add(2, 3), "2 + 3 should equal 5"); } @Test void testMultiplication() { assertAll( () -> assertEquals(6, calculator.multiply(2, 3), "2 * 3 should equal 6"), () -> assertEquals(0, calculator.multiply(0, 5), "0 * 5 should equal 0") ); } @AfterEach void tearDown() { // Clean up resources, if necessary calculator = null; } } Dynamic Tests in JUnit 5 JUnit 5 introduced a powerful feature called dynamic tests. Unlike static tests, which are defined at compile-time using the @Test annotation, dynamic tests are created at runtime. This allows for more flexibility and dynamism in test creation. Why Use Dynamic Tests? Parameterized testing: This allows you to create a set of tests that execute the same code but with different parameters. Dynamic data sources: Create tests based on data that may not be available at compile-time (e.g., data from external sources). Adaptive testing: Tests can be generated based on the environment or system conditions. Creating Dynamic Tests JUnit provides the DynamicTest class for creating dynamic tests. You also need to use the @TestFactory annotation to mark the method that returns the dynamic tests. Example of Dynamic Tests Java import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import java.util.Arrays; import java.util.Collection; import java.util.stream.Stream; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.DynamicTest.dynamicTest; class DynamicTestsExample { @TestFactory Stream<DynamicTest> dynamicTestsFromStream() { return Stream.of("apple", "banana", "lemon") .map(fruit -> dynamicTest("Test for " + fruit, () -> { assertEquals(5, fruit.length()); })); } @TestFactory Collection<DynamicTest> dynamicTestsFromCollection() { return Arrays.asList( dynamicTest("Positive Test", () -> assertEquals(2, 1 + 1)), dynamicTest("Negative Test", () -> assertEquals(-2, -1 + -1)) ); } } Creating Parameterized Tests In JUnit 5, you can create parameterized tests using the @ParameterizedTest annotation. You'll need to use a specific source annotation to supply the parameters. Here's an overview of the commonly used sources: @ValueSource: Supplies a single array of literal values. @CsvSource: Supplies data in CSV format. @MethodSource: Supplies data from a factory method. @EnumSource: Supplies data from an Enum. Example of Parameterized Tests Using @ValueSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; import static org.junit.jupiter.api.Assertions.assertTrue; class ValueSourceTest { @ParameterizedTest @ValueSource(strings = {"apple", "banana", "orange"}) void testWithValueSource(String fruit) { assertTrue(fruit.length() > 4); } } Using @CsvSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import static org.junit.jupiter.api.Assertions.assertEquals; class CsvSourceTest { @ParameterizedTest @CsvSource({ "test,4", "hello,5", "JUnit,5" }) void testWithCsvSource(String word, int expectedLength) { assertEquals(expectedLength, word.length()); } } Using @MethodSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.MethodSource; import java.util.stream.Stream; import static org.junit.jupiter.api.Assertions.assertTrue; class MethodSourceTest { @ParameterizedTest @MethodSource("stringProvider") void testWithMethodSource(String word) { assertTrue(word.length() > 4); } static Stream<String> stringProvider() { return Stream.of("apple", "banana", "orange"); } } Best Practices for Parameterized Tests Use descriptive test names: Leverage @DisplayName for clarity. Limit parameter count: Keep the number of parameters manageable to ensure readability. Reuse methods for data providers: For @MethodSource, use static methods that provide the data sets. Combine data sources: Use multiple source annotations for comprehensive test coverage. Tagging in JUnit 5 The other salient feature in JUnit 5 is tagging: it allows for assigning their own custom tags to tests. Tags allow, therefore, a way to group tests and later execute groups selectively by their tag. This would be very useful for managing large test suites. Key Features of Tagging Flexible grouping: Multiple tags can be applied to a single test method or class, so flexible grouping strategies can be defined. Selective execution: Sometimes it may be required to execute only the desired group of tests by adding tags. Improved organization: Provides an organized way to set up tests for improved clarity and maintainability. Using Tags in JUnit 5 To use tags, you annotate your test methods or test classes with the @Tag annotation, followed by a string representing the tag name. Example Usage of @Tag Java import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; @Tag("fast") class FastTests { @Test @Tag("unit") void fastUnitTest() { // Test logic for a fast unit test } @Test void fastIntegrationTest() { // Test logic for a fast integration test } } @Tag("slow") class SlowTests { @Test @Tag("integration") void slowIntegrationTest() { // Test logic for a slow integration test } } Running Tagged Tests You can run tests with specific tags using: Command line: Run the tests by passing the -t (or --tags) argument to specify which tags to include or exclude.mvn test -Dgroups="fast" IDE: Most modern IDEs like IntelliJ IDEA and Eclipse allow selecting specific tags through their graphical user interfaces. Build tools: Maven and Gradle support specifying tags to include or exclude during the build and test phases. Best Practices for Tagging Consistent tag names: Use a consistent naming convention across your test suite for tags, such as "unit", "integration", or "slow". Layered tagging: Apply broader tags at the class level (e.g., "integration") and more specific tags at the method level (e.g., "slow"). Avoid over-tagging: Do not add too many tags to a single test, which can reduce clarity and effectiveness. JUnit 5 Extensions The JUnit 5 extension model allows developers to extend and otherwise customize test behavior. They provide a mechanism for extending tests with additional functionality, modifying the test execution lifecycle, and adding new features to your tests. Key Features of JUnit 5 Extensions Customization: Modify the behavior of test execution or lifecycle methods. Reusability: Create reusable components that can be applied to different tests or projects. Integration: Integrate with other frameworks or external systems to add functionality like logging, database initialization, etc. Types of Extensions Test Lifecycle Callbacks BeforeAllCallback, BeforeEachCallback, AfterAllCallback, AfterEachCallback. Allow custom actions before and after test methods or test classes. Parameter Resolvers ParameterResolver. Inject custom parameters into test methods, such as mock objects, database connections, etc. Test Execution Condition ExecutionCondition. Enable or disable tests based on custom conditions (e.g., environment variables, OS type). Exception Handlers TestExecutionExceptionHandler. Handle exceptions thrown during test execution. Others TestInstancePostProcessor, TestTemplateInvocationContextProvider, etc. Customize test instance creation, template invocation, etc. Implementing Custom Extensions To create a custom extension, you need to implement one or more of the above interfaces and annotate the class with @ExtendWith. Example: Custom Parameter Resolver A simple parameter resolver that injects a string into the test method: Java import org.junit.jupiter.api.extension.*; public class CustomParameterResolver implements ParameterResolver { @Override public boolean supportsParameter(ParameterContext parameterContext, ExtensionContext extensionContext) { return parameterContext.getParameter().getType().equals(String.class); } @Override public Object resolveParameter(ParameterContext parameterContext, ExtensionContext extensionContext) { return "Injected String"; } } Using the Custom Extension in Tests Java import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @ExtendWith(CustomParameterResolver.class) class CustomParameterTest { @Test void testWithCustomParameter(String injectedString) { System.out.println(injectedString); // Output: Injected String } } Best Practices for Extensions Separation of concerns: Extensions should have a single, well-defined responsibility. Reusability: Design extensions to be reusable across different projects. Documentation: Document how the extension works and its intended use cases. Unit testing and Test-Driven Development (TDD) offer significant benefits that positively impact software development processes and outcomes. Benefits of Unit Testing Improved Code Quality Detection of bugs: Unit tests detect bugs early in the development cycle, making them easier and cheaper to fix. Code integrity: Tests verify that code changes don't break existing functionality, ensuring continuous code integrity. Simplifies Refactoring Tests serve as a safety net during code refactoring. If all tests pass after refactoring, developers can be confident that the refactoring did not break existing functionality. Documentation Tests serve as live documentation that illustrates how the code is supposed to be used. They provide examples of the intended behavior of methods, which can be especially useful for new team members. Modularity and Reusability Writing testable code encourages modular design. Code that is easily testable is generally also more reusable and easier to understand. Reduces Fear of Changes A comprehensive test suite helps developers make changes confidently, knowing they will be notified if anything breaks. Regression Testing Unit tests can catch regressions, where previously working code stops functioning correctly due to new changes. Encourages Best Practices Developers tend to write cleaner, well-structured, and decoupled code when unit tests are a priority. Benefits of Test-Driven Development (TDD) Ensures test coverage: TDD ensures that every line of production code is covered by at least one test. This provides comprehensive coverage and verification. Focus on requirements: Writing tests before writing code forces developers to think critically about requirements and expected behavior before implementation. Improved design: The incremental approach of TDD often leads to better system design. Code is written with testing in mind, resulting in loosely coupled and modular systems. Reduces debugging time: Since tests are written before the code, bugs are caught early in the development cycle, reducing the amount of time spent debugging. Simplifies maintenance: Well-tested code is easier to maintain because the tests provide instant feedback when changes are introduced. Boosts developer confidence: Developers are more confident in their changes knowing that tests have already validated the behavior of their code. Facilitates collaboration: A comprehensive test suite enables multiple developers to work on the same codebase, reducing integration issues and conflicts. Helps identify edge cases: Thinking through edge cases while writing tests helps to identify unusual conditions that could be overlooked otherwise. Reduces overall development time: Although TDD may initially seem to slow development due to the time spent writing tests, it often reduces the total development time by preventing bugs and reducing the time spent on debugging and refactoring. Conclusion By leveraging unit testing and TDD in Java with JUnit, developers can produce high-quality software that's easier to maintain and extend over time. These practices are essential for any professional software development workflow, fostering confidence and stability in your application's codebase.

Queuing theory is a branch of mathematics that analyzes how waiting lines (queues) form and behave in systems. In the context of non-functional software testing, it provides a valuable tool for understanding how a system performs under varying loads. By analyzing queue lengths, waiting times, and server utilization, queuing models can help predict potential bottlenecks and performance issues before they occur in real-world use. This article starts with the basics of queuing theory for non-functional software testing. Benefits and limitations will be addressed. As a case study, we will explore a sample of queuing models that could be appropriate for mobile gaming applications. Finally, a set of tools that can be used will be explored with their pros and cons. Key Concepts in Queuing Theory Queuing theory provides mathematical models that can be used for non-functional testing. First, we will explain basic queuing concepts to understand how to use them. Arrival rate (λ): This refers to the average number of tasks or requests entering the system per unit of time. For example, it could represent the number of customers arriving at a bank per minute or the number of network packets arriving at a router per second. Service time (μ): This represents the average time it takes for a resource to complete a task. In a bank, it might be the average time a teller spends with a customer. In a network, it could be the average time it takes to process a data packet. Queue length (L): This refers to the number of tasks waiting for service at any given time. It is directly related to the arrival rate, service time, and the number of available resources. Number of servers (S): This refers to the resources available to handle tasks. In a bank, it's the number of tellers available to serve customers. In a network, it could be the number of processing cores in a server or the number of available network channels. Queue discipline: This defines how tasks are selected from the queue for service. Some common disciplines include: First-In-First-Out (FIFO): The first task to enter the queue is the first to be served. This is often used in situations like checkout lines or waiting rooms. Priority queue: Certain tasks are assigned a higher priority and are served before lower-priority tasks, even if they arrive earlier. This is used in situations where certain tasks are critical and require immediate attention. Shortest Processing Time (SPT): The task with the shortest expected service time is served first. This can be beneficial for minimizing average waiting times overall. Applications in Non-Functional Testing A sample of applications in non-functional software testing includes the following: Load Testing By simulating realistic user loads with specific arrival rates and service times based on queuing models, load testing tools can evaluate system performance under stress. This helps identify potential bottlenecks like overloaded servers, slow database queries, or inefficient code execution. By analyzing queue lengths and waiting times during the load test, you can pinpoint areas where the system struggles and implement improvements before deployment. Capacity Planning Queuing theory models can be integrated with testing tools to determine the system's breaking point or optimal resource allocation. The model can predict how the system behaves with different numbers of servers, allowing you to find the sweet spot between adequate performance and cost-effective resource utilization. This helps ensure the system can handle expected user traffic without compromising performance or incurring unnecessary costs for over-provisioning. Performance Benchmarking Queuing models can be used to compare the performance of different system configurations or architectures. By simulating the same workload on different system setups, you can evaluate which configuration delivers the best performance in terms of waiting times and server utilization. This can be particularly helpful when choosing between different hardware or software options. Other Useful Concepts in Queuing Theory Little's Law This fundamental relationship in queuing theory states that the average number of tasks in the system (L) is equal to the average arrival rate (λ) multiplied by the average waiting time (W). This allows you to estimate one of these values if you know the other two. Kendall-Lee Notation This notation is a standardized way to describe queuing systems based on arrival distribution, service distribution, number of servers, and queue discipline. Understanding this notation helps classify different queuing models and choose the appropriate one for analysis. Open vs. Closed Queuing Systems Open queuing systems allow tasks to enter and leave the system. Closed queuing systems have a fixed number of tasks circulating within the system. Choosing the right model depends on the system being analyzed. Limitations of Using Queuing Theory As with all theories, queuing theory is based on assumptions. The benefits that we can enjoy by employing queuing theory in non-functional testing are largely affected by how realistic are those assumptions. Simplified Assumptions Queuing models often rely on simplifying assumptions to make the math tractable. These assumptions include: Stable arrival rates and service times: Real-world systems might experience fluctuations in arrival rates and service times. Queuing models might not accurately reflect such dynamic behavior. Infinite queues: In reality, queues might have a finite capacity. If the queue becomes full, new arrivals might be rejected, leading to system instability. Queuing models with finite queues can be more complex but offer a more realistic representation. The Case of Mobile Gaming Mobile games, especially those with online multiplayer components or microtransaction systems, often involve interactions that we can model using queuing theory. We will analyze a list of possible queuing models appropriate for mobile games. The list is not exhaustive but it can explain the reasoning for using different models and their benefits. 1. M/M/1 Queuing System With Network Latency In mobile games with online multiplayer features, players may connect to a central server to interact with each other. This scenario can be modeled as an M/M/1 queuing system, where players are the arriving entities, and the server acts as the single server. Incorporating network latency into the model allows developers to analyze the impact of delays on player experience and design strategies to mitigate them. Understanding the queuing behavior helps optimize server capacity and network infrastructure to minimize latency and enhance the gameplay experience. 2. M/G/1 Queuing System for In-Game Purchases Mobile games often include in-game stores where players can make purchases using real or virtual currency. The arrival of purchase requests and the service times for processing these transactions may not follow exponential distributions typical of M/M/1 systems. An M/G/1 queuing system, where the service time distribution is generalized, can be more appropriate for modeling in-game purchase transactions. Analyzing this model helps game developers optimize the payment processing system, streamline transaction flows, and manage server resources efficiently. 3. Finite Source Queuing Models for Limited Resources Many mobile games have limited resources, such as virtual items, game levels, or server capacity. Players may need to wait in a queue to access these resources, especially during peak usage periods. Finite source queuing models, such as the M/M/c/K model (with c servers and a finite queue of size K), are suitable for analyzing scenarios where there are constraints on the availability of resources. By understanding queue dynamics and resource utilization, developers can implement strategies to balance resource allocation, reduce wait times, and optimize player satisfaction. 4. Dynamic Queueing Models for Matchmaking Matchmaking algorithms are crucial for ensuring balanced and enjoyable gameplay experiences in multiplayer mobile games. These algorithms often involve queuing mechanisms to match players with similar skill levels or preferences. Dynamic queueing models, such as the M/M/1/K queue with dynamic arrival rates or the Erlang queuing model with variable service rates, can be employed to optimize matchmaking systems. By adjusting queue parameters dynamically based on player behavior, game developers can achieve faster and fairer matchmaking results. This may lead to increased player engagement and retention. Queuing Theory Saves the Launch Day A mobile game development company was gearing up for the highly anticipated launch of their latest game. Based on pre-registration numbers and social media buzz, they were expecting a massive influx of players on launch day. Their concern was twofold: ensuring smooth gameplay for all users and avoiding server crashes due to overload. The development team decided to use queuing theory to create a model of their game server infrastructure. Model Derivation They identified the game server system as an M/M/c queuing system, meaning: M: Player arrivals follow a Poisson distribution (random and independent). M: The time it takes to process a player's request (e.g., joining a game, updating the game state) follows a Poisson distribution (random and independent). c: Represents the number of available game servers (acting as multiple queues) Key Performance Metrics Using queuing theory formulas, they calculated the following metrics: Arrival rate (λ): Estimated based on pre-registration data and industry benchmarks for similar game launches Service time (μ): Measured by analyzing the average time it takes to process player requests during internal testing Server utilization (ρ): ρ = λ / (c * μ). This metric indicates how busy each server is on average. Model Analysis The key aspect of the model was understanding how server utilization (ρ) would change with different server configurations (number of servers, "c"). High server utilization (ρ > 0.8): Signifies servers are overloaded, leading to potential queueing delays, slow gameplay, and increased risk of crashes Low server utilization (ρ < 0.5): Indicates underutilized servers, which might be cost-inefficient, but ensures smooth gameplay Taking Action Using the queuing model, the team conducted a series of tests: Scenario 1: Existing Server Configuration The model predicted a server utilization exceeding 80% during peak launch hours, potentially leading to performance issues and frustrated players. Scenario 2: Adding 20% More Servers The model showed a utilization dropping to around 65%, offering a significant performance improvement while maintaining some buffer for unexpected player surges. Scenario 3: Doubling the Servers Utilization dropped further to around 40%, but the additional server cost might not be justified if player growth was slower than anticipated. The Decision Based on the model's predictions, the team decided to add 20% more servers to their existing infrastructure. This provided a significant performance boost without incurring excessive costs. Additionally, they implemented auto-scaling rules to automatically provision additional servers if player traffic exceeded pre-defined thresholds. The Outcome Launch day arrived, and the company saw a record number of players joining the new game. However, thanks to the queuing model and proactive server scaling, the servers handled the load efficiently. Players experienced smooth gameplay without major delays or crashes. Tool Selection There are various testing tools available that incorporate queuing theory principles. Choosing the right tool depends on the complexity of the system, the desired level of detail, and the specific testing goals. The following list is by no means exhaustive. Microsoft Excel with queuing model add-ins: Pros: Free, readily available for most users, easy to learn basic formulas Cons: Limited functionality, error-prone with complex models, not ideal for large-scale testing Online queuing model calculators: Pros: Free, user-friendly interface, good for quick estimations Cons: Limited model options, may not capture specific system details, limited customization JMeter: Pros: Open-source, robust load testing capabilities, supports basic queuing theory integration for user load simulation Cons: Setting up queuing models can be complex, requires scripting knowledge for advanced features Apache JMeter plugin - queueing theory: Pros: Extends JMeter functionalities with queuing theory models, allows for analyzing server utilization and wait times Cons: Relies on JMeter's learning curve, additional configuration needed for queuing features AppDynamics Pros: Commercial tool with a good user interface, offers performance monitoring with queuing theory insights (queue lengths, wait times) Cons: Subscription-based cost, may require training for advanced features AnyLogic: Pros: Powerful simulation software, integrates with queuing models to create complex scenarios, provides detailed performance reports Cons: Steeper learning curve, requires modeling expertise, significant cost for commercial licenses Wrapping Up Queuing theory is a valuable option for optimizing performance and resource allocation in various software development scenarios. By understanding the core queuing models and their limitations, development teams can leverage testing tools. They can analyze server utilization, identify potential bottlenecks, and make data-driven decisions. Our task could be to ensure smooth gameplay for a mobile game launch, to optimize infrastructure for a rapidly growing company, or to simply choose the best cloud platform for an application. In any case, queuing theory may empower developers to navigate the complexities of system load and create a seamless user experience.

By Stelios Manioudakis, PhD CORE

Wireshark, the free, open-source packet sniffer and network protocol analyzer, has cemented itself as an indispensable tool in network troubleshooting, analysis, and security (on both sides). This article delves into the features, uses, and practical tips for harnessing the full potential of Wireshark, expanding on aspects that may have been glossed over in discussions or demonstrations. Whether you're a developer, security expert, or just curious about network operations, this guide will enhance your understanding of Wireshark and its applications. Introduction to Wireshark Wireshark was initially developed by Eric Rescorla and Gerald Combs, and designed to capture and analyze network packets in real-time. Its capabilities extend across various network interfaces and protocols, making it a versatile tool for anyone involved in networking. Unlike its command-line counterpart, tcpdump, Wireshark's graphical interface simplifies the analysis process, presenting data in a user-friendly "proto view" that organizes packets in a hierarchical structure. This facilitates quick identification of protocols, ports, and data flows. The key features of Wireshark are: Graphical User Interface (GUI): Eases the analysis of network packets compared to command-line tools Proto view: Displays packet data in a tree structure, simplifying protocol and port identification Compatibility: Supports a wide range of network interfaces and protocols Browser Network Monitors FireFox and Chrome contain a far superior network monitor tool built into them. It is superior because it is simpler to use and works with secure websites out of the box. If you can use the browser to debug the network traffic you should do that. In cases where your traffic requires low-level protocol information or is outside of the browser, Wireshark is the next best thing. Installation and Getting Started To begin with Wireshark, visit their official website for the download. The installation process is straightforward, but attention should be paid to the installation of command-line tools, which may require separate steps. Upon launching Wireshark, users are greeted with a selection of network interfaces as seen below. Choosing the correct interface, such as the loopback for local server debugging, is crucial for capturing relevant data. When debugging a Local Server (localhost), use the loopback interface. Remote servers will probably fit with the en0 network adapter. You can use the activity graph next to the network adapter to identify active interfaces for capture. Navigating Through Noise With Filters One of the challenges of using Wireshark is the overwhelming amount of data captured, including irrelevant "background noise" as seen in the following image. Wireshark addresses this with powerful display filters, allowing users to hone in on specific ports, protocols, or data types. For instance, filtering TCP traffic on port 8080 can significantly reduce unrelated data, making it easier to debug specific issues. Notice that there is a completion widget on top of the Wireshark UI that lets you find out the values more easily. In this case, we filter by port tcp.port == 8080 which is the port used typically in Java servers (e.g., Spring Boot/tomcat). But this isn't enough as HTTP is more concise. We can filter by protocol by adding http to the filter which narrows the view to HTTP requests and responses as shown in the following image. Deep Dive Into Data Analysis Wireshark excels in its ability to dissect and present network data in an accessible manner. For example, HTTP responses carrying JSON data are automatically parsed and displayed in a readable tree structure as seen below. This feature is invaluable for developers and analysts, providing insights into the data exchanged between clients and servers without manual decoding. Wireshark parses and displays JSON data within the packet analysis pane. It offers both hexadecimal and ASCII views for raw packet data. Beyond Basic Usage While Wireshark's basic functionalities cater to a wide range of networking tasks, its true strength lies in advanced features such as ethernet network analysis, HTTPS decryption, and debugging across devices. These tasks, however, may involve complex configuration steps and a deeper understanding of network protocols and security measures. There are two big challenges when working with Wireshark: HTTPS decryption: Decrypting HTTPS traffic requires additional configuration but offers visibility into secure communications. Device debugging: Wireshark can be used to troubleshoot network issues on various devices, requiring specific knowledge of network configurations. The Basics of HTTPS Encryption HTTPS uses the Transport Layer Security (TLS) or its predecessor, Secure Sockets Layer (SSL), to encrypt data. This encryption mechanism ensures that any data transferred between the web server and the browser remains confidential and untouched. The process involves a series of steps including handshake, data encryption, and data integrity checks. Decrypting HTTPS traffic is often necessary for developers and network administrators to troubleshoot communication errors, analyze application performance, or ensure that sensitive data is correctly encrypted before transmission. It's a powerful capability in diagnosing complex issues that cannot be resolved by simply inspecting unencrypted traffic or server logs. Methods for Decrypting HTTPS in Wireshark Important: Decrypting HTTPS traffic should only be done on networks and systems you own or have explicit permission to analyze. Unauthorized decryption of network traffic can violate privacy laws and ethical standards. Pre-Master Secret Key Logging One common method involves using the pre-master secret key to decrypt HTTPS traffic. Browsers like Firefox and Chrome can log the pre-master secret keys to a file when configured to do so. Wireshark can then use this file to decrypt the traffic: Configure the browser: Set an environment variable (SSLKEYLOGFILE) to specify a file where the browser will save the encryption keys. Capture traffic: Use Wireshark to capture the traffic as usual. Decrypt the traffic: Point Wireshark to the file with the pre-master secret keys (through Wireshark's preferences) to decrypt the captured HTTPS traffic. Using a Proxy Another approach involves routing traffic through a proxy server that decrypts HTTPS traffic and then re-encrypts it before sending it to the destination. This method might require setting up a dedicated decryption proxy that can handle the TLS encryption/decryption: Set up a decryption proxy: Tools like Mitmproxy or Burp Suite can act as an intermediary that decrypts and logs HTTPS traffic. Configure network to route through proxy: Ensure the client's network settings route traffic through the proxy. Inspect Traffic: Use the proxy's tools to inspect the decrypted traffic directly. Integrating tcpdump With Wireshark for Enhanced Network Analysis While Wireshark offers a graphical interface for analyzing network packets, there are scenarios where using it directly may not be feasible due to security policies or operational constraints. tcpdump, a powerful command-line packet analyzer, becomes invaluable in these situations, providing a flexible and less intrusive means of capturing network traffic. The Role of tcpdump in Network Troubleshooting tcpdump allows for the capture of network packets without a graphical user interface, making it ideal for use in environments with strict security requirements or limited resources. It operates under the principle of capturing network traffic to a file, which can then be analyzed at a later time or on a different machine using Wireshark. Key Scenarios for tcpdump Usage High-security environments: In places like banks or government institutions where running network sniffers might pose a security risk, tcpdump offers a less intrusive alternative. Remote servers: Debugging issues on a cloud server can be challenging with Wireshark due to the graphical interface; tcpdump captures can be transferred and analyzed locally. Security-conscious customers: Customers may be hesitant to allow third-party tools to run on their systems; tcpdump's command-line operation is often more palatable. Using tcpdump Effectively Capturing traffic with tcpdump involves specifying the network interface and an output file for the capture. This process is straightforward but powerful, allowing for detailed analysis of network interactions: Command syntax: The basic command structure for initiating a capture involves specifying the network interface (e.g., en0 for wireless connections) and the output file name. Execution: Once the command is run, tcpdump silently captures network packets. The capture continues until it's manually stopped, at which point the captured data can be saved to the specified file. Opening captures in Wireshark: The file generated by tcpdump can be opened in Wireshark for detailed analysis, utilizing Wireshark's advanced features for dissecting and understanding network traffic. The following shows the tcpdump command and its output: $ sudo tcpdump -i en0 -w output Password: tcpdump: listening on en, link-type EN10MB (Ethernet), capture size 262144 bytes ^C3845 packets captured 4189 packets received by filter 0 packets dropped by kernel Challenges and Considerations Identifying the correct network interface for capture on remote systems might require additional steps, such as using the ifconfig command to list available interfaces. This step is crucial for ensuring that relevant traffic is captured for analysis. Final Word Wireshark stands out as a powerful tool for network analysis, offering deep insights into network traffic and protocols. Whether it's for low-level networking work, security analysis, or application development, Wireshark's features and capabilities make it an essential tool in the tech arsenal. With practice and exploration, users can leverage Wireshark to uncover detailed information about their networks, troubleshoot complex issues, and secure their environments more effectively. Wireshark's blend of ease of use with profound analytical depth ensures it remains a go-to solution for networking professionals across the spectrum. Its continuous development and wide-ranging applicability underscore its position as a cornerstone in the field of network analysis. Combining tcpdump's capabilities for capturing network traffic with Wireshark's analytical prowess offers a comprehensive solution for network troubleshooting and analysis. This combination is particularly useful in environments where direct use of Wireshark is not possible or ideal. While both tools possess a steep learning curve due to their powerful and complex features, they collectively form an indispensable toolkit for network administrators, security professionals, and developers alike. This integrated approach not only addresses the challenges of capturing and analyzing network traffic in various operational contexts but also highlights the versatility and depth of tools available for understanding and securing modern networks. Videos Wireshark tcpdump

By Shai Almog CORE

Unit testing is an essential practice in software development that involves testing individual codebase components to ensure they function correctly. In Spring-based applications, developers often use Aspect-Oriented Programming (AOP) to separate cross-cutting concerns, such as logging, from the core business logic, thus enabling modularization and cleaner code. However, testing aspects in Spring AOP pose unique challenges due to their interception-based nature. Developers need to employ appropriate strategies and best practices to facilitate effective unit testing of Spring AOP aspects. This comprehensive guide aims to provide developers with detailed and practical insights on effectively unit testing Spring AOP aspects. The guide covers various topics, including the basics of AOP, testing the pointcut expressions, testing around advice, testing before and after advice, testing after returning advice, testing after throwing advice, and testing introduction advice. Moreover, the guide provides sample Java code for each topic to help developers understand how to effectively apply the strategies and best practices. By following the guide's recommendations, developers can improve the quality of their Spring-based applications and ensure that their code is robust, reliable, and maintainable. Understanding Spring AOP Before implementing effective unit testing strategies, it is important to have a comprehensive understanding of Spring AOP. AOP, or Aspect-Oriented Programming, is a programming paradigm that enables the separation of cross-cutting concerns shared across different modules in an application. Spring AOP is a widely used aspect-oriented framework that is primarily implemented using runtime proxy-based mechanisms. The primary objective of Spring AOP is to provide modularity and flexibility in designing and implementing cross-cutting concerns in a Java-based application. The key concepts that one must understand in Spring AOP include: Aspect: An aspect is a module that encapsulates cross-cutting concerns that are applied across multiple objects in an application. Aspects are defined using aspects-oriented programming techniques and are typically independent of the application's core business logic. Join point: A join point is a point in the application's execution where the aspect can be applied. In Spring AOP, a join point can be a method execution, an exception handler, or a field access. Advice: Advice is an action that is taken when a join point is reached during the application's execution. In Spring AOP, advice can be applied before, after, or around a join point. Pointcut: A pointcut is a set of joint points where an aspect's advice should be applied. In Spring AOP, pointcuts are defined using expressions that specify the join points based on method signatures, annotations, or other criteria. By understanding these key concepts, developers can effectively design and implement cross-cutting concerns in a Java-based application using Spring AOP. Challenges in Testing Spring AOP Aspects Unit testing Spring AOP aspects can be challenging compared to testing regular Java classes, due to the unique nature of AOP aspects. Some of the key challenges include: Interception-based behavior: AOP aspects intercept method invocations or join points, which makes it difficult to test their behavior in isolation. To overcome this challenge, it is recommended to use mock objects to simulate the behavior of the intercepted objects. Dependency Injection: AOP aspects may rely on dependencies injected by the Spring container, which requires special handling during testing. It is important to ensure that these dependencies are properly mocked or stubbed to ensure that the aspect is being tested in isolation and not affected by other components. Dynamic proxying: Spring AOP relies on dynamic proxies, which makes it difficult to directly instantiate and test aspects. To overcome this challenge, it is recommended to use Spring's built-in support for creating and configuring dynamic proxies. Complex pointcut expressions: Pointcut expressions can be complex, making it challenging to ensure that advice is applied to the correct join points. To overcome this challenge, it is recommended to use a combination of unit tests and integration tests to ensure that the aspect is being applied correctly. Transaction management: AOP aspects may interact with transaction management, introducing additional complexity in testing. To overcome this challenge, it is recommended to use a combination of mock objects and integration tests to ensure that the aspect is working correctly within the context of the application. Despite these challenges, effective unit testing of Spring AOP aspects is crucial for ensuring the reliability, maintainability, and correctness of the application. By understanding these challenges and using the recommended testing approaches, developers can ensure that their AOP aspects are thoroughly tested and working as intended. Strategies for Unit Testing Spring AOP Aspects Unit testing Spring AOP Aspects can be challenging, given the system's complexity and the multiple pieces of advice involved. However, developers can use various strategies and best practices to overcome these challenges and ensure effective unit testing. One of the most crucial strategies is to isolate aspects from dependencies when writing unit tests. This isolation ensures that the tests focus solely on the aspect's behavior without interference from other modules. Developers can accomplish this by using mocking frameworks such as Mockito, EasyMock, or PowerMockito, which allow them to simulate dependencies' behavior and control the test environment. Another best practice is to test each piece of advice separately. AOP Aspects typically consist of multiple pieces of advice, such as "before," "after," or "around" advice. Testing each piece of advice separately ensures that the behavior of each piece of advice is correct and that it functions correctly in isolation. It's also essential to verify that the pointcut expressions are correctly configured and target the intended join points. Writing tests that exercise different scenarios helps ensure the correctness of point-cut expressions. Aspects in Spring-based applications often rely on beans managed by the ApplicationContext. Mocking the ApplicationContext allows developers to provide controlled dependencies to the aspect during testing, avoiding the need for a fully initialized Spring context. Developers should also define clear expectations for the behavior of the aspect and use assertions to verify that the aspect behaves as expected under different conditions. Assertions help ensure that the aspect's behavior aligns with the intended functionality. Finally, if aspects involve transaction management, developers should consider testing transactional behavior separately. This can be accomplished by mocking transaction managers or using in-memory databases to isolate the transactional aspect of the code for testing. By employing these strategies and best practices, developers can ensure effective unit testing of Spring AOP Aspects, resulting in robust and reliable systems. Sample Code: Testing a Logging Aspect To gain a better understanding of testing Spring AOP aspects, let's take a closer look at the sample code. We will analyze the testing process step-by-step, emphasizing important factors to take into account, and providing further information to ensure clarity. Let's assume that we will be writing unit tests for the following main class: Java import org.aspectj.lang.JoinPoint; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Before; import org.springframework.stereotype.Component; @Aspect @Component public class LoggingAspect { @Before("execution(* com.example.service.*.*(..))") public void logBefore(JoinPoint joinPoint) { System.out.println("Logging before " + joinPoint.getSignature().getName()); } } The LoggingAspect class logs method executions with a single advice method, logBefore, which executes before methods in the com.example.service package. The LoggingAspectTest class contains unit tests for the LoggingAspect. Let's examine each part of the test method testLogBefore() in detail: Java import org.aspectj.lang.JoinPoint; import org.aspectj.lang.Signature; import org.junit.jupiter.api.Test; import static org.mockito.Mockito.*; public class LoggingAspectTest { @Test void testLogBefore() { // Given LoggingAspect loggingAspect = new LoggingAspect(); // Creating mock objects JoinPoint joinPoint = mock(JoinPoint.class); Signature signature = mock(Signature.class); // Configuring mock behavior when(joinPoint.getSignature()).thenReturn(signature); when(signature.getName()).thenReturn("methodName"); // When loggingAspect.logBefore(joinPoint); // Then // Verifying interactions with mock objects verify(joinPoint, times(1)).getSignature(); verify(signature, times(1)).getName(); // Additional assertions can be added to ensure correct logging behavior } } In the above code, there are several sections that play a vital role in testing. Firstly, the Given section sets up the test scenario. We do this by creating an instance of the LoggingAspect and mocking the JoinPoint and Signature objects. By doing so, we can control the behavior of these objects during testing. Next, we create mock objects for the JoinPoint and Signature using the Mockito mocking framework. This allows us to simulate behavior without invoking real instances, providing a controlled environment for testing. We then use Mockito's when() method to specify the behavior of the mock objects. For example, we define that when thegetSignature() method of the JoinPoint is called, it should return the mock Signature object we created earlier. In the When section, we invoke the logBefore() method of the LoggingAspect with the mocked JoinPoint. This simulates the execution of the advice before a method call, which triggers the logging behavior. Finally, we use Mockito's verify() method to assert that specific methods of the mocked objects were called during the execution of the advice. For example, we verify that the getSignature() and getName() methods were called once each. Although not demonstrated in this simplified example, additional assertions can be added to ensure the correctness of the aspect's behavior. For instance, we could assert that the logging message produced by the aspect matches the expected format and content. Additional Considerations Testing pointcut expressions: Pointcut expressions define where advice should be applied within the application. Writing tests to verify the correctness of pointcut expressions ensures that the advice is applied to the intended join points. Testing aspect behavior: Aspects may perform more complex actions beyond simple logging. Unit tests should cover all aspects of the aspect's behavior to ensure its correctness, including handling method parameters, logging additional information, or interacting with other components. Integration testing: While unit tests focus on isolating aspects, integration tests may be necessary to verify the interactions between aspects and other components of the application, such as service classes or controllers. By following these principles and best practices, developers can create thorough and reliable unit tests for Spring AOP aspects, ensuring the stability and maintainability of their applications. Conclusion Unit testing Spring AOP aspects is crucial for reliable and correct aspect-oriented code. To create robust tests, isolate aspects, use mocking frameworks, test each advice separately, verify pointcut expressions, and assert expected behavior. Sample code provided as a starting point for Java applications. With proper testing strategies in place, developers can confidently maintain and evolve AOP-based functionalities in their Spring app.

By Reza Ganji CORE

Tech teams do their best to develop amazing software products. They spent countless hours coding, testing, and refining every little detail. However, even the most carefully crafted systems may encounter issues along the way. That's where reliability models and metrics come into play. They help us identify potential weak spots, anticipate failures, and build better products. The reliability of a system is a multidimensional concept that encompasses various aspects, including, but not limited to: Availability: The system is available and accessible to users whenever needed, without excessive downtime or interruptions. It includes considerations for system uptime, fault tolerance, and recovery mechanisms. Performance: The system should function within acceptable speed and resource usage parameters. It scales efficiently to meet growing demands (increasing loads, users, or data volumes). This ensures a smooth user experience and responsiveness to user actions. Stability: The software system operates consistently over time and maintains its performance levels without degradation or instability. It avoids unexpected crashes, freezes, or unpredictable behavior. Robustness: The system can gracefully handle unexpected inputs, invalid user interactions, and adverse conditions without crashing or compromising its functionality. It exhibits resilience to errors and exceptions. Recoverability: The system can recover from failures, errors, or disruptions and restore normal operation with minimal data loss or impact on users. It includes mechanisms for data backup, recovery, and rollback. Maintainability: The system should be easy to understand, modify, and fix when necessary. This allows for efficient bug fixes, updates, and future enhancements. This article starts by analyzing mean time metrics. Basic probability distribution models for reliability are then highlighted with their pros and cons. A distinction between software and hardware failure models follows. Finally, reliability growth models are explored including a list of factors for how to choose the right model. Mean Time Metrics Some of the most commonly tracked metrics in the industry are MTTA (mean time to acknowledge), MTBF (mean time before failure), MTTR (mean time to recovery, repair, respond or resolve), and MTTF (mean time to failure). They help tech teams understand how often incidents occur and how quickly the team bounces back from those incidents. The acronym MTTR can be misleading. When discussing MTTR, it might seem like a singular metric with a clear definition. However, it actually encompasses four distinct measurements. The 'R' in MTTR can signify repair, recovery, response, or resolution. While these four metrics share similarities, each carries its own significance and subtleties. Mean Time To Repair: This focuses on the time it takes to fix a failed component. Mean Time To Recovery: This considers the time to restore full functionality after a failure. Mean Time To Respond: This emphasizes the initial response time to acknowledge and investigate an incident. Mean Time To Resolve: This encompasses the entire incident resolution process, including diagnosis, repair, and recovery. While these metrics overlap, they provide a distinct perspective on how quickly a team resolves incidents. MTTA, or Mean Time To Acknowledge, measures how quickly your team reacts to alerts by tracking the average time from alert trigger to initial investigation. It helps assess both team responsiveness and alert system effectiveness. MTBF or Mean Time Between Failures, represents the average time a repairable system operates between unscheduled failures. It considers both the operating time and the repair time. MTBF helps estimate how often a system is likely to experience a failure and require repair. It's valuable for planning maintenance schedules, resource allocation, and predicting system uptime. For a system that cannot or should not be repaired, MTTF, or Mean Time To Failure, represents the average time that the system operates before experiencing its first failure. Unlike MTBF, it doesn't consider repair times. MTTF is used to estimate the lifespan of products that are not designed to be repaired after failing. This makes MTTF particularly relevant for components or systems where repair is either impossible or not economically viable. It's useful for comparing the reliability of different systems or components and informing design decisions for improved longevity. An analogy to illustrate the difference between MTBF and MTTF could be a fleet of delivery vans. MTBF: This would represent the average time between breakdowns for each van, considering both the driving time and the repair time it takes to get the van back on the road. MTTF: This would represent the average lifespan of each van before it experiences its first breakdown, regardless of whether it's repairable or not. Key Differentiators Feature MTBF MTTF Repairable System Yes No Repair Time Considered in the calculation Not considered in the calculation Failure Focus Time between subsequent failures Time to the first failure Application Planning maintenance, resource allocation Assessing inherent system reliability The Bigger Picture MTTR, MTTA, MTTF, and MTBF can also be used all together to provide a comprehensive picture of your team's effectiveness and areas for improvement. Mean time to recovery indicates how quickly you get systems operational again. Incorporating mean time to respond allows you to differentiate between team response time and alert system efficiency. Adding mean time to repair further breaks down how much time is spent on repairs versus troubleshooting. Mean time to resolve incorporates the entire incident lifecycle, encompassing the impact beyond downtime. But the story doesn't end there. Mean time between failures reveals your team's success in preventing or reducing future issues. Finally, incorporating mean time to failure provides insights into the overall lifespan and inherent reliability of your product or system. Probability Distributions for Reliability The following probability distributions are commonly used in reliability engineering to model the time until the failure of systems or components. They are often employed in reliability analysis to characterize the failure behavior of systems over time. Exponential Distribution Model This model assumes a constant failure rate over time. This means that the probability of a component failing is independent of its age or how long it has been operating. Applications: This model is suitable for analyzing components with random failures, such as memory chips, transistors, or hard drives. It's particularly useful in the early stages of a product's life cycle when failure data might be limited. Limitations: The constant failure rate assumption might not always hold true. As hardware components age, they might become more susceptible to failures (wear-out failures), which the Exponential Distribution Model wouldn't capture. Weibull Distribution Model This model offers more flexibility by allowing dynamic failure rates. It can model situations where the probability of failure increases over time at an early stage (infant mortality failures) or at a later stage (wear-out failures). Infant mortality failures: This could represent new components with manufacturing defects that are more likely to fail early on. Wear-out failures: This could represent components like mechanical parts that degrade with use and become more likely to fail as they age. Applications: The Weibull Distribution Model is more versatile than the Exponential Distribution Model. It's a good choice for analyzing a wider range of hardware components with varying failure patterns. Limitations: The Weibull Distribution Model requires more data to determine the shape parameter that defines the failure rate behavior (increasing, decreasing, or constant). Additionally, it might be too complex for situations where a simpler model like the Exponential Distribution would suffice. The Software vs Hardware Distinction The nature of software failures is different from that of hardware failures. Although both software and hardware may experience deterministic as well as random failures, their failures have different root causes, different failure patterns, and different prediction, prevention, and repair mechanisms. Depending on the level of interdependence between software and hardware and how it affects our systems, it may be beneficial to consider the following factors: 1. Root Cause of Failures Hardware: Hardware failures are physical in nature, caused by degradation of components, manufacturing defects, or environmental factors. These failures are often random and unpredictable. Consequently, hardware reliability models focus on physical failure mechanisms like fatigue, corrosion, and material defects. Software: Software failures usually stem from logical errors, code defects, or unforeseen interactions with the environment. These failures may be systematic and can be traced back to specific lines of code or design flaws. Consequently, software reliability models do not account for physical degradation over time. 2. Failure Patterns Hardware: Hardware failures often exhibit time-dependent behavior. Components might be more susceptible to failures early in their lifespan (infant mortality) or later as they wear out. Software: The behavior of software failures in time can be very tricky and usually depends on the evolution of our code, among others. A bug in the code will remain a bug until it's fixed, regardless of how long the software has been running. 3. Failure Prediction, Prevention, Repairs Hardware: Hardware reliability models that use MTBF often focus on predicting average times between failures and planning preventive maintenance schedules. Such models analyze historical failure data from identical components. Repairs often involve the physical replacement of components. Software: Software reliability models like Musa-Okumoto and Jelinski-Moranda focus on predicting the number of remaining defects based on testing data. These models consider code complexity and defect discovery rates to guide testing efforts and identify areas with potential bugs. Repair usually involves debugging and patching, not physical replacement. 4. Interdependence and Interaction Failures The level of interdependence between software and hardware varies for different systems, domains, and applications. Tight coupling between software and hardware may cause interaction failures. There can be software failures due to hardware and vice-versa. Here's a table summarizing the key differences: Feature Hardware Reliability Models Software Reliability Models Root Cause of Failures Physical Degradation, Defects, Environmental Factors Code Defects, Design Flaws, External Dependencies Failure Patterns Time-Dependent (Infant Mortality, Wear-Out) Non-Time Dependent (Bugs Remain Until Fixed) Prediction Focus Average Times Between Failures (MTBF, MTTF) Number of Remaining Defects Prevention Strategies Preventive Maintenance Schedules Code Review, Testing, Bug Fixes By understanding the distinct characteristics of hardware and software failures, we may be able to leverage tailored reliability models, whenever necessary, to gain in-depth knowledge of our system's behavior. This way we can implement targeted strategies for prevention and mitigation in order to build more reliable systems. Code Complexity Code complexity assesses how difficult a codebase is to understand and maintain. Higher complexity often correlates with an increased likelihood of hidden bugs. By measuring code complexity, developers can prioritize testing efforts and focus on areas with potentially higher defect density. The following tools can automate the analysis of code structure and identify potential issues like code duplication, long functions, and high cyclomatic complexity: SonarQube: A comprehensive platform offering code quality analysis, including code complexity metrics Fortify: Provides static code analysis for security vulnerabilities and code complexity CppDepend (for C++): Analyzes code dependencies and metrics for C++ codebases PMD: An open-source tool for identifying common coding flaws and complexity metrics Defect Density Defect density illuminates the prevalence of bugs within our code. It's calculated as the number of defects discovered per unit of code, typically lines of code (LOC). A lower defect density signifies a more robust and reliable software product. Reliability Growth Models Reliability growth models help development teams estimate the testing effort required to achieve desired reliability levels and ensure a smooth launch of their software. These models predict software reliability improvements as testing progresses, offering insights into the effectiveness of testing strategies and guiding resource allocation. They are mathematical models used to predict and improve the reliability of systems over time by analyzing historical data on defects or failures and their removal. Some models exhibit characteristics of exponential growth. Other models exhibit characteristics of power law growth while there exist models that exhibit both exponential and power law growth. The distinction is primarily based on the underlying assumptions about how the fault detection rate changes over time in relation to the number of remaining faults. While a detailed analysis of reliability growth models is beyond the scope of this article, I will provide a categorization that may help for further study. Traditional growth models encompass the commonly used and foundational models, while the Bayesian approach represents a distinct methodology. The advanced growth models encompass more complex models that incorporate additional factors or assumptions. Please note that the list is indicative and not exhaustive. Traditional Growth Models Musa-Okumoto Model It assumes a logarithmic Poisson process for fault detection and removal, where the number of failures observed over time follows a logarithmic function of the number of initial faults. Jelinski-Moranda Model It assumes a constant failure intensity over time and is based on the concept of error seeding. It postulates that software failures occur at a rate proportional to the number of remaining faults in the system. Goel-Okumoto Model It incorporates the assumption that the fault detection rate decreases exponentially as faults are detected and fixed. It also assumes a non-homogeneous Poisson process for fault detection. Non-Homogeneous Poisson Process (NHPP) Models They assume the fault detection rate is time-dependent and follows a non-homogeneous Poisson process. These models allow for more flexibility in capturing variations in the fault detection rate over time. Bayesian Approach Wall and Ferguson Model It combines historical data with expert judgment to update reliability estimates over time. This model considers the impact of both defect discovery and defect correction efforts on reliability growth. Advanced Growth Models Duane Model This model assumes that the cumulative MTBF of a system increases as a power-law function of the cumulative test time. This is known as the Duane postulate and it reflects how quickly the reliability of the system is improving as testing and debugging occur. Coutinho Model Based on the Duane model, it extends to the idea of an instantaneous failure rate. This rate involves the number of defects found and the number of corrective actions made during testing time. This model provides a more dynamic representation of reliability growth. Gooitzen Model It incorporates the concept of imperfect debugging, where not all faults are detected and fixed during testing. This model provides a more realistic representation of the fault detection and removal process by accounting for imperfect debugging. Littlewood Model It acknowledges that as system failures are discovered during testing, the underlying faults causing these failures are repaired. Consequently, the reliability of the system should improve over time. This model also considers the possibility of negative reliability growth when a software repair introduces further errors. Rayleigh Model The Rayleigh probability distribution is a special case of the Weibull distribution. This model considers changes in defect rates over time, especially during the development phase. It provides an estimation of the number of defects that will occur in the future based on the observed data. Choosing the Right Model There's no single "best" reliability growth model. The ideal choice depends on the specific project characteristics and available data. Here are some factors to consider. Specific objectives: Determine the specific objectives and goals of reliability growth analysis. Whether the goal is to optimize testing strategies, allocate resources effectively, or improve overall system reliability, choose a model that aligns with the desired outcomes. Nature of the system: Understand the characteristics of the system being analyzed, including its complexity, components, and failure mechanisms. Certain models may be better suited for specific types of systems, such as software, hardware, or complex systems with multiple subsystems. Development stage: Consider the stage of development the system is in. Early-stage development may benefit from simpler models that provide basic insights, while later stages may require more sophisticated models to capture complex reliability growth behaviors. Available data: Assess the availability and quality of data on past failures, fault detection, and removal. Models that require extensive historical data may not be suitable if data is limited or unreliable. Complexity tolerance: Evaluate the complexity tolerance of the stakeholders involved. Some models may require advanced statistical knowledge or computational resources, which may not be feasible or practical for all stakeholders. Assumptions and limitations: Understand the underlying assumptions and limitations of each reliability growth model. Choose a model whose assumptions align with the characteristics of the system and the available data. Predictive capability: Assess the predictive capability of the model in accurately forecasting future reliability levels based on past data. Flexibility and adaptability: Consider the flexibility and adaptability of the model to different growth patterns and scenarios. Models that can accommodate variations in fault detection rates, growth behaviors, and system complexities are more versatile and applicable in diverse contexts. Resource requirements: Evaluate the resource requirements associated with implementing and using the model, including computational resources, time, and expertise. Choose a model that aligns with the available resources and capabilities of the organization. Validation and verification: Verify the validity and reliability of the model through validation against empirical data or comparison with other established models. Models that have been validated and verified against real-world data are more trustworthy and reliable. Regulatory requirements: Consider any regulatory requirements or industry standards that may influence the choice of reliability growth model. Certain industries may have specific guidelines or recommendations for reliability analysis that need to be adhered to. Stakeholder input: Seek input and feedback from relevant stakeholders, including engineers, managers, and domain experts, to ensure that the chosen model meets the needs and expectations of all parties involved. Wrapping Up Throughout this article, we explored a plethora of reliability models and metrics. From the simple elegance of MTTR to the nuanced insights of NHPP models, each instrument offers a unique perspective on system health. The key takeaway? There's no single "rockstar" metric or model that guarantees system reliability. Instead, we should carefully select and combine the right tools for the specific system at hand. By understanding the strengths and limitations of various models and metrics, and aligning them with your system's characteristics, you can create a comprehensive reliability assessment plan. This tailored approach may allow us to identify potential weaknesses and prioritize improvement efforts.

By Stelios Manioudakis, PhD CORE