A testing methodology for managing the permutations of parameters while testing an IoT based product.
The emergence of the Internet of Things (IoT) poses some interesting testing challenges that will make many Quality Analysts rethink their traditional processes.
For instance, I recently worked on a product, where a mobile app talks to a connected machine. The various states that these two devices could be in made it particularly challenging while coming up with test scenarios. I would like to present the framework that I found useful when testing the IoT-based product. This framework, which I’ve called the IoT Testing Atlas helps manage the various permutations of states that can get extremely complex in IoT deployments.
When we consider some common states or variants that is tested in a simple web application, we see four basic results:
When testing any internet application, we need to be mindful of these four states. Now, consider a mobile application. It’s imperative that we keep in mind, the additional set of states and variants that arise from operating in a mobile environment. These states include:
Now, if we then look at the variety of states that ‘connected machines’ introduce, we see four new states:
This means that even with the given set of example states, there are about 96 (4 times 6 times 4) states that the entire system can be in, at any point in time.
Each of these states cannot be treated as a standalone entity, as the state transition within a system introduces additional constraints. For example, the state change from ‘offline’ to ‘online’ is likely to trigger a set of events.The above set of parameters is only the tip of the proverbial iceberg. As we go deeper into specifics, connecting the different states to logical scenarios could become overwhelming.
When I tried using the existing web-based techniques like all pairs, equivalence partitioning, boundary value and similar, I found that they did a good job at deriving scenarios with a variable data set for a static system. These techniques apply elimination logic to arrive at the most optimum data set for testing.
For example, the all pairs technique advocates eliminating the repetitive data pair combination. But when we apply the same technique to the variable states of the system to derive scenarios, the discarded system states may leave us with a non-communicable system, making it unreliable. Nevertheless, these techniques will still work well, inside a single unit of the IoT system.
Which is why I saw the need for an IoT Testing Atlas.
Most of us would have flipped through an atlas in geography classes. The atlas, in my context, outlines all of the potential system parameters and derives meaningful scenarios that are required to test a feature.
Every system within the product is captured as a circle with its n states. The logical next states are placed close to each other. These circles are rotatable entities. I have included a separate circle for the NFRs too, because they easily slip out of our minds when we test such complex integrations.
The following image is my view of the IoT Testing Atlas:
Let’s look at a few scenarios of using the Atlas, and derive logical scenarios for some features that involve only mobile and machine interaction. This means our focus circles are Device, Machine and Network.
Similarly, the Atlas can be extended to multiple scenarios with multiple rotations. While some scenarios may not fit a current feature and some may not even be relevant to the business needs, the atlas is nonetheless exhaustive. This framework can also be extended to many systems in a product.
On a practical level, when a team that’s testing an IoT product has more than one QA, this Atlas should provide a common point of reference. The Atlas accomplishes such efficient collaboration because it covers the unique requirements for testing—the permutation of tools, devices, scenarios and protocols in a comprehensive manner.
This book provides a solid, high-level overview of how devices use Ble to…
Using the Web to Build the IoT is a free book built as a collection of six…
Whether it’s called physical computing, ubiquitous computing, or the…
Internet is huge! Help us to find great IoT resources and products.
The ioTStash is a collection of all the best resources and products related to the Internet of Things.
Discover the Internet of Things and beyond.
Leave a Reply