"Protocol testing is the process of verifying different features of a protocol used in data communication and tele communication devices like routers, switches etc to ensure that it is working as per the rules and regulations mentioned in the standard"
The programs which does different functions of these communication devices are generally called as protocols and hence the term protocol testing is used for the testing of these programs. Protocol testing can be considered as a subdomain of software testing where the software that you are testing is the operating system of the DUT.
In manual testing, a tester will be responsible for different tasks such as configuring the protocols, traffic generation/capturing, checking the log files, verification of the expected results etc.
In automated testing a scripting language like Python, TCL or Perl will be used to do all the above mentioned tasks. Automated testing can really boost the speed of test execution and thus reduce the overall time for project completion. Protocol testing involves configuration of the devices either by CLI or GUI. These scripting languages will be useful for automated testing when the devices are configured using CLI. If GUI is used for configuration, testing tools like selenium will be used for automated testing. Though automated testing reduces test duration, protocol testing done manually gets more bugs and thus cannot be completely replaced by automation.
In protocol testing, Smoke testing and Sanity testing are synonymous. This involves checking the basic features of the protocol in a quick and broad manner. This type of testing is done normally before a build or release is accepted for system testing, to check whether it is possible and reasonable to continue the testing process .
Conformance testing is the checking of the program's compliance with the standards. Protocols or standards are defined by organizations like IEEE, IETF, ANSI, ETSI etc. Conformance testing involves checking whether the protocol's behavior is exactly same as mentioned in the standard document. Protocol testing projects normally starts with conformance testing process.
In the context of protocol testing, verification of different features or functionalities of a protocol to ensure that it is working as per the functional specification is called as functional testing.
Performance testing is the verification of the protocol's / DUT's stability and efficiency under high work load. CPU utilization, Memory utilization, Bandwidth utilization etc are normally monitored during performance testing. Protocol testing tools like IxNetwork, Ixload, IxExplorer, Smartbits, Router tester etc are commonly used performance testing.
Scalability testing is checking the scalability of the protocol's features by increasing work load more than expected. This is used to check the user experience, resource utilization and any unexpected issues like system crashes. Performance testing , load testing and scalability testing are normally grouped together in protocol testing projects.
Testing your DUT with devices from other vendors is known as interoperability testing. If protocol testing is carried out only by using devices of the same vendor you will not be able to find bugs that will happen in real time scenarios where your device will be connected with devices from other vendors. Interoperability testing doesn't include all cases of functional testing, it will be broad testing of the features where interactions with two devices are required.
An existing working program can get effected when it is updated by using patches or when interfaced with a new program and can introduce new bugs to the system. Regression testing is used ensure that new bugs are not emerged as part of a system update. In protocol testing projects, regression testing is normally done by using automated test scripts.
Common user activities , accidental events and some negative events are tested a part of user-scenario testing.
Stress testing is the testing the features of a protocol beyond the limits of normal operation. Traffic generating tools and simulators are normally used to create abnormal circumstances to understand the breaking point of the system.
Negative testing used to check the behavior of the protocol when invalid inputs or unexpected scenarios are encountered. Negative testing is also used to break the system intentionally and check whether it is recovering gracefully.
Benchmark testing is a type of protocol testing which is used to find out the performance metrics of a DUT with given set of industry standard test cases. This can be used to determine the performance of your DUT and compare with the industry bests.
Rather than checking the predefined test cases with the expected results, exploratory testing gives more freedom to the tester to investigate and explore different applications of the protocol. An experienced tester will be able to check all nooks and cranny of a program and can uncover new bugs which might not happen with normal scenario testing. Exploratory testing require more time and effort but at the same time it does not guarantee to dig up new bugs. So this type of testing is kept as optional in most of the protocol testing projects.
Competitive testing is done by executing the same test cases with devices of different companies and comparing the results to understand the differences in performance, feature and usability. Comparing the results of your device with a competitor's provides valuable insights to new features and tweaking the performance.
In the traditional waterfall model , when the development process is over, an independent team of testers will do the testing according to a well defined schedule. These protocol testing projects will be normally of durations of 2 to 4 months. The risk with this method is that if testers find many bugs during testing cycle, developers will not get enough time to fix the issues before the scheduled release date.
In agile testing method, testing is not considered as a different phase, but it is done together with the coding process. In this model of protocol testing, bugs are found immediately and developers will fix the issues before moving to the next feature. Regression testing will be done when each module is added to check whether any new bugs are introduced because of the modifications. Agile testing model is recently introduced to protocol testing and is gaining popularity .
Though companies does not strictly follow a common standard approach to protocol testing projects , following are the commonly executed phases
Like in other testing process , protocol testing projects also starts with the requirement analysis process of the SDLC. In this phase testers has to understand the requirements and decide what has to be tested, which features has to be included or excluded, which types of testing are required and document the questions if any clarification is required.
Test planning phase is a where a test strategy is developed. The test objectives, scope, resources required, risk involved, effort planning etc are defined in this stage. This is considered as most import phase of a protocol testing project because any mistake in this phase can lead to schedule delays and cost overruns.
Once the objectives and scopes are decided , tester can develop test cases according to the features mentioned in the functional specification. To develop test cases for protocol testing , testers should refer the standard documents to understand the exact working of the protocol.
Manual testing involves topology setup, configuration of the devices/testing tools and verification of expected results.
Devices used in protocol testing like routers, switches, firewalls etc are normally configured using command line interface. Test automation is done with the help of automation frameworks using scripting languages like Python, TCL and Perl.
Though regression testing can be done in manual testing and automation phases, it is common to have a regression testing phase at the end of every protocol testing project to make sure that modifications done during the testing phase has not introduced new bugs.
Protocol testing projects are considered as completed when the exit criteria is met. Test closure phase involves documentation of results, important bugs and issues happened during the project. This document will be used a guide line for the next project.