Summarized test Report for 1st NGN Interoperability testing event
(Sintesio test site, Bled, 15 – 19 May 2006)
ISUP/SIP Profile A and B interworking tests
(ETSI DTS 06014-1: Interworking between SIP and BICC
Protocol or ISDN user part, Profile A and B)
Scope
The 1st NGN interop event hosted by Sintesio test lab, in cooperation with ETSI Plugtest took place May 15-19, 2006. While the overall test scope is focused on PSTN/ISDN emulation service scenarios, performing end-to-end network integration test and SIP/ISUP profile A and B, the document provides an overview of test results for SIP/ISUP profile A/B test campaign.
The report includes schematic presentation of test-bed, list of tests and corresponding results. Detailed listings and printings of test results are available on request.
Table of Contents
Test campaign
Test-bed
The figure below provides an overview of test-bed with the list of required equipment needed for testing activities.
Figure 1 - schematic overview of test-bed configuration
Required hardware consists of:
-
System under test (Vendor#1 softswitch )
-
Catapult p300, which plays a role of IP <-> ISUP/E1 signaling converter.
-
Testing server with TTWorkbench, where TTCN-3 based test cases are implemented, managed and executed; also the computer, where encoding/decoding SW adapters are deployed.
SIP and ISUP signaling connections are generated between testing server and system under test.
Test execution
Tests were executed automatically using TTCN-3 test campaign tool (TTWorkbench) provided by Testing Tech. Test scope was based on the TISPAN DTS 06014-1 (“Interworking between Session Initiation Protocol (SIP) and Bearer Independent Call Control Protocol or ISDN User Part, Profile A and B”).
Test approach
Test cases are described with TISPAN DTS 06014-1 document and realized in TTCN3 code. TTCN3 code with all test cases should be validated and officially announced.
Test cases were chosen and done randomly. Result of single test case is described with overall verdict, which can be:
-
Pass (test is passed successful)
-
Fail (test isn’t passed )
-
Error (test is stopped manually because of some problems)
Recommendation of ETSI standards are satisfied with passed tests. If the test is not passed, then it should be discovered about the origin of the problem. Sometimes the problem is solved with re-setting of right parameters value (PISC or PIXIT parameters) or different settings on system under test. Otherwise the exchange doesn’t work right.
Results
General report
The test bed configuration was prepared on Vendor#1 test bed prior the start of the event. While the Vendor#1 softswitch on Sintesio test bed was replaced by the new one, a configuration database on Vendor#1 softswitch was copied.
One working day was needed to set up the test bed hardware in Sintesio lab, but additional working day was needed to start executing tests using testing server, due to the problems with configuring Vendor#1 softswitch to support SIP subscribers when the database configuration was copied among both Vendor#1 softswitch.
The test campaign itself last 24 hours. All test cases were executed automatically using TTWorkbench.
The testing team was set-up by:
-
2 testers with the knowledge of TTCN-3, TTWorkbench, Java programming and SIP/ISUP
-
1 person responsible for the maintenance and configuration support on Vendor#1 softswitch
Results on executing test cases
Following table (see below) contains a list of executed test cases and corresponding results, including:
| Test case Id. |
Verdict |
Observation |
| 101_010 |
Fail |
Continuity check indicator = 00 instead 10 |
| 101_011 |
Fail |
Continuity check indicator = 00 instead 01 |
| 101_023 |
Pass |
|
| 101_027 |
Pass |
|
| 101_028 |
Pass |
|
| 101_031 |
Pass |
|
| 104_002 |
Pass |
|
| 106_001 |
Pass |
|
| 107_001 |
Pass |
|
| 108_002 |
Pass |
|
| 108_003 |
Fail |
603 instead of 480 |
| 108_003 |
Fail |
603 instead of 480 |
| 108_006 |
Pass |
|
| 108_007 |
Fail |
404 without reason |
| 108_008 |
Fail |
486 without reason |
| 109_001 |
Pass |
|
| 109_002 |
Pass |
|
| 110_001 |
Pass |
|
| 110_004 |
Pass |
|
| 110_006 |
Fail |
Wrong CallID in message BYE |
| 110_010 |
Fail |
503 Service Unavailable |
| 301_001 |
Pass |
|
| 301_002 |
Pass |
|
| 301_003 |
Pass |
|
| 301_004 |
Pass |
|
| 301_007 |
Pass |
|
| 301_037 |
Pass |
|
| 301_038 |
Pass |
|
| 301_039 |
Pass |
|
| 301_040 |
Pass, |
|
Total of 30 test cases are executed; among these, 22 passed and 8 failed, as seen on the figure below:
Figure 2 - Graphical presentation of test results
Example of the result of the test verdict as passed can be seen on the figure below.
Figure 3 - Sample of test execution result as provided by TTWorkbench.
The following reasons were the major cause of unsuccessful verdicts during the test campaign:
Problems observed during the test campaign
In first tryouts, the ALERT message was not received on testing server; this was due to the fact that the database configuration for Vendor#1 softswitch was not copied correctly from the one on original test bed in Iskratel to the one located in Sintesio test lab.
Conclusion
This testing concept is very good because tester can execute a lot of tests in short time; our experience is that the test bed was full configured and prepared prior to starting test campaign; at least 15 test cases can be executed per day.
Problem is only with setting of some parameters but when they are defined and saved, tests can be automatically executed. It is obvious that the usage of advanced TTCN-3 tools, like TTWorkbench, enables the analyses of test cases “on-site” because all messages with appropriate reference parameters are already located there.
|
