Automated Testing: A Practical Approach for ERP product
Kishore C.S.Manager Technical
Ramco Systems
Chennai
Email: cs@rsi.ramco.com
Abstract
Objectives of introducing automation of Testing for ERP product are to reduce costs of testing, ensure improved quality through repeated logical test suite execution. Data driven applications need a careful approach during automation of testing. Constrained with immature state of testing practices in Software Industry and limited automation tools, the attempt to automated ERP product testing has been a learning experience. The paper discusses automation experience of ERP Product with existing constraints to add value to the testing process. First Utility of Automation has been to reduce data upload time for a test setup of ERP Product. Second Utility of Automation has been to perform Installation testing of ERP product. Usage of Software Development Principles ensured that quality of Test Automation is good and gives adequate return on Investment. Test Script generation based on functional specification of the product has been used to reduce test script development time in addition to programming of test scripts.
Introduction
Ramco ERP product is a comprising of applications covering Logistics, Sales, Financials, Production, Maintenance, Human Resources, Treasury and Quality Control Modules. The completely integrated suite of applications requires dedicated effort of testing. Over the years, the company has invested man-months in ensuring quality of product versions. Automation of Testing has been initiated more than 6 years ago. The task has been championed by Quality Assurance Group. GUI Testing Tool has been developed to ensure conformance of product with GUI Standards. An attempt to automate functional testing of the product has been launched 3 years back. The exercise started with evaluation of available automated testing tools.
Initial Experience
Ramco ERP Version 3.0 has been subjected to two cycles of automated functional testing. A team of 20 members was involved in generating test scripts covering all functional areas. Around 5000 test scripts/test data files were prepared during this exercise of 2 months.
Areas of improvement identified from this Initial Exercise are:
End of the exercise we had a test suite that is difficult to be maintained or limited reuse for next verson of the product. Return on Investment was not satisfactory.
Learning from Experience
When next version of Ramco Marshal Suite was being developed the team involved in automated testing went through a learning experience. The team realised the mistakes in initial experience. This resulted in a search for best practices in test automation. Summary of the lessons learned from experts in automation [1,2,3], is given below:
Record and Playback approach to creation of test scripts and test suites is easy to develop but difficult to maintain. Especially in data driven tests achieving reusability of test scripts will be very limited. Error recovery cannot be incorporated by just recording a test script. ERP Applications depend on consistent data. Creation of test data and integration of the same with test scripts is the time consuming part. Our experience showed that record and playback could be used only to a limited extent. A sample recording for a smoke test of application can be more than 200 lines long. When the same is coded with data input file, it will be less than 40 lines. This forced us to adopt ‘test development’ approach instead of relying on ‘test script recording’.
Test Automation is Software developmentIt is necessary to design test programs, develop reusable programs, review test scripts and test the test scripts to ensure the quality of test automation. Bugs in Test automation can upset basic expectations from test automation. Test developers need to have high quality consciousness.
It is not possible to achieve automated testig in all types of testingInitial management expectations on automation were aimed at replacing manual testing efforts gradually. It is necessary to understand that automation can aid manual testing effort but cannot replace manual testing. What machines are good at and humans are slow at should be chosen for automation instead of choosing what is difficult for human as the tests to be automated. Setting realistic goals in early stages of test automation is important for achieving long-term success. Typical test that cannot be automated is usability testing.
Error Recovery routine is very important for automated test suite to be successful.Error Recovery routine enables the test suite to run continuously unattended. The function of this routine is to anticipate errors, decide on corrective action, log the error and proceed further with next test, if possible. E.g. if unexpected termination of application under test happens, the routine should recognize the interruption and restart the application. This prevent cascading effect or reporting wrong defects after a test suite execution.
Test Automation will have bugs.Test scripts should be tested before they are used in a test suite. Testing of all test scripts should be planned in test automation activity. This learning forced us to recruit best people for the job, ensure test script development is of good quality, adequate tests are performed for each test script. When Test Error simulation and rectification is difficult and time-consuming process reporting false errors can cost more to the organization and defeat the objective. The goal for the automation team has to be that a test program should never report a false problem.
Not all Developers take testing activity seriously. Developers focus more on software development tools. Motivating testers is a difficult task. Testers are worried about their career chart and their skills set in various development tools and tend to lose interest on ‘Testing’ and ‘Test Automation’ as a career.
Test scripts developed in Initial exercise were reviewed and coding standards for test scripts were prepared. Checklists have been developed for review of test scripts. However, the time investment in this activity turned out to be on higher than the expected time. So efforts were spent to attempt ‘Test Script generation’ which is discussed later.
Functionality of test scripts that was repetitive was identified and around 50 reusable test functions were developed and reused
This ensured that failures in test execution are effectively trapped, interpreted and suite continues to run without failures. Without such an error recovery system, automated test suite runs will never take off. Manual presence will become a necessity during test suite execution. Despite our best efforts, our experience has been that there were very few test suites that run successfully. The interruption of test suite by test application or test script bug effected test suite executions frequently.
These scripts are executed before any other functionality test can be executed on the product. Typical data setup for ERP Suite testing manually used to be a time consuming exercise. Definition of 400 Item codes, 500 Account Codes, 500 Customer Codes etc. used to span around 10 days of effort. Automation has reduced the time taken for setup. Creation of data takes 3 days of effort once automation has been done resulting in faster setup of test data for further functionality testing.
To reduce the efforts required for test script development and to ensure quality of test scripts, a ‘script generator’ has been developed. The script generator takes information on user control of application interface, properties of each control, data conditions for which functional testing needs to be performed. The tool generates necessary tests for the given interface. This has reduced the test script development time from 2 days to 0.5 days per unit. Moreover, quality of test automation has improved. However, not all functional tests could be generated using the script generator. The tool was able to generate 80% of test cases in some application units and only 20% in some application units.
Tests that are repeated most often in product development cycle have been identified. These tests include Installation testing, Stability Testing. Special test suites have been prepared to cover these test areas.
Stability Testing: Periodic Product builds are tested for Product Stability by running the test suites. These tests were not highly dependent on data and hence could be executed in shorter time frame. Over a period of 3 months, around 80 product stability defects were discovered through repeated stability tests that could not be uncovered during manual testing.
Installation Testing: ERP product installation is an elaborate process that needs to be verified for accuracy by running installation test suite. This suite will start after a product installation as per a pre-identified configuration. The test suite will go through a normal sequence of Product startup and cover functional areas of various critical masters and transactions of all modules. During recent release of Ramco ERP Suite Product version, this installation test suite has been executed more than 15 times and critical showstopper functional defects have been uncovered. The same has been used to test service packs of the product also.
Limitations
The following are limitations of Automation that should be considered while planning for test automation.
- Immature testing Processes and methods
Software testing practices are still evolving. We continue to grope for better testing methods to ensure better quality products. In such scenario, ensuring test automation can cover all new methods of testing is not possible. It should be realised that some of the test cases cannot be automated or is not worth automation also. E.g., Usability testing cannot be automated and tests that need to be executed only once in a product life cycle are not worth the effort. So Test planning for automation must focus on areas that are worth automation.
- Technical Limitations of tools
Tools will have technical limitations that prevent efficient automation. E.g. in our Product Suite Spreadsheet data manipulation could not be achieved using any of the testing tools available. This limited the number of test cases that can be automated. So it is necessary to evaluate testing tools for critical interfaces of Application that need to be automated. Identification of technical limitations after purchase of tool can upset the automation program.
- Need for development/training of ‘Test Developers’
Indian Software training institutes offer extensive training on development tools. They are not focusing adequately on software engineering principles. There is lot of scope for improvement in the attitude of ‘the trained’ towards quality and testing methods. This state of affairs makes it very difficult to allocate resources to testing and keep them motivated. So to meet the resource requirement inadequately skilled people should not be deployed in test automation. In such scenario test automation cannot meet expected requirements. Indian software community needs to make conscious efforts to give adequate focus to Testers and Test Developers as career options and pay them at par with Software Developers. If the discrimination in pay continues, Industry will continue to recruit testers who will never meet our expectations and testing will continue to be a concern area for organizations.
- Difficulty in measuring Return on Investment of Test Automation
Quantification of return on investment will vary from situation to situation. There is no easy method to determine this. If number of test cases that are automated can be quanitified, then it is possible to quanity percentage of automation and make an estimate of efforts saved. To achieve this test case quantification should be done first. In our experience we quantified the effectiveness of test automation by the number of times the test suites were executed and the number of critical defects that were uncovered during the test suite execution.
Conclusions
Organizations realize that automation is the only long-term solution for reduced costs in software testing and better quality products. In eagerness to achieve fast results, basics of testing and test automation tend to be overlooked. This will only result in wasted effort and frustration. Sometimes it may even result in doubts about the test tools that are used. However, the problem does not lie with the available test tools. All Test tools provide basic features of test recording and test scripting. Methods and Practices used for test automation will determine success of test automation and not changing tools or people involved in the activity. Limitations mentioned in this paper also need to be taken into consideration while setting goals for automation. Take positive steps towards automation with reality in focus and achieve success.
"Take care with test automation. When done well, it can bring many benefits. When not, it can be a very expensive exercise in frustration". [4]
References