Feature Test Summary: : Instant On Testing For WsSecurity
**Feature Test Summary: Instant On Testing for WsSecurity**
Test Strategy
Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.
For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.
If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?
List of FAT projects affected
- com.ibm.ws.wssecurity_fat.wsscxf.1
Test strategy
- What functionality is new or modified by this feature?
- What are the positive and negative tests for that functionality? (Tell me the specific scenarios you tested. What kind of tests do you have for when everything ends up working (positive tests)? What about tests that verify we fail gracefully when things go wrong (negative tests)? See the Positive and negative tests section of the Feature Test Summary Process wiki for more detail.)
- What manual tests are there (if any)? (Note: Automated testing is expected for all features with manual testing considered an exception to the rule.)
Confidence Level
Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.
Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:
0 - No automated testing delivered
1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.
2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths
3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.
4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.
5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.
confidence level - 4
The FATs attempt to cover as much as possible such that we can release the initial support for InstantOn for WsSecurity feature.
Based on your answer above, for any answer other than a 4 or 5 please provide details of what drove your answer. Please be aware, it may be perfectly reasonable in some scenarios to deliver with any value above. We may accept no automated testing is needed for some features, we may be happy with low levels of testing on samples for instance so please don't feel the need to drive to a 5. We need your honest assessment as a team and the reasoning for why you believe shipping at that level is valid. What are the gaps, what is the risk etc. Please also provide links to the follow on work that is needed to close the gaps (should you deem it needed)
Q&A
Q: What is the purpose of the Feature Test Summary?
A: The Feature Test Summary is a document that provides a summary of the testing performed for a specific feature. It outlines the test strategy, approach, and results, and helps to ensure that the feature is thoroughly tested before it is released.
Q: What is the difference between FAT, unit, and BVT tests?
A: FAT (Functional Automated Testing) tests are automated tests that verify the functionality of a feature. Unit tests are automated tests that verify the functionality of a single unit of code. BVT (Build Verification Test) tests are automated tests that verify the build process and ensure that the feature is properly integrated into the product.
Q: What is the significance of the confidence level in the Feature Test Summary?
A: The confidence level indicates the team's confidence in the testing delivered for the feature. It helps to identify any gaps or risks in the testing and provides a clear understanding of the feature's readiness for release.
Q: What is the purpose of the cognitive testing database?
A: The cognitive testing database is a repository that stores the results of automated tests. It helps to track the progress of testing and provides a centralized location for viewing test results.
Q: What is the expected outcome of the Feature Test Summary process?
A: The expected outcome of the Feature Test Summary process is a comprehensive summary of the testing performed for the feature, including the test strategy, approach, and results. It should provide a clear understanding of the feature's readiness for release and identify any gaps or risks in the testing.
Q: What is the role of the CSI Team in the Feature Test Summary process?
A: The CSI Team provides advice and guidance on the Feature Test Summary process and helps to ensure that the results are properly recorded and stored in the cognitive testing database.
Q: What is the significance of the Positive and Negative tests in the Feature Test Summary?
A: Positive tests verify that the feature works as expected, while negative tests verify that the feature fails gracefully when things go wrong. They help to ensure that the feature is thoroughly tested and provides a clear understanding of its behavior under different scenarios.
Q: What is the expected outcome of the manual tests in the Feature Test Summary?
A: The expected outcome of the manual tests is to verify that the feature works as expected and provides a clear understanding of its behavior under different scenarios. They help to ensure that the feature is thoroughly tested and provides a high level of confidence in its readiness for release.
Q: What is the significance of the links to follow-on work in the Feature Test Summary?
A: The links to follow-on work provide a clear understanding of the gaps or risks in the testing and help to identify the next steps needed to close these gaps. They provide a clear understanding of the feature's readiness for release and help to ensure that any remaining issues are properly addressed.