Table of Contents

1.     Introduction to QA Statement of Work.

1.1       Purpose and Background.

1.2       Mission understanding.

1.3       Requirements for project success.

1.4       Test strategy for integrative test phases.

1.5       Traceability of requirements (requirements management)

1.6       Agile approach in the overarching test activities – methods and mindset.

1.7       API Testing Approach.

1.8       API Automation.

1.9       Quality Management Approach.

1.10     Overview of QA Contractor Tasks & Deliverable Process.

2.     Statement of Work.

2.1       Task 1:  Quality Management Planning.

2.2       Task 2:  Quality Control (QC)

2.3       Task 3:  Quality Assurance (QA)

2.4       Task 4:  Independent Verification and Validation (IV&V) Testing.

2.5       Task 5:  Risk Assessment.

3.     Deliverable and Payment Schedule.

4.     Appendices.

Introduction to QA Statement of Work

1.1 Purpose and background

Contractor shall perform Quality Management services (“QA”) for the Authorized Purchaser (hereinafter, “Agency”) in relation to Agency’s XYZ Project (henceforth “Project” or “XYZ Project”).  The purpose of the Contractor’s work is to assure that appropriate levels of Quality Management activities are performed throughout the Project lifecycle and/or for a specific portion of the Project lifecycle.  These activities shall provide Agency with appropriate visibility into the (type of) processes being used and the products being built in the Project, especially by its Design, Development, and Implementation Contractor (henceforth “Development Contractor”).  These Quality Management activities must be sufficient to assure that the project satisfies the needs for which it was undertaken and that Project risks are well understood and appropriately mitigated or managed.

(Provide a brief description of the project and its purpose, or refer to an attachment that provides a detailed description of the project and its purpose, or both.)

1.1 Mission understanding

<client name> is planning a fundamental (re)design of <project> to achieve substantial and sustainable cost reductions and efficiency gains

<client name> opted for a hybrid project methodology that combines the advantages of project management methods with the agile methodology of SAFe Essential. A distinction is made between a control level and an implementation level.

The quality is ensured by the test activities. For this purpose, a test, integration, and release management (TIRM) is provided on the control level, which is responsible, among other things, for the test strategy for the overall program and the tooling for test automation and has the corresponding guideline competence for testing.

For the implementation level, support is provided by test specialists who accompany the operational test activities within the feature teams

Tests are carried out risk-based according to the agile procedure – Essential SAFe – including test automation and continuous testing at the implementation level before the final integration and acceptance test phase

The three test levels (components, system integration, pre-prod) are carried out in three test environments (development, integration, pre-prod)

1.3 Requirements for project success

Commitment and acceptance for the professional, technical and organizational target image as well as the TIRM framework

Active and continuous cooperation of the future users in the project

The acceptance of test cases and test results is the responsibility of internal resources

Continuous change management process (communication plan, regular communication in the organization to stakeholders and all affected employees, roadshows)

Close involvement of software manufacturers to address functional deficits quickly, directly, and to define robust release planning for the project

Resilient, realistic roadmap and planning with a perfect balance of scope and affordability as well as a high level of “robustness” against disruptions through the planning of buffers on the critical path. Open project culture with clear risk management and transparent and timely reporting. The aim is real-time reporting from the SDLC (Software Development Life Cycle) systems.

The responsibility for test preparations (test case creation, test script creation, modifications, etc.) and implementation lies with the feature teams.

1.4 Test Strategy for integrative test phases

We will create a central test strategy including a comprehensive test concept for all projects and feature teams. The necessary processes, the tools to be used, and the quality goals of the test execution and documentation are communicated in a mandatory manner for everyone.

1.5. Traceability of requirements (requirements management)

With the following assignments, we enable the traceability of the work packages through the software development lifecycle: From business processes to user stories, coverage through test cases, and the recursive traceability of error states to user stories and requirements. Requirements, test cases, and errors are automatically assigned. The synchronization between the software development lifecycle tool Jira for requirements for work packages will play a central role.

1.6 Agile approach in the overarching test activities – methods and mindset

In order to be able to establish an efficient test procedure, all team members must be involved in the test procedure. In addition, the test preparations must be coordinated and planned with all parties involved

  1. In order to be able to achieve the success of an agile approach (Scrum), the planned test procedure (test goals, test planning, test model, test environments, and test tools) and the ideas are presented to all feature teams in kick-off events (sprint zero) conveyed this agile approach. The feature teams are closely involved in the planned procedure and are involved in the decision on the adopted procedure. Towards the end of a sprint, retrospectives are used to analyze and assess what can be improved in the test process.
  2. By using an automation framework (framework specifications for continuous integration), timely feedback to the respective feature team on the developed code is possible. This allows errors to be identified and corrected in the short term. The quality of the software to be generated is continuously improved and the risk of many or serious errors at the end of the sprint is reduced. For this purpose, the expected results are defined and the test cases are created parallel to the software development and carried out automatically on a daily basis.

1.7 API Testing Approach

API Testing Approach is a predefined strategy or a method that the QA team will perform in order to conduct the API testing after the build is ready. This testing does not include the source code. The API testing approach helps to:

  1.     Understanding the functionality of the API program and clearly define the scope of the program
  2.     Apply testing techniques such as equivalence classes, boundary value analysis, and error guessing and write test cases for the API
  3.     Input Parameters for the API need to be planned and defined appropriately
  4.     Execute the test cases and compare expected and actual results.

1.8 API Automation

Either it will be done using REST Assured or Postman. Depending on the automation tool used for automating other testing features a decision will be made.

i.e. If the testing of the GUI part will be also automated, then REST Assured java libraries will be used. If the goal is only to automate API, then the Postman tool will be used.

1.9 Quality Management Approach

The approach to ensure that the appropriate quality management and risk management activities are conducted shall be based on the International Software Testing Qualifications Board’s (ISTQB) Standard as described in the Advanced Level Syllabus Test Manager, version 2012 (or later) and International Standards Organization standard ISO12207 for system life cycle management.  This includes activities of Quality Management that determine the quality policy, objectives, and responsibilities, and implements them by means such as quality planning, quality control, and quality assurance.

Quality Management is defined as “a subset of project management that includes the process required to assure that the Project shall satisfy the needs for which it was undertaken.”  Quality Management consists of activities in quality planning, quality assurance, and quality control.  Where appropriate, quality control activities shall incorporate independent verification and validation (IV&V) activities, which are independent in the sense that they are conducted independently of the Development Contractor.  The contractor shall apply a risk management approach, in accordance with the standard identified in the last paragraph, in all aspects of quality management. 

For the purpose of this document, the term “quality standards” shall refer to both Project “process” and “product” quality standards.   “Process” quality standards shall cover organizational influences, management support, decision drivers, project management, schedule, resourcing, experience, and others.   “Product” quality standards shall cover product content, design, development, deployment, environment, technology, security, maintainability, and others.

The XYZ Project’s Quality Management approach shall adhere to and include, but not be limited to, work activities in support of the following quality management and related processes, all of which together comprise the State’s independent verification and validation requirements:

Quality Planning – Identifying and/or verifying quality standards that are relevant to the project and determining how to satisfy them.  Quality Planning shall include the preparation of a Quality Management Plan based on the review of the Project’s Quality Management Guide, Agency policy and standards, statewide policy and standards, and other Project-specific materials to identify quality standards and checklists relevant to the Project.  Any such matter is relevant if its exclusion from the project may likely result in low-quality outcomes creating significant risk to the Project’s success.   In addition to identifying the relevant quality standards and checklists, quality planning involves determining how to satisfy each quality standard within the constraints of a project’s schedule, available resources, and internal policies and procedures. The Quality Management Plan must address how these quality standards will be implemented, inspected, controlled, and reported.  Comprehensive quality planning includes the following companion deliverables: 

1.   Identification of relevant quality standards, beginning with the Standards set out in Appendix A to this SOW, compiled within the context of operational definitions related to the risk status of a standard’s fulfillment at a given point in the project; i.e., low, medium, or high risk.  See, Task 1, Deliverable 1.1.

2.   Compilation of quality checklists with focal points related to project standards and particular project deliverables identified for quality control review.  See, Task 1, Deliverable 1.2.

3.   Development of a quality management plan that sets out the manner and means of the QA Contractor’s performance of all required quality management services.  See, Task 1, Deliverable 1.3.

4.   Development of a Baseline Plan for implementing the Agency-accepted, comprehensive quality management planning system.  See, Task 1, Deliverable 1.4.

Quality Control – Quality Control involves monitoring both the process and the products, to determine if the project is meeting relevant quality standards and identifying ways to mitigate risk or eliminate causes of unsatisfactory results at the work product level.  Examples of Quality Control techniques include:

  1. Initial Assessment – review of key project documentation (i.e., business case, project charter, business requirements, technical documentation, management plans, the work breakdown structure, project schedule and budget – original and current baseline,  and project reports, etc.) and interviews with key business and technical staff.
  2. Peer Review / Work Product Review – a methodical examination of work products to identify defects and other needed changes. Examples of work product review methods include inspections, structured walkthroughs, and active reviews.
  3. IV&V Testing – independent verification and validation, typically on a sampling basis, to ascertain that the products for each phase in the software development life-cycle satisfies relevant standards, practices, and requirements for correctness, completeness, consistency, and accuracy.  IV&V augments the Development Contractor’s internal software testing and QC activities.

Quality Assurance – Periodic executive review and evaluation of the management processes as well as overall project performance to assure that the Project will satisfy relevant quality standards.    Quality assurance includes the periodic review of key project processes, documentation (i.e., business case, project charter, project schedule and budget – original and current baseline, requirements, designs, and project reports, etc.), and interviews with key business and technical staff. Quality assurance also includes evaluating, identifying, and recommending adjustments to the activities or tasks (and associated resources) that must be performed in the project to provide confidence that the project will satisfy the relevant quality standards.

Risk Management – In all aspects of quality management (i.e. quality planning, quality control, and quality assurance), risk management methodology shall be applied to characterize risks at the level of work product, process, and the overall Project. Risk management includes the identification of risks, the thorough assessment of the probability and the impact for the occurrence of risks, and the planning of viable responses that include, but are not limited to, mitigation, contingency, and avoidance strategies.

1.10 Overview of QA Contractor Tasks & Deliverable Process

The QA Contractor shall perform five primary Tasks:

  1. Task 1 – Quality Management Planning
  2. Task 2 – Quality Control
  3. Task 3 – Quality Assurance
  4. Task 4 – Independent Verification and Validation Testing
  5. Task 5 – Risk Assessment

Together, these Tasks help Agency assure that the Development Contractor applies best practices in project management, including quality management, and delivers technical work products that meet or exceed Agency requirements in respect to schedule, cost, functionality, reliability, security, and other relevant quality standards. 

Deliverable Requirements:

The QA Contractor shall meet deadlines, and provide required Deliverables within the QA Contract requirements. Each task has one or more Deliverables. 

Deliverable Process: 

The Development Contractor may implement the XYZ Project in phases by sub-systems where appropriate, with corresponding QA Contractor Deliverables conforming to the delivery schedule of the Development Contractor.  As a result, the QA Contractor must align its work schedule to that of the Development Contractor as approved by the Agency Project Manager. Approved Development Contractor project plans, Development Contract statement of work and related Development Contract terms & conditions shall serve as the basis for identifying Development Contractor activities.  Certain QA Contractor Deliverables and their due dates are dependent on Development Contractor deliverables.  Subject to the Agency Project Manager’s approval, the QA Contractor must adjust its Project plan on an on-going basis.  For further details on the Development Contractor procurement, see Attachment X, Development Contractor Statement of Work.

Acceptance Process:

The QA Contractor will have full responsibility for the Deliverables and Tasks listed in this SOW and shall treat the Deliverables and associated Tasks as formal work requirements. QA Contractor shall describe how deliverable due dates are met in the Baseline Project Plan.  

All QA Contractor Deliverables and work products shall be submitted to the Agency Project Manager for acceptance and approval. (Note: The “Agency Project Manager” means Agency representative, or it’s duly authorized delegate, who coordinates Agency responsibilities under the Contract with Contractor and oversee all aspects of the Work.) The Agency Project Manager may request that a Deliverable outline be submitted for approval prior to work commencing on the Deliverable.  All correspondence and documentation shall be delivered in both paper and electronic format with a signed cover letter.

Due dates for Deliverables refer to the first submittal of Deliverables to the Agency Project Manager.  The Agency Project Manager shall review the Deliverable and provide written comments or provide direction or provisional approval with or without comments to the QA Contractor within xx (number) business days.   

The QA Contractor shall address the Agency Project Manager’s evaluation comments, and submit the final Deliverable within yy (number) business days from QA Contractor’s receipt of Agency Project Manager’s comments.

2. Statement of Work

2.1 Task 1: Quality Management Planning

The QA Contractor will work with the Project’s management to produce a Quality Management Plan (QMP), including all required quality standards, checklists, report templates, and processes.  The QA Contractor will define the Project work tasks and resourcing that will add value and/or reduce risk subject to the constraints of available Project resources and organizational policies, Agency rules, and Revised Statutes, Administrative Rules, and statewide policies and standards. 

The QA Contractor’s work must adhere to generally accepted industry practices for project management and system life cycle management.  In addition, the QA Contractor will utilize the Project’s Quality Management Guide as a reference.  Deliverables, unless otherwise noted, shall conform to the International Software Testing Qualifications Board’s (ISTQB) Standard and International Standards Organization standard ISO12207 for software development life cycle management.

Based on the QMP, the QA Contractor shall define the Baseline Project Plan and subsequent Project management-related Deliverables.  The QA Contractor will ensure that the Project plans, standards, processes, and work tasks fit the Project’s needs and verify that they will be usable for performing quality control reviews, inspections, and uncovering quality risks throughout the life cycle of the Project.  The emphasis of this task is to confirm that quality is planned into the Project versus a reactive quality approach that measures quality through audits completed after the work is done.  Additionally – at the end of each Project phase and/or at the Project’s completion – the QA Contractor will facilitate a meeting to identify the lessons learned and record the results in the lessons learned and Project evaluation report.

Specifically, the QA Contractor shall document the required quality planning activity in Deliverables 1.1 through 1.4:

Deliverable 1.1:  Quality Standards – Operational Definitions

Develop and submit a written operational definition of the quality standards (“Quality Standards”) for the Project products and processes, which describe in very specific terms, what the standards are, and how each will be measured by the quality control process.  The QA Contractor will use the Generic Software Quality Standards Template (see Appendix A) with suitable customization or tailoring to the Project’s quality management and risk assessment needs by performing the following services:

  1. Identifying from the template the relevant process and product quality standards, risk cues or measurements which are appropriate for the Project based on the Project’s current business and technical complexity assessment.
  2. Recommending additional quality standards based on the type of project, current phase of the project, or expert opinion where the absence of a quality standard would present high risk to the project; e.g. information security.
  3. Recommending elimination of unnecessary quality standards based on the Project’s business and technical complexity assessment or current phase of the Project. 

However, the following Process and Product Quality Standards shall not be eliminated:

Process Quality Standards:

  1. Definition of the project, development schedule, delivery commitment, cost controls, budget and resource size, project management approach (standards #10,11,12, 23, 25, 26, 27)
  2. Political influences, organizational stability (standard #6, 37)
  3. Leadership, project management authority, executive involvement (standard #12,17, 41)
  4. Team productivity, team member availability, user involvement, user justification (standard #28,34,44,48)

Product Quality Standards:

  1. An appropriate, useful, and maintainable set of requirements (standard #50)
  2. An appropriate, useful, and maintainable set of engineering and design specifications (standard #58)
  3. Appropriate, useful, and maintainable code, utility, object, etc., construction (standard #80)
  4. Appropriate, useful, and maintainable test planning, test execution, and test corrective actions (standard #51)
  5. The product or application is fit for use (standard #46, 76)
  6. Alternatives Analysis, Commitment Process, Lessons Learned (standard #55, 56, 63)
  7. Appropriate, thorough and maintainable implementation (standard #53,81)

Deliverable 1.2:  Quality Checklists

For each of the Development Contractor deliverables identified under Deliverable 2.1, the QA Contractor shall determine how the Quality Standards will be monitored and measured.  At a minimum, for each major subsystem, the QA Contractor shall develop a checklist that combines the selected quality standards with the expected monitoring activities described in: (1) the Quality Control section of the Quality Management Approach (see above); and (2) Task 2, Quality Control.  The QA Contractor shall deliver checklists for each Development Contractor deliverable to be reviewed under Task 2. These checklists must have the following attributes:

  1. Indicate the schedule for reviews, including the deliverable(s) under review, the person(s) responsible for the deliverable(s), and the person(s) responsible for conducting the reviews.
  2. Indicate a specific review procedure to follow, e.g. a procedure for schedule analysis, code walk-through, peer review, interviews, lessons learned, test methodology, or if the review will be performed informally.
  3. Verify and record that a set of required inspections have been performed.
  4. Indicate that the minimum quality standard has been met.
  5. Record the measurements.
  6. Identify the expected risk cue or measurement.
  7. Indicate the expected acceptability or tolerance.
  8. Indicate the rank of the quality standards where risk was found unacceptable.
  9. Indicate change in risk rank since previous review.
  10. Indicate a reference that will describe what was reviewed, who was interviewed, and the information or reasoning that non-adherence to a specific quality standard causes risk.

The process of measurement, setting tolerance, and risk ranking referred to above may be based on the QA Contractor’s subject matter knowledge, domain experience, and expert opinions.  These checklists shall conform to the specific choice of system life cycle model chosen by the Development Contractor or Agency, especially in respect to detailed requirements definition, general and detailed system design, test design, test planning and execution, and system implementation.

Deliverable 1.3:  Quality Management Plan (QMP)

The QA Contractor shall develop and submit a Quality Management Plan (QMP) that defines how Quality Assurance (QA), Quality Control (QC), Independent Verification and Validation (IV&V), and Risk Assessment will be conducted in the Project.  When developing the QMP, the QA Contractor shall review and use as key input the following: all available Project planning artifacts including any guidance provided by the Agency in writing or orally; the ISTQB, version 2012 (or later); and ISO 12207.  The QA Contractor shall adhere to generally accepted industry practices for project quality management and software development quality management.   At a minimum, the QMP shall include the following:

  1. Definition of QA Contractor, Development Contractor, Office of the –  and procedures and resources needed from the QA Contractor, Development Contractor and Agency staff to implement the Quality Management Plan.
  2. Identification of all the work tasks to be performed by the QA Contractor, Development Contractor and the Agency staff to provide confidence that the Project will satisfy the specified and relevant process and product quality standards. At a minimum the QMP shall include:
    1. Detailed plan for QA Review, including scope, criteria, and methodology to be employed
    2. Detailed plan for QC, including scope, criteria, and methodology, including activities relating to IV&V
    3. Quality Management Tools:
      1. Quality standards
      2. Quality checklists
      3. Templates for all reporting types required during the lifecycle of the Project; e.g. QA Status & Improvement Report, QC Report, Monthly Status Report, Weekly Status Report, On-Going Risk Notification Report, Project Assessment Report, Project Budget and Schedule Variance Report, etc.
      4. Additional quality tools and methods as needed
    4. Risk Assessment Methodology
      1. Verification that the Quality Management Plan relates to and references the other Deliverables supporting the Quality Management Planning Task in this SOW.

Templates for quality standards, QA Status and Improvement Reports, Project Assessment Report, and Project Budget & Schedule Variance Report are provided in the Appendices.

Deliverable 1.4:  Baseline Project Plan

The QA Contractor shall deliver a Baseline Project Plan.  The plan shall address each QA Contractor Deliverable and establish the baseline against which subsequent QA Contractor Deliverables are to be measured.  Upon acceptance by Authorized Purchaser, the QA Contractor shall conduct a walk-through presentation with the Agency Project team and -. 

To accomplish this task, the QA Contractor must be well versed in all aspects of the Project.  The QA Contractor shall review the following documents as input to its work:

  1. The approved business case on file with the -, as documented in the Information Resource Request signed by the State Chief Information Officer (CIO) or designee.
  2. All available Agency planning artifacts, e.g. Integrated Project Plan (IPP), supporting plans, and quality management guidance.
  3. Agency or statewide standards, including technology and security standards.
  4. Documented requirements, including functional, non-functional, and security requirements.
  5. The Development Contractor’s Proposal and contract SOW, especially the Project plan, software development methodology, data conversion & interface plan, and training plan where available.

Deliverable 1.4 shall include at a minimum the following elements:

  1. General approach to achieve Contract requirements.
  2. Schedule
  3. Detailed work breakdown structure, with key milestones, critical path elements, and Deliverables identified
  4. Estimated resource requirements
  5. Staffing plan with key persons identified
  6. Assessment of all levels of assistance needed from the Development Contractor, including but not limited to hands-on participation, facilities, and infrastructure
  7. Assessment of all levels of assistance needed from State personnel, including but not limited to hands-on participation, facilities, and infrastructure
  8. Demonstrate a clear link to the Development Contractor’s Project Plan and Deliverables

The accepted Baseline Project Plan shall be updated quarterly thereafter and be submitted to Agency for review and approval.

At its option, the Agency may authorize the QA Contractor to satisfy special requests or evaluate the project through the development of a lessons learned report, or both, as follows:

Deliverable 1.5:  Internal/External Presentations and Special Requests

At the Agency Project Manager’s request, the Contractor shall be required to appear before various committees and/or individuals to discuss the overall strategic direction and progress of the Project.  The QA Contractor shall accompany Agency staff at these appearances to make presentations, help answer questions, and provide its assessment of confidence on whether the Project will satisfy the needs it was undertaken to address. 

The QA Contractor shall, at the Agency Project Manager’s request, help prepare, attend, and participate in the management or other Project meetings, and for consulting advice as requested by the Agency Project Manager, through the Contract term.  The QA Contractor shall participate on-site unless otherwise directed by the Agency Project Manager.

During the Contract Period, the Agency Project Manager may submit written requests for internal/external presentation requests and special work.  Special work includes independent analysis, recommendations on unforeseen problems, opportunities for improvement, research and recommendations on alternative courses of action, and additional quality control reviews or risk management activities.  Additionally, consulting assistance may be requested for the QA Contractor to develop and review documents.  All requests will be within the original Scope of Work for this opportunity.

Within five (5) business days of QA Contractor’s receipt of a work request, or within such time frame within the Contract Period as the Agency Project Manager and the QA Contractor shall mutually agree, the QA Contractor shall provide the Agency Project Manager with a written fixed cost proposal detailing a proposed schedule.  The fixed-cost proposal shall reflect the skill level of position categories and associated fully loaded hourly rates identified in the contract.

Upon receiving the Agency Project Manager’s written approval and notice to proceed, the QA Contractor shall develop and submit, to the Agency Project Manager, written presentations or materials that meet the requirements of the request to the Agency Project Manager.  Upon the Agency Project Manager’s approval of such written presentations or materials, and if they are part of the Agency Project Manager’s request, the QA Contractor shall perform the presentation or tasks remaining as described in the approved proposal.

Deliverable 1.6:  Lessons Learned Report – Project Evaluation

The Contractor shall facilitate a “Project Evaluation/Lessons Learned” session at the end of each Project phase or at the close of the Project, and create a report of the findings.  The session will utilize as inputs, the Project’s QA Quarterly Status and Improvement Reports and other reports deemed suitable by the Agency Project Manager, to create the Project Evaluation Report (Appendix B). At a minimum, the session shall examine the progress made, planned vs. accomplished tasks, the risks, problems, and delays encountered, mitigation actions taken or not taken, successes and mistakes made, and shall include recommended actions.  Further, the session shall specifically examine whether the original (or current baseline) business case & ROI targets (if they exist) will be achieved.

Within 30 months from the contract start, the QA Contractor shall prepare and conduct a “Project Evaluation/Lessons Learned” session.  Those persons to be in attendance will be identified by the Agency Project Manager and the -. Within 15 business days following the Project Evaluation/Lessons Learned session, the QA Contractor will deliver a written Project Evaluation Report.

The QA Contractor shall prepare Deliverable number 1.6 “Lessons Learned Report – Project Evaluation” within 30 days of the termination of the Design, Development and Implementation (DDI) contract in the event that the Development Contract is terminated early.  In such case, Deliverable 1.6 will be considered the final Deliverable for this contract.

Task 1 Deliverables:

Deliverable NumberDeliverable DescriptionDue Date
1.1Quality Standards – Operational Definitions ReportContract Start or Agency acceptance of Task 5, Deliverable 5.1+ 40 Business Days
1.2Quality ChecklistsContract Start + 40 Business Days
1.3Quality Management PlanContract Start or Agency acceptance of Task 5, Deliverable 5.1 + 40 Business Days
1.4Baseline Project PlanContract Start or Agency acceptance of Task 5, Deliverable 5.1+ 40 Business Days for the first version; quarterly update thereafter.
1.5Internal/External presentations and Special RequestsTBD
1.6Lessons Learned Report – Project EvaluationTBD

2.2 Task 2: Quality Control (QC)

This task monitors Project results to determine if they comply with stated Project requirements. Project results include both work product results, notably deliverables, and project management results, notably schedule and cost performance.

The QA Contractor shall:

  1. Review Project activities and inspect Agency and Development Contractor work product deliverables throughout the life of the Project in terms of their ability to meet the performance requirements for scope, schedule, and cost.  This evaluation shall be done at the level of a specific work product deliverable, as well as its impact on subsequent deliverables to verify compliance and provide the Agency Project Manager, Project Team, others in Agency management, and the – with visibility as to whether the Project is adhering to its established plans, process and product standards, and work tasks.
  2. Review work product compliance with Agency security standards, especially in relation to system access.
  3. Review Development Contractor deliverables for completeness and accuracy, as well as identifying and assessing issues and risks at the work product level.  In ascertaining whether the Development Contractor work products meet Agency requirements, the QA Contractor shall pay particular attention to the scope, execution, and results of the Development Contractor’s Test Plan.
  4. Address issues within the Project for resolution, if possible.  Issues not resolvable within the Project will be escalated to the Project’s Steering Committee or in accordance with the Project’s governance processes.

Specifically, the QA Contractor shall:

Deliverable 2.1:  Quality Control Reviews 

Detailed Quality Control (QC) Reviews and Reports are required for the following Development Contractor deliverables and Agency foundational project management documents (Agency must align with Stage Gate Review Process requirements):

  1. Development Contractor Project Plan, Quarterly
  2. Major Deliverable 1
  3. Major Deliverable 2
  4. Major Deliverable 3
  5. Major Deliverable 4
  6. Major Deliverable 5
  7. Major Deliverable 6
  8. Major Deliverable 7
  9. Major Deliverable 8
  10. Major Deliverable 9
  11. Etc

For each Development Contractor deliverable, the QC Review shall evaluate major components or subsystems as necessary to establish acceptable workmanship.  Each QC report will be a separate Deliverable with all applicable quality checklists completed and attached.

QA Contractor shall conduct scheduled QC reviews by following the Quality Management Plan, and the approved Quality Standards, and Quality Checklists.  The Quality Management Plan, Quality Standards, and Quality Checklists will indicate the type of review expected, e.g. data analysis, schedule analysis, standards or process conformance analysis, and additional analysis based on subject matter expertise.  At a minimum, QA Contractor shall complete checklists indicating that the reviews for the process and products in the current period have been completed, measured, ranked, and reported.  The completed checklists will be provided with the QC Deliverables and shall become input to the Quarterly QA Status and Improvements Report (Deliverable 3.1), the – Project Assessment Report, and the – Project Budget & Schedule Variance Report.(See Appendix E for the – report templates.)

The precise nature of QC review shall be adjusted for the specific Development Contractor deliverable under review, as well the XYZ Project subsystem within a Development Contractor deliverable under review.  It shall also accommodate the specific choice of software development life cycle model chosen by the Development Contractor or Agency.

Deliverable 2.2: Security Code Review and Sampling

The QA Contractor shall review specific code samples for security standards compliance.  The QA Contractor shall also participate in major security code reviews by Agency for the modules that are developed specifically for the XYZ system, ensuring that relevant code adheres to statewide and Agency information security standard(s) where applicable.    The list of areas to be examined includes, but is not limited to the following:

  1. User Authentication
  2. Role Based Security
  3. Database Connectivity
  4. Password Validation
  5. Encryption

The QA Contractor shall work with the Development Contractor, the Agency Project Manager and the Agency Information Security Office to identify modules for code review and appropriate sampling rates.  The QA Contractor shall use the Project Plan, Project Schedule and the Development Contractor’s deliverable schedule when scheduling security code review and sampling.

Deliverable 2.3:  Quality Status Reporting and Tracking

Status Reporting.  The QA Contractor shall attend Project status meetings weekly and shall prepare a Monthly Quality Status Report that documents Project status with the following sections:

  1. Project Quality Status
    1. For the overall Project
    2. For each element on the work breakdown structure in the Baseline Project Plan being executed
    3. Tracking of recommendations made through QA Status and Improvements Report or other channels designated by the Agency Project manager or -.
  2. On-Going Risk Notification Report per task 5.2.

Task 2 Deliverables:

Deliverable NumberDeliverable DescriptionDue Date
2.1aProject Plan Report, QuarterlyDocument Delivery + 10 Business Days
2.1bMajor Deliverable 1Document Delivery + 10 Business Days
2.1cMajor Deliverable 2Document Delivery + 10 Business Days
2.1dMajor Deliverable 3Document Delivery + 10 Business Days
2.1eMajor Deliverable 4Document Delivery + 10 Business Days
2.1fMajor Deliverable 5Document Delivery + 10 Business Days
2.1gMajor Deliverable 6Document Delivery + 10 Business Days
2.1hMajor Deliverable 7Document Delivery + 10 Business Days
2.1iMajor Deliverable 8Document Delivery + 10 Business Days
2.1jMajor Deliverable 9Document Delivery + 10 Business Days
2.2aSecurity Code Review and Sampling PlanIn accordance with Development Contractor Project Plan and Schedule
2.2bSecurity Code Review and Sampling ReportTask 2.2.a acceptance + 10 Business Days
2.3aMonthly Quality Status Report FormatContract Start + 10 Business Days
2.3bMonthly Quality Status ReportTask 2.1a acceptance, first day of each calendar month

2.3 Task 3: Quality Assurance (QA)

The QA Contractor shall provide overall Project quality review; periodically examine quality control review results, checklists, change requests and tracking; and summarize the results for executive review and oversight throughout the life of the Project.  The QA Contractor will create and deliver quarterly Quality Assurance Status and Improvements Reports and Presentations summarizing the overall Project status, performance, risks and recommendations for process improvement to the Agency Project Manager, the Project’s Steering Committee and other relevant governance bodies, and the -.

Specifically the QA Contractor shall:

Deliverable 3.1:  QA Status and Improvement Reports / Presentations

The QA Contractor shall use the QA Status and Improvements Report template, Project Assessment Report, Project Budget and Schedule Variance Report, and other reporting vehicle or format as approved by the -, to conduct quarterly reviews, prepare reports, and deliver presentations.  (See Appendix C for the template and Appendix D for a sample.) 

In order to ensure that State and Development Contractor personnel are coordinated in a manner that minimizes the impact of the review on the overall Project schedule, the QA Contractor shall coordinate the review with the Agency Project Manager five (5) business days prior to conducting the review.

The Deliverable – QA Status and Improvement Report – shall contain, at a minimum, the following:

  1. Overall risk rank for the Project.
  2. Executive Summary paragraph of the entire report.  It shall contain information such as: A brief description of the primary achievement in the last period; A brief description of the highest ranked primary (if any) quality risks covered in the following detail of the report; A brief description of the impact or consequence of the risk if left unresolved.  At a minimum, the Executive Summary shall include:
    1. A summary of Project progress and accomplishments since the last report
    2. A summary of expenditures to date compared to budgeted Project dollars
    3. Identification and resolution of all problems encountered
    4. Plans, milestones and deliverables for the coming period
    5. A 3-month rolling risk matrix by assessment area
    6. Assessment of overall risk to the Project
    7. Evaluation of the overall Project status, Project budget and schedule variance, schedule and stage of completion, indicating the causal factors, mitigation efforts and recommendations
    8. Evaluation of the Project’s ability to deliver the benefits/results stated in the approved business case on file with the -, as documented in the Information Resource Request signed by the State CIO.
  3. Summary of the current progress followed by major milestones.  At a minimum the report shall contain a progress, status and risk analysis for the following assessment areas, in addition to other areas determined by the Agency Project Manager to be critical for monitoring purposes:
    1. Project Management
    2. Customer Involvement
    3. Technology
    4. Facilities and Support
    5. Project Scope
    6. Business Impact
    7. Deliverable Quality
    8. Development and Implementation
    9. Schedule
    10. Resources and Staff
    11. Actual expenditures compared to original (or current baseline) Project budget
    12. Project schedule performance compared to original (or current baseline) schedule
    13. Security
  4. Brief Project effectiveness statements on all the high-level categories from the approved Quality Standards:
    1. Categories where no risks were identified shall simply be indicated in one line, for example: Decision Drivers – no risks currently identified, or not evaluated at this time.
    1. Categories that contained a significant risk shall describe the risk, describe the factors used to determine the risk, estimate probability of occurrence, describe the potential impact, indicate the risk severity rating, and provide a recommended resolution strategy.  At a minimum the resolution strategy shall include:
      1. Status of risks that must be monitored, identifying probability of occurrence and potential impacts
      2. Identification of trigger points for intervention for each risk
      3. Recommended and alternative correction actions plans, with options identified, presented for each identified risk to prevent and/or reduce potential for the risk occurring or the danger escalating
      4. Intervention strategies to address any risk(s) that exceed a trigger point, to include a definition of options available to address the risk, the potential affects and costs of implementing the strategy, a comparative summary of the alternative strategies and identification of a recommended strategy
      5. Implications for the Development Contractor’s Project Plan
  5. Risks resolved since last period.  Provide the previous risk rank, and brief status or the actions taken on risks and results.
  6. Completed the – Project Assessment Reports and Project Budget & Schedule Variance Reports since Project start. (See Appendix E.)

The QA Status and Improvement Report Template will assist with the executive summary.  (See Appendix C.)  The QA Status and Improvement Report EXAMPLE will assist with how to create the report.  (See Appendix D.) The – requires the completion of a Project Assessment Report approximately once every 6 weeks. (See Appendix E.)

Task 3 Deliverables:

Deliverable NumberDeliverable DescriptionDue Date
3.1Quarterly QA Status and Improvement Reports / PresentationsEnd of Quarter + 10 Business Days*

*Note:  The exact dates of the End of Quarter will be specified in the Baseline Project Plan.  These dates will take into account and include but not be limited to, Contractor start date, and Joint Legislative Committee on Information Management Technology hearing dates.

2.4 Task 4:  Independent Verification and Validation (IV&V) Testing

This task ensures that the system and its components, as delivered by the Development Contractor, are functional, stable, and secured as defined by approved requirements of the XYZ Project.  The QA Contractor shall perform IV&V testing independent of the Development Contractor.

In Deliverable 4.1, the QA Contractor shall prepare an IV&V Master Test Plan (MTP).  The plan shall call for IV&V of the Development Contractor’s work products on a sampling basis. As the Development Contractor’s deliverables may contain software elements that have been previously deployed and proven stable, the QA Contractor shall focus IV&V efforts primarily on high-risk and newly developed code.  In Deliverable 4.2, the QA Contractor shall execute the IV&V MTP and report testing status and results.  The QA Contractor shall plan and perform security related IV&V separately under Deliverable 4.3.

Specifically, the QA Contractor shall:

Deliverable 4.1:  IV&V Master Test Plan (MTP)

The QA Contractor shall develop an IV&V Master Test Plan (MTP).  The MTP shall include plans, methodologies, development procedures, and bug tracking.  Emphasis shall be on the testing of high risk and new code areas.  High Risk areas will include but not be limited to sub-system integration, interfaces to other data systems, and provider claims processing systems.  At a minimum, the MTP shall have the following elements:

  1. Identification of potential high risk or new code areas
  2. Test Definition and Test Matrix
  3. Detailed Test Script Development Procedure
  4. Detailed Configuration and Build Control Procedure
  5. Testing Procedure
  6. Testing Environment
  7. Test Scripts
  8. Testing Metric and Reporting

The nature of IV&V testing to be conducted may vary greatly depending on the Development Contractor selected by the State.

Deliverable 4.2:  Test Execution and Status Report

The QA Contractor shall prepare for and conduct IV&V testing.  The QA Contractor will provide a Status Report on a bi-weekly basis, not to exceed 20 bi-weekly reports, from the initiation of IV&V through completion of the Project report to update the Agency Project Manager on the status of IV&V activities.  (Note:  Testing will not begin until the Agency Project Manager has accepted the Master Test Plan, Deliverable 4.1). 

At a minimum, this report shall include the following:

  1. Number of total tests to be conducted for the Project
  2. Number of tests per high risk or new code areas of the system
  3. Number of tests conducted, completed, and awaiting retest
  4. Types of bugs correlated to the areas of test
  5. Summary of the results of each security test

This Deliverable shall have a total not-to-exceed cost of EURxyz.

Task 4 Deliverables:

Deliverable NumberDeliverable DescriptionDue Date
4.1IV&V Master Test Plan (MTP)Developer Contractor Test Plan acceptance + 10 Business Days
4.2Bi-Weekly Test Execution and Status ReportIV&V Initiation + 2 weeks, then bi-weekly
4.3  Separate Security-related TestingTBD by subsequent task release order or WOC amendment.

2.5 Task 5:  Risk Assessment

The Risk Assessment Task defines the QA Contractor tasks to support the XYZ Project’s overall risk management efforts. The QA Contractor will refer to the Project’s Risk Management Plan if one exists.  The XYZ Project Team has the primary responsibility for executing the Project’s risk management activities with the QA Contractor providing a supporting function.  Within the QA Contractor’s role of completing quality management, quality assurance, and quality control on the Development Contractor’s plans, processes, and products, the QA Contractor will identify risks and provide recommendations for risk avoidance and mitigation strategies.  Similarly, the QA Contractor will identify risks and provide recommendations for risk avoidance and mitigation strategies on the Agency project management’s plans, processes, and products. 

The QA Contractor shall perform an initial risk assessment (Deliverable 5.1) on the XYZ Project Integrated Project Plan and/or related supporting plans, and the Development Contractor’s proposed project plan.  After the initial assessment, the QA Contractor will provide On-Going Risk Notification (Deliverable 5.2) for the life of the Contract.

Identification of Information Security risks will be an essential part of the scope of the QA Contractor’s initial and ongoing risk assessment.

Specifically, the Contractor shall:

Deliverable 5.1:  Initial Risk Assessment

The QA Contractor shall conduct an Initial Risk Assessment of the Project to identify the current status of the project, identify risks and their likelihood of occurring, and provide an independent evaluation of the planned schedule, resources (budget and staffing) and processes.

The QA Contractor shall evaluate the XYZ Project Project’s Integrated Project Plan (IPP), its components and processes, for reasonableness, validity, thoroughness and accuracy with emphasis on the following:

  1. Project Plan including State Resource Plan
  2. Project Requirements Management Plan
  3. Project Change Management Plan
  4. Project Issue Management Procedure
  5. Communication Plan
  6. Other critical Project processes

The QA Contractor shall perform a risk assessment on the Development Contractor’s most recent Project Plans including the Project Schedule, Software Development Plan, Facility Plan, Configuration Management Plan, Data Conversion Strategy, Testing Strategy, Post-Implementation Support Strategy (for on-going operations and maintenance), Training Strategy, and Certification Checklist where applicable.   This risk assessment shall take into account work product and Project level considerations, including at a minimum:

  1. Feasibility of the technical solution
  2. Sufficiency of security controls to safeguard protected or confidential information and to meet security requirements
  3. Sufficiency of Project components and processes
  4. Sufficiency of Project budget, schedule and resources

To accomplish this task, the QA Contractor must be well versed in all aspects of the XYZ Project.  To gain basic project knowledge, the QA Contractor shall review the following documents:

  1. XYZ Business Case, Project Charter, Project Work Plan and Work Breakdown Structure, Approved (original or current) baseline project budget and schedule, Integrated Project Plan and/or related supporting plans, the Development Contractor’s Proposal, Statement of Work, and Project Plan, etc.
  2. The Development Contract including the XYZ Project Requirements that specifies functional, non-functional, and security requirements.

The Initial Risk Assessment Deliverable (5.1) will include:

  1. Risk Identification – Provide a description, level of impact, probability of occurrence, and measurable threshold to trigger the risk.
  2. Risk Avoidance Plan – Recommend specific solutions to avoid the triggering of each risk.
  3. Risk Mitigation Plan – Recommend specific risk mitigation solutions for each risk that is identified.  Solutions shall include cost/benefit analysis for each mitigation option along with specific recommendations.
  4. The – Project Assessment Report – QA Contractor will complete “forward view” column for all metrics identified therein. (See Appendix E.)

Deliverable 5.2:  On-Going Risk Notification

The QA Contractor shall immediately verbally report to the Agency Project Manager the discovery of problems, new risks, or previously known risks that have increased in risk probability or potential impact, which pose a risk of failure or danger to the success of the project.  The QA Contractor shall develop and submit a written On-Going Risk Notification Report within three business days.

The on-going risk notification will follow the same requirements as defined for the initial risk assessment with an emphasis on managing risks that occur and change during the development and implementation process.  On-going risk assessment will be conducted while conducting the quality assurance and quality control tasks.  The on-going risk assessment will as a minimum:

  1. Monitor the replacement XYZ system’s hardware, software, infrastructure, and data security requirements and risks as they apply to the project’s progress to ensure that the XYZ system meets Agency requirements.   
  2. Review where applicable the Business Continuity Plan (BCP), the Disaster Recovery Plan (DRP) and XYZ audit controls to ensure the security of Agency protected or confidential information.
  3. Review of implementation tasks and activities to assure that the confidentiality, integrity and availability of the XYZ system are not compromised. 

Task 5 Deliverables:

Deliverable NumberDeliverable DescriptionDue Date
5.1Initial Project Risk Assessment ReportContract Start + 40 Business Days
5.2On-Going Risk Notification ReportAs needed. Written report three (3) days after verbal notification.

3. Deliverable and Payment Schedule

Deliverable NumberDeliverable DescriptionCost

4. Appendices

QA Contractor shall use the appendices provided as reference materials for developing a proposal and performing work specified in the Statement of Work.  In addition, the QA Contractor should reference material documented in the Development Contract Statement of Work, other project planning artifacts, and related supporting documents as available and applicable.