DevSecOps Initiative

Revision as of 12:01, 7 April 2021 by Greggory.elton (talk | contribs) (Created page with "<div style="float: right; z-index: 10; position: absolute; right: 0; top: 1;">File:JoinusonGCconnex.png|link=http://gcconnex.gc.ca/groups/profile/2785549/gc-enterprise-secur...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
JoinusonGCconnex.png
ESAcontactus.png
GOC ESA.jpg

Purpose

This site provides an overview of the Government of Canada (GC) DevSecOps conceptual framework and outlines the building blocks to enable a continuous security approach in the GC.

Overview

DevSecOps is a transformational shift which incorporates secure culture, practices, and tools to drive visibility, collaboration and agility of security into each phase of the Development and Operations (DevOps) pipeline. It includes 4 key building blocks and pillars:

Governance

  • Establish security ‘guardrails’ and monitor results
  • Redesign the operational & compliance framework
  • Establish shared metrics to evaluate progress

People

  • Break down silos between security and DevOps teams and instill cyber awareness
  • Incorporate security staff in DevOps teams
  • Have security teams brief dev and ops teams on current threats/exploits/breaches

Process

  • Orchestrate an integrated process flow and drive “in-line’ risk rationalized feedback
  • Asset inventory and risk awareness
  • Integrated back-log and pipeline

Technology

  • Automate recurring security tasks and harden the development pipeline
  • Automate secure application development
  • Protect the toolchain and infrastructure

What is DevOps?

Before diving into DevSecOps it is important to understand the concept behind DevOps itself. DevOps is a set of practices that automates the processes between development and operation teams to build, test and release software quickly and reliably.

Process automation and integration reduces the time and effort spent on manual tasks such as application testing, build, and deploying. This allows an organization to deliver product updates in short cycles.

Drivers and Benefits

DevSecOps uses a "security by design" by using automation to review and test code for security issues. This allows security issues and flaws to be remediated quickly and automatically in some cases.In DevSecOps compliance, auditing and notification systems are integrated to provide developers with instant insight into the development of a project.

1. Improve Security and Quality

  • Increase deployment success rates
  • Reduce meantime to resolve incidents
  • Reduce number of open security defects
  • Sample metrics: Change failure rate, Mean time to recover, CVSS Score

2. Improve Time to Market

  • Increase production deployment frequency
  • Greater speed of deployment
  • Sample metrics: Lead time & Cycle time

3. Improve Productivity

  • Controlled production access
  • More story points per sprint
  • Increase pipeline velocity
  • Sample metrics: Deployment frequency, average time for change approval

4. Improve Compliance Feedback

  • Reduction in open compliance findings
  • Decrease time from audit request to evidence delivery
  • Sample metrics: Security benchmark deviation, passed security test ratio

DevSecOps Strategy

DevSecOps is a transformational shift which incorporates secure culture, practices, and tools to drive visibility, collaboration and agility of security into each phase of the DevOps pipeline.

Component Approach
Strategic Goals Provide guidance and enable a culture change to foster collaboration between teams, improving time to market and productivity.

1.Establish DevSecOps policies to provide guidance around strategic drivers

2.Develop well-defined DevSecOps roles and responsibilities to enable an effective and organized team

3.Purposefully enable a culture shift toward security awareness to initiate culture change and foster collaboration between security and other teams

Architecture and Operations Design a DevSecOps operating model that includes designing data flows, developing standards, and mapping technologies and processes to core security operations.

1.Establish DevSecOps processes to help organizations to keep up with the pace of development

2.Enable security automation to reduce vulnerability and security flaws that are created by human error.

Monitor Enable automated monitoring and reviews, improving compliance feedback.

1.Enable automated audit evidence collection through utilizing security monitoring and notification systems that allow for an automated audit trail. This facilitates compliance reporting.

2.Monitor security metrics for continuous feedback, allowing for general oversight by management and to guide teams around improving their security decisions

Goals

The implementation of DevSecOps puts an focus on both meeting business demands and security considerations throughout the development lifecycle. However, the execution of this process requires strategic governance and personnel collaboration in order to be successful. This includes:

  • Policies: Specific elements should be integrated into organizational policies to communicate strategic DevSecOps drivers and goals. The statement of intent should help guide decisions in line with business priorities, which will then be supported by actionable processes at the tactical and operational level. Management must also develop metrics to monitor the level of integration and efficiency between teams in order to monitor developments and identify where additional assistance is required.
  • Roles and responsibilities: Responsibility and accountability for security should be clearly defined for all development, operations, and security team members. Personnel should understand who is responsible to review the results of automated security testing or complete manual security tasks, along with how they are responsible to liaise and integrate with the other teams.
  • Culture shift: Targeted training and awareness initiatives should be put in place to initiate culture change toward agile and continuous collaboration between teams, along with empowering teams and emphasizing security ownership by all individuals. There is a need to bridge various team cultures and existing ways of working in order to create ‘one unit’ that will successfully execute DevSecOps together.

Architecture and Operations

The integration of security into DevOps requires changes across the usual DevOps process. In a mature DevSecOps organization, automated build, test and deployment are organized as a series of steps in a workflow (known as a ‘pipeline’).

Illustrated below, there are four key processes required to support the development of a DevSecOps pipeline.

The aforementioned four key processes required for a DevSecOps are:

  • Continuous Integration (CI): A series of iterative processes where code changes are regularly merged, integrated, compiled and tested automatically across the “Design”, “Develop”, and “Test” phases. Artifacts are archived in a central artifact repository to make them available for the deployment process.
  • Continuous Deployment (CD): A series of iterative processes allowing for revisions to be deployed automatically across all environments where possible in the “Release” phase, while allowing for additional gates and manual approvals for production deployments.
  • Continuous Testing (CT): Run across the CI/CD cycles, test automation allows for immediate detection and resolution of quality-related problems while automatically evaluating whether application features meet the acceptance criteria automatically.
  • Continuous Monitoring (CM): Involving automatically monitoring running applications and infrastructure for anomalies and deviation from standard behavior across all phases. Infrastructure, network, and applications must be monitored and logs should be aggregated and monitored using a central database.

Security Activities

In order to enable DevSecOps, a number of ‘security’ activities must be integrated into the DevOps process.

The image below details different security activities that are integrated into the DevOps process.

These security activities include:

Stage Activity Description
Plan DevSecOps Framework Design The DevSecOps process workflow should be designed based on project requirements, enabling the creation of process flowcharts, tool selection, an implementation plan, and deployment/release management and operation platform selection. Training for personnel will be required to support the execution of this framework.
Develop Code Scan and Review Source code changes should be reviewed and scanned by a secondary source before it is integrated into the pipeline. Code can be analyzed automatically by a Static Application Security Testing (SAST) tool.
Develop Hardening Unnecessary ports, protocols, and services should be disabled to improve the security of the infrastructure or application. Automated security compliance or container security tools can be leveraged to identify vulnerabilities and recommended remediation actions.
Build Static Application Security Testing (SAST) As mentioned above, automated scanning of source code can help identify application source code, byte code, and binaries for conditions that are indicative of vulnerabilities. This can be conducted at both the Develop and Build stages.
Build Vulnerability Checking Automated tools can be used to identify vulnerabilities in internal and external dependencies.
Build Build Configuration Control and Audit An automated tool can track build histories, unit test execution results, SAST reports, dependency vulnerability reports, deployment playbooks, and release notes to

identify any security actions items before go/no-go decisions.

Test Dynamic Application Security Testing An automated tool is used to test running applications through the web interface, identifying potential security vulnerabilities in the web application and architecture
Test Penetration Testing Mimicking an attacker, security tests are performed at all levels of the application to identify vulnerabilities. The “attacker” can test the application with no prior knowledge, or be provided with a base set of credentials to imitate an insider attack. Automated tools and manual methods are used for penetration testing.
Release Post-Deployment Verification Automated or manual security tests can be run in the production environment to verify that there is no known vulnerability in the system before workload is redirected to the environment. This can be supported by security compliance tools.
Operate Security Monitoring and Auditing Automated tools are used to continuously monitor the security status of the system, while identifying potential risks and vulnerabilities. A Security Incident and Event Management solution will be used to aggregate and correlate logs from different components and provide alerts when suspicious or specific (previously defined) activity is detected. While this is not specific to DevSecOps, this is crucial for any deployment.
Operate System Secrets Rotation An automated configuration tool can be used to update and refresh secrets (credentials, access tokens, keys, certificates) across all systems components
Operate Incident management Track and manage incidents across the system. While this is not specific to DevSecOps, this is crucial for any deployment

Most security tools can be executed using automated tools at various stages of the process enabling continuous and faster development cycles.

Technical Building Blocks

Technical building blocks are required to support the development of a DevSecOps pipeline, illustrated below.

The aforementioned technical building blocks required for a DevSecOps pipeline are explained below along with accompanying security implications. These include:

Technical Building Block Description Benefits Security Implications
Code Repository A centrally managed location where source code is stored and managed. This can be hosted locally or in cloud-based offerings. flexible collaboration workflows, complete traceability of changes, and use of “open source practices” in the organization. Secure code is centrally stored, allowing developers to retrieve and build on top of trusted source code
CI/CD Server A server used to perform both integration and deployment, allowing for automated build (code compilation, testing, and archiving), test, and deployment by the same machine. more robust and efficient pipeline, increasing organizational output. A CI/CD server controls automated builds, integrating and deploying secure components (as configured)
Test Automation Framework A set of tools and guidelines that enable writing, execution, and reporting of functional/non-functional automated tests reducing testing effort and maintenance costs, while allowing for higher code coverage and more frequent testing. Security tests can be run automatically, increasing code coverage and reducing the chance manual error
Artifact Repository A storage location for binary files, container, virtual machine images, external dependencies, and their metadata. The repository can store the output of the build process and act as a proxy store to download external dependencies. the ability to check for licenses, popularity, and security ratings of external artifacts that were downloaded. Trusted elements are integrated into the artifact repository, allowing for storage and retrieval of secure files and images
Container and Container Registry Software that packages the executable code and dependencies into an image. The registry archives container images with the necessary metadata.  the ability to run the application (container) in any environment, while the registry continually scans images for security vulnerabilities. Containers provide segregation and self-containment, adding additional resiliency between application components
Environment The basic runtime zone where build gets deployed, containing the necessary infrastructure, networking, and security components required by an application to run. The minimum environment setup should include on production environment and two non-production for development and Quality Assurance (QA) purposes. enabling separate instances between various activities (development and individual testing, automated testing and integration, and live services provided to users). Environments provide segregation between

various activities conducted at different stages of the DevSecOps lifecycle, allowing testing activities and “in-progress” fixes for vulnerabilities to be kept separate from live applications and user data

Operations Dashboard Used to aggregate data from various sources, including centralized logging systems, incident management tools, or Application Performance Monitoring (APM) tools. providing insights that can be used for early detection, root cause analysis, and assessment of application health. A centralized dashboard allows team members to monitor and view trends related to security data (and any identified anomalies)

Monitoring

Monitoring of processes, personnel, and tools should occur at both the strategic and operational levels to closely measure the effectiveness and efficiency of DevSecOps operations, along with the level of culture adoption among team members. A number of metrics have been suggested to monitor status at various stages of the DevSecOps process:

Planning

  • Change Lead Time: Average time between the initiation and completion of a change request/task
  • Change Cycle Time: Average time between start of work to end of work on a change request/task
  • Average Time for Change Approval/Rejection: Average time from requesting a change to time the request is approved/rejected
  • Passed Security Tests Ratio: Rate of passed security tests for the change
  • Security Benchmark Deviation: Rate of deviation from security benchmarks
  • Vulnerability Patching Lead Time: Average time from identifying patching vulnerabilities to the time it is complete
  • Version Controlled Assets Coverage: Percentage of assets maintained under version control
  • Employee Satisfaction: Average employee satisfaction about process, engineering practices, and organizational culture

Develop

  • Code Complexity Score: Static code analysis tools provide a way to capture degree to maintainability, complexity, and adherence to best practices
  • Unit Test Code Coverage: Percentage of code covered by unit tests
  • Test Requirement Coverage: Percentage of requirements covered by tests

Build

  • Build Pass Rate: Percentage of builds that pass compilation, unit testing, and static code analysis
  • Vulnerability Creation Rate: Rate of creating new vulnerabilities by change or time period
  • Unit Test Code Coverage: Percentage of code covered by unit tests
  • Test Requirement Coverage: Percentage of requirements covered by tests

Test

  • Test Automation Percentage: Percentage of automated functional, integration, end-to-end and regression tests
  • Unit Test Code Coverage: Percentage of code covered by unit tests
  • Test Requirement Coverage: Percentage of requirements covered by tests

Release

  • Deployment Frequency: Number of production deployment per day/week
  • Deployment Time: Average time to complete a production deployment
  • Change Failure Rate: Rate of production failures due to changes
  • User Product / Service Rating: Rate of service or product from end users
  • Application Performance: Response time of a particular service from customer service

Operate

  • Security Incidents Rate: Rate of new security related incidents per given time period
  • Mean Time to Acknowledge: Average time from when an incident occurs until resolution process starts
  • Mean Time to Recover: Average time from when an incident occurs until resolution completion
  • Incident Resolution Effort: Average effort (of active work) required to resolve incidents
  • Application Availability / SLA Compliance: Percentage of time the application is available and functional to end users
  • Unresolved Ticket Volume: Average number of unresolved customer tickets or formal complaints per week
  • Number of Unauthorized Changes Detected Automatically: Number of automatically detected changes that are done by unauthorized person(s) or violate the approved scope of change

DevSecOps Success Criteria in the GC

Measurable DevSecOps success criteria for consideration by the Government of Canada include:

  • Open Collaboration
    • Set shared expectations and metrics for measuring success
    • Align Security architects and focus on activities based on business priorities
  • Security at the source
    • Create consumable, self-service security capabilities
    • Establish security ‘guardrails’ and monitor results
  • Reinforce and evaluation through automation
    • Create integrated process flows by automating recurring tasks
    • Embed preventative operational controls and audit trails
  • Risk oriented operations
    • Utilize operations insights and threat intelligence to drive process flow, prioritization and remediation recommendations
    • Don’t just rely on scans and take a risk based approach to testing
  • Holistic approach to security
    • Integrate framework to secure both the pipeline and application
    • End-to-end security implementation and defense in depth with production environments
  • Proactive monitoring
    • Continuous testing to identify problems before they become issues
    • Leverage logging to drive learning and innovation

References