DevOps and CI/CD

I am an avid reader of technical books, specifically those focused on Cloud, DevOps, and Site Reliability Engineering (SRE). In this post, I will share a list of books that I believe are essential for anyone looking to start or advance their career in Cloud, DevOps, or SRE. These books will help you build a strong foundation in the top skills required in these fields. While this post focuses primarily on Amazon AWS for public cloud, I will also include a few vendor-neutral books. Note: This is my honest opinion, and I am not affiliated with any of these book authors or publishers. How Linux Works, 3rd Edition: What Every Superuser Should Know by Brian Ward and the Linux Command Line, 2nd Edition by William Shotts Learning Linux is the first step before acquiring any other skills in DevOps. These books are excellent for building a strong foundation in Linux internals and getting familiar with the Linux command line, which is essential for excelling in the DevOps space. Python Programming is the second most important skill after Linux for DevOps or SRE. I recommend starting with Python Cookbook: Recipes for Mastering Python 3. Begin with the basics, then move on to object-oriented concepts, databases, APIs, and scripting. Eventually, you should learn about MVC and other design patterns to build comprehensive products, not just scripts. As a production engineer, you will need to develop many infrastructure tools. Solutions Architect's Handbook — Third Edition and AWS Cookbook These books provide a comprehensive view of what an AWS engineer needs to know. They were particularly helpful in preparing for the AWS Solution Associate exam, covering topics such as MVC architecture, domain-driven design, container-based application architecture, cloud-native design patterns, and performance considerations. The AWS Cookbook is excellent for practical labs and contains useful topics like secure web content delivery, dynamic access with security groups, automated password rotation for RDS, and big data solutions. Terraform up and Running by Yevgeniy Brikman Terraform is a widely used infrastructure automation tool in DevOps. This book covers the basics, and intermediate topics like managing state files, creating reusable modules, Terraform syntax, and more. It also addresses the challenge of managing secrets and provides options for integrating with Docker and Kubernetes. The book concludes with strategies for managing Terraform code within a team. Continuous Integration and Deployment This skill is crucial for developers, DevOps, SRE, or any engineer involved in development or operations. Tools like Jenkins, GitLab, and GitHub Actions are commonly used. For Kubernetes environments, GitOps tools like Flux and ArgoCD are popular. I recommend Automating DevOps with GitLab CI/CD Pipelines by Christopher Cowell for those starting with CI/CD. Kubernetes This technology has been trending and growing rapidly. For theoretical knowledge, the Kubernetes documentation is sufficient, but for hands-on learning, I recommend The Kubernetes Bible: The Definitive Guide to Deploying and Managing Kubernetes Across Cloud and On-Prem Environments. Microservice development and deployment are on the rise. For AWS, small microservices-based products can be deployed with ECS but for large-scale products, Kubernetes is required. System Design This is a vendor-neutral skill. I recommend Designing Data-Intensive Applications by Martin Kleppmann to learn how to build reliable and scalable systems. Finally, I acknowledge that merely reading books won't make you an expert. You need to engage in extensive hands-on labs to excel in this field.

Twenty years ago, software was eating the world. Then around a decade ago, containers started eating software, heralded by the arrival of open source OCI standards. Suddenly, developers were able to package an application artifact in a container — sometimes all by themselves. And each container image could technically run anywhere — especially in cloud infrastructure. No more needing to buy VM licenses, look for Rackspace and spare servers, and no more contacting the IT Ops department to request provisioning. Unfortunately, the continuing journey of deploying containers throughout all enterprise IT estates hasn’t been all smooth sailing. Dev teams are confronted with an ever-increasing array of options for building and configuring multiple container images to support unique application requirements and different underlying flavors of commercial and open-source platforms. Even if a developer becomes an expert in docker build, and the team has enough daily time to keep track of changes across all components and dependencies, they are likely to see functional and security gaps appearing within their expanding container fleet. Fortunately, we are seeing a bright spot in the evolution of Cloud Native Buildpacks, an open-source implementation project pioneered at Heroku and adopted early at Pivotal, which is now under the wing of the CNCF. Paketo Buildpacks is an open-source implementation of Cloud Native Buildpacks currently owned by the Cloud Foundry Foundation. Paketo automatically compiles and encapsulates developer application code into containers. Here’s how this latest iteration of buildpacks supports several important developer preferences and development team initiatives. Open Source Interoperability Modern developers appreciate the ability to build on open-source technology whenever they can, but it’s not always that simple to decide between open-source solutions when vendors and end-user companies have already made architectural decisions and set standards. Even in an open-source-first shop, many aspects of the environment will be vendor-supported and offer opinionated stacks for specific delivery platforms. Developers love to utilize buildpacks because they allow them to focus on coding business logic, rather than the infinite combinations of deployment details. Dealing with both source and deployment variability is where Paketo differentiates itself from previous containerization approaches. So, it doesn’t matter whether the developer codes in Java, Go, nodeJS, or Python, Paketo can compile ready-to-run containers. And, it doesn’t matter which cloud IaaS resource or on-prem server it runs on. “I think we're seeing a lot more developers who have a custom platform with custom stacks, but they keep coming back to Paketo Buildpacks because they can actually plug them into a modular system,” said Forest Eckhardt, contributor and maintainer to the Paketo project. “I think that adoption is going well, a lot of the adopters that we see are DevOps or Operations leaders who are trying to deliver applications for their clients and external teams.” Platform Engineering With Policy Platform engineering practices give developers shared, self-service resources and environments for development work, reducing setup costs and time, and encouraging code, component, and configuration reuse. These common platform engineering environments can be offered within a self-service internal portal or an external partner development portal, sometimes accompanied by support from a platform team that curates and reviews all elements of the platform. If the shared team space has too many random uploads, developers will not be able to distinguish the relative utility or safety of various unvalidated container definitions and packages. Proper governance means giving developers the ability to build to spec — without having to slog through huge policy checklists. Buildpacks take much of the effort and risk out of the ‘last mile’ of platform engineering. Developers can simply bring their code, and Paketo Buildpacks detects the language, gathers dependencies, and builds a valid container image that fits within the chosen methodology and policies of the organization. DevOps-Speed Automation In addition to empowering developers with self-service resources, automating everything as much as possible is another core tenet of the DevOps movement. DevOps is usually represented as a continuous infinity loop, where each change the team promotes in the design/development/build/deploy lifecycle should be executed by automated processes, including production monitoring and feedback to drive the next software delivery cycle. Any manual intervention in the lifecycle should be looked at as the next potential constraint to be addressed. If developers are spending time setting up Dockerfiles and validating containers, that’s less time spent creating new functionality or debugging critical issues. Software Supply Chain Assurance Developers want to move fast, so they turn to existing code and infrastructure examples that are working for peers. Heaps of downloadable packages and source code snippets are ready to go on npm StackOverflow and DockerHub – many with millions of downloads and lots of upvotes and review stars. The advent of such public development resources and git-style repositories offers immense value for the software industry as a whole, but by nature, it also provides an ideal entry point for software supply chain (or SSC) attacks. Bad actors can insert malware and irresponsible ones can leave behind vulnerabilities. Scanning an application once exploits are baked in can be difficult. It’s about time the software industry started taking a page from other discrete industries like high-tech manufacturing and pharmaceuticals that rely on tight governance of their supply chains to maximize customer value with reduced risk. For instance, an automotive brand would want to know the provenance of every part that goes into a car they manufacture, a complete bill-of-materials (or BOM) including both its supplier history and its source material composition. Paketo Buildpacks automatically generates an SBOM (software bill-of-materials) during each build process, attached to the image, so there’s no need to rely on external scanning tools. The SBOM documents information about every component in the packaged application, for instance, that it was written with Go version 1.22.3, even though that original code was compiled. The Intellyx Take Various forms of system encapsulation routines have been around for years, well before Docker appeared. Hey, containers even existed on mainframes. But there’s something distinct about this current wave of containerization for a cloud-native world. Paketo Buildpacks provides application delivery teams with total flexibility in selecting their platforms and open-source components of choice, with automation and reproducibility. Developers can successfully build the same app, in the same way, thousands of times in a row, even if underlying components are updated. That’s why so many major development shops are moving toward modern buildpacks, and removing the black box around containerization — no matter what deployment platform and methodology they espouse. ©2024 Intellyx B.V. Intellyx is editorially responsible for this document. At the time of writing, Cloud Foundry Foundation is an Intellyx customer. No AI bots were used to write this content. Image source: Adobe Express AI.

Introduction to Secrets Management In the world of DevSecOps, where speed, agility, and security are paramount, managing secrets effectively is crucial. Secrets, such as passwords, API keys, tokens, and certificates, are sensitive pieces of information that, if exposed, can lead to severe security breaches. To mitigate these risks, organizations are turning to secret management solutions. These solutions help securely store, access, and manage secrets throughout the software development lifecycle, ensuring they are protected from unauthorized access and misuse. This article aims to provide an in-depth overview of secrets management in DevSecOps, covering key concepts, common challenges, best practices, and available tools. Security Risks in Secrets Management The lack of implementing secrets management poses several challenges. Primarily, your organization might already have numerous secrets stored across the codebase. Apart from the ongoing risk of exposure, keeping secrets within your code promotes other insecure practices such as reusing secrets, employing weak passwords, and neglecting to rotate or revoke secrets due to the extensive code modifications that would be needed. Here below are some of the risks highlighting the potential risks of improper secrets management: Data Breaches If secrets are not properly managed, they can be exposed, leading to unauthorized access and potential data breaches. Example Scenario A Software-as-a-Service (SaaS) company uses a popular CI/CD platform to automate its software development and deployment processes. As part of their DevSecOps practices, they store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue Unfortunately, the CI/CD platform they use experiences a security vulnerability that allows attackers to gain unauthorized access to the secrets management tool's API. This vulnerability goes undetected by the company's security monitoring systems. Consequence Attackers exploit the vulnerability and gain access to the secrets stored in the management tool. With these credentials, they are able to access the company's production systems and databases. They exfiltrate sensitive customer data, including personally identifiable information (PII) and financial records. Impact The data breach leads to significant financial losses for the company due to regulatory fines, legal fees, and loss of customer trust. Additionally, the company's reputation is tarnished, leading to a decrease in customer retention and potential business partnerships. Preventive Measures To prevent such data breaches, the company could have implemented the following preventive measures: Regularly auditing and monitoring access to the secrets management tool to detect unauthorized access. Implementing multi-factor authentication (MFA) for accessing the secrets management tool. Ensuring that the secrets management tool is regularly patched and updated to address any security vulnerabilities. Limiting access to secrets based on the principle of least privilege, ensuring that only authorized users and systems have access to sensitive credentials. Implementing strong encryption for storing secrets to mitigate the impact of unauthorized access. Conducting regular security assessments and penetration testing to identify and address potential security vulnerabilities in the CI/CD platform and associated tools. Credential Theft Attackers may steal secrets, such as API keys or passwords, to gain unauthorized access to systems or resources. Example Scenario A fintech startup uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as database passwords and API keys, in a secrets management tool integrated with their pipelines. Issue An attacker gains access to the company's internal network by exploiting a vulnerability in an outdated web server. Once inside the network, the attacker uses a variety of techniques, such as phishing and social engineering, to gain access to a developer's workstation. Consequence The attacker discovers that the developer has stored plaintext files containing sensitive credentials, including database passwords and API keys, on their desktop. The developer had mistakenly saved these files for convenience and had not securely stored them in the secrets management tool. Impact With access to the sensitive credentials, the attacker gains unauthorized access to the company's databases and other systems. They exfiltrate sensitive customer data, including financial records and personal information, leading to regulatory fines and damage to the company's reputation. Preventive Measures To prevent such credential theft incidents, the fintech startup could have implemented the following preventive measures: Educating developers and employees about the importance of securely storing credentials and the risks of leaving them in plaintext files. Implementing strict access controls and auditing mechanisms for accessing and managing secrets in the secrets management tool. Using encryption to store sensitive credentials in the secrets management tool, ensures that even if credentials are stolen, they cannot be easily used without decryption keys. Regularly rotating credentials and monitoring for unusual or unauthorized access patterns to detect potential credential theft incidents early. Misconfiguration Improperly configured secrets management systems can lead to accidental exposure of secrets. Example Scenario A healthcare organization uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as database passwords and API keys, in a secrets management tool integrated with their pipelines. Issue A developer inadvertently misconfigures the permissions on the secrets management tool, allowing unintended access to sensitive credentials. The misconfiguration occurs when the developer sets overly permissive access controls, granting access to a broader group of users than intended. Consequence An attacker discovers the misconfigured access controls and gains unauthorized access to the secrets management tool. With access to sensitive credentials, the attacker can now access the healthcare organization's databases and other systems, potentially leading to data breaches and privacy violations. Impact The healthcare organization suffers reputational damage and financial losses due to the data breach. They may also face regulatory fines for failing to protect sensitive information. Preventive Measures To prevent such misconfiguration incidents, the healthcare organization could have implemented the following preventive measures: Implementing least privilege access controls to ensure that only authorized users and systems have access to sensitive credentials. Regularly auditing and monitoring access to the secrets management tool to detect and remediate misconfigurations. Implementing automated checks and policies to enforce proper access controls and configurations for secrets management. Providing training and guidance to developers and administrators on best practices for securely configuring and managing access to secrets. Compliance Violations Failure to properly manage secrets can lead to violations of regulations such as GDPR, HIPAA, or PCI DSS. Example Scenario A financial services company uses a popular CI/CD platform to automate their software development and deployment processes. They store sensitive credentials, such as encryption keys and API tokens, in a secrets management tool integrated with their pipelines. Issue The financial services company fails to adhere to regulatory requirements for managing and protecting sensitive information. Specifically, they do not implement proper encryption for storing sensitive credentials and do not maintain proper access controls for managing secrets. Consequence Regulatory authorities conduct an audit of the company's security practices and discover compliance violations related to secrets management. The company is found to be non-compliant with regulations such as PCI DSS (Payment Card Industry Data Security Standard) and GDPR (General Data Protection Regulation). Impact The financial services company faces significant financial penalties for non-compliance with regulatory requirements. Additionally, the company's reputation is damaged, leading to a loss of customer trust and potential legal consequences. Preventive Measures To prevent such compliance violations, the financial services company could have implemented the following preventive measures: Implementing encryption for storing sensitive credentials in the secrets management tool to ensure compliance with data protection regulations. Implementing strict access controls and auditing mechanisms for managing and accessing secrets to prevent unauthorized access. Conducting regular compliance audits and assessments to identify and address any non-compliance issues related to secrets management. Lack of Accountability Without proper auditing and monitoring, it can be difficult to track who accessed or modified secrets, leading to a lack of accountability. Example Scenario A technology company uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue The company does not establish clear ownership and accountability for managing and protecting secrets. There is no designated individual or team responsible for ensuring that proper security practices are followed when storing and accessing secrets. Consequence Due to the lack of accountability, there is no oversight or monitoring of access to sensitive credentials. As a result, developers and administrators have unrestricted access to secrets, increasing the risk of unauthorized access and data breaches. Impact The lack of accountability leads to a data breach where sensitive credentials are exposed. The company faces financial losses due to regulatory fines, legal fees, and loss of customer trust. Additionally, the company's reputation is damaged, leading to a decrease in customer retention and potential business partnerships. Preventive Measures To prevent such lack of accountability incidents, the technology company could have implemented the following preventive measures: Designating a specific individual or team responsible for managing and protecting secrets, including implementing and enforcing security policies and procedures. Implementing access controls and auditing mechanisms to monitor and track access to secrets, ensuring that only authorized users have access. Providing regular training and awareness programs for employees on the importance of secrets management and security best practices. Conducting regular security audits and assessments to identify and address any gaps in secrets management practices. Operational Disruption If secrets are not available when needed, it can disrupt the operation of DevSecOps pipelines and applications. Example Scenario A financial institution uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as encryption keys and API tokens, in a secrets management tool integrated with their pipelines. Issue During a routine update to the secrets management tool, a misconfiguration occurs that causes the tool to become unresponsive. As a result, developers are unable to access the sensitive credentials needed to deploy new applications and services. Consequence The operational disruption leads to a delay in deploying critical updates and features, impacting the financial institution's ability to serve its customers effectively. The IT team is forced to troubleshoot the issue, leading to downtime and increased operational costs. Impact The operational disruption results in financial losses due to lost productivity and potential revenue. Additionally, the financial institution's reputation is damaged, leading to a loss of customer trust and potential business partnerships. Preventive Measures To prevent such operational disruptions, the financial institution could have implemented the following preventive measures: Implementing automated backups and disaster recovery procedures for the secrets management tool to quickly restore service in case of a failure. Conducting regular testing and monitoring of the secrets management tool to identify and address any performance issues or misconfigurations. Implementing a rollback plan to quickly revert to a previous version of the secrets management tool in case of a failed update or configuration change. Establishing clear communication channels and escalation procedures to quickly notify stakeholders and IT teams in case of operational disruption. Dependency on Third-Party Services Using third-party secrets management services can introduce dependencies and potential risks if the service becomes unavailable or compromised. Example Scenario A software development company uses a popular CI/CD platform to automate its software development and deployment processes. They rely on a third-party secrets management tool to store sensitive credentials, such as API keys and database passwords, used in their pipelines. Issue The third-party secrets management tool experiences a service outage due to a cyber attack on the service provider's infrastructure. As a result, the software development company is unable to access the sensitive credentials needed to deploy new applications and services. Consequence The dependency on the third-party secrets management tool leads to a delay in deploying critical updates and features, impacting the software development company's ability to deliver software on time. The IT team is forced to find alternative ways to manage and store sensitive credentials temporarily. Impact The dependency on the third-party secrets management tool results in financial losses due to lost productivity and potential revenue. Additionally, the software development company's reputation is damaged, leading to a loss of customer trust and potential business partnerships. Preventive Measures To prevent such dependencies on third-party services, the software development company could have implemented the following preventive measures: Implementing a backup plan for storing and managing sensitive credentials locally in case of a service outage or disruption. Diversifying the use of secrets management tools by using multiple tools or providers to reduce the impact of a single service outage. Conducting regular reviews and assessments of third-party service providers to ensure they meet security and reliability requirements. Implementing a contingency plan to quickly switch to an alternative secrets management tool or provider in case of a service outage or disruption. Insider Threats Malicious insiders may abuse their access to secrets for personal gain or to harm the organization. Example Scenario A technology company uses a popular CI/CD platform to automate their software development and deployment processes. They store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue An employee with privileged access to the secrets management tool decides to leave the company and maliciously steals sensitive credentials before leaving. The employee had legitimate access to the secrets management tool as part of their job responsibilities but chose to abuse that access for personal gain. Consequence The insider threat leads to the theft of sensitive credentials, which are then used by the former employee to gain unauthorized access to the company's systems and data. This unauthorized access can lead to data breaches, financial losses, and damage to the company's reputation. Impact The insider threat results in financial losses due to potential data breaches and the need to mitigate the impact of the stolen credentials. Additionally, the company's reputation is damaged, leading to a loss of customer trust and potential legal consequences. Preventive Measures To prevent insider threats involving secrets management, the technology company could have implemented the following preventive measures: Implementing strict access controls and least privilege principles to limit the access of employees to sensitive credentials based on their job responsibilities. Conducting regular audits and monitoring of access to the secrets management tool to detect and prevent unauthorized access. Providing regular training and awareness programs for employees on the importance of data security and the risks of insider threats. Implementing behavioral analytics and anomaly detection mechanisms to identify and respond to suspicious behavior or activities involving sensitive credentials. Best Practices for Secrets Management Here are some best practices for secrets management in DevSecOps pipelines: Use a dedicated secrets management tool: Utilize a specialized tool or service designed for securely storing and managing secrets. Encrypt secrets at rest and in transit: Ensure that secrets are encrypted both when stored and when transmitted over the network. Use strong access controls: Implement strict access controls to limit who can access secrets and what they can do with them. Regularly rotate secrets: Regularly rotate secrets (e.g., passwords, API keys) to minimize the impact of potential compromise. Avoid hardcoding secrets: Never hardcode secrets in your code or configuration files. Use environment variables or a secrets management tool instead. Use environment-specific secrets: Use different secrets for different environments (e.g., development, staging, production) to minimize the impact of a compromised secret. Monitor and audit access: Monitor and audit access to secrets to detect and respond to unauthorized access attempts. Automate secrets retrieval: Automate the retrieval of secrets in your CI/CD pipelines to reduce manual intervention and the risk of exposure. Regularly review and update policies: Regularly review and update your secrets management policies and procedures to ensure they are up-to-date and effective. Educate and train employees: Educate and train employees on the importance of secrets management and best practices for handling secrets securely. Use-Cases of Secrets Management For Different Tools Here are the common use cases for different tools of secrets management: IBM Cloud Secrets Manager Securely storing and managing API keys Managing database credentials Storing encryption keys Managing certificates Integrating with CI/CD pipelines Compliance and audit requirements by providing centralized management and auditing of secrets usage. Ability to dynamically generate and rotate secrets HashiCorp Vault Centralized secrets management for distributed systems Dynamic secrets generation and management Encryption and access controls for secrets Secrets rotation for various types of secrets AWS Secrets Manager Securely store and manage AWS credentials Securely store and manage other types of secrets used in AWS services Integration with AWS services for seamless access to secrets Automatic secrets rotation for supported AWS services Azure Key Vault Centralized secrets management for Azure applications Securely store and manage secrets, keys, and certificates Encryption and access policies for secrets Automated secrets rotation for keys, secrets, and certificates CyberArk Conjur Secrets management and privileged access management Secrets retrieval via REST API for integration with CI/CD pipelines Secrets versioning and access controls Automated secrets rotation using rotation policies and scheduled tasks Google Cloud Secret Manager Centralized secrets management for Google Cloud applications Securely store and manage secrets, API keys, and certificates Encryption at rest and in transit for secrets Automated and manual secrets rotation with integration with Google Cloud Functions These tools cater to different cloud environments and offer various features for securely managing and rotating secrets based on specific requirements and use cases. Implement Secrets Management in DevSecOps Pipelines Understanding CI/CD in DevSecOps CI/CD in DevSecOps involves automating the build, test, and deployment processes while integrating security practices throughout the pipeline to deliver secure and high-quality software rapidly. Continuous Integration (CI) CI is the practice of automatically building and testing code changes whenever a developer commits code to the version control system (e.g., Git). The goal is to quickly detect and fix integration errors. Continuous Delivery (CD) CD extends CI by automating the process of deploying code changes to testing, staging, and production environments. With CD, every code change that passes the automated tests can potentially be deployed to production. Continuous Deployment (CD) CD goes one step further than continuous delivery by automatically deploying every code change that passes the automated tests to production. This requires a high level of automation and confidence in the automated tests. Continuous Compliance (CC) CC refers to the practice of integrating compliance checks and controls into the automated CI/CD pipeline. It ensures that software deployments comply with relevant regulations, standards, and internal policies throughout the development lifecycle. DevSecOps DevSecOps integrates security practices into the CI/CD pipeline, ensuring that security is built into the software development process from the beginning. This includes performing security testing (e.g., static code analysis, dynamic application security testing) as part of the pipeline and managing secrets securely. The following picture depicts the DevSecOps lifecycles: Picture courtesy Implement Secrets Management Into DevSecOps Pipelines Implementing secrets management into DevSecOps pipelines involves securely handling and storing sensitive information such as API keys, passwords, and certificates. Here's a step-by-step guide to implementing secrets management in DevSecOps pipelines: Select a Secrets Management Solution Choose a secrets management tool that aligns with your organization's security requirements and integrates well with your existing DevSecOps tools and workflows. Identify Secrets Identify the secrets that need to be managed, such as database credentials, API keys, encryption keys, and certificates. Store Secrets Securely Use the selected secrets management tool to securely store secrets. Ensure that secrets are encrypted at rest and in transit and that access controls are in place to restrict who can access them. Integrate Secrets Management into CI/CD Pipelines Update your CI/CD pipeline scripts and configurations to integrate with the secrets management tool. Use the tool's APIs or SDKs to retrieve secrets securely during the pipeline execution. Implement Access Controls Implement strict access controls to ensure that only authorized users and systems can access secrets. Use role-based access control (RBAC) to manage permissions. Rotate Secrets Regularly Regularly rotate secrets to minimize the impact of potential compromise. Automate the rotation process as much as possible to ensure consistency and security. Monitor and Audit Access Monitor and audit access to secrets to detect and respond to unauthorized access attempts. Use logging and monitoring tools to track access and usage. Best Practices for Secrets Management Into DevSecOps Pipelines Implementing secrets management in DevSecOps pipelines requires careful consideration to ensure security and efficiency. Here are some best practices: Use a secrets management tool: Utilize a dedicated to store and manage secrets securely. Encrypt secrets: Encrypt secrets both at rest and in transit to protect them from unauthorized access. Avoid hardcoding secrets: Never hardcode secrets in your code or configuration files. Use environment variables or secrets management tools to inject secrets into your CI/CD pipelines. Rotate secrets: Implement a secrets rotation policy to regularly rotate secrets, such as passwords and API keys. Automate the rotation process wherever possible to reduce the risk of human error. Implement access controls: Use role-based access controls (RBAC) to restrict access to secrets based on the principle of least privilege. Monitor and audit access: Enable logging and monitoring to track access to secrets and detect any unauthorized access attempts. Automate secrets retrieval: Automate the retrieval of secrets in your CI/CD pipelines to reduce manual intervention and improve security. Use secrets injection: Use tools or libraries that support secrets injection (e.g., Kubernetes secrets, Docker secrets) to securely inject secrets into your application during deployment. Conclusion Secrets management is a critical aspect of DevSecOps that cannot be overlooked. By implementing best practices such as using dedicated secrets management tools, encrypting secrets, and implementing access controls, organizations can significantly enhance the security of their software development and deployment pipelines. Effective secrets management not only protects sensitive information but also helps in maintaining compliance with regulatory requirements. As DevSecOps continues to evolve, it is essential for organizations to prioritize secrets management as a fundamental part of their security strategy.

By Josephine Eskaline Joyce CORE

Ansible is one of the fastest-growing Infrastructure as Code (IaC) and automation tools in the world. Many of us use Ansible for Day 1 and Day 2 operations. One of the best analogies to understand the phases/stages/operations is defined on RedHat's website: "Imagine you're moving into a house. If Day 1 operations are moving into the house (installation), Day 2 operations are the 'housekeeping' stage of a software’s life cycle." Simply put, in a software lifecycle: Day 0: Design/planning phase - This phase involves preparation, initial planning, brainstorming, and preparing for the project. Typical activities in this phase are defining the scope, gathering requirements, assembling the development team, and setting up the development environments. For example, the team discusses the CI/CD platform to integrate the project with, the strategy for project management, etc. Day 1: Development/deployment phase - This phase marks the actual development activities such as coding, building features, and implementation based on the requirements gathered in the planning phase. Additionally, testing will begin to ensure early detection of issues (in development lingo, "bugs"). Day 2: Maintenance phase - This phase in which your project/software goes live and you keep a tap on the health of the project. You may need to patch or update the software and file feature requests/issues based on user feedback for your development team to work on. This is the phase where monitoring and logging (observability) play a crucial role. Ansible is an open-source tool written in Python and uses YAML to define the desired state of configuration. Ansible is used for configuration management, application deployment, and orchestration. It simplifies the process of managing and deploying software across multiple servers, making it one of the essential tools for system administrators, developers, and IT operations teams. With AI, generating Ansible code has become simpler and more efficient. Check out the following article to learn how Ansible is bringing AI tools to your Integrated Development Environment: "Automation, Ansible, AI." RedHat Ansible Lightspeed with IBM Watsonx code assistant At its core, Ansible employs a simple, agentless architecture, relying on SSH to connect to remote servers and execute tasks. This eliminates the need for installing any additional software or agents on target machines, resulting in a lightweight and efficient automation solution. Key Features of Ansible Here is a list of key features that Ansible offers: Infrastructure as Code (IaC) Ansible allows you to define your infrastructure and configuration requirements in code, enabling you to version control, share, and replicate environments with ease. For example, say you plan to move your on-premises application to a cloud platform. Instead of provisioning the cloud services and installing the dependencies manually, you can define the required cloud services and dependencies for your application like compute, storage, networking, security, etc., in a configuration file. That desired state is taken care of by Ansible as an Infrastructure as Code tool. In this way, setting up your development, test, staging, and production environments will easily avoid repetition. Playbooks Ansible playbooks are written in YAML format and define a series of tasks to be executed on remote hosts. Playbooks offer a clear, human-readable way to describe complex automation workflows. Using playbooks, you define the required dependencies and desired state for your application. Modules Ansible provides a vast collection of modules for managing various aspects of systems, networks, cloud services, and applications. Modules are idempotent, meaning they ensure that the desired state of the system is achieved regardless of its current state. For example, ansible.bultin.command is a module that helps you to execute commands on a remote machine. You can either use modules that are built in, like dnf, yum, etc., as part of Ansible Core, or you can develop your own modules in Ansible. To further understand the Ansible modules, check out this topic on RedHat. Inventory Management Ansible uses an inventory file to define the hosts it manages. This inventory can be static or dynamic, allowing for flexible configuration management across different environments. An inventory file (.ini or .yaml) is a list of hosts or nodes on which you install, configure, or set up a software, add a user, or change the permissions of a folder, etc. Refer to how to build an inventory for best practices. Roles Roles in Ansible provide a way to organize and reuse tasks, variables, and handlers. They promote code reusability and help maintain clean and modular playbooks. You can group tasks that are repetitive as a role to reuse or share with others. One good example is pinging a remote server, you can move the tasks, variables, etc., under a role to reuse. Below is an example of a role directory structure with eight main standard directories. You will learn about a tool to generate this defined structure in the next section of this article. Shell roles/ common/ # this hierarchy represents a "role" tasks/ # main.yml # <-- tasks file can include smaller files if warranted handlers/ # main.yml # <-- handlers file templates/ # <-- files for use with the template resource ntp.conf.j2 # <------- templates end in .j2 files/ # bar.txt # <-- files for use with the copy resource foo.sh # <-- script files for use with the script resource vars/ # main.yml # <-- variables associated with this role defaults/ # main.yml # <-- default lower priority variables for this role meta/ # main.yml # <-- role dependencies library/ # roles can also include custom modules module_utils/ # roles can also include custom module_utils lookup_plugins/ # or other types of plugins, like lookup in this case webtier/ # same kind of structure as "common" was above, done for the webtier role monitoring/ # "" fooapp/ Beyond Automation Ansible finds applications in several areas. Configuration management: Ansible simplifies the management of configuration files, packages, services, and users across diverse IT infrastructures. Application deployment: Ansible streamlines the deployment of applications by automating tasks such as software installation, configuration, and version control. Continuous Integration/Continuous Deployment (CI/CD): Ansible integrates seamlessly with CI/CD pipelines, enabling automated testing, deployment, and rollback of applications. Orchestration: Ansible orchestrates complex workflows involving multiple servers, networks, and cloud services, ensuring seamless coordination and execution of tasks. Security automation: Ansible helps enforce security policies, perform security audits, and automate compliance checks across IT environments. Cloud provisioning: Ansible's cloud modules facilitate the provisioning and management of cloud resources on platforms like IBM Cloud, AWS, Azure, Google Cloud, and OpenStack. The list is not exhaustive, so only a subset of applications is included above. Ansible can act as a security compliance manager by enforcing security policies and compliance standards across infrastructure and applications through patch management, configuration hardening, and vulnerability remediation. Additionally, Ansible can assist in setting up monitoring and logging, automating disaster recovery procedures (backup and restore processes, failovers, etc.,), and integrating with a wide range of tools and services, such as version control systems, issue trackers, ticketing systems, and configuration databases, to create end-to-end automation workflows. Tool and Project Ecosystem Ansible provides a wide range of tools and programs like Ansible-lint, Molecule for testing Ansible plays and roles, yamllint, etc. Here are additional tools that are not mentioned in the Ansible docs: Ansible Generator: Creates the necessary folder/directory structure; comes in handy when you create Ansible roles AWX: Provides a web-based user interface, REST API, and task engine built on top of Ansible; Comes with an awx-operator if you are planning to set up on a container orchestration platform like RedHat OpenShift Ansible VS Code extension by Red Hat: Syntax highlighting, validation, auto-completion, auto-closing Jinja expressions ("{{ my_variable }") etc. The Ansible ecosystem is very wide. This article gives you just a glimpse of the huge set of tools and frameworks. You can find the projects in the Ansible ecosystem on Ansible docs. Challenges With Ansible Every tool or product comes with its own challenges. Learning curve: One of the major challenges with Ansible is the learning curve. Mastering the features and best practices can be time-consuming, especially for users new to infrastructure automation or configuration. Complexity: Initially, understanding the terminology, folder structure, and hierarchy challenges the user. Terms like inventory, modules, plugins, tasks, playbooks, etc., are hard to understand in the beginning. As the number of nodes/hosts increases, the complexity of managing the playbooks, and orchestrating increases. Troubleshooting and error handling: For beginners, troubleshooting errors and debugging playbooks can be challenging. Especially, understanding error messages and identifying the root cause of failures requires familiarity with Ansible's syntax and modules, etc. Conclusion In this article, you learned that Ansible as an open-source tool can be used not only for automation but also for configuration, deployment, and security enablement. You also understood the features, and challenges and learned about the tools Ansible and the community offers. Ansible will become your go-to Infrastructure as Code tool once you pass the initial learning curve. To overcome the initial complexity, here's a GitHub repository with Ansible YAML code snippets to start with. Happy learning. If you like this article, please like and share it with your network.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Hello! My name is Roman Burdiuzha. I am a Cloud Architect, Co-Founder, and CTO at Gart Solutions. I have been working in the IT industry for 15 years, a significant part of which has been in management positions. Today I will tell you how I find specialists for my DevSecOps and AppSec teams, what I pay attention to, and how I communicate with job seekers who try to embellish their own achievements during interviews. Starting Point I may surprise some of you, but first of all, I look for employees not on job boards, but in communities, in general chats for IT specialists, and through acquaintances. This way you can find a person with already existing recommendations and make a basic assessment of how suitable he is for you. Not by his resume, but by his real reputation. And you can already know him because you are spinning in the same community. Building the Ideal DevSecOps and AppSec Team: My Hiring Criteria There are general chats in my city (and not only) for IT specialists, where you can simply write: "Guys, hello, I'm doing this and I'm looking for cool specialists to work with me." Then I send the requirements that are currently relevant to me. If all this is not possible, I use the classic options with job boards. Before inviting for an interview, I first pay attention to the following points from the resume and recommendations. Programming Experience I am sure that any security professional in DevSecOps and AppSec must know the code. Ideally, all security professionals should grow out of programmers. You may disagree with me, but DevSecOps and AppSec specialists should work with code to one degree or another, be it some YAML manifests, JSON, various scripts, or just a classic application written in Java, Go, and so on. It is very wrong when a security professional does not know the language in which he is looking for vulnerabilities. You can't look at one line that the scanner highlighted and say: "Yes, indeed, this line is exploitable in this case, or it's false." You need to know the whole project and its structure. If you are not a programmer, you simply will not understand this code. Taking Initiative I want my future employees to be proactive — I mean people who work hard enough, do big tasks, have ambitions, want to achieve, and spend a lot of time on specific tasks. I support people's desire to develop in their field, to advance in the community, and to look for interesting tasks and projects for themselves, including outside of work. And if the resume indicates the corresponding points, I will definitely highlight it as a plus. Work-Life Balance I also pay a lot of attention to this point and I always talk about it during the interview. The presence of hobbies and interests in a person indicates his ability to switch from work to something else, his versatility and not being fixated on one job. It doesn't have to be about active sports, hiking, walking, etc. The main thing is that a person's life has not only work but also life itself. This means that he will not burn out in a couple of years of non-stop work. The ability to rest and be distracted acts as a guarantee of long-term employment relationships. In my experience, there have only been a couple of cases when employees had only work in their lives and nothing more. But I consider them to be unique people. They have been working in this rhythm for a long time, do not burn out, and do not fall into depression. You need to have a certain stamina and character for this. But in 99% of cases, overwork and inability to rest are a guaranteed departure and burnout of the employee in 2-3 years. At the moment, he can do a lot, but I don't need to change people like gloves every couple of years. Education I graduated from postgraduate studies myself, and I think this is more a plus than a minus. You should check the availability of certificates and diplomas of education specified in the resume. Confirmation of qualifications through certificates can indicate the veracity of the declared competencies. It is not easy to study for five years, but at the same time, when you study, you are forced to think in the right direction, analyze complex situations, and develop something that has scientific novelty at present and can be used in the future with benefit for people. And here, in principle, it is the same: you combine common ideas with colleagues and create, for example, progressive DevOps, which allows you to further help people; in particular, in the security of the banking sector. References and Recommendations I ask the applicant to provide contacts of previous employers or colleagues who can give recommendations on his work. If a person worked in the field of information security, then there are usually mutual acquaintances with whom I also communicate and who can confirm his qualifications. What I Look for in an Interview Unfortunately, not all aspects can be clarified at the stage of reading the resume. The applicant may hide some things in order to present themselves in a more favorable light, but more often it is simply impossible to take into account all the points needed by the employer when compiling a resume. Through leading questions in a conversation with the applicant and his stories from previous jobs, I find out if the potential employee has the qualities listed below. Ability To Read It sounds funny, but in fact, it is not such a common quality. A person who can read and analyze can solve almost any problem. I am absolutely convinced of this because I have gone through it myself more than once. Now I try to look for information from many sources, I actively use the same ChatGPT and other similar services just to speed up the work. That is, the more information I push through myself, the more tasks I will solve, and, accordingly, I will be more successful. Sometimes I ask the candidate to find a solution to a complex problem online and provide him with material for analysis, I look at how quickly he can read and conduct a qualitative analysis of the provided article. Analytical Mind There are two processes: decomposition and composition. Programmers usually use the second part. They conduct compositional analysis, that is, they assemble some artifact from the code that is needed for further work. An information security analyst or security specialist uses decomposition. That is, on the contrary, it disassembles the artifact into its components and looks for vulnerabilities. If a programmer creates, then a security specialist disassembles. An analytical mind is needed in the part that is related to how someone else's code works. In the 90s, for example, we talked about disassembling if the code was written in assembler. That is, you have a binary file, and you need to understand how it works. And if you do not analyze all entry and exit points, all processes, and functions that the programmer has developed in this code, then you cannot be sure that the program works as intended. There can be many pitfalls and logical things related to the correct or incorrect operation of the program. For example, there is a function that can be passed a certain amount of data. The programmer can consider this function as some input numerical data that can be passed to it, or this data can be limited by some sequence or length. For example, we enter the card number. It seems like the card number has a certain length. But, at the same time, any analyst and you should understand that instead of a number there can be letters or special characters, and the length may not be the same as the programmer came up with. This also needs to be checked, and all hypotheses need to be analyzed, to look at everything much wider than what is embedded in the business logic and thinking of the programmer who wrote it all. How do you understand that the candidate has an analytical mind? All this is easily clarified at the stage of "talking" with the candidate. You can simply ask questions like: "There is a data sample for process X, which consists of 1000 parameters. You need to determine the most important 30. The analysis task will be solved by 3 groups of analysts. How will you divide these parameters to obtain high efficiency and reliability of the analysis?" Experience Working in a Critical Situation It is desirable that the applicant has experience working in a crunch; for example, if he worked with servers with some kind of large critical load and was on duty. Usually, these are night shifts, evening shifts, on a weekend, when you have to urgently raise and restore something. Such people are very valuable. They really know how to work and have personally gone through different "pains." They are ready to put out fires with you and, most importantly, are highly likely to be more careful than others. I worked for a company that had a lot of students without experience. They very often broke a lot of things, and after that, it was necessary to raise all this. This is, of course, partly a consequence of mentoring. You have to help, develop, and turn students into specialists, but this does not negate the "pain" of correcting mistakes. And until you go through all this with them, they do not become cool. If a person participated in these processes and had the strength and ability to raise and correct, this is very cool. You need to select and take such people for yourself because they clearly know how to work. How To Avoid Being Fooled by Job Seekers Job seekers may overstate their achievements, but this is fairly easy to verify. If a person has the necessary experience, you need to ask them practical questions that are difficult to answer without real experience. For example, I ask about the implementation of a particular practice from DevSecOps, that is, what orchestrator he worked in. In a few words, the applicant should write, for example, a job in which it was all performed, and what tool he used. You can even suggest some keys from this vulnerability scanner and ask what keys and in what aspect you would use to make everything work. Only a specialist who has worked with this can answer these questions. In my opinion, this is the best way to check a person. That is, you need to give small practical tasks that can be solved quickly. It happens that not all applicants have worked and are working with the same as me, and they may have more experience and knowledge. Then it makes sense to find some common questions and points of contact with which we worked together. For example, just list 20 things from the field of information security and ask what the applicant is familiar with, find common points of interest, and then go through them in detail. When an applicant brags about having developments in interviews, it is also better to ask specific questions. If a person tells without hesitation what he has implemented, you can additionally ask him some small details about each item and direction. For example, how did you implement SAST verification, and with what tools? If he tells in detail and, possibly, with some additional nuances related to the settings of a particular scanner, and this fits into the general concept, then the person lived by this and used what he is talking about. Wrapping Up These are all the points that I pay attention to when looking for new people. I hope this information will be useful both for my Team Lead colleagues and for job seekers who will know what qualities they need to develop to successfully pass the interview.

In an era where the pace of software development and deployment is accelerating, the significance of having a robust and integrated DevOps environment cannot be overstated. Azure DevOps, Microsoft's suite of cloud-based DevOps services, is designed to support teams in planning work, collaborating on code development, and building and deploying applications with greater efficiency and reduced lead times. The objective of this blog post is twofold: first, to introduce Azure DevOps, shedding light on its components and how they converge to form a powerful DevOps ecosystem, and second, to provide a balanced perspective by delving into the advantages and potential drawbacks of adopting Azure DevOps. Whether you're contemplating the integration of Azure DevOps into your workflow or seeking to optimize your current DevOps practices, this post aims to equip you with a thorough understanding of what Azure DevOps has to offer, helping you make an informed decision tailored to your organization's unique requirements. What Is Azure DevOps? Azure DevOps represents the evolution of Visual Studio Team Services, capturing over 20 years of investment and learning in providing tools to support software development teams. As a cornerstone in the realm of DevOps solutions, Azure DevOps offers a suite of tools catering to the diverse needs of software development teams. Microsoft provides this product in the Cloud with Azure DevOps Services or on-premises with Azure DevOps Server. It offers integrated features accessible through a web browser or IDE client. At its core, Azure DevOps comprises five key components, each designed to address specific aspects of the development process. These components are not only powerful in isolation but also offer enhanced benefits when used together, creating a seamless and integrated experience for users. Azure Boards It offers teams a comprehensive solution for project management, including agile planning, work item tracking, and visualization tools. It enables teams to plan sprints, track work with Kanban boards, and use dashboards to gain insights into their projects. This component fosters enhanced collaboration and transparency, allowing teams to stay aligned on goals and progress. Azure Repos It is a set of version control tools designed to manage code efficiently. It provides Git (distributed version control) or Team Foundation Version Control (centralized version control) for source code management. Developers can collaborate on code, manage branches, and track version history with complete traceability. This component ensures streamlined and accessible code management, allowing teams to focus on building rather than merely managing their codebase. Azure Pipelines Azure Pipelines automates the stages of the application's lifecycle, from continuous integration and continuous delivery to continuous testing, build, and deployment. It supports any language, platform, and cloud, offering a flexible solution for deploying code to multiple targets such as virtual machines, various environments, containers, on-premises, or PaaS services. With Azure Pipelines, teams can ensure that code changes are automatically built, tested, and deployed, facilitating faster and more reliable software releases. Azure Test Plans Azure Test Plans provide a suite of tools for test management, enabling teams to plan and execute manual, exploratory, and automated testing within their CI/CD pipelines. Furthermore, Azure Test Plans ensure end-to-end traceability by linking test cases and suites to user stories, features, or requirements. They facilitate comprehensive reporting and analysis through configurable tracking charts, test-specific widgets, and built-in reports, empowering teams with actionable insights for continuous improvement. Thus providing a framework for rigorous testing to ensure that applications meet the highest standards before release. Azure Artifacts It allows teams to manage and share software packages and dependencies across the development lifecycle, offering a streamlined approach to package management. This feature supports various package formats, including npm, NuGet, Python, Cargo, Maven, and Universal Packages, fostering efficient development processes. This service not only accelerates development cycles but also enhances reliability and reproducibility by providing a reliable source for package distribution and version control, ultimately empowering teams to deliver high-quality software products with confidence. Below is an example of architecture leveraging various Azure DevOps services: Image captured from Microsoft Benefits of Leveraging Azure DevOps Azure DevOps presents a compelling array of benefits that cater to the multifaceted demands of modern software development teams. Its comprehensive suite of tools is designed to streamline and optimize various stages of the development lifecycle, fostering efficiency, collaboration, and quality. Here are some of the key advantages: Seamless Integration One of Azure DevOps' standout features is its ability to seamlessly integrate with a plethora of tools and platforms, whether they are from Microsoft or other vendors. This interoperability is crucial for anyone who uses a diverse set of tools in their development processes. Scalability and Flexibility Azure DevOps is engineered to scale alongside your business. Whether you're working on small projects or large enterprise-level solutions, Azure DevOps can handle the load, providing the same level of performance and reliability. This scalability is a vital attribute for enterprises that foresee growth or experience fluctuating demands. Enhanced Collaboration and Visibility Collaboration is at the heart of Azure DevOps. With features like Azure Boards, teams can have a centralized view of their projects, track progress, and coordinate efforts efficiently. This visibility is essential for aligning cross-functional teams, managing dependencies, and ensuring that everyone is on the same page. Continuous Integration and Deployment (CI/CD) Azure Pipelines provides robust CI/CD capabilities, enabling teams to automate the building, testing, and deployment of their applications. This automation is crucial to accelerate their time-to-market and improve the quality of their software. By automating these processes, teams can detect and address issues early, reduce manual errors, and ensure that the software is always in a deployable state, thereby enhancing operational efficiency and software reliability. Drawbacks of Azure DevOps While Azure DevOps offers a host of benefits, it's essential to acknowledge and understand its potential drawbacks. Like any tool or platform, it may not be the perfect fit for every organization or scenario. Here are some of the disadvantages that one might encounter: Vendor Lock-In By adopting Azure DevOps services for project management, version control, continuous integration, and deployment, organizations may find themselves tightly integrated into the Microsoft ecosystem. This dependency could limit flexibility and increase reliance on Microsoft's tools and services, making it challenging to transition to alternative platforms or technologies in the future. Integration Challenges Although Azure DevOps boasts impressive integration capabilities, there can be challenges when interfacing with certain non-Microsoft or legacy systems. Some integrations may require additional customization or the use of third-party tools, potentially leading to increased complexity and maintenance overhead. For organizations heavily reliant on non-Microsoft products, this could pose integration and workflow continuity challenges. Cost Considerations Azure DevOps operates on a subscription-based pricing model, which, while flexible, can become significant at scale, especially for larger teams or enterprises with extensive requirements. The cost can escalate based on the number of users, the level of access needed, and the use of additional features and services. For smaller teams or startups, the pricing may be a considerable factor when deciding whether Azure DevOps is the right solution for their needs. Potential for Over-Complexity With its myriad of features and tools, there's a risk of over-complicating workflows and processes within Azure DevOps. Teams may find themselves navigating through a plethora of options and configurations, which, if not properly managed, can lead to inefficiency rather than improved productivity. Organizations must strike a balance between leveraging Azure DevOps' capabilities and maintaining simplicity and clarity in their processes. While these disadvantages are noteworthy, they do not necessarily diminish the overall value that Azure DevOps can provide to an organization. It's crucial for enterprises and organizations to carefully assess their specific needs, resources, and constraints when considering Azure DevOps as their solution. By acknowledging these potential drawbacks, organizations can plan effectively, ensuring that their adoption of Azure DevOps is strategic, well-informed, and aligned with their operational goals and challenges. Conclusion In the landscape of modern software development, Azure DevOps stands out as a robust and comprehensive platform, offering a suite of tools designed to enhance and streamline the DevOps process. Its integration capabilities, scalability, and extensive features make it an attractive choice for any organization or enterprise. However, like any sophisticated platform, Azure DevOps comes with its own set of challenges and considerations. The vendor lock-in, integration complexities, cost factors, and potential for over-complexity are aspects that organizations need to weigh carefully. It's crucial for enterprises to undertake a thorough analysis of their specific needs, resources, and constraints when evaluating Azure DevOps as a solution. The decision to adopt Azure DevOps should be guided by a strategic assessment of how well its advantages align with the organization's goals and how its disadvantages might impact operations. For many enterprises, the benefits of streamlined workflows, enhanced collaboration, and improved efficiency will outweigh the drawbacks, particularly when the adoption is well-planned and aligned with the organization's objectives.

The acronym "Ops" has rapidly increased in IT operations in recent years. IT operations are turning towards the automation process to improve customer delivery. Traditional application development uses DevOps implementation for Continued Integration (CI) and Continued Deployment (CD). The exact delivery and deployment process may not be suitable for data-intensive Machine Learning and Artificial Intelligence (AI) applications. This article will define different "Ops" and explain their work for the following: DevOps, DataOps, MLOps, and AIOps. DevOps This practice automates the collaboration between Development (Dev) and Operations (Ops). The main goal is to deliver the software product more rapidly and reliably and continue delivery with software quality. DevOps complements the agile software development process/agile way of working. DataOps DataOps is a practice or technology that combines integrated and process-oriented data with automation to improve data quality, collaboration, and analytics. It mainly deals with the cooperation between data scientists, data engineers, and other data professionals. MLOps MLOps is a practice or technology that develops and deploys machine learning models reliably and efficiently. MLOps is the set of practices at the intersection of DevOps, ML, and Data Engineering. AIOps AIOps is the process of capabilities to automate and streamline operations workflows for natural language processing and machine learning models. Machine Learning and Big Data are major aspects of AIOps because AI needs data from different systems and processes using ML models. AI is driven by machine learning models to create, deploy, train, and analyze the data to get accurate results. As per the IBM Developer, below are the typical “Ops” work together: Image Source: IBM Collective Comparison The table below describes the comparison between DevOps, DataOps, MLOps, and AIOps: Aspect DevOps DataOps MLOps AIOps Focus on: IT operations and software development with Agile way of working Data quality, collaboration, and analytics Machine Learning models IT operations Key Technologies/Tools: Jenkins, JIRA, Slack, Ansible, Docker, Git, Kubernetes, and Chef Apache Airflow, Databricks, Data Kitchen, High Byte Python, TensorFlow, PyTorch, Jupyter, and Notebooks Machine learning, AI algorithms, Big Data, and monitoring tools Key Principles: IT process automation Team collaboration and communication Continuous integration and continuous delivery (CI/CD) Collaboration between data Data pipeline automation and optimization Version control for data artifacts Data scientists and operations teams collaborate. Machine learning models, version control Continuous monitoring and feedback Automated analysis and response to IT incidents Proactive issue resolution using analytics IT management tools integration Continuous improvement using feedback Primary Users Software and DevOps engineers Data and DataOps engineers Data scientists and MLOps engineers Data scientists, Big Data scientists, and AIOps engineers Use Cases Microservices, containerization, CI/CD, and collaborative development Ingestion of data, processing and transforming data, and extraction of data into other platforms Machine learning (ML) and data science projects for predictive analytics and AI IT AI operations to enhance network, system, and infrastructure Summary In summary, managing a system from a single project team is at the end of its life due to business processes becoming more complex and IT systems changing dynamically with new technologies. The detailed implementation involves a combination of collaborative practices, automation, monitoring, and a focus on continuous improvement as part of DevOps, DataOps, MLOps, and AIOps processes. DevOps focuses primarily on IT processes and software development, and the DataOps and MLOps approaches focus on improving IT and business collaborations as well as overall data use in organizations. DataOps workflows leverage DevOps principles to manage the data workflows. MLOps also leverages the DevOps principles to manage applications built-in machine learning.

By Ravi Kiran Mallidi CORE

Overview of Azure DevOps Azure DevOps is a set of tools and services for software development that covers everything from planning and coding to testing and deployment. Developed by Microsoft and based in the cloud Azure DevOps facilitates collaboration and project management efficiency offering features tailored to developers and operations teams alike. This platform enables organizations to deliver top-notch software products by simplifying workflows and promoting teamwork among teams. Figure courtesy from Microsoft An essential aspect of Azure DevOps is Azure Repositories, which offer robust source control management. Developers can work together on projects, manage code versions, and maintain a record of changes. With support for branching and merging strategies teams can experiment with features without jeopardizing the stability of the codebase. Another critical element within Azure DevOps is Azure Boards, which provides a suite of tools for project management and tracking work items. Teams can create tasks, user stories and bugs using boards and backlogs to prioritize work and plan sprints efficiently to keep projects on schedule. By integrating methodologies, like Scrum and Kanban, teams can adopt industry practices while continuously enhancing their processes. Azure Pipelines serves as the engine for Continuous Integration and Continuous Deployment (CI/CD) in Azure DevOps. It automates tasks like builds, tests, and deployments making the release process smoother and reducing errors. Developers can set up pipeline configurations using YAML files to define the steps and environments involved in building and deploying applications. Azure Pipelines is versatile supporting a variety of programming languages, platforms, and cloud services making it suitable for project needs. Azure Artifacts functions as a package management service that enables teams to manage dependencies across projects. Developers can create, share, and use packages to ensure consistency in their development processes. The service supports package formats such as NuGet, npm, Maven, and PyPI to cater to project requirements. Azure Test Plans provide a suite of testing tools for testing and exploratory testing activities. Teams can effectively manage test cases, execute tests, and track bugs within the Azure DevOps environment. This integration ensures that thorough testing is seamlessly integrated into the development lifecycle to help identify issues. Moreover, Azure DevOps integrates seamlessly, with third-party tools and services to expand its capabilities and empower teams to tailor their workflows based on requirements. Some common tools integrated with Azure DevOps include Jenkins, GitHub, Docker, and Kubernetes. This versatility enables teams to make the most of their existing tools while taking advantage of Azure DevOps's strong features. One of the benefits of Azure DevOps is its ability to scale up or down based on project size and complexity. As a cloud-based solution, it can cater to projects ranging from development teams to enterprise endeavors. This scalability allows teams to focus on their development tasks without having to worry about managing infrastructure resources. Moreover, Azure DevOps provides analytics and reporting functionalities that offer insights into project performance and progress, for teams. Dashboards and reports are useful for teams to monitor metrics, like the success rates of building and deploying completion of work items and code coverage. This data-focused approach enables teams to make informed decisions and continually enhance their methods. Simply put Azure DevOps is a platform that supports the software development cycle. With features for source control, project management, CI/CD, package management, and testing Azure DevOps simplifies. Encourages teamwork among groups. Its ability to integrate with tools and services coupled with its emphasis on security and scalability positions it as a robust option for organizations seeking to enhance their software development processes. Understanding Continuous Integration (CI) Continuous Integration (CI) is a development practice that focuses on automating the process of combining code modifications from contributors into a shared repository reliably. This approach helps in the detection and resolution of integration issues during the development phase leading to stable software releases and a smoother development journey. CI plays a role in software development practices and is commonly linked with Continuous Delivery (CD) or Continuous Deployment to establish a seamless transition from code creation to production deployment. Essentially CI entails the merging of code changes made by team members into a central repository followed by automated building and testing processes. This enables developers to promptly identify and resolve integration conflicts and problems thereby minimizing the chances of introducing bugs or other issues into the codebase. Through the integration of changes, teams can uphold a level of code quality and uniformity. A standard CI workflow comprises stages. Initially, developers commit their code alterations to a version control system (VCS) like Git. The CI server keeps an eye on the VCS repository for any commits. Triggers an automated build once it detects changes. Throughout the construction phase, the server compiles the code. Executes a series of automated tests, which include unit tests, integration tests, and other forms of testing, like static code analysis or security scans. If all goes well with the build and tests the alterations are deemed integrated. The build is labeled as successful. In case any issues arise, such as test failures or build glitches the CI server promptly notifies developers for resolution. This quick feedback loop stands out as an advantage of CI enabling teams to catch problems and prevent development delays. CI also fosters collaboration and communication among team members. With frequent code integrations happening developers can regularly. Discuss each other's work. This practice promotes peer review culture and ongoing improvement efforts helping teams uphold standards of code quality and adhere to practices. A significant benefit of CI lies in its ability to thwart the integration hell" scenario where substantial changes are infrequently merged leading to an integration process that consumes time. By integrating changes, through CI practices teams can mitigate risks effectively and maintain a consistent development pace Another crucial aspect of Continuous Integration (CI) involves utilizing automation tools to oversee the build and testing procedures. CI servers, like Jenkins, GitLab CI/CD, and Azure DevOps Pipelines offer automation functionalities that streamline workflows and maintain consistency across builds. These tools can be customized to execute tasks, such as code compilation, test execution, and report generation based on the team's needs. In summary, Continuous Integration plays a role in software development by promoting high standards of code quality and efficiency. By integrating code changes, automating builds and tests, and providing feedback CI helps teams identify issues early on and avoid integration difficulties. This enables teams to deliver software products while maintaining a smooth development workflow. Establishing an Azure DevOps Pipeline With Continuous Integration Initiating a New Azure DevOps Project Sign in to Azure DevOps. Click on "Create New Project". Specify a project name. Choose the desired visibility setting (public or private). Create the project. Configuring Source Code Repositories Within your project navigate to "Repositories" to establish your source code repository. Create a repository. Replicate an existing one from an external origin. Setting up Build Processes Navigate to the "Pipelines" section in your Azure DevOps project. Select "New Pipeline". Indicate the source of your code (Azure Repos, GitHub, etc.). Opt for a pipeline template. Craft a new one from scratch. Outline the steps for constructing your application (compiling code and executing tests). Save your settings. Initiate the pipeline execution. Setting up Deployment Workflows Navigate to the "Pipelines" section. Choose "Releases." Select "New release pipeline". Pick a source pipeline (build pipeline) for your deployment. Outline the stages for your deployment workflow (e.g., Development, Staging, Production). Include tasks for deploying, configuring, and any follow-up steps after deployment. Save. Execute the workflow. Benefits of Optimizing Azure DevOps Pipeline Optimizing Azure DevOps pipelines brings advantages that can enhance the effectiveness and quality of software development and deployment processes. By streamlining workflows and promoting collaboration organizations can achieve more software delivery. Here are some key advantages of optimizing Azure DevOps pipelines, Quicker Feedback Loops Optimized pipelines offer feedback on code modifications through automated builds and tests enabling developers to promptly detect and address issues. Rapid feedback aids in reducing the time needed to resolve bugs and enhancing code quality. Enhanced Code Quality Automated testing, encompassing unit, integration, and end-to-end tests ensures that code alterations do not introduce problems or setbacks. Incorporating AI-driven code quality assessment tools can help spot issues like code irregularities, security susceptibilities, and undesirable patterns. Improved Developer Efficiency By automating tasks, like builds, tests, and deployments developers can concentrate on crafting top-notch code and creating features. Efficient pipelines diminish involvement. Decrease the likelihood of human errors. Boosted Dependability Consistent and automated testing guarantees that the software stays stable and functional throughout the development cycle. Automated deployments can be authenticated against predefined acceptance criteria to lessen deployment complications. Efficient Use of Resources Improving workflows can help manage the distribution and utilization of resources reducing resource consumption and expenses. Utilizing features, like processing and data caching can accelerate the building and deployment procedures while minimizing infrastructure costs. Scalability and Adaptability Azure DevOps pipelines can be easily expanded to support projects of sizes and complexities catering to both development teams and large corporate ventures. The platform offers support for programming languages, frameworks, and cloud services providing flexibility in tool selection and customization options. Enhanced Collaboration and Communication Functions such as requests, code evaluations, and threaded discussions facilitate teamwork among members by enabling collaboration on code modifications. Optimized workflows promote a culture of enhancement and knowledge exchange among team members. Improved Monitoring and Analysis Azure DevOps provides tools for monitoring performance metrics and project advancement offering insights into pipeline efficiency. Interactive dashboards and detailed reports help teams monitor indicators such as success rates, in building/deployment processes, test coverage levels, and task completion progress. Continuous Enhancement Streamlined workflows empower teams to iterate rapidly while continuously enhancing their development practices. By pinpointing areas needing improvement and bottlenecks, teams can enhance their workflows. Embrace strategies. Embracing DevOps Principles Azure DevOps pipelines facilitate the adoption of DevOps principles, like Infrastructure as Code (IaC) automated testing and continuous delivery. These principles play a role in streamlining development processes to become more agile and efficient. To sum up, streamlining Azure DevOps pipelines brings about advantages that lead to more dependable and superior software releases. Through the utilization of automation, AI-driven tools, and best practices teams can elevate their development procedures, for increased productivity and effectiveness. AI in Azure DevOps Pipelines With Example AI can bring significant enhancements to Azure DevOps pipelines, making them more efficient, reliable, and productive. By leveraging AI, you can improve code quality, optimize testing, automate various tasks, and gain insights from data analysis. One of the useful ways you can use AI in Azure DevOps pipelines is to enable automatic issue detection and resolution. Let's look into it, Automated Issue Detection and Resolution AI can automatically detect and even resolve common issues in the pipeline, such as build failures or flaky tests. Using AI to detect and resolve common issues in the pipeline, such as build failures or flaky tests, can improve the stability and reliability of your development workflow. Here's an example that demonstrates how you can use AI in an Azure DevOps pipeline to detect and resolve common issues: 1. Integrate AI-Based Monitoring and Insights Start by integrating AI-based monitoring and insights into your pipeline. This will enable you to gather data on pipeline performance and identify potential issues. Use Azure monitor: Integrate Azure Monitor with your pipeline to collect logs, metrics, and traces from your builds and tests. Configure AI-based anomaly detection: Use AI-based anomaly detection to monitor the pipeline for unusual patterns or deviations from expected performance. 2. Detecting Pipeline Issues With AI AI can be used to monitor the pipeline in real-time and detect common issues such as build failures or flaky tests. Analyze build logs: Use AI to analyze build logs and identify patterns that indicate build failures or flaky tests. Monitor test results: AI can monitor test results for inconsistencies, such as tests that pass intermittently (flaky tests). 3. Resolving Common Issues Automatically Once AI detects an issue, you can configure automated actions to resolve the problem. Automatic retry: If a build failure is detected, configure the pipeline to automatically retry the build to see if the issue persists. Flaky test management: If flaky tests are detected, AI can tag them for further investigation and potentially quarantine them to prevent them from impacting the pipeline. Rollbacks: If an issue occurs during deployment, AI can automatically trigger a rollback to the previous stable version. 4. Example Pipeline Configuration Here is an example Azure DevOps pipeline configuration (azure-pipelines.yml) that demonstrates how you might integrate with Azure OpenAI to "Generate code comments." YAML trigger: - main pr: - main pool: vmImage: 'ubuntu-latest' jobs: - job: GenerateCodeComments displayName: 'Generate Code Comments with Azure OpenAI' steps: - checkout: self displayName: 'Checkout Code' - task: AzureCLI@2 displayName: 'Generate Code and Comments with Azure OpenAI' inputs: azureSubscription: 'Your Azure Subscription' scriptLocation: 'inlineScript' inlineScript: | # Set the endpoint and API key for Azure OpenAI Service OPENAI_ENDPOINT="https://YOUR_AZURE_OPENAI_ENDPOINT.azure.com" OPENAI_API_KEY="YOUR_AZURE_OPENAI_API_KEY" # Prepare the prompt for code completion and comment generation # This example uses a placeholder. In practice, dynamically extract relevant code snippets or provide context. PROMPT="Extracted code snippet for analysis" # Make a REST API call to Azure OpenAI Service response=$(curl -X POST "$OPENAI_ENDPOINT/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ --data "{ \"model\": \"code-davinci-002\", \"prompt\": \"$PROMPT\", \"temperature\": 0.7, \"max_tokens\": 150, \"top_p\": 1.0, \"frequency_penalty\": 0.0, \"presence_penalty\": 0.0 }") echo "Generated code and comments:" echo $response # The response will contain the generated code completions and comments. # Consider parsing this response and integrating suggestions into the codebase manually or through automated scripts. # Optional: Add steps for reviewing or applying the generated suggestions # - script: echo "Review and integrate suggestions" # displayName: 'Review Suggestions' - Key Points Trigger and PR: This pipeline is triggered by commits to the main branch and pull requests targeting the main branch, ensuring that code comments and suggestions are generated for the most current and relevant changes. AzureCLI task: The core of this pipeline is the AzureCLI task, which makes a REST API call to the Azure OpenAI Service, passing a code snippet (the PROMPT) and receiving AI-generated code comments and suggestions. Dynamic prompt extraction: The example uses a static prompt. In a real-world scenario, you would dynamically extract relevant code snippets from your repository to use as prompts. This might involve additional scripting or tools to analyze your codebase and select meaningful snippets for comment generation. Review and integration: The optional step at the end hints at a manual or automated process for reviewing and integrating the AI-generated suggestions into your codebase. The specifics of this step would depend on your team's workflow and the tools you use for code review and integration. 5. Configure AI-Based Analysis Custom AI model: Use Azure Cognitive Services or another AI model to analyze build logs and test results for patterns indicative of common issues. Trigger actions: Based on the analysis results, trigger automated actions such as retrying builds, quarantining flaky tests, or rolling back deployments. 6. Review and Improve Monitor and adjust: Continuously monitor the AI-based analysis and automated actions to ensure they are effective in resolving issues. Feedback loop: Incorporate feedback from the AI analysis into your development process to continuously improve the pipeline's reliability and stability. By leveraging AI to detect and resolve common issues in the pipeline, you can minimize downtime, reduce manual intervention, and create a more robust and efficient development process. Conclusion By optimizing Azure DevOps pipelines with AI and Continuous Integration you can greatly boost the development process by enhancing efficiency, code quality, and reliability. This guide offers instructions on configuring and optimizing Azure DevOps pipelines with AI and CI.

By Naga Santhosh Reddy Vootukuri CORE

With modern tools and QAOps methodologies, infrastructure as code and configuration as code is taking development practices in an operational context to a whole new level. As a result, you get a much more rigid, streamlined process that’s much faster, better automated, and drastically reduces errors, not to mention giving you very consistent output. This is what configuration as code provides us. An application’s codebase and server deployment configuration are usually separated during software development and deployment. The Ops team often creates the configuration settings and tools necessary to build and deploy your app across various server instances and environments. Using configuration as code entails treating configuration settings similarly to your application code. For configuration settings, you should take advantage of version control. What Is Configuration as Code? An approach to managing your software called “configuration as code” advocates for configuration settings (such as environmental settings, resources provisioning, etc.) to be defined in code. This entails committing your software configuration settings to a version control repository and handling them the same way you would the rest of your code. This contrasts with having your configuration located elsewhere other than the repository or possibly needing to create and customize the configuration for each deployment. As a result, it becomes way easier to synchronize configuration changes across different deployments or instances. You can publish server change updates to the repository like any other commit, which can subsequently be picked up and sent to the server like any other update, saving you from having to configure server changes or use another out-of-code solution manually. Infrastructure as Code vs. Configuration as Code The approach of treating infrastructure as though it were software is known as infrastructure as code (IaC). You can write code to specify how your infrastructure should seem if you consider it another application in your software stack. Once tested, you may use that description to create or destroy infrastructure automatically. IaC and CaC both automate the provisioning and configuration of your software, but they do so in various ways. In infrastructure as a code, you codify your infrastructure so a machine can manage it. Before deploying your system, you build scripts that specify how you want it to be configured and how it should look. IaC is frequently used to automate the deployment and configuration of both physical and virtual servers. Before deploying an application, CaC requires you to model its configuration. When you implement new software configurations, your application configuration settings are updated without requiring manual involvement. CaC applies to containers, microservices, and other application types. Merge requests, CI/CD and IaC are essential GitOps techniques. Git is the only source of truth in GitOps, a method of controlling declarative infrastructure. Infrastructure updates are a crucial part of the software integration and delivery process with GitOps, and you can incorporate them into the same CI/CD pipeline. This integration simplifies config updates. Simply creating and pushing the configuration modifications to the source control repository is all that is required from a developer. Before making changes to the underlying infrastructure, the code in this repository is tested using CI/CD technologies. Why Use Configuration as Code? Teams can benefit from implementing configuration as code in several ways. Scalability Handling configuration changes as code, like IaC, enables teams to create, update, and maintain config files from a single centralized location while leveraging a consistent deployment approach. For instance, you require configuration files for each storage option if you are developing USB devices. You may create thousands of configurations by combining these files with the required software. To handle these variations, you need a robust, centralized source control that can be accessed from different levels in your CI/CD pipeline. Standardization When the configuration is written like source code, you can use your development best practices, such as linting and security scanning. Before they are committed, config files must be reviewed and tested to guarantee that modifications adhere to your team’s standards. Your configurations can be maintained stable and consistent via a complicated microservices architecture. Services function more effectively together when a set process is in place. Traceability Setting up configuration as code requires version control. You require a robust system that can conveniently save and track changes to your configuration and code files. This could improve the level of quality of your release. You can locate its source if a bug still slips through and rapidly identify/fix an issue by comparing the versioned configuration files. Increased Productivity You may streamline your build cycle by turning configurations into managed code. Both IT and end users are more productive as a result. Your administrators may incorporate everything into a release or build from a single version control system. Developers are confident in the accuracy of their changes because every component of your workflow has been tested in concert. When To Use Configuration as Code? Configuration as code is used to manage settings for packages and components. This works across a wide range of industries. During the development of an app, configurations might be utilized to support several operating systems. By maintaining configuration as code, you may track hundreds or even thousands of hardware schematics and testing information for embedded development. How Teams Implement Configuration as Code You must decide how to save the configuration files you create or refactor in your code in your version control system. Teams can accomplish this in various ways: Put configuration files and code in the same repository (monorepo). Keep configuration files and code together based on your needs. component-based development and microservices. Keep configurations and code in separate repositories. Monorepo Strategy Your workflow may be simpler if all your files are in one repository. However, if you treat configuration files as source code, any change to a setting can result in a fresh build. This might not be necessary and might make your team work more slowly. Not every config update demands a build. Your system’s administrators would have to configure it to enable the merging of changes to configuration files. They might then be deployed to one of your pre-production environments to do further testing. Because everything is code, it might be challenging to distinguish between configuration files and source code when performing an audit. Establishing a naming convention that is uniform across teams is crucial. Microservices/Component-Based Development Teams often separate their code into several repos for various reasons. According to this architecture, configuration files are kept and versioned alongside the particular microservice or component. Even though you might get a similar problem with trigger builds, it might be simpler to handle. Collaborate with your DevOps teams if you plan to version config files with their microservice or component. Plan how configuration changes will be distributed. Separate Repos for Configuration Files Whatever method you use to save your source code, some teams prefer to keep their configuration files in a separate repository. Although it sounds like an excellent idea, this is rarely viable. Even the most complicated projects may contain fewer than a thousand configuration files. As a result, they would occupy a relatively small space within a repository. The setup of your build pipeline would require time from your administrators. You might wish to consider alternative solutions, even if this paradigm can be useful for audits, rollbacks, and reviews. Config as Code: Use Cases What does “Config as Code” mean in practice? It can be put into effect in several different ways, not all of which are appropriate for every organization. See if the broad strokes below meet your particular needs: Making use of unique configuration source control repositories. Creating a custom build and deployment procedure. Establishing test environments with a focus on configuration. Making sure there are procedures for approval and quality control. Secrets management within configurations. Creating Test Environments for Configuration Maybe setting up a complete testing environment for application code is not necessary for a simple configuration modification. A company can save time and money by limiting the scope of a test environment to the requirements of the configuration deployment process. Additionally, this might imply that various changes can co-occur. During the testing of a configuration change, application developers can test their code. You improve environmental management and operation efficiency with this capacity for parallel testing. Conclusion Your development team can reap significant advantages by incorporating configuration as code into your process. Applying updates and ensuring that everything works as intended is made simpler by automating the deployment of configurations across environments. Changes are simple to manage and track because it uses a single repository. While enhancing the development and deployment of code, configuration as code is a valuable tool for managing and controlling complex infrastructure and pipelines. As a result, you have the visibility and control you need to speed up development without compromising the security of your deployments.

With the rise of high-frequency application deployment, CI/CD has been adopted by the modern software development industry. But many organizations are still looking for a solution that will give them more control over the delivery of their applications such as the Canary deployment method or even Blue Green. Called Progressive Delivery, this process will give organizations the ability to run multiple versions of their application and reduce the risk of pushing a bad release. In this post, we will focus on Canary deployment as there’s a high demand for organizations to run testing in production with real users and real traffic which Blue Green deployment cannot do. ArgoCD vs. Flagger: Overview A Canary deployment will be triggered by ArgoCD Rollout and Flagger if one of these changes ais applied: Deployment PodSpec (container images, commands, ports, env, resources, etc) ConfigMaps mounted as volumes or mapped to environment variables Secrets mounted as volumes or mapped to environment variables Why Not Use Kubernetes RollingUpdate? Kubernetes offers by default their RollingUpdate deployment strategy, but it can be limited due to: No fine-grained control over the speed of a new release, by default Kubernetes will wait for the new pod to get into a ready state and that’s it. Can’t manage traffic flow, without traffic split, it is impossible to send a percentage of the traffic to a newer release and adjust its percentage. No ability to verify external metrics such as Prometheus custom metrics to verify the status of a new release. Unable to automatically abort or rollback the update What Is ArgoCD Rollout? Just a year after ArgoCD creation, in 2019 the group behind the popular ArgoCD decided to overcome these limitations from Kubernetes by creating ArgoCD Rollout as a Kubernetes Controller used to achieve Canary, Blue Green, Canary analysis, experimentation, and progressive delivery features to Kubernetes with the most popular service mesh and ingress controllers. What Is Flagger? Created in 2018 by the FluxCD community, FluxCD has been growing massively since its creation and offers Flagger as one of its GitOps components to deal with progressive delivery on Kubernetes. Flagger helps developers solidify their production releases by applying canary, A/B testing, and Blue Green deployment strategies. It has direction integration with service mesh such as Istio and Linkerd but also ingress controllers like NGINX or even Traefik. How ArgoCD Rollout and Flagger Work With Istio If you are using Istio as a service mesh to deal with traffic management and want to use Canary as a deployment strategy: ArgoCD Rollout will transform your Kubernetes Deployment as a ReplicaSet. To start, you would need to create the Istio DestinationRule and Virtual Service but also the two Kubernetes Services (stable and canary) The next step would be creating your rollout, ArgoCD Rollout will manage the Virtual Service to match with the current desired canary weight and your DestionationRule that will contain the label for the canary ReplicaSet. Example: YAML apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: reviews-rollout namespace: default spec: replicas: 1 selector: matchLabels: app: reviews version: stable template: metadata: labels: app: reviews version: stable service.istio.io/canonical-revision: stable spec: serviceAccountName: bookinfo-reviews containers: - name: reviews image: docker.io/istio/examples-bookinfo-reviews-v1:1.18.0 imagePullPolicy: IfNotPresent env: - name: LOG_DIR value: "/tmp/logs" ports: - containerPort: 9080 volumeMounts: - name: tmp mountPath: /tmp - name: wlp-output mountPath: /opt/ibm/wlp/output securityContext: runAsUser: 1000 volumes: - name: wlp-output emptyDir: {} - name: tmp emptyDir: {} strategy: canary: canaryService: reviews-canary stableService: reviews-stable trafficRouting: istio: virtualService: name: reviews destinationRule: name: reviews canarySubsetName: canary stableSubsetName: stable steps: - setWeight: 20 - pause: {} # pause indefinitely - setWeight: 40 - pause: {duration: 10s} - setWeight: 60 - pause: {duration: 10s} - setWeight: 80 - pause: {duration: 10s} Here’s a documentation link for the Istio ArgoCD Rollout integration. Flagger relies on a k8s custom resource called Canary, example below: YAML apiVersion: flagger.app/v1beta1 kind: Canary metadata: name: reviews namespace: default spec: # deployment reference targetRef: apiVersion: apps/v1 kind: Deployment name: reviews # the maximum time in seconds for the canary deployment # to make progress before it is rollback (default 600s) progressDeadlineSeconds: 60 service: # service port number port: 9080 analysis: # schedule interval (default 60s) interval: 15s # max number of failed metric checks before rollback threshold: 5 # max traffic percentage routed to canary # percentage (0-100) maxWeight: 50 # canary increment step # percentage (0-100) stepWeight: 10 As seen on L11, you don’t have to define your deployment but can call its name so the k8s deployment is managed outside of the Canary custom resource. Once you apply this, Flagger will automatically create the Canary resources: # generated deployment.apps/reviews-primary service/reviews service/reviews-canary service/reviews-primary destinationrule.networking.istio.io/reviews-canary destinationrule.networking.istio.io/reviews-primary virtualservice.networking.istio.io/reviews As you can see, it created the Istio Destinationrule and Virtual service to achieve traffic management for canary deployment. How Does ArgoCD Rollout Compare to Flagger? Both solutions support the same service mesh and share a very similar analysis process but there are a few features that can make the difference in choosing your progressive delivery tool for Kubernetes ArgoCD Rollout Flagger + - Great UI/dashboard to manage releases - ArgoCD dashboard (not Rollout dashboard) can interact with ArgoCD Rollout to approve promotions. - Kubectl plugin which makes it easy to query via a CLI rollout status. - Automatically creates the Kubernetes Services, Istio DestinationRule, and Virtual Service. - Load tester can run advanced testing scenarios. - - ArgoCD Rollout needs you to create Kubernetes Services, Istio DestinationRules, and Vertical Services manually. - No authentication or RBAC for the Rollout dashboard. - CLI only, no UI/dashboard. - Logs can lack information, in addition to being difficult to visualize. - No Kubectl plugin to easily fetch deployment information. - Documentation may not be as detailed as ArgoCD Rollout. Conclusion Both solutions are backed up by strong communities so there’s not a bad option that really stands out. You may already be using FluxCD. In this case, Flagger makes sense as an option to achieve progressive delivery and the same goes for ArgoCD and ArgoCD Rollout We hope this helps you get an idea of how ArgoCD Rollout and Flagger work with Canary deployments and Istio, in addition to giving you a general overview of the two solutions.