Using YAML aliases to DRY your configurations

When projects reach any level of complexity the ability to change the projects configuration arises. Often YAML is used as a human and machine readable syntax to store configuration. Kubernetes, Java Spring Boot, GitLab, Docker, Ansible, AWS CloudFormation, and many other projects use YAML. YAML is a corner stone syntax in modern applications.

Understanding that YAML supports aliases can go a long way towards reducing repeated configuration. Lets take a look at a GitLab pipeline that is not using aliases.

include:
  - project: devops/pipelines/gitlab-ci
    file: qa-checkov.yml
  - project: devops/pipelines/gitlab-ci
    file: qa-conftest.yml

image:
  name: us.gcr.io/my-org/container-name:1.2.3

cache:
  paths:
    - .helm
    - .terraform
    - /root
    - ~/.ssh

stages:
  - config
  - init
  - qa
  - plan
  - apply
  - commit

terraform:
  except:
    - pipelines
  script:
    - echo "Configuration started..."
    - git config --global user.email "terradrone@tenerum.io"
    - git config --global user.name "terradrone"
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
    - echo "Configuration completed..."
    - terraform init
  stage: init

terraform-plan:
  artifacts:
    paths:
      - planfile-$CI_COMMIT_REF_NAME
  except:
    - pipelines
  script:
    - echo "Configuration started..."
    - git config --global user.email "terradrone@tenerum.io"
    - git config --global user.name "terradrone"
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
    - echo "Configuration completed..."
    - HOST="" terraform plan -out "planfile-$CI_COMMIT_REF_NAME"
  stage: plan

terraform-apply:
  except:
    - pipelines
  only:
    - master
  script:
    - echo "Configuration started..."
    - git config --global user.email "terradrone@tenerum.io"
    - git config --global user.name "terradrone"
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
    - echo "Configuration completed..."
    - HOST="" terraform apply "planfile-$CI_COMMIT_REF_NAME"
  stage: apply

tag:
  only:
    - pipelines
  script:
    - echo "Configuration started..."
    - git config --global user.email "terradrone@tenerum.io"
    - git config --global user.name "terradrone"
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
    - echo "Configuration completed..."
    - eval $(ssh-agent -s) && ssh-add <(terraform output ssh-key)
    - git clone git@${CI_SERVER_HOST}:${CI_PROJECT_PATH}.git
  stage: apply

commit:
  except:
    - tags
    - pipelines
  only:
    - master
  script:
    - echo "Configuration started..."
    - git config --global user.email "terradrone@tenerum.io"
    - git config --global user.name "terradrone"
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
    - echo "Configuration completed..."
    - eval $(ssh-agent -s) && ssh-add <(terraform output ssh-key)
    - cd ${CI_PROJECT_NAME}
    - git checkout ${CI_COMMIT_REF_NAME}
  stage: commit

Look at all that repeated configuration code. This pipeline would me a pain to manage.

Now lets take a look at the same pipeline with YAML alias pattern applied.

include:
  - project: devops/pipelines/gitlab-ci
    file: qa-checkov.yml
  - project: devops/pipelines/gitlab-ci
    file: qa-conftest.yml

image:
  name: us.gcr.io/my-org/container-name:1.2.3

cache:
  paths:
    - .helm
    - .terraform
    - /root
    - ~/.ssh

stages:
  - config
  - init
  - qa
  - plan
  - apply
  - commit

# This is the anchor of the alias, the definition
.config: &config
- echo "Configuration started..."
- git config --global user.email "terradrone@tenerum.io"
- git config --global user.name "terradrone"
- eval $(ssh-agent -s)
- echo "$DEPLOY_SSH" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts
- eval $(ssh-agent -s) && ssh-add <(terraform output ssh-key)
- echo "Configuration completed..."

terraform:
  except:
    - pipelines
  script:
    # this is a YAML reference to the anchor defined above
    - *config
    - terraform init
  stage: init

terraform-plan:
  artifacts:
    paths:
      - planfile-$CI_COMMIT_REF_NAME
  except:
    - pipelines
  script:
    - *config
    - HOST="" terraform plan -out "planfile-$CI_COMMIT_REF_NAME"
  stage: plan

terraform-apply:
  except:
    - pipelines
  only:
    - master
  script:
    - *config
    - HOST="" terraform apply "planfile-$CI_COMMIT_REF_NAME"
  stage: apply

tag:
  only:
    - pipelines
  script:
    - *config
    - git clone git@${CI_SERVER_HOST}:${CI_PROJECT_PATH}.git
  stage: apply

commit:
  except:
    - tags
    - pipelines
  only:
    - master
  script:
    - *config
    - cd ${CI_PROJECT_NAME}
    - git checkout ${CI_COMMIT_REF_NAME}
  stage: commit

That looks so much better, much more pragmatic, and much easier to maintain.

Conclusion

Using YAML alias we are now able to reduce complexity, increase maintainability, and DRY yaml configurations.

Additional References

• Yes, I am aware GitLab- CI supports a .pre-script configuration.
• https://dzone.com/articles/popular-tools-supporting-yaml-data-format
• https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet
• https://codebeautify.org/yaml-editor-online