ExternalMirrors
/
apt-cacher-formula
espelhamento de https://github.com/saltstack-formulas/apt-cacher-formula
Observar 1
Favorito 0
Fork 1
Código Issues 0 Versões 6 Wiki Atividade
Ver código fonte

Merge pull request #19 from n-rodriguez/wip/tofs 

Add TOFS pattern
tags/v0.7.0
Imran Iqbal 5 anos atrás
pai
7f0806443d d08ed97aaf
commit
c9dae39e64
Nenhuma conta vinculada ao e-mail do autor do commit
27 arquivos alterados com 995 adições e 122 exclusões
Visão dividida Mostrar estatísticas do Diff
  1. +0
    -14
      apt-cacher/ng/client.sls
  2. +27
    -0
      apt-cacher/ng/client/config/file.sls
  3. +5
    -0
      apt-cacher/ng/client/config/init.sls
  4. +5
    -0
      apt-cacher/ng/client/init.sls
  5. +5
    -1
      apt-cacher/ng/files/default/client.conf
  6. +4
    -4
      apt-cacher/ng/files/default/security.conf
  7. +5
    -1
      apt-cacher/ng/files/default/server.conf
  8. +112
    -0
      apt-cacher/ng/libtofs.jinja
  9. +48
    -11
      apt-cacher/ng/map.jinja
  10. +35
    -0
      apt-cacher/ng/osarchmap.yaml
  11. +9
    -2
      apt-cacher/ng/osfamilymap.yaml
  12. +13
    -0
      apt-cacher/ng/osfingermap.yaml
  13. +13
    -0
      apt-cacher/ng/osmap.yaml
  14. +0
    -88
      apt-cacher/ng/server.sls
  15. +75
    -0
      apt-cacher/ng/server/config/file.sls
  16. +5
    -0
      apt-cacher/ng/server/config/init.sls
  17. +7
    -0
      apt-cacher/ng/server/init.sls
  18. +5
    -0
      apt-cacher/ng/server/package/init.sls
  19. +10
    -0
      apt-cacher/ng/server/package/install.sls
  20. +5
    -0
      apt-cacher/ng/server/service/init.sls
  21. +33
    -0
      apt-cacher/ng/server/service/running.sls
  22. +15
    -0
      docs/README.rst
  23. +518
    -0
      docs/TOFS_pattern.rst
  24. +1
    -0
      kitchen.yml
  25. +31
    -1
      pillar.example
  26. +6
    -0
      test/integration/default/controls/config_spec.rb
  27. +3
    -0
      test/integration/default/controls/service_spec.rb

+ 0
- 14
apt-cacher/ng/client.sls Ver arquivo

@@ -1,14 +0,0 @@
{% if grains['os_family'] == 'Debian' %}
{% from "apt-cacher/ng/map.jinja" import apt_cacher_ng with context %}
{%- if apt_cacher_ng.server_address %}

{{ apt_cacher_ng.client_config }}:
file:
- managed
- user: root
- group: root
- mode: '644'
- source: salt://apt-cacher/ng/files/client.conf
- template: jinja
{%- endif %}
{% endif %}

+ 27
- 0
apt-cacher/ng/client/config/file.sls Ver arquivo

@@ -0,0 +1,27 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/ng/map.jinja" import apt_cacher_ng with context %}
{%- from tplroot ~ "/ng/libtofs.jinja" import files_switch with context %}

{%- if grains['os_family'] == 'Debian' %}
{%- if apt_cacher_ng.server_address %}

apt-cacher/ng/client/config/file:
file.managed:
- name: {{ apt_cacher_ng.client_config }}
- user: root
- group: root
- mode: '644'
- source: {{ files_switch(['client.conf'],
lookup='apt-cacher/ng/client/config/file',
use_subpath=True
)
}}
- template: jinja
- context:
apt_cacher_ng: {{ apt_cacher_ng | json }}
{%- endif %}
{%- endif %}

+ 5
- 0
apt-cacher/ng/client/config/init.sls Ver arquivo

@@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .file

+ 5
- 0
apt-cacher/ng/client/init.sls Ver arquivo

@@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .config

apt-cacher/ng/files/client.conf → apt-cacher/ng/files/default/client.conf Ver arquivo

@@ -1,4 +1,8 @@
{% from "apt-cacher/ng/map.jinja" import apt_cacher_ng with context -%}
########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################

Acquire::http::Proxy "http://{{ apt_cacher_ng.server_address }}:{{ apt_cacher_ng.server_port }}";
Acquire::https::Proxy "{{ apt_cacher_ng.https_proxy }}";
{% for host in apt_cacher_ng.local_mirrors -%}

apt-cacher/ng/files/security.conf → apt-cacher/ng/files/default/security.conf Ver arquivo

@@ -1,3 +1,7 @@
########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################

# This file contains confidential data and should be protected with file
# permissions from being read by untrusted users.
@@ -7,10 +11,6 @@

# Basic authentication with username and password, required to
# visit pages with administrative functionality. Format: username:password
{%- set cfg = salt['pillar.get']('apt_cacher_ng', {}) %}
{%- set admin_account = cfg.get('admin_account', False) %}
{%- set admin_passwd = cfg.get('admin_passwd', False) %}
{%- if admin_account and admin_passwd %}
AdminAuth: {{ admin_account }}:{{ admin_passwd }}
{%- endif %}


apt-cacher/ng/files/server.conf → apt-cacher/ng/files/default/server.conf Ver arquivo

@@ -1,4 +1,8 @@
{% from "apt-cacher/ng/map.jinja" import apt_cacher_ng with context %}
########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################

BindAddress: {{ apt_cacher_ng.server_bind_address }}
CacheDir: {{ apt_cacher_ng.server_cache_dir }}
LogDir: {{ apt_cacher_ng.server_log_dir }}

+ 112
- 0
apt-cacher/ng/libtofs.jinja Ver arquivo

@@ -0,0 +1,112 @@
{%- macro files_switch(source_files,
lookup=None,
default_files_switch=['id', 'os_family'],
indent_width=6,
use_subpath=False) %}
{#-
Returns a valid value for the "source" parameter of a "file.managed"
state function. This makes easier the usage of the Template Override and
Files Switch (TOFS) pattern.

Params:
* source_files: ordered list of files to look for
* lookup: key under '<tplroot>:tofs:source_files' to prepend to the
list of source files
* default_files_switch: if there's no config (e.g. pillar)
'<tplroot>:tofs:files_switch' this is the ordered list of grains to
use as selector switch of the directories under
"<path_prefix>/files"
* indent_witdh: indentation of the result value to conform to YAML
* use_subpath: defaults to `False` but if set, lookup the source file
recursively from the current state directory up to `tplroot`

Example (based on a `tplroot` of `xxx`):

If we have a state:

Deploy configuration:
file.managed:
- name: /etc/yyy/zzz.conf
- source: {{ files_switch(['/etc/yyy/zzz.conf', '/etc/yyy/zzz.conf.jinja'],
lookup='Deploy configuration'
) }}
- template: jinja

In a minion with id=theminion and os_family=RedHat, it's going to be
rendered as:

Deploy configuration:
file.managed:
- name: /etc/yyy/zzz.conf
- source:
- salt://xxx/files/theminion/etc/yyy/zzz.conf
- salt://xxx/files/theminion/etc/yyy/zzz.conf.jinja
- salt://xxx/files/RedHat/etc/yyy/zzz.conf
- salt://xxx/files/RedHat/etc/yyy/zzz.conf.jinja
- salt://xxx/files/default/etc/yyy/zzz.conf
- salt://xxx/files/default/etc/yyy/zzz.conf.jinja
- template: jinja
#}
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set path_prefix = salt['config.get'](tplroot ~ ':tofs:path_prefix', tplroot) %}
{%- set files_dir = salt['config.get'](tplroot ~ ':tofs:dirs:files', 'files') %}
{%- set files_switch_list = salt['config.get'](
tplroot ~ ':tofs:files_switch',
default_files_switch
) %}
{#- Lookup source_files (v2), files (v1), or fallback to an empty list #}
{%- set src_files = salt['config.get'](
tplroot ~ ':tofs:source_files:' ~ lookup,
salt['config.get'](tplroot ~ ':tofs:files:' ~ lookup, [])
) %}
{#- Append the default source_files #}
{%- set src_files = src_files + source_files %}
{#- Only add to [''] when supporting older TOFS implementations #}
{%- set path_prefix_exts = [''] %}
{%- if use_subpath and tplroot != tpldir %}
{#- Walk directory tree to find {{ files_dir }} #}
{%- set subpath_parts = tpldir.lstrip(tplroot).lstrip('/').split('/') %}
{%- for path in subpath_parts %}
{%- set subpath = subpath_parts[0:loop.index] | join('/') %}
{%- do path_prefix_exts.append('/' ~ subpath) %}
{%- endfor %}
{%- endif %}
{%- for path_prefix_ext in path_prefix_exts|reverse %}
{%- set path_prefix_inc_ext = path_prefix ~ path_prefix_ext %}
{#- For older TOFS implementation, use `files_switch` from the config #}
{#- Use the default, new method otherwise #}
{%- set fsl = salt['config.get'](
tplroot ~ path_prefix_ext|replace('/', ':') ~ ':files_switch',
files_switch_list
) %}
{#- Append an empty value to evaluate as `default` in the loop below #}
{%- if '' not in fsl %}
{%- do fsl.append('') %}
{%- endif %}
{%- for fs in fsl %}
{%- for src_file in src_files %}
{%- if fs %}
{%- set fs_dirs = salt['config.get'](fs, fs) %}
{%- else %}
{%- set fs_dirs = salt['config.get'](tplroot ~ ':tofs:dirs:default', 'default') %}
{%- endif %}
{#- Force the `config.get` lookup result as a list where necessary #}
{#- since we need to also handle grains that are lists #}
{%- if fs_dirs is string %}
{%- set fs_dirs = [fs_dirs] %}
{%- endif %}
{%- for fs_dir in fs_dirs %}
{%- set url = [
'- salt:/',
path_prefix_inc_ext.strip('/'),
files_dir.strip('/'),
fs_dir.strip('/'),
src_file.strip('/'),
] | select | join('/') %}
{{ url | indent(indent_width, true) }}
{%- endfor %}
{%- endfor %}
{%- endfor %}
{%- endfor %}
{%- endmacro %}

+ 48
- 11
apt-cacher/ng/map.jinja Ver arquivo

@@ -1,12 +1,49 @@
{% import_yaml "apt-cacher/ng/defaults.yaml" as defaults %}
{% import_yaml "apt-cacher/ng/osfamilymap.yaml" as osfamilymap %}

{%- set apt_cacher_ng = salt['grains.filter_by'](
defaults,
merge=salt['grains.filter_by'](
osfamilymap,
grain='os_family',
merge=salt['pillar.get']('apt_cacher_ng', {}),
),
base='apt_cacher_ng')
# -*- coding: utf-8 -*-
# vim: ft=jinja

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{#- Start imports as #}
{%- import_yaml tplroot ~ "/ng/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/ng/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/ng/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/ng/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/ng/osfingermap.yaml" as osfingermap %}

{#- Retrieve the config dict only once #}
{%- set _config = salt['config.get']('apt_cacher_ng', default={}) %}

{%- set defaults = salt['grains.filter_by'](
default_settings,
default='apt_cacher_ng',
merge=salt['grains.filter_by'](
osarchmap,
grain='osarch',
merge=salt['grains.filter_by'](
osfamilymap,
grain='os_family',
merge=salt['grains.filter_by'](
osmap,
grain='os',
merge=salt['grains.filter_by'](
osfingermap,
grain='osfinger',
merge=salt['grains.filter_by'](
_config,
default='lookup'
)
)
)
)
)
)
%}

{%- set config = salt['grains.filter_by'](
{'defaults': defaults},
default='defaults',
merge=_config
)
%}

{%- set apt_cacher_ng = config %}

+ 35
- 0
apt-cacher/ng/osarchmap.yaml Ver arquivo

@@ -0,0 +1,35 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['osarch'] based logic.
# You just need to add the key:values for an `osarch` that differ
# from `defaults.yaml`.
# Only add an `osarch` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `osarch` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osarch: {}
---
amd64:
arch: amd64

x86_64:
arch: amd64

386:
arch: 386

arm64:
arch: arm64

armv6l:
arch: armv6l

armv7l:
arch: armv7l

ppc64le:
arch: ppc64le

s390x:
arch: s390x

+ 9
- 2
apt-cacher/ng/osfamilymap.yaml Ver arquivo

@@ -1,8 +1,15 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['os_family'] based logic.
# You just need to add the key:values for an `os_family` that differ
# from `defaults.yaml` + `osarch.yaml`.
# Only add an `os_family` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os_family` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osfamilymap: {}
---
Debian: {}

FreeBSD:
credentials: /usr/local/etc/apt-cacher-ng/security.conf
root_group: wheel

+ 13
- 0
apt-cacher/ng/osfingermap.yaml Ver arquivo

@@ -0,0 +1,13 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['osfinger'] based logic.
# You just need to add the key:values for an `osfinger` that differ
# from `defaults.yaml` + `osarch.yaml` + `os_family.yaml` + `osmap.yaml`.
# Only add an `osfinger` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os_finger` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osfingermap: {}
---
osfingermap: {}

+ 13
- 0
apt-cacher/ng/osmap.yaml Ver arquivo

@@ -0,0 +1,13 @@
# -*- coding: utf-8 -*-
# vim: ft=yaml
#
# Setup variables using grains['os'] based logic.
# You just need to add the key:values for an `os` that differ
# from `defaults.yaml` + `osarch.yaml` + `os_family.yaml`.
# Only add an `os` which is/will be supported by the formula.
#
# If you do not need to provide defaults via the `os` grain,
# you will need to provide at least an empty dict in this file, e.g.
# osmap: {}
---
osmap: {}

+ 0
- 88
apt-cacher/ng/server.sls Ver arquivo

@@ -1,88 +0,0 @@
{% from "apt-cacher/ng/map.jinja" import apt_cacher_ng with context %}

{%- if 'include' in apt_cacher_ng %}
include:
{%- for include_line in apt_cacher_ng.include %}
- {{ include_line }}
{%- endfor %}
{%- endif %}

apt-cacher-ng-group:
group.present:
- name: {{ apt_cacher_ng.group }}
- require:
- pkg: apt-cacher-ng
- watch_in:
- service: apt-cacher-ng

apt-cacher-ng-user:
user.present:
- name: {{ apt_cacher_ng.user }}
- require:
- group: apt-cacher-ng-group
- watch_in:
- service: apt-cacher-ng

{%- if grains['os_family'] == 'FreeBSD' %}
apt-cacher-ng-sysrc-user:
sysrc.managed:
- name: apt_cacher_ng_user
- value: "{{ apt_cacher_ng.user }}"
- watch_in:
- service: apt-cacher-ng
{%- endif %}

apt-cacher-ng:
pkg.installed:
- name: {{ apt_cacher_ng.pkg }}
service.running:
- name: {{ apt_cacher_ng.service }}
- enable: True
- watch:
- pkg: {{ apt_cacher_ng.pkg }}
- file: {{ apt_cacher_ng.credentials }}
- file: {{ apt_cacher_ng.server_config }}
- file: {{ apt_cacher_ng.server_cache_dir }}
- file: {{ apt_cacher_ng.server_log_dir }}
{%- if 'require' in apt_cacher_ng %}
- require:
{%- for require in apt_cacher_ng.require %}
- {{ require }}
{%- endfor %}
{%- endif %}
{%- if 'require_in' in apt_cacher_ng %}
- require_in:
{%- for require_in in apt_cacher_ng.require_in %}
- {{ require_in }}
{%- endfor %}
{%- endif %}

{{ apt_cacher_ng.server_config }}:
file.managed:
- user: root
- group: {{ apt_cacher_ng.root_group }}
- mode: '644'
- source: salt://apt-cacher/ng/files/server.conf
- template: jinja

{{ apt_cacher_ng.server_cache_dir }}:
file.directory:
- makedirs: True
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '2755'

{{ apt_cacher_ng.server_log_dir }}:
file.directory:
- makedirs: True
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '2755'

{{ apt_cacher_ng.credentials }}:
file.managed:
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '600'
- source: salt://apt-cacher/ng/files/security.conf
- template: jinja

+ 75
- 0
apt-cacher/ng/server/config/file.sls Ver arquivo

@@ -0,0 +1,75 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_package_install = tplroot ~ '.ng.server.package.install' %}
{%- from tplroot ~ "/ng/map.jinja" import apt_cacher_ng with context %}
{%- from tplroot ~ "/ng/libtofs.jinja" import files_switch with context %}

include:
- {{ sls_package_install }}

apt-cacher/ng/server/config/group/create:
group.present:
- name: {{ apt_cacher_ng.group }}

apt-cacher/ng/server/config/user/create:
user.present:
- name: {{ apt_cacher_ng.user }}

{%- if grains['os_family'] == 'FreeBSD' %}
apt-cacher/ng/server/config/user/sysrc:
sysrc.managed:
- name: apt_cacher_ng_user
- value: "{{ apt_cacher_ng.user }}"
{%- endif %}

apt-cacher/ng/server/config/file:
file.managed:
- name: {{ apt_cacher_ng.server_config }}
- user: root
- group: {{ apt_cacher_ng.root_group }}
- mode: '644'
- source: {{ files_switch(['server.conf'],
lookup='apt-cacher/ng/server/config/file',
use_subpath=True
)
}}
- template: jinja
- require:
- sls: {{ sls_package_install }}
- context:
apt_cacher_ng: {{ apt_cacher_ng | json }}

apt-cacher/ng/server/config/server_cache_dir:
file.directory:
- name: {{ apt_cacher_ng.server_cache_dir }}
- makedirs: true
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '2755'

apt-cacher/ng/server/config/server_log_dir:
file.directory:
- name: {{ apt_cacher_ng.server_log_dir }}
- makedirs: true
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '2755'

apt-cacher/ng/server/config/credentials:
file.managed:
- name: {{ apt_cacher_ng.credentials }}
- user: {{ apt_cacher_ng.user }}
- group: {{ apt_cacher_ng.group }}
- mode: '600'
- source: {{ files_switch(['security.conf'],
lookup='apt-cacher/ng/server/config/credentials',
use_subpath=True
)
}}
- template: jinja
- context:
admin_account: {{ apt_cacher_ng.get('admin_account', False) }}
admin_passwd: {{ apt_cacher_ng.get('admin_passwd', False) }}

+ 5
- 0
apt-cacher/ng/server/config/init.sls Ver arquivo

@@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .file

+ 7
- 0
apt-cacher/ng/server/init.sls Ver arquivo

@@ -0,0 +1,7 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .package
- .config
- .service

+ 5
- 0
apt-cacher/ng/server/package/init.sls Ver arquivo

@@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .install

+ 10
- 0
apt-cacher/ng/server/package/install.sls Ver arquivo

@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/ng/map.jinja" import apt_cacher_ng with context %}

apt-cacher/ng/server/package/install:
pkg.installed:
- name: {{ apt_cacher_ng.pkg }}

+ 5
- 0
apt-cacher/ng/server/service/init.sls Ver arquivo

@@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
- .running

+ 33
- 0
apt-cacher/ng/server/service/running.sls Ver arquivo

@@ -0,0 +1,33 @@
# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_config_file = tplroot ~ '.ng.server.config.file' %}
{%- from tplroot ~ "/ng/map.jinja" import apt_cacher_ng with context %}

include:
- {{ sls_config_file }}
{%- if 'include' in apt_cacher_ng %}
{%- for include_line in apt_cacher_ng.include %}
- {{ include_line }}
{%- endfor %}
{%- endif %}

apt-cacher/ng/server/service/running:
service.running:
- name: {{ apt_cacher_ng.service }}
- enable: true
- require:
- sls: {{ sls_config_file }}
{%- if 'require' in apt_cacher_ng %}
{%- for require in apt_cacher_ng.require %}
- {{ require }}
{%- endfor %}
{%- endif %}
{%- if 'require_in' in apt_cacher_ng %}
- require_in:
{%- for require_in in apt_cacher_ng.require_in %}
- {{ require_in }}
{%- endfor %}
{%- endif %}

+ 15
- 0
docs/README.rst Ver arquivo

@@ -63,6 +63,21 @@ Install and configure apt-cacher-ng.

Supports Debian(ish) distributions and FreeBSD.

``apt-cacher.ng.server.package``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Installs the apt-cacher-ng package.

``apt-cacher.ng.server.config``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Installs the apt-cacher-ng config.

``apt-cacher.ng.server.service``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Manages the startup and running state of the apt-cacher-ng service.

``apt-cacher.ng.client``
^^^^^^^^^^^^^^^^^^^^^^^^


+ 518
- 0
docs/TOFS_pattern.rst Ver arquivo

@@ -0,0 +1,518 @@
.. _tofs_pattern:

TOFS: A pattern for using SaltStack
===================================

.. list-table::
:name: tofs-authors
:header-rows: 1
:stub-columns: 1
:widths: 2,2,3,2

* -
- Person
- Contact
- Date
* - Authored by
- Roberto Moreda
- moreda@allenta.com
- 29/12/2014
* - Modified by
- Daniel Dehennin
- daniel.dehennin@baby-gnu.org
- 07/02/2019
* - Modified by
- Imran Iqbal
- https://github.com/myii
- 23/02/2019

All that follows is a proposal based on my experience with `SaltStack <http://www.saltstack.com/>`_. The good thing of a piece of software like this is that you can "bend it" to suit your needs in many possible ways, and this is one of them. All the recommendations and thoughts are given "as it is" with no warranty of any type.

.. contents:: **Table of Contents**

Usage of values in pillar vs templates in ``file_roots``
--------------------------------------------------------

Among other functions, the *master* (or *salt-master*) serves files to the *minions* (or *salt-minions*). The `file_roots <http://docs.saltstack.com/en/latest/ref/file_server/file_roots.html>`_ is the list of directories used in sequence to find a file when a minion requires it: the first match is served to the minion. Those files could be `state files <http://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html>`_ or configuration templates, among others.

Using SaltStack is a simple and effective way to implement configuration management, but even in a `non-multitenant <http://en.wikipedia.org/wiki/Multitenancy>`_ scenario, it is not a good idea to generally access some data (e.g. the database password in our `Zabbix <http://www.zabbix.com/>`_ server configuration file or the private key of our `Nginx <http://nginx.org/en/>`_ TLS certificate).

To avoid this situation we can use the `pillar mechanism <http://docs.saltstack.com/en/latest/topics/pillar/>`_, which is designed to provide controlled access to data from the minions based on some selection rules. As pillar data could be easily integrated in the `Jinja <http://docs.saltstack.com/en/latest/topics/tutorials/pillar.html>`_ templates, it is a good mechanism to store values to be used in the final rendering of state files and templates.

There are a variety of approaches on the usage of pillar and templates as seen in the `saltstack-formulas <https://github.com/saltstack-formulas>`_' repositories. `Some <https://github.com/saltstack-formulas/nginx-formula/pull/18>`_ `developments <https://github.com/saltstack-formulas/php-formula/pull/14>`_ stress the initial purpose of pillar data into a storage for most of the possible variables for a determined system configuration. This, in my opinion, is shifting too much load from the original template files approach. Adding up some `non-trivial Jinja <https://github.com/saltstack-formulas/nginx-formula/blob/f74254c07e188bd448eaf1c5f9c802d78c4c005e/nginx/files/default/nginx.conf>`_ code as essential part of composing the state file definitely makes SaltStack state files (hence formulas) more difficult to read. The extreme of this approach is that we could end up with a new render mechanism, implemented in Jinja, storing everything needed in pillar data to compose configurations. Additionally, we are establishing a strong dependency with the Jinja renderer.

In opposition to the *put the code in file_roots and the data in pillars* approach, there is the *pillar as a store for a set of key-values* approach. A full-blown configuration file abstracted in pillar and jinja is complicated to develop, understand and maintain. I think a better and simpler approach is to keep a configuration file templated using just a basic (non-extensive but extensible) set of pillar values.

On the reusability of SaltStack state files
-------------------------------------------

There is a brilliant initiative of the SaltStack community called `salt-formulas <https://github.com/saltstack-formulas>`_. Their goal is to provide state files, pillar examples and configuration templates ready to be used for provisioning. I am a contributor for two small ones: `zabbix-formula <https://github.com/saltstack-formulas/zabbix-formula>`_ and `varnish-formula <https://github.com/saltstack-formulas/varnish-formula>`_.

The `design guidelines <http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_ for formulas are clear in many aspects and it is a recommended reading for anyone willing to write state files, even non-formulaic ones.

In the next section, I am going to describe my proposal to extend further the reusability of formulas, suggesting some patterns of usage.

The Template Override and Files Switch (TOFS) pattern
-----------------------------------------------------

I understand a formula as a **complete, independent set of SaltStack state and configuration template files sufficient to configure a system**. A system could be something as simple as an NTP server or some other much more complex service that requires many state and configuration template files.

The customization of a formula should be done mainly by providing pillar data used later to render either the state or the configuration template files.

Example: NTP before applying TOFS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Let's work with the NTP example. A basic formula that follows the `design guidelines <http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_ has the following files and directories tree:

.. code-block::

/srv/saltstack/salt-formulas/ntp-saltstack-formula/
ntp/
map.jinja
init.sls
conf.sls
files/
default/
etc/
ntp.conf.jinja

In order to use it, let's assume a `masterless configuration <http://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html>`_ and this relevant section of ``/etc/salt/minion``:

.. code-block:: yaml

pillar_roots:
base:
- /srv/saltstack/pillar
file_client: local
file_roots:
base:
- /srv/saltstack/salt
- /srv/saltstack/salt-formulas/ntp-saltstack-formula

.. code-block:: jinja

{#- /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/map.jinja #}
{%- set ntp = salt['grains.filter_by']({
'default': {
'pkg': 'ntp',
'service': 'ntp',
'config': '/etc/ntp.conf',
},
}, merge=salt['pillar.get']('ntp:lookup')) %}

In ``init.sls`` we have the minimal states required to have NTP configured. In many cases ``init.sls`` is almost equivalent to an ``apt-get install`` or a ``yum install`` of the package.

.. code-block:: sls

## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/init.sls
{%- from 'ntp/map.jinja' import ntp with context %}

Install NTP:
pkg.installed:
- name: {{ ntp.pkg }}

Enable and start NTP:
service.running:
- name: {{ ntp.service }}
- enabled: True
- require:
- pkg: Install NTP package

In ``conf.sls`` we have the configuration states. In most cases, that is just managing configuration file templates and making them to be watched by the service.

.. code-block:: sls

## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
include:
- ntp

{%- from 'ntp/map.jinja' import ntp with context %}

Configure NTP:
file.managed:
- name: {{ ntp.config }}
- template: jinja
- source: salt://ntp/files/default/etc/ntp.conf.jinja
- watch_in:
- service: Enable and start NTP service
- require:
- pkg: Install NTP package

Under ``files/default``, there is a structure that mimics the one in the minion in order to avoid clashes and confusion on where to put the needed templates. There you can find a mostly standard template for the configuration file.

.. code-block:: jinja

{#- /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/default/etc/ntp.conf.jinja #}
{#- Managed by saltstack #}
{#- Edit pillars or override this template in saltstack if you need customization #}
{%- set settings = salt['pillar.get']('ntp', {}) %}
{%- set default_servers = ['0.ubuntu.pool.ntp.org',
'1.ubuntu.pool.ntp.org',
'2.ubuntu.pool.ntp.org',
'3.ubuntu.pool.ntp.org'] %}

driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

{%- for server in settings.get('servers', default_servers) %}
server {{ server }}
{%- endfor %}

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

restrict 127.0.0.1
restrict ::1

With all this, it is easy to install and configure a simple NTP server by just running ``salt-call state.sls ntp.conf``: the package will be installed, the service will be running and the configuration should be correct for most of cases, even without pillar data.

Alternatively, you can define a highstate in ``/srv/saltstack/salt/top.sls`` and run ``salt-call state.highstate``.

.. code-block:: sls

## /srv/saltstack/salt/top.sls
base:
'*':
- ntp.conf

**Customizing the formula just with pillar data**, we have the option to define the NTP servers.

.. code-block:: sls

## /srv/saltstack/pillar/top.sls
base:
'*':
- ntp

.. code-block:: sls

## /srv/saltstack/pillar/ntp.sls
ntp:
servers:
- 0.ch.pool.ntp.org
- 1.ch.pool.ntp.org
- 2.ch.pool.ntp.org
- 3.ch.pool.ntp.org

Template Override
^^^^^^^^^^^^^^^^^

If the customization based on pillar data is not enough, we can override the template by creating a new one in ``/srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja``

.. code-block:: jinja

{#- /srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja #}
{#- Managed by saltstack #}
{#- Edit pillars or override this template in saltstack if you need customization #}

{#- Some bizarre configurations here #}
{#- ... #}

{%- for server in settings.get('servers', default_servers) %}
server {{ server }}
{%- endfor %}

This way we are locally **overriding the template files** offered by the formula in order to make a more complex adaptation. Of course, this could be applied as well to any of the files, including the state files.

Files Switch
^^^^^^^^^^^^

To bring some order into the set of template files included in a formula, as we commented, we suggest having a similar structure to a normal final file system under ``files/default``.

We can make different templates coexist for different minions, classified by any `grain <http://docs.saltstack.com/en/latest/topics/targeting/grains.html>`_ value, by simply creating new directories under ``files``. This mechanism is based on **using values of some grains as a switch for the directories under** ``files/``.

If we decide that we want ``os_family`` as switch, then we could provide the formula template variants for both the ``RedHat`` and ``Debian`` families.

.. code-block::

/srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/
default/
etc/
ntp.conf.jinja
RedHat/
etc/
ntp.conf.jinja
Debian/
etc/
ntp.conf.jinja

To make this work we need a ``conf.sls`` state file that takes a list of possible files as the configuration template.

.. code-block:: sls

## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
include:
- ntp

{%- from 'ntp/map.jinja' import ntp with context %}

Configure NTP:
file.managed:
- name: {{ ntp.config }}
- template: jinja
- source:
- salt://ntp/files/{{ grains.get('os_family', 'default') }}/etc/ntp.conf.jinja
- salt://ntp/files/default/etc/ntp.conf.jinja
- watch_in:
- service: Enable and start NTP service
- require:
- pkg: Install NTP package

If we want to cover the possibility of a special template for a minion identified by ``node01`` then we could have a specific template in ``/srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja``.

.. code-block:: jinja

{#- /srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja #}
{#- Managed by saltstack #}
{#- Edit pillars or override this template in saltstack if you need customization #}

{#- Some crazy configurations here for node01 #}
{#- ... #}

To make this work we could write a specially crafted ``conf.sls``.

.. code-block:: sls

## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
include:
- ntp

{%- from 'ntp/map.jinja' import ntp with context %}

Configure NTP:
file.managed:
- name: {{ ntp.config }}
- template: jinja
- source:
- salt://ntp/files/{{ grains.get('id') }}/etc/ntp.conf.jinja
- salt://ntp/files/{{ grains.get('os_family') }}/etc/ntp.conf.jinja
- salt://ntp/files/default/etc/ntp.conf.jinja
- watch_in:
- service: Enable and start NTP service
- require:
- pkg: Install NTP package

Using the ``files_switch`` macro
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We can simplify the ``conf.sls`` with the new ``files_switch`` macro to use in the ``source`` parameter for the ``file.managed`` state.

.. code-block:: sls

## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls
include:
- ntp

{%- set tplroot = tpldir.split('/')[0] %}
{%- from 'ntp/map.jinja' import ntp with context %}
{%- from 'ntp/libtofs.jinja' import files_switch %}

Configure NTP:
file.managed:
- name: {{ ntp.config }}
- template: jinja
- source: {{ files_switch(['/etc/ntp.conf.jinja'],
lookup='Configure NTP'
)
}}
- watch_in:
- service: Enable and start NTP service
- require:
- pkg: Install NTP package


* This uses ``config.get``, searching for ``ntp:tofs:source_files:Configure NTP`` to determine the list of template files to use.
* If this returns a result, the default of ``['/etc/ntp.conf.jinja']`` will be appended to it.
* If this does not yield any results, the default of ``['/etc/ntp.conf.jinja']`` will be used.

In ``libtofs.jinja``, we define this new macro ``files_switch``.

.. literalinclude:: ../template/libtofs.jinja
:caption: /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/libtofs.jinja
:language: jinja

How to customise the ``source`` further
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The examples below are based on an ``Ubuntu`` minion called ``theminion`` being configured via. pillar.

Using the default settings of the ``files_switch`` macro above,
the ``source`` will be:

.. code-block:: sls

- source:
- salt://ntp/files/theminion/etc/ntp.conf.jinja
- salt://ntp/files/Debian/etc/ntp.conf.jinja
- salt://ntp/files/default/etc/ntp.conf.jinja

Customise ``files``
~~~~~~~~~~~~~~~~~~~

The ``files`` portion can be customised:

.. code-block:: sls

ntp:
tofs:
dirs:
files: files_alt

Resulting in:

.. code-block:: sls

- source:
- salt://ntp/files_alt/theminion/etc/ntp.conf.jinja
- salt://ntp/files_alt/Debian/etc/ntp.conf.jinja
- salt://ntp/files_alt/default/etc/ntp.conf.jinja

Customise the use of grains
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Grains can be customised and even arbitrary paths can be supplied:

.. code-block:: sls

ntp:
tofs:
files_switch:
- any/path/can/be/used/here
- id
- os
- os_family

Resulting in:

.. code-block:: sls

- source:
- salt://ntp/files/any/path/can/be/used/here/etc/ntp.conf.jinja
- salt://ntp/files/theminion/etc/ntp.conf.jinja
- salt://ntp/files/Ubuntu/etc/ntp.conf.jinja
- salt://ntp/files/Debian/etc/ntp.conf.jinja
- salt://ntp/files/default/etc/ntp.conf.jinja

Customise the ``default`` path
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``default`` portion of the path can be customised:

.. code-block:: sls

ntp:
tofs:
dirs:
default: default_alt

Resulting in:

.. code-block:: sls

- source:
...
- salt://ntp/files/default_alt/etc/ntp.conf.jinja

Customise the list of ``source_files``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The list of ``source_files`` can be given:

.. code-block:: sls

ntp:
tofs:
source_files:
Configure NTP:
- '/etc/ntp.conf_alt.jinja'

Resulting in:

.. code-block:: sls

- source:
- salt://ntp/files/theminion/etc/ntp.conf_alt.jinja
- salt://ntp/files/theminion/etc/ntp.conf.jinja
- salt://ntp/files/Debian/etc/ntp.conf_alt.jinja
- salt://ntp/files/Debian/etc/ntp.conf.jinja
- salt://ntp/files/default/etc/ntp.conf_alt.jinja
- salt://ntp/files/default/etc/ntp.conf.jinja

Note: This does *not* override the default value.
Rather, the value from the pillar/config is prepended to the default.

Using sub-directories for ``components``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If your formula is composed of several components, you may prefer to provides files under sub-directories, like in the `systemd-formula <https://github.com/saltstack-formulas/systemd-formula>`_.

.. code-block::

/srv/saltstack/systemd-formula/
systemd/
init.sls
libtofs.jinja
map.jinja
networkd/
init.sls
files/
default/
network/
99-default.link
resolved/
init.sls
files/
default/
resolved.conf
timesyncd/
init.sls
files/
Arch/
resolved.conf
Debian/
resolved.conf
default/
resolved.conf
Ubuntu/
resolved.conf

For example, the following ``formula.component.config`` SLS:

.. code-block:: sls

{%- from "formula/libtofs.jinja" import files_switch with context %}

formula configuration file:
file.managed:
- name: /etc/formula.conf
- user: root
- group: root
- mode: 644
- template: jinja
- source: {{ files_switch(['formula.conf'],
lookup='formula',
use_subpath=True
)
}}

will be rendered on a ``Debian`` minion named ``salt-formula.ci.local`` as:

.. code-block:: sls

formula configuration file:
file.managed:
- name: /etc/formula.conf
- user: root
- group: root
- mode: 644
- template: jinja
- source:
- salt://formula/component/files/salt-formula.ci.local/formula.conf
- salt://formula/component/files/Debian/formula.conf
- salt://formula/component/files/default/formula.conf
- salt://formula/files/salt-formula.ci.local/formula.conf
- salt://formula/files/Debian/formula.conf
- salt://formula/files/default/formula.conf

+ 1
- 0
kitchen.yml Ver arquivo

@@ -199,6 +199,7 @@ suites:
base:
'*':
- apt-cacher.ng.server
- apt-cacher.ng.client
pillars:
top.sls:
base:

+ 31
- 1
pillar.example Ver arquivo

@@ -45,7 +45,6 @@ apt_cacher_ng:
- 192.168.0.1
- host.example.test

# yamllint disable-line rule:comments-indentation
# Example for require/require_in/include
# See: https://github.com/saltstack-formulas/apt-cacher-formula/pull/12 for details
# apt_cacher_ng:
@@ -62,3 +61,34 @@ apt_cacher_ng:
# # custom states: i.e. Debian package repos
# - 'pkgrepo: deb jessie-backports'
# - 'pkgrepo: deb-src jessie-backports'

tofs:
# The files_switch key serves as a selector for alternative
# directories under the formula files directory. See TOFS pattern
# doc for more info.
# Note: Any value not evaluated by `config.get` will be used literally.
# This can be used to set custom paths, as many levels deep as required.
# files_switch:
# - any/path/can/be/used/here
# - id
# - roles
# - osfinger
# - os
# - os_family
# All aspects of path/file resolution are customisable using the options below.
# This is unnecessary in most cases; there are sensible defaults.
# Default path: salt://< path_prefix >/< dirs.files >/< dirs.default >
# I.e.: salt://template/files/default
# path_prefix: template_alt
# dirs:
# files: files_alt
# default: default_alt
# The entries under `source_files` are prepended to the default source files
# given for the state
source_files:
apt-cacher/ng/client/config/file:
- 'alt_client.conf'
apt-cacher/ng/server/config/file:
- 'alt_server.conf'
apt-cacher/ng/server/config/credentials:
- 'alt_security.conf'

+ 6
- 0
test/integration/default/controls/config_spec.rb Ver arquivo

@@ -8,4 +8,10 @@ control 'AptCacherNG configuration' do
# Custom config
its('content') { should include "Port: 9999" }
end

describe file('/etc/apt/apt.conf.d/80proxy') do
# Default config
its('content') { should include 'Acquire::http::Proxy "http://localhost:9999";' }
its('content') { should include 'Acquire::https::Proxy "DIRECT";' }
end
end

+ 3
- 0
test/integration/default/controls/service_spec.rb Ver arquivo

@@ -8,5 +8,8 @@ control 'AptCacherNG service' do

describe port(9999) do
it { should be_listening }
its('processes') { should include 'apt-cacher-ng' }
its('protocols') { should include 'tcp' }
its('addresses') { should include '0.0.0.0' }
end
end

Escrever Pré-visualização
Carregando…
Cancelar
Salvar