ExternalMirrors
/
syslog-ng-formula
kopia lustrzana https://github.com/saltstack-formulas/syslog-ng-formula
Obserwuj 1
Polub 0
Forkuj 1
Kod Zgłoszenia 0 Wydania 7 Wiki Aktywność
Saltstack Official Syslog-NG Formula
Nie możesz wybrać więcej, niż 25 tematów Tematy muszą się zaczynać od litery lub cyfry, mogą zawierać myślniki ('-') i mogą mieć do 35 znaków.
239 Commity
3 Gałęzie
Gałąź: master
Gałęzie Tagi
${ item.name }
Utwórz gałąź ${ searchTerm }
z 'master'
${ noResults }
syslog-ng-formula/docs/TOFS_pattern.rst

519 lines
19KB
Czysty Bezpośredni odnośnik Wina Historia 

  1. .. _tofs_pattern:

  2. 

  3. TOFS: A pattern for using SaltStack

  4. ===================================

  5. 

  6. .. list-table::

  7.    :name: tofs-authors

  8.    :header-rows: 1

  9.    :stub-columns: 1

  10.    :widths: 2,2,3,2

  11. 

  12.    * -

  13.      - Person

  14.      - Contact

  15.      - Date

  16.    * - Authored by

  17.      - Roberto Moreda

  18.      - moreda@allenta.com

  19.      - 29/12/2014

  20.    * - Modified by

  21.      - Daniel Dehennin

  22.      - daniel.dehennin@baby-gnu.org

  23.      - 07/02/2019

  24.    * - Modified by

  25.      - Imran Iqbal

  26.      - https://github.com/myii

  27.      - 23/02/2019

  28. 

  29. All that follows is a proposal based on my experience with `SaltStack <http://www.saltstack.com/>`_. The good thing of a piece of software like this is that you can "bend it" to suit your needs in many possible ways, and this is one of them. All the recommendations and thoughts are given "as it is" with no warranty of any type.

  30. 

  31. .. contents:: **Table of Contents**

  32. 

  33. Usage of values in pillar vs templates in ``file_roots``

  34. --------------------------------------------------------

  35. 

  36. Among other functions, the *master* (or *salt-master*) serves files to the *minions* (or *salt-minions*). The `file_roots <http://docs.saltstack.com/en/latest/ref/file_server/file_roots.html>`_ is the list of directories used in sequence to find a file when a minion requires it: the first match is served to the minion. Those files could be `state files <http://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html>`_ or configuration templates, among others.

  37. 

  38. Using SaltStack is a simple and effective way to implement configuration management, but even in a `non-multitenant <http://en.wikipedia.org/wiki/Multitenancy>`_ scenario, it is not a good idea to generally access some data (e.g. the database password in our `Zabbix <http://www.zabbix.com/>`_ server configuration file or the private key of our `Nginx <http://nginx.org/en/>`_ TLS certificate).

  39. 

  40. To avoid this situation we can use the `pillar mechanism <http://docs.saltstack.com/en/latest/topics/pillar/>`_, which is designed to provide controlled access to data from the minions based on some selection rules. As pillar data could be easily integrated in the `Jinja <http://docs.saltstack.com/en/latest/topics/tutorials/pillar.html>`_ templates, it is a good mechanism to store values to be used in the final rendering of state files and templates.

  41. 

  42. There are a variety of approaches on the usage of pillar and templates as seen in the `saltstack-formulas <https://github.com/saltstack-formulas>`_' repositories. `Some <https://github.com/saltstack-formulas/nginx-formula/pull/18>`_ `developments <https://github.com/saltstack-formulas/php-formula/pull/14>`_ stress the initial purpose of pillar data into a storage for most of the possible variables for a determined system configuration. This, in my opinion, is shifting too much load from the original template files approach. Adding up some `non-trivial Jinja <https://github.com/saltstack-formulas/nginx-formula/blob/f74254c07e188bd448eaf1c5f9c802d78c4c005e/nginx/files/default/nginx.conf>`_ code as essential part of composing the state file definitely makes SaltStack state files (hence formulas) more difficult to read. The extreme of this approach is that we could end up with a new render mechanism, implemented in Jinja, storing everything needed in pillar data to compose configurations. Additionally, we are establishing a strong dependency with the Jinja renderer.

  43. 

  44. In opposition to the *put the code in file_roots and the data in pillars* approach, there is the *pillar as a store for a set of key-values* approach. A full-blown configuration file abstracted in pillar and jinja is complicated to develop, understand and maintain. I think a better and simpler approach is to keep a configuration file templated using just a basic (non-extensive but extensible) set of pillar values.

  45. 

  46. On the reusability of SaltStack state files

  47. -------------------------------------------

  48. 

  49. There is a brilliant initiative of the SaltStack community called `salt-formulas <https://github.com/saltstack-formulas>`_. Their goal is to provide state files, pillar examples and configuration templates ready to be used for provisioning. I am a contributor for two small ones: `zabbix-formula <https://github.com/saltstack-formulas/zabbix-formula>`_ and `varnish-formula <https://github.com/saltstack-formulas/varnish-formula>`_.

  50. 

  51. The `design guidelines <http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_ for formulas are clear in many aspects and it is a recommended reading for anyone willing to write state files, even non-formulaic ones.

  52. 

  53. In the next section, I am going to describe my proposal to extend further the reusability of formulas, suggesting some patterns of usage.

  54. 

  55. The Template Override and Files Switch (TOFS) pattern

  56. -----------------------------------------------------

  57. 

  58. I understand a formula as a **complete, independent set of SaltStack state and configuration template files sufficient to configure a system**. A system could be something as simple as an NTP server or some other much more complex service that requires many state and configuration template files.

  59. 

  60. The customization of a formula should be done mainly by providing pillar data used later to render either the state or the configuration template files.

  61. 

  62. Example: NTP before applying TOFS

  63. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  64. 

  65. Let's work with the NTP example. A basic formula that follows the `design guidelines <http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_ has the following files and directories tree:

  66. 

  67. .. code-block:: console

  68. 

  69.    /srv/saltstack/salt-formulas/ntp-saltstack-formula/

  70.      ntp/

  71.        map.jinja

  72.        init.sls

  73.        conf.sls

  74.        files/

  75.          default/

  76.            etc/

  77.              ntp.conf.jinja

  78. 

  79. In order to use it, let's assume a `masterless configuration <http://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html>`_ and this relevant section of ``/etc/salt/minion``:

  80. 

  81. .. code-block:: yaml

  82. 

  83.    pillar_roots:

  84.      base:

  85.        - /srv/saltstack/pillar

  86.    file_client: local

  87.    file_roots:

  88.      base:

  89.        - /srv/saltstack/salt

  90.        - /srv/saltstack/salt-formulas/ntp-saltstack-formula

  91. 

  92. .. code-block:: jinja

  93. 

  94.    {#- /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/map.jinja #}

  95.    {%- set ntp = salt['grains.filter_by']({

  96.      'default': {

  97.        'pkg': 'ntp',

  98.        'service': 'ntp',

  99.        'config': '/etc/ntp.conf',

  100.      },

  101.    }, merge=salt['pillar.get']('ntp:lookup')) %}

  102. 

  103. In ``init.sls`` we have the minimal states required to have NTP configured. In many cases ``init.sls`` is almost equivalent to an ``apt-get install`` or a ``yum install`` of the package.

  104. 

  105. .. code-block:: sls

  106. 

  107.    ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/init.sls

  108.    {%- from 'ntp/map.jinja' import ntp with context %}

  109. 

  110.    Install NTP:

  111.      pkg.installed:

  112.        - name: {{ ntp.pkg }}

  113. 

  114.    Enable and start NTP:

  115.      service.running:

  116.        - name: {{ ntp.service }}

  117.        - enabled: True

  118.        - require:

  119.          - pkg: Install NTP package

  120. 

  121. In ``conf.sls`` we have the configuration states. In most cases, that is just managing configuration file templates and making them to be watched by the service.

  122. 

  123. .. code-block:: sls

  124. 

  125.    ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls

  126.    include:

  127.      - ntp

  128. 

  129.    {%- from 'ntp/map.jinja' import ntp with context %}

  130. 

  131.    Configure NTP:

  132.      file.managed:

  133.        - name: {{ ntp.config }}

  134.        - template: jinja

  135.        - source: salt://ntp/files/default/etc/ntp.conf.jinja

  136.        - watch_in:

  137.          - service: Enable and start NTP service

  138.        - require:

  139.          - pkg: Install NTP package

  140. 

  141. Under ``files/default``, there is a structure that mimics the one in the minion in order to avoid clashes and confusion on where to put the needed templates. There you can find a mostly standard template for the configuration file.

  142. 

  143. .. code-block:: jinja

  144. 

  145.    {#- /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/default/etc/ntp.conf.jinja #}

  146.    {#- Managed by saltstack #}

  147.    {#- Edit pillars or override this template in saltstack if you need customization #}

  148.    {%- set settings = salt['pillar.get']('ntp', {}) %}

  149.    {%- set default_servers = ['0.ubuntu.pool.ntp.org',

  150.                              '1.ubuntu.pool.ntp.org',

  151.                              '2.ubuntu.pool.ntp.org',

  152.                              '3.ubuntu.pool.ntp.org'] %}

  153. 

  154.    driftfile /var/lib/ntp/ntp.drift

  155.    statistics loopstats peerstats clockstats

  156.    filegen loopstats file loopstats type day enable

  157.    filegen peerstats file peerstats type day enable

  158.    filegen clockstats file clockstats type day enable

  159. 

  160.    {%- for server in settings.get('servers', default_servers) %}

  161.    server {{ server }}

  162.    {%- endfor %}

  163. 

  164.    restrict -4 default kod notrap nomodify nopeer noquery

  165.    restrict -6 default kod notrap nomodify nopeer noquery

  166. 

  167.    restrict 127.0.0.1

  168.    restrict ::1

  169. 

  170. With all this, it is easy to install and configure a simple NTP server by just running ``salt-call state.sls ntp.conf``: the package will be installed, the service will be running and the configuration should be correct for most of cases, even without pillar data.

  171. 

  172. Alternatively, you can define a highstate in ``/srv/saltstack/salt/top.sls`` and run ``salt-call state.highstate``.

  173. 

  174. .. code-block:: sls

  175. 

  176.    ## /srv/saltstack/salt/top.sls

  177.    base:

  178.      '*':

  179.        - ntp.conf

  180. 

  181. **Customizing the formula just with pillar data**, we have the option to define the NTP servers.

  182. 

  183. .. code-block:: sls

  184. 

  185.    ## /srv/saltstack/pillar/top.sls

  186.    base:

  187.      '*':

  188.        - ntp

  189. 

  190. .. code-block:: sls

  191. 

  192.    ## /srv/saltstack/pillar/ntp.sls

  193.    ntp:

  194.      servers:

  195.        - 0.ch.pool.ntp.org

  196.        - 1.ch.pool.ntp.org

  197.        - 2.ch.pool.ntp.org

  198.        - 3.ch.pool.ntp.org

  199. 

  200. Template Override

  201. ^^^^^^^^^^^^^^^^^

  202. 

  203. If the customization based on pillar data is not enough, we can override the template by creating a new one in ``/srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja``

  204. 

  205. .. code-block:: jinja

  206. 

  207.    {#- /srv/saltstack/salt/ntp/files/default/etc/ntp.conf.jinja #}

  208.    {#- Managed by saltstack #}

  209.    {#- Edit pillars or override this template in saltstack if you need customization #}

  210. 

  211.    {#- Some bizarre configurations here #}

  212.    {#- ... #}

  213. 

  214.    {%- for server in settings.get('servers', default_servers) %}

  215.    server {{ server }}

  216.    {%- endfor %}

  217. 

  218. This way we are locally **overriding the template files** offered by the formula in order to make a more complex adaptation. Of course, this could be applied as well to any of the files, including the state files.

  219. 

  220. Files Switch

  221. ^^^^^^^^^^^^

  222. 

  223. To bring some order into the set of template files included in a formula, as we commented, we suggest having a similar structure to a normal final file system under ``files/default``.

  224. 

  225. We can make different templates coexist for different minions, classified by any `grain <http://docs.saltstack.com/en/latest/topics/targeting/grains.html>`_ value, by simply creating new directories under ``files``. This mechanism is based on **using values of some grains as a switch for the directories under** ``files/``.

  226. 

  227. If we decide that we want ``os_family`` as switch, then we could provide the formula template variants for both the ``RedHat`` and ``Debian`` families.

  228. 

  229. .. code-block:: console

  230. 

  231.    /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/files/

  232.      default/

  233.        etc/

  234.          ntp.conf.jinja

  235.      RedHat/

  236.        etc/

  237.          ntp.conf.jinja

  238.      Debian/

  239.        etc/

  240.          ntp.conf.jinja

  241. 

  242. To make this work we need a ``conf.sls`` state file that takes a list of possible files as the configuration template.

  243. 

  244. .. code-block:: sls

  245. 

  246.    ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls

  247.    include:

  248.      - ntp

  249. 

  250.    {%- from 'ntp/map.jinja' import ntp with context %}

  251. 

  252.    Configure NTP:

  253.      file.managed:

  254.        - name: {{ ntp.config }}

  255.        - template: jinja

  256.        - source:

  257.          - salt://ntp/files/{{ grains.get('os_family', 'default') }}/etc/ntp.conf.jinja

  258.          - salt://ntp/files/default/etc/ntp.conf.jinja

  259.        - watch_in:

  260.          - service: Enable and start NTP service

  261.        - require:

  262.          - pkg: Install NTP package

  263. 

  264. If we want to cover the possibility of a special template for a minion identified by ``node01`` then we could have a specific template in ``/srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja``.

  265. 

  266. .. code-block:: jinja

  267. 

  268.    {#- /srv/saltstack/salt/ntp/files/node01/etc/ntp.conf.jinja #}

  269.    {#- Managed by saltstack #}

  270.    {#- Edit pillars or override this template in saltstack if you need customization #}

  271. 

  272.    {#- Some crazy configurations here for node01 #}

  273.    {#- ... #}

  274. 

  275. To make this work we could write a specially crafted ``conf.sls``.

  276. 

  277. .. code-block:: sls

  278. 

  279.    ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls

  280.    include:

  281.      - ntp

  282. 

  283.    {%- from 'ntp/map.jinja' import ntp with context %}

  284. 

  285.    Configure NTP:

  286.      file.managed:

  287.        - name: {{ ntp.config }}

  288.        - template: jinja

  289.        - source:

  290.          - salt://ntp/files/{{ grains.get('id') }}/etc/ntp.conf.jinja

  291.          - salt://ntp/files/{{ grains.get('os_family') }}/etc/ntp.conf.jinja

  292.          - salt://ntp/files/default/etc/ntp.conf.jinja

  293.        - watch_in:

  294.          - service: Enable and start NTP service

  295.        - require:

  296.          - pkg: Install NTP package

  297. 

  298. Using the ``files_switch`` macro

  299. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  300. 

  301. We can simplify the ``conf.sls`` with the new ``files_switch`` macro to use in the ``source`` parameter for the ``file.managed`` state.

  302. 

  303. .. code-block:: sls

  304. 

  305.    ## /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/conf.sls

  306.    include:

  307.      - ntp

  308. 

  309.    {%- set tplroot = tpldir.split('/')[0] %}

  310.    {%- from 'ntp/map.jinja' import ntp with context %}

  311.    {%- from 'ntp/libtofs.jinja' import files_switch %}

  312. 

  313.    Configure NTP:

  314.      file.managed:

  315.        - name: {{ ntp.config }}

  316.        - template: jinja

  317.        - source: {{ files_switch(['/etc/ntp.conf.jinja'],

  318.                                  lookup='Configure NTP'

  319.                     )

  320.                  }}

  321.        - watch_in:

  322.          - service: Enable and start NTP service

  323.        - require:

  324.          - pkg: Install NTP package

  325. 

  326. 

  327. * This uses ``config.get``, searching for ``ntp:tofs:source_files:Configure NTP`` to determine the list of template files to use.

  328. * If this returns a result, the default of ``['/etc/ntp.conf.jinja']`` will be appended to it.

  329. * If this does not yield any results, the default of ``['/etc/ntp.conf.jinja']`` will be used.

  330. 

  331. In ``libtofs.jinja``, we define this new macro ``files_switch``.

  332. 

  333. .. literalinclude:: ../template/libtofs.jinja

  334.    :caption: /srv/saltstack/salt-formulas/ntp-saltstack-formula/ntp/libtofs.jinja

  335.    :language: jinja

  336. 

  337. How to customise the ``source`` further

  338. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  339. 

  340. The examples below are based on an ``Ubuntu`` minion called ``theminion`` being configured via. pillar.

  341. 

  342. Using the default settings of the ``files_switch`` macro above,

  343. the ``source`` will be:

  344. 

  345. .. code-block:: sls

  346. 

  347.          - source:

  348.            - salt://ntp/files/theminion/etc/ntp.conf.jinja

  349.            - salt://ntp/files/Debian/etc/ntp.conf.jinja

  350.            - salt://ntp/files/default/etc/ntp.conf.jinja

  351. 

  352. Customise ``files``

  353. ~~~~~~~~~~~~~~~~~~~

  354. 

  355. The ``files`` portion can be customised:

  356. 

  357. .. code-block:: sls

  358. 

  359.    ntp:

  360.      tofs:

  361.        dirs:

  362.          files: files_alt

  363. 

  364. Resulting in:

  365. 

  366. .. code-block:: sls

  367. 

  368.          - source:

  369.            - salt://ntp/files_alt/theminion/etc/ntp.conf.jinja

  370.            - salt://ntp/files_alt/Debian/etc/ntp.conf.jinja

  371.            - salt://ntp/files_alt/default/etc/ntp.conf.jinja

  372. 

  373. Customise the use of grains

  374. ~~~~~~~~~~~~~~~~~~~~~~~~~~~

  375. 

  376. Grains can be customised and even arbitrary paths can be supplied:

  377. 

  378. .. code-block:: sls

  379. 

  380.    ntp:

  381.      tofs:

  382.        files_switch:

  383.          - any/path/can/be/used/here

  384.          - id

  385.          - os

  386.          - os_family

  387. 

  388. Resulting in:

  389. 

  390. .. code-block:: sls

  391. 

  392.          - source:

  393.            - salt://ntp/files/any/path/can/be/used/here/etc/ntp.conf.jinja

  394.            - salt://ntp/files/theminion/etc/ntp.conf.jinja

  395.            - salt://ntp/files/Ubuntu/etc/ntp.conf.jinja

  396.            - salt://ntp/files/Debian/etc/ntp.conf.jinja

  397.            - salt://ntp/files/default/etc/ntp.conf.jinja

  398. 

  399. Customise the ``default`` path

  400. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  401. 

  402. The ``default`` portion of the path can be customised:

  403. 

  404. .. code-block:: sls

  405. 

  406.    ntp:

  407.      tofs:

  408.        dirs:

  409.          default: default_alt

  410. 

  411. Resulting in:

  412. 

  413. .. code-block:: sls

  414. 

  415.          - source:

  416.            ...

  417.            - salt://ntp/files/default_alt/etc/ntp.conf.jinja

  418. 

  419. Customise the list of ``source_files``

  420. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  421. 

  422. The list of ``source_files`` can be given:

  423. 

  424. .. code-block:: sls

  425. 

  426.    ntp:

  427.      tofs:

  428.        source_files:

  429.          Configure NTP:

  430.            - '/etc/ntp.conf_alt.jinja'

  431. 

  432. Resulting in:

  433. 

  434. .. code-block:: sls

  435. 

  436.          - source:

  437.            - salt://ntp/files/theminion/etc/ntp.conf_alt.jinja

  438.            - salt://ntp/files/theminion/etc/ntp.conf.jinja

  439.            - salt://ntp/files/Debian/etc/ntp.conf_alt.jinja

  440.            - salt://ntp/files/Debian/etc/ntp.conf.jinja

  441.            - salt://ntp/files/default/etc/ntp.conf_alt.jinja

  442.            - salt://ntp/files/default/etc/ntp.conf.jinja

  443. 

  444. Note: This does *not* override the default value.

  445. Rather, the value from the pillar/config is prepended to the default.

  446. 

  447. Using sub-directories for ``components``

  448. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  449. 

  450. If your formula is composed of several components, you may prefer to provides files under sub-directories, like in the `systemd-formula <https://github.com/saltstack-formulas/systemd-formula>`_.

  451. 

  452. .. code-block:: console

  453. 

  454.    /srv/saltstack/systemd-formula/

  455.      systemd/

  456.        init.sls

  457.        libtofs.jinja

  458.        map.jinja

  459.        networkd/

  460.          init.sls

  461.          files/

  462.            default/

  463.              network/

  464.                99-default.link

  465.        resolved/

  466.          init.sls

  467.          files/

  468.            default/

  469.              resolved.conf

  470.        timesyncd/

  471.          init.sls

  472.          files/

  473.            Arch/

  474.              resolved.conf

  475.            Debian/

  476.              resolved.conf

  477.            default/

  478.              resolved.conf

  479.            Ubuntu/

  480.              resolved.conf

  481. 

  482. For example, the following ``formula.component.config`` SLS:

  483. 

  484. .. code-block:: sls

  485. 

  486.    {%- from "formula/libtofs.jinja" import files_switch with context %}

  487. 

  488.    formula configuration file:

  489.      file.managed:

  490.        - name: /etc/formula.conf

  491.        - user: root

  492.        - group: root

  493.        - mode: 644

  494.        - template: jinja

  495.        - source: {{ files_switch(['formula.conf'],

  496.                                  lookup='formula',

  497.                                  use_subpath=True

  498.                     )

  499.                  }}

  500. 

  501. will be rendered on a ``Debian`` minion named ``salt-formula.ci.local`` as:

  502. 

  503. .. code-block:: sls

  504. 

  505.    formula configuration file:

  506.      file.managed:

  507.        - name: /etc/formula.conf

  508.        - user: root

  509.        - group: root

  510.        - mode: 644

  511.        - template: jinja

  512.        - source:

  513.          - salt://formula/component/files/salt-formula.ci.local/formula.conf

  514.          - salt://formula/component/files/Debian/formula.conf

  515.          - salt://formula/component/files/default/formula.conf

  516.          - salt://formula/files/salt-formula.ci.local/formula.conf

  517.          - salt://formula/files/Debian/formula.conf

  518.          - salt://formula/files/default/formula.conf