Pillar openssh.known_hosts_salt_ssh (#128)

* Pillar openssh.known_hosts_salt_ssh

* Dropped ill-named file

* Fixed aliasing of host names

* Improved pillar.example

* Opt-in to include localhost

* pillar/known_hosts_salt_ssh: clear cache in run()

* Dropped forgotten debugging output

README.rst

 
 Manages the side-wide ssh_known_hosts file and fills it with the
 public SSH host keys of your minions (collected via the Salt mine)
-and of hosts listed in you pillar data. You can restrict the set of minions
+and of hosts listed in you pillar data. It's possible to include
+minions managed via ``salt-ssh`` by using the ``known_hosts_salt_ssh`` renderer.
+
+You can restrict the set of minions
 whose keys are listed by using the pillar data ``openssh:known_hosts:target``
 and ``openssh:known_hosts:tgt_type`` (those fields map directly to the
 corresponding attributes of the ``mine.get`` function).
 
-The Salt mine is used to share the public SSH host keys, you must thus
+The **Salt mine** is used to share the public SSH host keys, you must thus
 configure it accordingly on all hosts that must export their keys. Two
 mine functions are required, one that exports the keys (one key per line,
 as they are stored in ``/etc/ssh/ssh_host_*_key.pub``) and one that defines
         mine_function: cmd.run
         cmd: cat /etc/ssh/ssh_host_*_key.pub
         python_shell: True
-      public_ssh_hostname:
+      public_ssh_host_names:
         mine_function: grains.get
         key: id
 
 official hostname of a minion, the alternate DNS name will be associated to the
 minion's public SSH host key.
 
-To add public keys of hosts not among your minions list them under the
+To **include minions managed via salt-ssh** install the ``known_hosts_salt_ssh`` renderer::
+
+    # in pillar.top:
+    '*':
+      - openssh.known_hosts_salt_ssh
+
+    # In your salt/ directory:
+    # Link the pillar file:
+    mkdir pillar/openssh
+    ln -s ../../formulas/openssh-formula/_pillar/known_hosts_salt_ssh.sls pillar/openssh/known_hosts_salt_ssh.sls
+
+Pillar ``openssh:known_hosts:salt_ssh`` overrides the Salt Mine.
+
+The pillar is fed by a host key cache. Populate it by applying ``openssh.gather_host_keys``
+to the salt master::
+
+    salt 'salt-master.example.test' state.apply openssh.gather_host_keys
+
+The state tries to fetch the SSH host keys via ``salt-ssh``. It calls the command as user
+``salt-master`` by default. The username can be changed via Pillar::
+
+    openssh:
+      known_hosts:
+        salt_ssh:
+          user: salt-master
+
+It's possible to define aliases for certain hosts::
+
+    openssh:
+      known_hosts:
+        salt_ssh:
+          public_ssh_host_names:
+            minion.id:
+              - minion.id
+              - alias.of.minion.id
+
+You can use a cronjob to populate a host key cache::
+
+    # crontab -e -u salt-master
+    0 1 * * * salt 'salt-master.example.test' state.apply openssh.gather_host_keys
+
+Or just add it to your salt master::
+
+    # states/top.sls:
+    base:
+      salt:
+        - openssh.known_hosts_salt_ssh
+
+You can also use a "golden" known hosts file. It overrides the keys fetched by the cronjob.
+This lets you re-use the trust estabished in the salt-ssh user's known_hosts file::
+
+    # In your salt/ directory: (Pillar expects the file here.)
+    ln -s /home/salt-master/.ssh/known_hosts ./known_hosts
+
+    # Test it:
+    salt-ssh 'minion' pillar.get 'openssh:known_hosts:salt_ssh'
+
+To add **public keys of hosts not among your minions** list them under the
 pillar key ``openssh:known_hosts:static``::
 
     openssh:
           github.com: 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq[...]'
           gitlab.com: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA[...]'
 
+Pillar ``openssh:known_hosts:static`` overrides ``openssh:known_hosts:salt_ssh``.
+
+To **include localhost** and local IP addresses (``127.0.0.1`` and ``::1``) use this Pillar::
+
+    openssh:
+      known_hosts:
+        include_localhost: True
 
 ``openssh.moduli``
 -----------------------

_pillar/known_hosts_salt_ssh.sls

+#!py
+
+import logging
+import os.path
+import re
+import subprocess
+
+cache = {}
+ssh_key_pattern = re.compile("^[^ ]+ (ssh-.+)$")
+log = logging.getLogger(__name__)
+
+def config_dir():
+    if '__master_opts__' in __opts__:
+        # run started via salt-ssh
+        return __opts__['__master_opts__']['config_dir']
+    else:
+        # run started via salt
+        return __opts__['config_dir']
+
+def cache_dir():
+    if '__master_opts__' in __opts__:
+        # run started via salt-ssh
+        return __opts__['__master_opts__']['cachedir']
+    else:
+        # run started via salt
+        return __opts__['cachedir']+'/../master'
+
+def minions():
+    if not 'minions' in cache:
+        cache['minions'] = __salt__.slsutil.renderer(config_dir() + '/roster')
+    return cache['minions']
+
+def host_variants(minion):
+    _variants = [minion]
+    def add_port_variant(host):
+        if 'port' in minions()[minion]:
+            _variants.append("[{}]:{}".format(host, minions()[minion]['port']))
+    add_port_variant(minion)
+    if 'host' in minions()[minion]:
+        host = minions()[minion]['host']
+        _variants.append(host)
+        add_port_variant(host)
+    return _variants
+
+def host_keys_from_known_hosts(minion, path):
+    '''
+    Fetches all host keys of the given minion.
+    '''
+    if not os.path.isfile(path):
+        return []
+    pubkeys = []
+    def fill_pubkeys(host):
+        for line in host_key_of(host, path).splitlines():
+            match = ssh_key_pattern.search(line)
+            if match:
+              pubkeys.append(match.group(1))
+    # Try the minion ID and its variants first
+    for host in host_variants(minion):
+        fill_pubkeys(host)
+    # When no keys were found ...
+    if not pubkeys:
+        # ... fetch IP addresses via DNS and try them.
+        for host in (salt['dig.A'](minion) + salt['dig.AAAA'](minion)):
+            fill_pubkeys(host)
+    # When not a single key was found anywhere:
+    if not pubkeys:
+        log.error("No SSH host key found for {}. "
+                  "You may need to add it to {}.".format(minion, path))
+    return "\n".join(pubkeys)
+
+def host_key_of(host, path):
+    cmd = ["ssh-keygen", "-H", "-F", host, "-f", path]
+    call = subprocess.Popen(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE
+    )
+    out, err = call.communicate()
+    if err == '':
+        return out
+    else:
+        log.error("{} failed:\nSTDERR: {}\nSTDOUT: {}".format(
+            " ".join(cmd),
+            err,
+            out
+        ))
+        return ""
+
+def host_keys(minion_id):
+    # Get keys from trusted known_hosts file
+    trusted_keys = host_keys_from_known_hosts(minion_id,
+        config_dir()+'/known_hosts')
+    if trusted_keys:
+        print "trusted_keys"
+        return trusted_keys
+    # Get keys from host key cache
+    cache_file = "{}/known_hosts_salt_ssh/{}.pub".format(cache_dir(), minion_id)
+    try:
+        with open(cache_file, 'r') as f:
+          return f.read()
+    except IOError:
+        return ''
+
+def run():
+    cache = {}  # clear the cache
+    config = {
+        'public_ssh_host_keys': {},
+        'public_ssh_host_names': {}
+    }
+    for minion in minions().keys():
+        config['public_ssh_host_keys'][minion] = host_keys(minion)
+        config['public_ssh_host_names'][minion] = minion
+    return {'openssh': {'known_hosts': {'salt_ssh': config}}}
+
+# vim: ts=4:sw=4:syntax=python

openssh/files/ssh_known_hosts

 #}
 
 {#- Generates one known_hosts entry per given key #}
-{%- macro known_host_entry(host, host_names, keys) %}
+{%- macro known_host_entry(host, host_names, keys, include_localhost) %}
 
 {#-   Get IPv4 and IPv6 addresses from the DNS #}
 {%-   set ip4 = salt['dig.A'](host) -%}
 
 {#-   The host names to use are to be found within the dict 'host_names'. #}
 {#-   If there are none, the host is used directly. #}
-{%-   set names = [host_names.get(host, host)] -%}
+{%-   set names = host_names.get(host, host) -%}
+{%-   set names = [names] if names is string else names %}
+{%-   if include_localhost and host == grains['id'] %}
+{%-     do names.append('localhost') %}
+{%-     do names.append('127.0.0.1') %}
+{%-     do names.append('::1') %}
+{%-   endif -%}
 
 {#-   Extract the hostname from the FQDN and add it to the names. #}
 {%-   if use_hostnames is iterable -%}
 {%- endmacro -%}
 
 {#- Pre-fetch pillar data #}
-{%- set target = salt['pillar.get']('openssh:known_hosts:target', '*') -%}
+{%- set target = salt['pillar.get']('openssh:known_hosts:target', "*.{}".format(grains['domain'])) -%}
 {%- set tgt_type = salt['pillar.get']('openssh:known_hosts:tgt_type', 'glob') -%}
 {%- set keys_function = salt['pillar.get']('openssh:known_hosts:mine_keys_function', 'public_ssh_host_keys') -%}
 {%- set hostname_function = salt['pillar.get']('openssh:known_hosts:mine_hostname_function', 'public_ssh_hostname') -%}
 {%- set hostnames_target_default = '*' if grains['domain'] == '' else "*.{}".format(grains['domain']) -%}
 {%- set hostnames_target = salt['pillar.get']('openssh:known_hosts:hostnames:target', hostnames_target_default) -%}
 {%- set hostnames_tgt_type = salt['pillar.get']('openssh:known_hosts:hostnames:tgt_type', 'glob') -%}
+{%- set include_localhost = salt['pillar.get']('openssh:known_hosts:include_localhost', False) -%}
 
 {#- Lookup IP of all aliases so that when we have a matching IP, we inject the alias name
     in the SSH known_hosts entry -#}
   {%- endfor -%}
 {%- endfor -%}
 
-{#- Loop over targetted minions -#}
+{#- Salt Mine #}
 {%- set host_keys = salt['mine.get'](target, keys_function, tgt_type=tgt_type) -%}
 {%- set host_names = salt['mine.get'](target, hostname_function, tgt_type=tgt_type) -%}
+
+{#- Salt SSH (if any) #}
+{%- for minion_id, minion_host_keys in salt['pillar.get'](
+  'openssh:known_hosts:salt_ssh:public_ssh_host_keys',
+  {}
+).items() -%}
+{%-   if salt["match.{}".format(tgt_type)](target, minion_id=minion_id) -%}
+{%      do host_keys.update({minion_id: minion_host_keys}) %}
+{%-   endif -%}
+{%- endfor -%}
+{%- for minion_id, minion_host_names in salt['pillar.get'](
+  'openssh:known_hosts:salt_ssh:public_ssh_host_names',
+  {}
+).items() -%}
+{%-   if salt["match.{}".format(tgt_type)](target, minion_id=minion_id) -%}
+{%      do host_names.update({minion_id: minion_host_names}) %}
+{%-   endif -%}
+{%- endfor %}
+
+{#- Static Pillar data #}
 {%- do host_keys.update(salt['pillar.get']('openssh:known_hosts:static',
     {}).items()) -%}
+
+{#- Loop over targetted minions -#}
 {%- for host, keys in host_keys| dictsort -%}
-{{ known_host_entry(host, host_names, keys) }}
+{{ known_host_entry(host, host_names, keys, include_localhost) }}
 {%- endfor -%}

openssh/gather_host_keys.sls

+{%- set minions = salt.slsutil.renderer(opts['config_dir'] + '/roster') %}
+{%- set cache_dir = opts['cachedir'] + '/../master/known_hosts_salt_ssh' %}
+{%- set cmd = "cat /etc/ssh/ssh_host_*_key.pub 2>/dev/null" %}
+
+{{ cache_dir }}:
+  file.directory:
+    - makedirs: True
+
+{%- for minion_id in minions %}
+{%-   set salt_ssh_cmd = "salt-ssh --out=json --static '{}' cmd.run_all '{}'".format(minion_id, cmd) %}
+{%-   set result = salt['cmd.run_all'](salt_ssh_cmd,
+          python_shell=True,
+          runas=salt['pillar.get']('openssh:known_hosts:salt_ssh:user', 'salt-master')
+        )
+%}
+
+{%-   set pubkeys = False %}
+{%-   if result['retcode'] == 0 %}
+{%-     load_json as inner_result %}
+{{        result['stdout'] }}
+{%-     endload %}
+{%-     set pubkeys = inner_result[minion_id]['stdout'].splitlines() | sort | join("\n") %}
+{%-   else %}
+{%-     do salt.log.error("{} failed: {}".format(salt_ssh_cmd, result)) %}
+{%-   endif %}
+
+{%-   if pubkeys %}
+{{ cache_dir }}/{{ minion_id }}.pub:
+  file.managed:
+    - contents: |
+        {{ pubkeys | indent(8) }}
+    - require:
+      - file: {{ cache_dir }}
+{%-   endif %}
+
+{%- endfor %}

pillar.example

     #hostnames:
     # Restrict wich hosts you want to use via their hostname
     # (i.e. ssh user@host instead of ssh user@host.example.com)
-    #  target: '*'  # Defaults to "*.{}".format(grains['domain']) with a fallback to '*'
+    #  target: '*'  # Defaults to "*.{{ grains['domain']}}"
     #  tgt_type: 'glob'
     # To activate the defaults you can just set an empty dict.
     #hostnames: {}
+    # Include localhost, 127.0.0.1 and ::1 (default: False)
+    include_localhost: False
+    # Host keys fetched via salt-ssh
+    salt_ssh:
+      # The salt-ssh user
+      user: salt-master
+      # specify public host names of a minion
+      public_ssh_host_names:
+        minion.id:
+          - minion.id
+          - alias.of.minion.id
+      # specify public host keys of a minion
+      public_ssh_host_keys:
+        minion.id: |
+          ssh-rsa [...]
+          ssh-ed25519 [...]
     # Here you can list keys for hosts which are not among your minions:
     static:
       github.com: 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGm[...]'