PROD-16375

Change-Id: I758d38701cdf0e1cc6fa99c38192de2f9052468c

Change-Id: Idfe764a5b8106e02614eb3eab78895629a6fc570

Change-Id: I1d09c5297440a6469cfaabbfc67d1f1c11decece

Exposing CA keys in a mine creates a security flaw, thus such
should be avoided.
This change removes code responsible for putting and retrieving
CA key from a mine and changes the ca.sls state to allow configuring
where CA cert and its key would be generated as well as their owners.

Fixes PROD-13439

Change-Id: I6d78b13dcb3754c51606edd7e2d8158e128244a4

Issues:
* cmd.wait doesn't work correctly with bg=True
* cmd.wait will be deprecated in next releases [1]
* watch/watch_in work differently than onchanges/onchanges_in as some
* module override mod_watch()

This patch does:
* Replaces cmd.wait to cmd.run with onchanges.
* On any config file changes salt_minion_service_restart will be
  triggered so salt master will get state immediately because of bg=True
* salt_minion_service is used only for service enablement

[1] https://docs.saltstack.com/en/latest/ref/states/all/salt.states.cmd.html

Change-Id: I7b87b3614708b861e1767566426c7a67c337ba01
Signed-off-by: Sergii Golovatiuk <sgolovatiuk@mirantis.com>

In cases  when a service whants to generate and sign a certificate
it requires a CA key along with a CA cert itself.
For example, Octavia needs it for signing a certificate it generates
for a newly spawned amphora.

This change add sending a CA key to the mine from where it can be
extracted in the cert.sls state.
Also allow managing permissions for a CA cert and key retrieved
from the mine.

Related PROD: PROD-11933

Change-Id: I911effb4a63ae048e348ed04b7aca33998e359aa

In case of trust_salt_ca usage, the salt.minion.cert state
generates broken certs body due to a space replacing.

Change-Id: Id49e42807ddbc2addaf59a4c4541b720bbf87527

Currently the CI job fails with module not found errror.
AttributeError: 'module' object has no attribute 'check_refresh

The reason is new version of salt tries to call
salt.utils.pkg.check_refresh which is not available in 2016.11.3 but is
available in 2016.11.6.
Reference:
1. https://github.com/saltstack/salt/blob/v2016.11.3/salt/states/pkg.py
2. https://github.com/saltstack/salt/blob/v2016.11.6/salt/states/pkg.py#L1819

The fix proposed here is to use pkg.installed instead of pkg.latest.
It's not a good idea to always update salt whenever the salt state is
run, this may introduce failures unknown to the user. There is a pipeline
to update packages which should be used for updating to latest.
pkg.latest generally isn't a good way to write idempotent formulas. See
official salt docs: "Generally it is better for the installed function
to be used, as latest will update the package whenever a new package is
available."

Change-Id: I8da5c36c1613e54768993080f2514afc920c49f8

1. Remove implicit creation of "local_trusted_symlink".
   To install a system-wide certificates the linux.system.cert
   state or 'trusted_ca_minion' option must be used.

2. A ca-cert file may exist on a file-system and
   not be pulled from mine. So, in this case
   the following state be incorrect:

   - watch:
     - x509: ca_file

   To support this case, we need to replace `watch` statement
   with `watch_in`.

Change-Id: If41d050b56913d72da1ef7981f30780fec5d6d95

Change-Id: Ie6cf0a8361524b594b60119b25f6d336b6797ece

This reverts commit 6b2a592dc9.

1:
In case of trust_salt_ca usage, the salt.minion.cert state
generates broken certs body due to a space replacing:

-----BEGINCERTIFICATE-----
  MIIFzzCCA7egAwIBAgIITiyuuFgl1S4wDQYJKoZIhvcNAQELBQAwSjELMAkGA1UE
  BhMCY3oxFzAVBgNVBAMMDlNhbHQgT
  ....

To fix it the "x509.pem_managed" is used.

2:
If a symlink to CA cert is already exists, then the state is failing.
The force=True (replace) is used now to avoid the issue.

Change-Id: I4a2bd7c882e179560657a3dc7edf18c7e5835492

Change-Id: I73a4a14734651f6483fae169a2290e9c5b377d76

Change-Id: I7b4dc2001e07b047964f4ebbbbe0b23db7819566

Change-Id: Ie1f1397817b47f299107e4f44dfb4c5ffe1c010c

Change-Id: I6f1292779858c45f9cf6f4caf3657ee000b2cf06

Change-Id: I1e1269268a81d8cd01b5fe9328f63e8bf85e941b

We can have this failing because of bad mine data:
salt['mine.get'](cert.host, 'x509.get_pem_entries')

Without this change, dependency between salt_minion_cert_*_all and
ca_file is just ignored and salt_minion_cert_*_all state fails because
it can't find appropriate file.

Change-Id: I2a5dd12e08159bf110ff0d9879ebf0ad5d9d97c1

Change-Id: I2aba1213b1dda46aee929b8ea583c41316e3eb0b

It was failing with:
	Rendering SLS 'base:salt.minion.cert' failed: Conflicting ID 'salt_ca_certificates_packages'

ca-certificates installation should be probably moved out of the loop in
the future.

Change-Id: I26aeae62cc1c1d407d36d1d6bf101db073d9e601

Change-Id: I89cec95e87db52fd59a84d57c485d8c938711ef3

Unfortunately this is not idempotent, however we surely want to sync
everything when salt.minion state is executed.

Change-Id: I0faaf606b57dbd7d009156abfe50d2e5f350190e

Change-Id: I1eaa9cf7a7f861cc5de604e03fba8a74436d99ad

This error occures when multiple certs signed by same CA are requested
on minion.

Change-Id: I6b20ab4e1795298c94f55fdc61af99f933d8491c

First run is made during salt-master cloud-init and thus it is onchanges
is not suitable here because ca.crt file is already generated.

It was responsible for

    [CRITICAL] Rendering SLS 'base:salt.minion.cert' failed: Jinja variable 'dict object' has no attribute 'cfg01.mk20-lab-advanced.local'
    [ERROR   ] Data passed to highstate outputter is not a valid highstate return: {'local': ["Rendering SLS 'base:salt.minion.cert' failed: Jinja variable 'dict object' has no attribute 'cfg01.mk20-lab-advanced.local'"]}

- allow defining custom key/cert path
- ensure key/cert directories
- set key/cert permissions by metadata

source:engine metadata created - defaults to pkg
installation, pip installation alternative added