Skip to main content

Kubernetes ETCD Data Store Cluster with TLS Certificate

This article is all about configuration of ETCD kubernetes data store with high security  measurement. We will use TLS/SSL certificate so that all communication within cluster will secure and private.
Environment:
3 RHEL 7 / Centos 7 Server with minimal installation.
etcd and etcdctl binary which we can download from https://github.com/coreos/etcd/releases/
openssl.conf will provide in tutorial.
Inventory:
Node1: etcd-01.syshunt.com 192.168.43.45
Node2: etcd-02.syshunt.com 192.168.43.46
Node3: etcd-03.syshunt.com 192.168.43.47
Certificate Generation:
We will use openssl tool for certificate generation as it required for secure communication.
Create Certificate Authority (CA):
Create “etcd-certificate” directory and switch in etcd-certificate directory, will use this directory in our tutorial for all certificate generation,
[root@etcd-01]# mkdir /root/etcd-certificate
[root@etcd-01]# cd /root/etcd-certificate
[root@etcd-01 etcd-certificate]# openssl genrsa -out ca-key.pem 2048
[root@etcd-01 etcd-certificate]# openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=etcd-ca"
ETCD node-1 certificate generation:
Create openssl.conf file with below given configuration,
[root@etcd-01 etcd-certificate]# vi openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-01
IP.1 = 192.168.43.45
IP.2 = 127.0.0.1
Set openssl.conf location variable in your current shell,
[root@etcd-01 etcd-certificate]# CONFIG=`echo $PWD/openssl.conf`
Generate Certificates:
[root@etcd-01 etcd-certificate]# openssl genrsa -out member-etcd-01-key.pem 2048
[root@etcd-01 etcd-certificate]# openssl req -new -key member-etcd-01-key.pem -out member-etcd-01.csr -subj "/CN=etcd-01" -config ${CONFIG}
[root@etcd-01 etcd-certificate]# openssl x509 -req -in member-etcd-01.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-01.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}
ETCD node-2 certificate generation:
Create openssl.conf file with below given configuration,
[root@etcd-01 etcd-certificate]# vi openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-02
IP.1 = 192.168.43.46
IP.2 = 127.0.0.1
Generate Certificates:
[root@etcd-01 etcd-certificate]# openssl genrsa -out member-etcd-02-key.pem 2048
[root@etcd-01 etcd-certificate]# openssl req -new -key member-etcd-02-key.pem -out member-etcd-02.csr -subj "/CN=etcd-02" -config ${CONFIG}
[root@etcd-01 etcd-certificate]# openssl x509 -req -in member-etcd-02.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-02.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}
ETCD node-3 certificate generation:
Create openssl.conf file with below given configuration,
[root@etcd-02 etcd-certificate]# vi openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd-03
IP.1 = 192.168.43.47
IP.2 = 127.0.0.1
Generate Certificates:
[root@etcd-01 etcd-certificate]# openssl genrsa -out member-etcd-03-key.pem 2048
[root@etcd-01 etcd-certificate]# openssl req -new -key member-etcd-03-key.pem -out member-etcd-03.csr -subj "/CN=etcd-03" -config ${CONFIG}
[root@etcd-01 etcd-certificate]# openssl x509 -req -in member-etcd-03.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-etcd-03.pem -days 3650 -extensions ssl_client -extfile ${CONFIG}

Copy all created certificates in every etcd nodes from “/etc/ssl/etcd-certificate” directory to “/etc/ssl/etcd/ssl/”:


[root@etcd-* etcd]# cp -rvp /root/etcd-certificate/*.pem /etc/ssl/etcd/ssl/




Install ETCD binary on all the 3 etcd Members:



Copy etcd and etcdctl binary on all 3 etcd members which you have downloaded form
etcd github project page ( https://github.com/coreos/etcd/releases/ ),


[root@etcd-* etcd]# cp -vp etcd etcdctl /usr/bin




Create Etcd data directory (on all 3 members):



Etcd data directory is very important as etcd utility saves all cluster data in data directory,


[root@etcd-* etcd]# mkdir /var/lib/etcd




Create etcd user on all 3 nodes:


[root@etcd-* etcd]# useradd etcd -s /sbin/nologin -r -d /var/lib/etcd




Fixing permissions on all etcd nodes at required directory and files:


[root@etcd-* etcd]# chmod -Rv 550 /etc/ssl/etcd/
[root@etcd-* etcd]# chmod 440 /etc/ssl/etcd/ssl/*.pem
[root@etcd-* etcd]# chown -Rv etcd:etcd /etc/ssl/etcd/
[root@etcd-* etcd]# chown -Rv etcd:etcd /etc/ssl/etcd/*
[root@etcd-* etcd]# chown etcd:etcd /var/lib/etcd/




Setting Up etcd service daemon files on all etcd nodes:



It is required to create etcd service files on every etcd nodes so that we can start and enable etcd as service daemon.



[root@etcd-* etcd]# vi /usr/lib/systemd/system/docker.service


[Unit]
Description=etcd
After=network.target

[Service]
Type=notify
User=etcd
EnvironmentFile=/etc/etcd.env
ExecStart=/usr/bin/etcd
NotifyAccess=all
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target




ETCD Configuration File on every etcd node:



For configuring etcd with our custom configuration it required to create configuration file in “/etc/etcd.env” location because we are calling this environment file in etcd service file.



Etcd configuration file for etcd node1:



[root@etcd-1 etcd]# vi /etc/etcd.env


ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.45:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.45:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.45:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.45:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380
#ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-01.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-01-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-01.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-01-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true




Etcd configuration file for etcd node2:



[root@etcd-2 etcd]# vi /etc/etcd.env


ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.46:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.46:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.46:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.46:2380
ETCD_NAME=etcd2
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380
#ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.46:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-02.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-02-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-02.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-02-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true




Etcd configuration file for etcd node3:



[root@etcd-* etcd]# vi /etc/etcd.env


ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.43.47:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.43.47:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_LISTEN_CLIENT_URLS=https://192.168.43.47:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_LISTEN_PEER_URLS=https://192.168.43.47:2380
ETCD_NAME=etcd3
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.43.45:2380,etcd2=https://192.168.43.46:2380,etcd3=https://192.168.43.47:2380

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-03.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-03-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-etcd-03.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-etcd-03-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true




Applying ETCD configuration on systemd for every etcd nodes:



In systemd based system it is required to reload daemon on every etcd nodes after service file change.


[root@etcd-* etcd]# systemctl daemon-reload




Start and enable ETCD service on every:



Now we can start our etcd service by systemd command, It is also required to enable daemon on system boot time so we will need not to start service every time manually on system startup.


[root@etcd-* etcd]# systemctl start etcd
[root@etcd-* etcd]# systemctl enable etcd




Verifying ETCD cluster status:



For verifying etcd cluster status we can run below command on any etcd nodes.


[root@etcd-01 ~]# etcdctl -C https://192.168.43.45:2379 --ca-file /etc/ssl/etcd/ssl/ca.pem cluster-health
member 649628565489a99c is healthy: got healthy result from https://192.168.43.45:2379
member caa56683e6af0137 is healthy: got healthy result from https://192.168.43.46:2379
member dc4795c6ff3e6627 is healthy: got healthy result from https://192.168.43.47:2379
cluster is healthy




If cluster is showing healthy that indicated we have configured everything properly so it’s time to use etcd based data store in our kubernetes cluster . In our future tutorials we will demonstrate how can we setup highly secure kubernetes cluster.

Comments

Post a Comment

Popular posts from this blog

Troubleshooting Tomcat using Catalina log

Troubleshooting Tomcat using Catalina log We can get the  catalina  logs in  $CATALINA_HOME/logs Here we'll have a  catalina.out We can see some huge info in that log file. If we want to what exactly happens since the start of tomcat then we need to log that into a new file. By default we have log rotation enabled on this logs. For now let us stop tomcat and nullify the catalina.out file and then start tomcat to have some new info in the catalina.out # service tomcat stop # cd $CATALINA_HOME/logs # :> catalina.out (or) cat /dev/null > catalina.out # ll catalina.out -rw-r----- 1 root root 0 Feb 18 16:19 catalina.out # service tomcat start # ll catalina.out -rw-r----- 1 root root 17341 Feb 18 16:21 catalina.out We can see the log being written after the start of tomcat. Now we can notice the instance giving some information about the starting of tomcat, about the configuration for the startup, path of the servlet instance for which the log is re...
Post a Comment
Read more

Managing tomcat application

Tomcat is a servlet application which is ready to use after download. There is no need of installing this application. We can directly start or stop it from the bin directory.  So in such case we should go to the bin directory in the CATALINA_HOME or else we need to use the startup script along with its absolute path. And the same is needed to shut it down. We can use the below script to manage the tomcat application and we neither need to go to the CATALINA_HOME nor to use the absolute path. All we need to do is to change the permissions to make it executable and copy that to /sbin and /etc/init.d/ directories. If we copy this into /etc/init.d/ then also we need to mention the absolute path but we can easily memorize it as we know all the startup scripts exists there. But if we copy that into /sbin then there is absolutely no need of mentioning the path of the file. We can use this like below: # tomcat start|stop|restart|status #!/bin/bash # Author : Arjun S...
Post a Comment
Read more

Red Hat Enterprise Virtualization Manager RHEL 7 – Part 2

I have just published  what is RHEV  is the first article to understand RHEV Better. In order to maintain multiple RHEV-Hosts from central place RHEV-Manager is required. RHEV-M will be used as central place to manage all RHEV-Hosts. In this Article We are going to see HowTo Install RHEV-M Red Hat Enterprise Virtualization Manager RHEL 7. Red Hat Enterprise Virtualization Management (RHEV-M) is a virtual management console built on Red Hat Enterprise Linux (RHEL). It interacts with individual RHEV-Hosts using the Virtual Desktop Server Manager (VDSM). A VDSM agent is running on each of the RHEV-H nodes. RHEV-M allows administrators to manage/control number of data centers and their network, compute and storage resources. In addition RHEV-M provides a central repository for storing virtual machines, disks, images and virtual machine snapshots. Install RHEV-M Hardware Requirements Hard Disk Space Minimum 50GB RAM Minimum 4GB – Max 2TB Dual Core or High Processor 1Gb...
Post a Comment
Read more