My previous efforts in creating a #devops environment have been entirely manual. When you start scaling and needing to setup things quickly and consistently or outside of production, there is need to automate the process. For infrastructure automation, I have turned to Ansible, which I will henceforth use to configure the entire environment and automate deployments with.

(1) To begin, designate a system that will be used as the host — from where Ansible will be launched. In this case, my development laptop (a MacBook Pro) fits the bill, given that Ansible is mostly a Linux tool. Then in addition to the tools you normally use for development work, you must also install Ansible, VirtualBox, and Vagrant. The idea is to use a VM for development before pushing the changes out to the actual #devops server.

# ansible --version
ansible 2.2.0.0
# vagrant --version
Vagrant 1.8.5
# vboxmanage --version
5.1.10r112026

(2) The second step is to create the Ansible project consisting of a minimal file layout and configuration options, as shown in the image below.

screenshot-2016-12-03-03-47-13

As can be observed in the image, the Ansible config and inventory files are both setup within the project, and I have elected to configure my servers in the “devops” role. The VM server I’ll use for development is called “devops_vm”, and all servers will be in the “devops_servers” group, to begin with.

(3) To ensure consistency and expediency of major dependencies, such as Java, I will have the archive available locally. In this case, it is under .roles/devops/files/java. I am also following the philosophy of gathering facts always and using Jinja2 (as Ansible is based on Python 2.x) to mint all configuration files. To manage secrets, I’ll maintain a password file locally at /etc/ansible/vault_password.

(4) The current playbook (./provision.yml) will ensure some basic settings, utilities, and services on the provisioned server, as well as install Java if it is not already installed. The source code for this project can be reviewed at GitHub. Ansible will do what I had manually done in #305: Install Java 8 on CentOS 7, only a lot simpler.

In the big leagues of #devops, manual installation and configuration of software environments is passée. Everyone uses some kind of automation to provision the systems, build the software, test it, and deploy it to various tiers. At my day job, we use Chef, but I am taking this opportunity to get into Ansible, a more modern tool in this space. Installation is a breeze.

# yum -y update
# yum -y install epel-release
# yum -y install ansible
# ansible --version
*****
ansible 2.1.1.0
 config file = /etc/ansible/ansible.cfg
 configured module search path = Default w/o overrides
*****

With Ansible installed, I will revisit previous setup until this point by developing provisioning scripts for this #devops environment.

Since I sometimes need to administer my Linux box remotely, I opened up the SSH port through the firewall and implemented user security right on the server. Whenever I logged in via SSH, I would see a message similar to this:

Last failed login: Thu Jul 23 03:04:09 MDT 2016 from 221.229.172.74 on ssh:notty
There were 162185 failed login attempts since the last successful login.
Last login: Sat Aug 16 07:47:21 2016 from xxxxxxxxxxxxxxxx.hsd1.co.comcast.net

Enter the very nifty fail2ban (http://www.fail2ban.org/), a log-parsing utility that monitors system logs for symptoms of an automated attack on your Linux machine. When an attempted compromise is detected, it adds a new rule to IPTABLES, and blocks the IP address of the attacker, either for a set amount of time or permanently. Fail2ban optionally alerts you through email that an attack is occurring. It is primarily focused on SSH attacks, although it can be further configured to work for any service that uses log files.

(1) Enable the EPEL repository

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# rpm -ivh epel-release-latest-7.noarch.rpm

(2) Install

# yum -y install fail2ban sendmail

(3) Permanently block repeat offenders

# vi /etc/fail2ban/action.d/iptables-repeater.conf
*****
[Definition]
actionstart = iptables -N fail2ban-REPEAT-<name>
 iptables -A fail2ban-REPEAT-<name> -j RETURN
 iptables -I INPUT -j fail2ban-REPEAT-<name>
 cat /etc/fail2ban/ip.blocklist.<name> |grep -v ^\s*#|awk '{print $1}' | while read IP; do iptables -I fail2ban-REPEAT-<name> 1 -s $IP -j DROP; done
actionstop = iptables -D INPUT -j fail2ban-REPEAT-<name>
 iptables -F fail2ban-REPEAT-<name>
 iptables -X fail2ban-REPEAT-<name>
actioncheck = iptables -n -L INPUT | grep -q fail2ban-REPEAT-<name>
actionban = iptables -I fail2ban-REPEAT-<name> 1 -s <ip> -j DROP
 ! grep -Fq <ip> /etc/fail2ban/ip.blocklist.<name> && echo "<ip> # fail2ban/$( date '+%%Y-%%m-%%d %%T' ): auto-add for repeat offender" >> /etc/fail2ban/ip.blocklist.<name>
actionunban = /bin/true
[Init]
name = REPEAT
*****

(4) Update the jail configuration

# vi /etc/fail2ban/jail.conf
// Update each of these settings
*****
backend = systemd
ignoreip = 127.0.0.1/8 192.168.1.0/24
*****
// Add the following lines at the bottom
*****
[ssh-repeater]
enabled = true
filter = sshd
action = iptables-repeater[name=ssh]
 sendmail-whois[name=SSH-repeater, dest=root, sender=root]
logpath = /var/log/secure
maxretry = 21
findtime = 31536000
bantime = 31536000
*****

(4) Starting the services and enable them to start at boot.

# systemctl start fail2ban
# systemctl enable fail2ban
# systemctl start sendmail
# systemctl enable sendmail

(5) Restarting the service, checking status

# fail2ban-client stop
# fail2ban-client start
# fail2ban-client status

(6) Check the security log for blacklist candidates

# tail /var/log/audit/audit.log
*****
type=CRYPTO_KEY_USER msg=audit(1471422089.906:10626420): pid=1298 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=57:e8:d7:22:87:e6:0e:dc:77:07:21:5b:ed:6b:d6:c5 direction=? spid=1299 suid=74 exe="/usr/sbin/sshd" hostname=? addr=116.31.116.48 terminal=? res=success'
*****

There were so many like the above that had successfully been authenticated. A quick check revealed that IP address 116.31.116.48 = China Telecom Guangdong, one of the major sources of probing bots and hacks. To ban specific addresses,

# sudo fail2ban-client set ssh-repeater banip 116.31.116.0/24

(7) Check your mail for notifications about IPs that have been banned.

# vi /var/spool/mail/root
# tail -50 /var/log/audit/audit.log

You should see a whole lot less attempts to log in from now on, and no unknown successful logins. It might be a good idea to kill the SSH keys right now and restart sshd.

The primary application (Geldzin) I’ll be managing in this devops environment requires an FTP location to post nightly backups. Therefore we need to setup SFTP services and a user that the application will use for this purpose.

(1) Directory, usually on a partition with a lot of space, where FTP files will exist.

# mkdir -p /data/ftp

(2) SFTP access should be restricted to users of the “sftpusers” group.

# groupadd sftpusers

(3) Update SSH configuration and setup the chroot jail.

# vi /etc/ssh/sshd_config
*****
Subsystem sftp internal-sftp
// Add these at the bottom
Match Group sftpusers
ChrootDirectory /data/ftp/%u
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp
*****

(4) SELinux permissions to allow sftpd to write to the FTP directories

# setsebool ftp_home_dir=1
# semanage fcontext -a -t public_content_t /data/
# restorecon -R -v /data/ftp
# setsebool -P allow_ftpd_anon_write on

(5) Restart the service

systemctl restart sshd

(6) Setup the user account the application will use.

# useradd -m geldzin -d /home/geldzin -s /sbin/nologin -g sftpusers
# passwd geldzin
# mkdir -p /data/ftp/geldzin/files
# semanage fcontext -a -t public_content_rw_t "/data/ftp(/.*)?"
# restorecon -R -v /data/ftp
# chown -R geldzin:sftpusers /data/ftp/geldzin/*
# chmod -R 700 /data/ftp/geldzin/*

The only catch here is that users cannot write to /data/ftp/%u directly (they don’t own it), but should be able to write to directories under that folder (such as /files). The chroot jail ensures users don’t escape their sandbox and write elsewhere on the system.

(7) Test the connection from a remote host or SFTP client

# sftp geldzin@workhorse

(8) You might experience issues connecting to the SFTP server, something like this:

packet_write_wait: Connection to 192.168.x.xxx: Broken pipe
Connection closed

And the security log () might show something similar to:

sshd[22909]: pam_unix(sshd:session): session opened for user geldzin by (uid=0)
sshd[22909]: fatal: bad ownership or modes for chroot directory component "/data/" [postauth]
sshd[22909]: pam_unix(sshd:session): session closed for user geldzin

This issue indicates an ownership problem. sshd will reject SFTP connections to accounts that are set to chroot into any directory that has ownership/permissions that sshd considers insecure. sshd’s strict ownership/permissions requirements dictate that every directory in the chroot path must be owned by root and only writable by the owner. So we fix the problem in this case with:

chown root:root /data/
# chmod 755 /data/

You should be in business.

For most of the Java web applications I have developed, my go-to web server has been Apache Tomcat. Although I have not had much success with the Tomcat plugin in my Maven projects (instead I use the Jetty Maven plugin during development), overall I have had little trouble using Tomcat for production.

(1) Setup where Tomcat will be installed on your local machine.

# cd /apps/
# mkdir apache-tomcat-8.0.32

(2) Create “tomcat” group and user.

# groupadd tomcat
# useradd -M -s /bin/nologin -g tomcat -d /apps/apache-tomcat-8.0.32 tomcat

(3) Download and install Tomcat (https://tomcat.apache.org/download-80.cgi).

# cd /data/downloads
# mkdir tomcat ; cd $_
# wget http://ftp.wayne.edu/apache/tomcat/tomcat-8/v8.0.32/bin/apache-tomcat-8.0.32.tar.gz
# tar xvf apache-tomcat-8.0.32.tar.gz -C /apps/apache-tomcat-8.0.32 --strip-components=1

(4) Configure permissions and access.

# cd /apps/apache-tomcat-8.0.32
# chgrp -R tomcat conf ; chmod g+rwx conf ; chmod g+r conf/*
# chown -R tomcat webapps/ work/ temp/ logs/

(5) Setup the service.

# vim /etc/systemd/system/tomcat8.service
=====
# Systemd unit file for tomcat
[Unit]
Description=Apache Tomcat 8 Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=JAVA_HOME=/apps/jdk1.8.0_66/jre
Environment=CATALINA_PID=/apps/apache-tomcat-8.0.32/temp/tomcat.pid
Environment=CATALINA_HOME=/apps/apache-tomcat-8.0.32
Environment=CATALINA_BASE=/apps/apache-tomcat-8.0.32
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/apps/apache-tomcat-8.0.32/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID

User=tomcat
Group=tomcat

[Install]
WantedBy=multi-user.target
=====
# systemctl daemon-reload

(6) Setup port security for Tomcat 8.

# vim conf/server.xml
=====
<Server port="38083" shutdown="SHUTDOWN">
<Connector port="18083" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="48083" />
<Connector port="28083" protocol="AJP/1.3" redirectPort="48083" />
=====
# vim /etc/sysconfig/iptables
=====
-A INPUT -p tcp -m tcp --dport 18083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 28083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 48083 -j ACCEPT
=====
# systemctl restart iptables

(7) Configure SELinux access

# semanage port -a -t http_port_t -p tcp 18083
# semanage port -a -t http_port_t -p tcp 28083
# semanage port -a -t http_port_t -p tcp 38083
# semanage port -a -t http_port_t -p tcp 48083
# semanage fcontext -a -t 'httpd_cache_t' '/apps/apache-tomcat-8.0.32(/.*)?'
# restorecon -Rv /apps/apache-tomcat-8.0.32
# setsebool -P httpd_builtin_scripting 1

(8) Start Tomcat 8 at boot.

# chkconfig tomcat8 on

(9) Start, restart, and stop Tomcat 8.

# systemctl start tomcat8
# systemctl restart tomcat8
# systemctl stop tomcat8

(10) Check the log

# cat /apps/apache-tomcat-8.0.32/logs/catalina.out

(11) Check that service is running

# netstat -tnlp | grep 8083

(12) Tunnel through firewall

# firewall-cmd --zone=public --add-port=18083/tcp --permanent
# firewall-cmd --zone=public --add-port=28083/tcp --permanent
# firewall-cmd --zone=public --add-port=38083/tcp --permanent
# firewall-cmd --zone=public --add-port=48083/tcp --permanent
# firewall-cmd --reload

(13) Configure the web management console.

# vim /apps/apache-tomcat-8.0.32/conf/tomcat-users.xml
=====
<tomcat-users>
 <user username="tomcat8" password="*****" roles="manager-gui,admin-gui"/>
</tomcat-users>
=====
# systemctl restart tomcat8

The manager application would then be accessible at http://localhost:18083/manager/html, and the virtual host manager at http://localhost:18083/host-manager/html.

One of the main reasons for DevOps is to manage delivery of software into production in a controlled manner. I use Jenkins to build the software from sources, test it, profile the builds, generate documentation, and stage it for deployment to any environment on schedule or on-demand. In fact, when all tests pass, the code is automatically promoted into subsequent environments via Github after a development code review. As a continuous integration tool, post-development tasks are now automated.

(1) Install or update the Jenkins repo

# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
# rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key

(2) Install Jenkins

# yum install jenkins

(3) I’m in the habit of not using default ports and locations

# vim /etc/sysconfig/jenkins
*****
JENKINS_HOME="/data/jenkins"
JENKINS_PORT="18082"
JENKINS_AJP_PORT="28082"
*****
# mkdir -p /data/jenkins
# chgrp -R jenkins /data/jenkins
# chown -R jenkins:jenkins /data/jenkins
# chmod -R 775 /data/jenkins

(4) Allow port for remote access

# vim /etc/sysconfig/iptables
*****
-A INPUT -p tcp -m tcp --dport 18082 -j ACCEPT
*****
# systemctl restart iptables

(5) Configure SELinux

# semanage port -a -t http_port_t -p tcp 18082
# semanage port -a -t http_port_t -p tcp 28082
# semanage fcontext -a -t 'httpd_cache_t' '/data/jenkins(/.*)?'
# restorecon -Rv /data/jenkins
# setsebool -P httpd_builtin_scripting 1

(6) Configure Jenkins to auto-start at boot

# chkconfig jenkins on

(7) Starting and stopping Jenkins

# service jenkins start
# service jenkins stop

(8) Know where the logs are

# cat /var/log/jenkins/jenkins.log

(9) Check that service is running

# netstat -tnlp | grep 8082

(10) Fix access through the system firewall, or Jenkins will not be accessible over HTTP

# firewall-cmd --zone=public --add-port=18082/tcp --permanent
# firewall-cmd --reload

Et voilà, c’est tout que vous devez faire!

A lot of the applications I develop these days require some kind of database, and for years, Oracle’s MySQL has been my go-to database application. Because of this need, it is fitting to install it in the DevOps environment.

(1) Install MySQL community repository

# yum install http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
# yum search 'mysql' | grep 'mysql-'
# yum info mysql-community-server // Shows version 5.6
# yum install mysql-community-server

yum will install MySQL and setup the log file at /var/log/mysqld.log, and the configuration file at /etc/my.cnf.

(2) Change the location of the data file (usually to a partition that has more space).

# vim /etc/my.cnf
*****
datadir=/data/mysql/data
*****

Update the permissions on the data directory.

# mkdir -p /data/mysql/data
# chgrp -R mysql /data/mysql
# chown -R mysql:mysql /data/mysql
# chmod -R 775 /data/mysql

(3) Allow MySQL to be accessed via its TCP port. Many applications use network URLs (such as jdbc:mysql://localhost:3306/sakila?profileSQL=true) to connect to local databases.

# vim /etc/sysconfig/iptables
*****
-A INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
*****
# systemctl restart iptables

(4) Additionally, SELinux is configured by default on CentOS 7. So we need to allow the use of port 3306 and the alternate data directory we set up earlier.

# semanage port -a -t mysqld_port_t -p tcp 3306
# semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
# restorecon -Rv /data/mysql/data

(5) Often you need MySQL to start automatically when the system boots up.

# systemctl enable mysqld.service

(6) How to start, restart, or stop the MySQL service.

# systemctl start mysqld
# systemctl restart mysqld
# systemctl stop mysqld

(7) Set the root password for the first time, or change the root password.

# mysqladmin -u root password <new-password>
# mysqladmin -u root -p <current-password> <new-password>

(8) Now you can log in locally, which will prompt for a password.

# mysql -u root -p

This is how simple the setup is. At this point, you can choose to further secure and tune the installation to your needs, but because this will only be a test/development database, I see not need for it here.

When I first started developing software, Apache Subversion was what small-scale developers like myself used for source code management (SCM). I then worked for a company that used IBM Rational ClearCase for SCM, in fact, doing configuration management. Just over a year ago, I decided to learn Git and it has become my de-facto SCM on recent projects. Thus I need it in my DevOps environment.

YUM repositories do not always have the latest Git, so you will need to install the latest yourself.

(1) Install required packages

# yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils-MakeMaker
# yum remove git

(2) Download the latest .tar, check https://www.kernel.org/pub/software/scm/

# mkdir /data/downloads/git
# cd /data/downloads/git
# wget https://www.kernel.org/pub/software/scm/git/git-2.7.0.tar.gz
# tar xzf git-2.7.0.tar.gz

(3) Install to location of choice

# cd git-2.7.0
# make prefix=/apps/git-2.7.0 all
# make prefix=/apps/git-2.7.0 install
# echo "export PATH=$PATH:/apps/git-2.7.0/bin" >> /etc/bashrc
# source /etc/bashrc
# git --version
*****
git version 2.7.0
*****

That’s it.

maven

The third tool in my home business DevOps is Apache Maven, a dependency management and project build tool. It is particularly well-suited for my development needs, since I develop mostly Java software. I need it on the DevOps machine because I’ll be doing nightly and on-demand builds of the software there.

To install Maven:
(1) Visit https://maven.apache.org/download.cgi and grab the URL for the latest binary package (3.3.9 at this writing) from a mirror.

(2) Prepare the local home for Maven, and download the package.

# mkdir /data/downloads/maven ; cd $_
# wget http://apache.claz.org/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

(3) Install Maven and setup the local repository.

# tar xvf apache-maven-3.3.9-bin.tar.gz
# mv apache-maven-3.3.9/ /apps/
# mkdir -p /data/maven/repo
# chmod 777 /data/maven/repo

(4) Configure Maven’s global environment variables.

# vim /etc/profile.d/maven.sh
==========
export M2_HOME=/apps/apache-maven-3.3.9
export M2_REPO=/data/maven/repo
export PATH=$M2_HOME/bin:$PATH
==========
# source /etc/profile.d/maven.sh

(5) Update the global Maven settings file with the repository information as well, for applications (such as Jenkins) that do not read it from environment variables.

# vim /apps/apache-maven-3.3.9/conf/settings.xml
==========
 <localRepository>/data/maven/repo</localRepository>
==========

Now you can check that Maven is installed.

# mvn -v
==========
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T09:41:47-07:00)
Maven home: /apps/apache-maven-3.3.9
Java version: 1.8.0_66, vendor: Oracle Corporation
Java home: /apps/jdk1.8.0_66/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.4.4.el7.x86_64", arch: "amd64", family: "unix"
==========

Done.

artifactory

The second tool in my DevOps arsenal (after Java 8) is Artifactory, a binary repository management system. Its purpose is to store and provide uniform access to versioned resources (files) needed during development, build, and deployments. In fact, I use it as an integration point between various stages of the software lifecycle. If you have used Maven, your dependency repositories probably run on a system powered by Artifactory.

(1) Download the .zip from https://www.jfrog.com/open-source/.

# mkdir -p /data/downloads/artifactory ; cd $_
# wget --output-document=artifactory-4.4.1.zip "http://bit.ly/Hqv9aj"

(2) Installing Artifactory is as simple as extracting its zipped file.

# unzip artifactory-4.4.1.zip -d /apps/artifactory

(3) Configure the home directory: move the following directories to the home directory {backup/, data/, etc/, logs/, run/, support/, webapps/}.

# mkdir -p /data/artifactory/
# mv /apps/artifactory-oss-4.4.1/<directory> /data/artifactory

(4) Update the ports on which Artifactory will run

# vim /apps/artifactory-oss-4.4.1/tomcat/conf/server.xml

(5) Install Artifactory as a service, having it automatically start when the system is booted.

# cd /apps/artifactory-oss-4.4.1/bin
# ./installService.sh

(6) Update the environment variables.

# vim /etc/opt/jfrog/artifactory/default
export ARTIFACTORY_HOME=/data/artifactory
export ARTIFACTORY_USER=artifactory
export JAVA_HOME=/apps/jdk1.8.0_66
export TOMCAT_HOME=/apps/artifactory-oss-4.4.1/tomcat
export ARTIFACTORY_PID=$ARTIFACTORY_HOME/run/artifactory.pid
export JAVA_OPTIONS="-server -Xms512m -Xmx2g -Xss256k -XX:+UseG1GC"
export JAVA_OPTIONS="$JAVA_OPTIONS -Djruby.compile.invokedynamic=false -Dfile.encoding=UTF8 -Dartdist=zip"

(7) SELinux: ensure ports can be accessed

# semanage port -a -t http_port_t -p tcp 18081
# semanage port -a -t http_port_t -p tcp 28081
# semanage port -a -t http_port_t -p tcp 38081

(8) Update IPTABLES so that Artfactory’s Tomcat can be accessible.

# systemctl stop firewalld ; systemctl mask firewalld
# vim /etc/sysconfig/iptables
-A INPUT -p tcp -m tcp --dport 18081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 28081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38081 -j ACCEPT
# systemctl restart iptables
# systemctl unmask firewalld ; systemctl start firewalld

(9) Start Artifactory

# chown -R artifactory: /data/artifactory/
# chmod -R u+rwx artifactory/
# service artifactory start

Now you can configure your repositories as needed. Remember to make them browsable, if possible, anonymously. Some build tools (such as the Maven dependency analyzers) assume as much. It is also a best practice to lock down repositories meant for deployment of internal products.