The primary application (Geldzin) I’ll be managing in this devops environment requires an FTP location to post nightly backups. Therefore we need to setup SFTP services and a user that the application will use for this purpose.

(1) Directory, usually on a partition with a lot of space, where FTP files will exist.

# mkdir -p /data/ftp

(2) SFTP access should be restricted to users of the “sftpusers” group.

# groupadd sftpusers

(3) Update SSH configuration and setup the chroot jail.

# vi /etc/ssh/sshd_config
*****
Subsystem sftp internal-sftp
// Add these at the bottom
Match Group sftpusers
ChrootDirectory /data/ftp/%u
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp
*****

(4) SELinux permissions to allow sftpd to write to the FTP directories

# setsebool ftp_home_dir=1
# semanage fcontext -a -t public_content_t /data/
# restorecon -R -v /data/ftp
# setsebool -P allow_ftpd_anon_write on

(5) Restart the service

systemctl restart sshd

(6) Setup the user account the application will use.

# useradd -m geldzin -d /home/geldzin -s /sbin/nologin -g sftpusers
# passwd geldzin
# mkdir -p /data/ftp/geldzin/files
# semanage fcontext -a -t public_content_rw_t "/data/ftp(/.*)?"
# restorecon -R -v /data/ftp
# chown -R geldzin:sftpusers /data/ftp/geldzin/*
# chmod -R 700 /data/ftp/geldzin/*

The only catch here is that users cannot write to /data/ftp/%u directly (they don’t own it), but should be able to write to directories under that folder (such as /files). The chroot jail ensures users don’t escape their sandbox and write elsewhere on the system.

(7) Test the connection from a remote host or SFTP client

# sftp geldzin@workhorse

(8) You might experience issues connecting to the SFTP server, something like this:

packet_write_wait: Connection to 192.168.x.xxx: Broken pipe
Connection closed

And the security log () might show something similar to:

sshd[22909]: pam_unix(sshd:session): session opened for user geldzin by (uid=0)
sshd[22909]: fatal: bad ownership or modes for chroot directory component "/data/" [postauth]
sshd[22909]: pam_unix(sshd:session): session closed for user geldzin

This issue indicates an ownership problem. sshd will reject SFTP connections to accounts that are set to chroot into any directory that has ownership/permissions that sshd considers insecure. sshd’s strict ownership/permissions requirements dictate that every directory in the chroot path must be owned by root and only writable by the owner. So we fix the problem in this case with:

chown root:root /data/
# chmod 755 /data/

You should be in business.

For most of the Java web applications I have developed, my go-to web server has been Apache Tomcat. Although I have not had much success with the Tomcat plugin in my Maven projects (instead I use the Jetty Maven plugin during development), overall I have had little trouble using Tomcat for production.

(1) Setup where Tomcat will be installed on your local machine.

# cd /apps/
# mkdir apache-tomcat-8.0.32

(2) Create “tomcat” group and user.

# groupadd tomcat
# useradd -M -s /bin/nologin -g tomcat -d /apps/apache-tomcat-8.0.32 tomcat

(3) Download and install Tomcat (https://tomcat.apache.org/download-80.cgi).

# cd /data/downloads
# mkdir tomcat ; cd $_
# wget http://ftp.wayne.edu/apache/tomcat/tomcat-8/v8.0.32/bin/apache-tomcat-8.0.32.tar.gz
# tar xvf apache-tomcat-8.0.32.tar.gz -C /apps/apache-tomcat-8.0.32 --strip-components=1

(4) Configure permissions and access.

# cd /apps/apache-tomcat-8.0.32
# chgrp -R tomcat conf ; chmod g+rwx conf ; chmod g+r conf/*
# chown -R tomcat webapps/ work/ temp/ logs/

(5) Setup the service.

# vim /etc/systemd/system/tomcat8.service
=====
# Systemd unit file for tomcat
[Unit]
Description=Apache Tomcat 8 Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=JAVA_HOME=/apps/jdk1.8.0_66/jre
Environment=CATALINA_PID=/apps/apache-tomcat-8.0.32/temp/tomcat.pid
Environment=CATALINA_HOME=/apps/apache-tomcat-8.0.32
Environment=CATALINA_BASE=/apps/apache-tomcat-8.0.32
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/apps/apache-tomcat-8.0.32/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID

User=tomcat
Group=tomcat

[Install]
WantedBy=multi-user.target
=====
# systemctl daemon-reload

(6) Setup port security for Tomcat 8.

# vim conf/server.xml
=====
<Server port="38083" shutdown="SHUTDOWN">
<Connector port="18083" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="48083" />
<Connector port="28083" protocol="AJP/1.3" redirectPort="48083" />
=====
# vim /etc/sysconfig/iptables
=====
-A INPUT -p tcp -m tcp --dport 18083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 28083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38083 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 48083 -j ACCEPT
=====
# systemctl restart iptables

(7) Configure SELinux access

# semanage port -a -t http_port_t -p tcp 18083
# semanage port -a -t http_port_t -p tcp 28083
# semanage port -a -t http_port_t -p tcp 38083
# semanage port -a -t http_port_t -p tcp 48083
# semanage fcontext -a -t 'httpd_cache_t' '/apps/apache-tomcat-8.0.32(/.*)?'
# restorecon -Rv /apps/apache-tomcat-8.0.32
# setsebool -P httpd_builtin_scripting 1

(8) Start Tomcat 8 at boot.

# chkconfig tomcat8 on

(9) Start, restart, and stop Tomcat 8.

# systemctl start tomcat8
# systemctl restart tomcat8
# systemctl stop tomcat8

(10) Check the log

# cat /apps/apache-tomcat-8.0.32/logs/catalina.out

(11) Check that service is running

# netstat -tnlp | grep 8083

(12) Tunnel through firewall

# firewall-cmd --zone=public --add-port=18083/tcp --permanent
# firewall-cmd --zone=public --add-port=28083/tcp --permanent
# firewall-cmd --zone=public --add-port=38083/tcp --permanent
# firewall-cmd --zone=public --add-port=48083/tcp --permanent
# firewall-cmd --reload

(13) Configure the web management console.

# vim /apps/apache-tomcat-8.0.32/conf/tomcat-users.xml
=====
<tomcat-users>
 <user username="tomcat8" password="*****" roles="manager-gui,admin-gui"/>
</tomcat-users>
=====
# systemctl restart tomcat8

The manager application would then be accessible at http://localhost:18083/manager/html, and the virtual host manager at http://localhost:18083/host-manager/html.

One of the main reasons for DevOps is to manage delivery of software into production in a controlled manner. I use Jenkins to build the software from sources, test it, profile the builds, generate documentation, and stage it for deployment to any environment on schedule or on-demand. In fact, when all tests pass, the code is automatically promoted into subsequent environments via Github after a development code review. As a continuous integration tool, post-development tasks are now automated.

(1) Install or update the Jenkins repo

# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
# rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key

(2) Install Jenkins

# yum install jenkins

(3) I’m in the habit of not using default ports and locations

# vim /etc/sysconfig/jenkins
*****
JENKINS_HOME="/data/jenkins"
JENKINS_PORT="18082"
JENKINS_AJP_PORT="28082"
*****
# mkdir -p /data/jenkins
# chgrp -R jenkins /data/jenkins
# chown -R jenkins:jenkins /data/jenkins
# chmod -R 775 /data/jenkins

(4) Allow port for remote access

# vim /etc/sysconfig/iptables
*****
-A INPUT -p tcp -m tcp --dport 18082 -j ACCEPT
*****
# systemctl restart iptables

(5) Configure SELinux

# semanage port -a -t http_port_t -p tcp 18082
# semanage port -a -t http_port_t -p tcp 28082
# semanage fcontext -a -t 'httpd_cache_t' '/data/jenkins(/.*)?'
# restorecon -Rv /data/jenkins
# setsebool -P httpd_builtin_scripting 1

(6) Configure Jenkins to auto-start at boot

# chkconfig jenkins on

(7) Starting and stopping Jenkins

# service jenkins start
# service jenkins stop

(8) Know where the logs are

# cat /var/log/jenkins/jenkins.log

(9) Check that service is running

# netstat -tnlp | grep 8082

(10) Fix access through the system firewall, or Jenkins will not be accessible over HTTP

# firewall-cmd --zone=public --add-port=18082/tcp --permanent
# firewall-cmd --reload

Et voilà, c’est tout que vous devez faire!

A lot of the applications I develop these days require some kind of database, and for years, Oracle’s MySQL has been my go-to database application. Because of this need, it is fitting to install it in the DevOps environment.

(1) Install MySQL community repository

# yum install http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
# yum search 'mysql' | grep 'mysql-'
# yum info mysql-community-server // Shows version 5.6
# yum install mysql-community-server

yum will install MySQL and setup the log file at /var/log/mysqld.log, and the configuration file at /etc/my.cnf.

(2) Change the location of the data file (usually to a partition that has more space).

# vim /etc/my.cnf
*****
datadir=/data/mysql/data
*****

Update the permissions on the data directory.

# mkdir -p /data/mysql/data
# chgrp -R mysql /data/mysql
# chown -R mysql:mysql /data/mysql
# chmod -R 775 /data/mysql

(3) Allow MySQL to be accessed via its TCP port. Many applications use network URLs (such as jdbc:mysql://localhost:3306/sakila?profileSQL=true) to connect to local databases.

# vim /etc/sysconfig/iptables
*****
-A INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
*****
# systemctl restart iptables

(4) Additionally, SELinux is configured by default on CentOS 7. So we need to allow the use of port 3306 and the alternate data directory we set up earlier.

# semanage port -a -t mysqld_port_t -p tcp 3306
# semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
# restorecon -Rv /data/mysql/data

(5) Often you need MySQL to start automatically when the system boots up.

# systemctl enable mysqld.service

(6) How to start, restart, or stop the MySQL service.

# systemctl start mysqld
# systemctl restart mysqld
# systemctl stop mysqld

(7) Set the root password for the first time, or change the root password.

# mysqladmin -u root password <new-password>
# mysqladmin -u root -p <current-password> <new-password>

(8) Now you can log in locally, which will prompt for a password.

# mysql -u root -p

This is how simple the setup is. At this point, you can choose to further secure and tune the installation to your needs, but because this will only be a test/development database, I see not need for it here.

When I first started developing software, Apache Subversion was what small-scale developers like myself used for source code management (SCM). I then worked for a company that used IBM Rational ClearCase for SCM, in fact, doing configuration management. Just over a year ago, I decided to learn Git and it has become my de-facto SCM on recent projects. Thus I need it in my DevOps environment.

YUM repositories do not always have the latest Git, so you will need to install the latest yourself.

(1) Install required packages

# yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils-MakeMaker
# yum remove git

(2) Download the latest .tar, check https://www.kernel.org/pub/software/scm/

# mkdir /data/downloads/git
# cd /data/downloads/git
# wget https://www.kernel.org/pub/software/scm/git/git-2.7.0.tar.gz
# tar xzf git-2.7.0.tar.gz

(3) Install to location of choice

# cd git-2.7.0
# make prefix=/apps/git-2.7.0 all
# make prefix=/apps/git-2.7.0 install
# echo "export PATH=$PATH:/apps/git-2.7.0/bin" >> /etc/bashrc
# source /etc/bashrc
# git --version
*****
git version 2.7.0
*****

That’s it.

maven

The third tool in my home business DevOps is Apache Maven, a dependency management and project build tool. It is particularly well-suited for my development needs, since I develop mostly Java software. I need it on the DevOps machine because I’ll be doing nightly and on-demand builds of the software there.

To install Maven:
(1) Visit https://maven.apache.org/download.cgi and grab the URL for the latest binary package (3.3.9 at this writing) from a mirror.

(2) Prepare the local home for Maven, and download the package.

# mkdir /data/downloads/maven ; cd $_
# wget http://apache.claz.org/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

(3) Install Maven and setup the local repository.

# tar xvf apache-maven-3.3.9-bin.tar.gz
# mv apache-maven-3.3.9/ /apps/
# mkdir -p /data/maven/repo
# chmod 777 /data/maven/repo

(4) Configure Maven’s global environment variables.

# vim /etc/profile.d/maven.sh
==========
export M2_HOME=/apps/apache-maven-3.3.9
export M2_REPO=/data/maven/repo
export PATH=$M2_HOME/bin:$PATH
==========
# source /etc/profile.d/maven.sh

(5) Update the global Maven settings file with the repository information as well, for applications (such as Jenkins) that do not read it from environment variables.

# vim /apps/apache-maven-3.3.9/conf/settings.xml
==========
 <localRepository>/data/maven/repo</localRepository>
==========

Now you can check that Maven is installed.

# mvn -v
==========
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T09:41:47-07:00)
Maven home: /apps/apache-maven-3.3.9
Java version: 1.8.0_66, vendor: Oracle Corporation
Java home: /apps/jdk1.8.0_66/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.4.4.el7.x86_64", arch: "amd64", family: "unix"
==========

Done.

artifactory

The second tool in my DevOps arsenal (after Java 8) is Artifactory, a binary repository management system. Its purpose is to store and provide uniform access to versioned resources (files) needed during development, build, and deployments. In fact, I use it as an integration point between various stages of the software lifecycle. If you have used Maven, your dependency repositories probably run on a system powered by Artifactory.

(1) Download the .zip from https://www.jfrog.com/open-source/.

# mkdir -p /data/downloads/artifactory ; cd $_
# wget --output-document=artifactory-4.4.1.zip "http://bit.ly/Hqv9aj"

(2) Installing Artifactory is as simple as extracting its zipped file.

# unzip artifactory-4.4.1.zip -d /apps/artifactory

(3) Configure the home directory: move the following directories to the home directory {backup/, data/, etc/, logs/, run/, support/, webapps/}.

# mkdir -p /data/artifactory/
# mv /apps/artifactory-oss-4.4.1/<directory> /data/artifactory

(4) Update the ports on which Artifactory will run

# vim /apps/artifactory-oss-4.4.1/tomcat/conf/server.xml

(5) Install Artifactory as a service, having it automatically start when the system is booted.

# cd /apps/artifactory-oss-4.4.1/bin
# ./installService.sh

(6) Update the environment variables.

# vim /etc/opt/jfrog/artifactory/default
export ARTIFACTORY_HOME=/data/artifactory
export ARTIFACTORY_USER=artifactory
export JAVA_HOME=/apps/jdk1.8.0_66
export TOMCAT_HOME=/apps/artifactory-oss-4.4.1/tomcat
export ARTIFACTORY_PID=$ARTIFACTORY_HOME/run/artifactory.pid
export JAVA_OPTIONS="-server -Xms512m -Xmx2g -Xss256k -XX:+UseG1GC"
export JAVA_OPTIONS="$JAVA_OPTIONS -Djruby.compile.invokedynamic=false -Dfile.encoding=UTF8 -Dartdist=zip"

(7) SELinux: ensure ports can be accessed

# semanage port -a -t http_port_t -p tcp 18081
# semanage port -a -t http_port_t -p tcp 28081
# semanage port -a -t http_port_t -p tcp 38081

(8) Update IPTABLES so that Artfactory’s Tomcat can be accessible.

# systemctl stop firewalld ; systemctl mask firewalld
# vim /etc/sysconfig/iptables
-A INPUT -p tcp -m tcp --dport 18081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 28081 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38081 -j ACCEPT
# systemctl restart iptables
# systemctl unmask firewalld ; systemctl start firewalld

(9) Start Artifactory

# chown -R artifactory: /data/artifactory/
# chmod -R u+rwx artifactory/
# service artifactory start

Now you can configure your repositories as needed. Remember to make them browsable, if possible, anonymously. Some build tools (such as the Maven dependency analyzers) assume as much. It is also a best practice to lock down repositories meant for deployment of internal products.

In CentOS7, IPTABLES is the firewall, yet another security layer you must consider when you install your applications, especially to non-standard ports. To install it:

# yum install iptables-services

To edit the IPTABLES policy, you need sudo access:

# sudo vim /etc/sysconfig/iptables

Other quick tasks you will need on hand include starting and stopping, and masking and unmasking:

# systemctl stop firewalld
# systemctl mask firewalld
# systemctl unmask firewalld
# systemctl start firewalld

Reference:
https://wiki.centos.org/HowTos/Network/IPTables
https://www.centos.org/docs/5/html/5.2/Deployment_Guide/ch-iptables.html
https://community.rackspace.com/products/f/25/t/4504

The latest serious distributions of Linux now come armed with SELinux. To most old-school Linux users, this is a nuisance akin to Microsoft’s User Account Control (UAC) that we all turned off right after installing Windows. But SELinux ought to be embraced, if at least for the common-sense security it provides out-of-the-box.

To check whether the SELinux module is enabled, flip the shell:

# getenforce

It should echo “Enforcing” if the module is enabled, or “Disabled” if not. With root privileges, you can turn it on and off by calling setenforce with 1 or 0, respectively. (see http://bit.ly/1Wmt5kD). On CentOS7 and probably other Linux distros, it is cumbersome to work directly with the SELinux module; you need a policy manager that makes it easy to configure the module. The most popular is SEMANAGE (see http://do.co/1Wmu7Nj on usage).

# yum -y install policycoreutils-python

Since my DevOps machine will be handing a fair amount of HTTP/web traffic, the first thing I check is which ports are allowed to do HTTP at all. It is not enough to bind some application server to a port; you must explicitly also allow it by SELinux policy (and as we’ll see later, by IPTABLES, yet another security layer).

# semanage port -l | grep http
*****
http_cache_port_t tcp 8080, 8118, 8123, 10001-10010
http_cache_port_t udp 3130
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988
pegasus_https_port_t tcp 5989

So the Linux administration best practice to remember is to review the security policies affecting resources used by applications running on your box. SELinux doesn’t just control ports; file systems and partitions too. In configuring my server, I created a couple of non-standard partitions that will host VMs and backups. But to use them for such required adding the appropriate policy to SELinux. It’s here to stay (for good reason), deal with it.

My home business develops mostly Java software, so we obviously need Java 8 on the DevOps machine (which runs CentOS 7). Although OpenJDK is virtually compatible with the Oracle version, and many businesses are moving to it (from the fallout of Google vs. Oracle over API copyrights), I still prefer the Oracle version of the JVM. So that is what I will install.

(1) There might already be Java installed on the system. Use yum to see installed Java packages.

# yum list installed 'java'

(2) yum makes it easy to find packages to install. You might need to install a package repository that is updated often and thus has the latest Oracle JDKs.

# yum search java | grep 'java-'

(3) If you want the very latest, download the binary from Oracle directly.

# cd /data/downloads/java 
# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u66-b17/jdk-8u66-linux-x64.tar.gz"

(4) Extract the TAR file in the location where Java will be installed. This manual method gives you more control of where things are installed.

# cd /apps/
# tar zxf /data/downloads/java/jdk-8u66-linux-x64.tar.gz

(5) Configure the Java installation.

# cd /apps/jdk1.8.0_66/
# alternatives --install /usr/bin/java java /apps/jdk1.8.0_66/bin/java 2
# alternatives --install /usr/bin/jar jar /apps/jdk1.8.0_66/bin/jar 2
# alternatives --install /usr/bin/javac javac /apps/jdk1.8.0_66/bin/javac 2
# alternatives --set java /apps/jdk1.8.0_66/bin/java
# alternatives --set jar /apps/jdk1.8.0_66/bin/jar
# alternatives --set javac /apps/jdk1.8.0_66/bin/javac

(6) Set your environment variables (bash profile, that is). You can also get this Java globally, but it is not advisable if you might have various JDKs installed, or some of your applications require specific versions (different from the globally declared).

# vi ~/.bash_profile

Add the following to the profile.

export JAVA_HOME=/apps/jdk1.8.0_66
export JRE_HOME=/apps/jdk1.8.0_66/jre
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin

(7) Test the Java version installed

# java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

That’s all. Keep tabs on the JAVA_HOME variable, as lots of Java applications will need to know it.