Jenkins distributed build (master-slave configuration)
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed how to install Jenkins in AWS EC2 instance.
https://cloudaffaire.com/how-to-install-jenkins-in-aws-ec2-instance/
In this blog post, we will discuss how to add additional nodes (slaves) in your Jenkins system to make a distributed build architecture.
It is pretty common when starting with Jenkins to have a single server which runs the master and all builds, however, Jenkins architecture is fundamental “Master+Agent”. The master is designed to do co-ordination and provide the GUI and API endpoints, and the Agents are designed to perform the work. The reason being that workloads are often best “farmed out” to distributed servers. This may be for scale, or to provide different tools, or build on different target platforms. Another common reason for remote agents is to enact deployments into secured environments
A “master” operating by itself is the basic installation of Jenkins and in this configuration, the master handles all tasks for your build system. In most cases installing an agent doesn’t change the behavior of the master. It will serve all HTTP requests, and it can still build projects on its own. Once you install a few agents you might find yourself removing the executors on the master in order to free up master resources (allowing it to concentrate resources on managing your build environment) but this is not a necessary step. If you start to use Jenkins a lot with just a master you will most likely find that you will run out of resources (memory, CPU, etc.). At this point, you can either upgrade your master or you can setup agents to pick up the load. As mentioned above you might also need several different environments to test your builds. In this case, using an agent to represent each of your required environments is almost a must.
An agent is a computer that is set up to offload build projects from the master and once setup this distribution of tasks is fairly automatic. The exact delegation behavior depends on the configuration of each project; some projects may choose to “stick” to a particular machine for a build, while others may choose to roam freely between agents. For people accessing your Jenkins system via the integrated website (http://yourjenkinsmaster:8080), things work mostly transparently. You can still browse JavaDoc, see test results, download build results from a master, without ever noticing that builds were done by agents. In other words, the master becomes a sort of “portal” to the entire build farm.
Since each agent runs a separate program called an “agent” there is no need to install the full Jenkins (package or compiled binaries) on an agent. There are various ways to start agents, but in the end, the agent and Jenkins master need to establish a bi-directional communication link (for example a TCP/IP socket) in order to operate.
Next, we are going to create a Jenkins distributed build configuration.
Jenkins master-slave configuration demo:
|
1 2 3 4 5 6 7 8 9 10 |
##---------------------------------------------------------- ## Jenkins Distributed Build (Master-Slave Configuration) ## ##---------------------------------------------------------- ## Systems used for this demo # hostnames ip os role # --------- ------------ -------- ------------ # system1 192.168.0.10 Centos 7 Master Node # system2 192.168.0.20 Centos 7 Slave Node One # system3 192.168.0.30 Centos 7 Slave Node Two |
Master node configuration:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
####################### ## Master Node Setup ## ####################### ## Login to the master node (192.168.0.10) ## Install Java JDK sudo yum install java-1.8.0-openjdk.x86_64 -y ## Set Java home path sudo cp /etc/profile /etc/profile_backup echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile source /etc/profile ## Check java version java -version ## Install Jenkins sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key sudo yum install jenkins -y ## Enable and start Jenkins service sudo systemctl start jenkins sudo systemctl enable jenkins sudo systemctl status jenkins ## Enable 8080 port in firewall sudo firewall-cmd --zone=public --permanent --add-port=8080/tcp sudo firewall-cmd --reload ## Get initial admin password sudo cat /var/lib/jenkins/secrets/initialAdminPassword # 25b1920c63864dcdae85a1a590461360 ## Open browser and navigate to 192.168.0.10:8080 ## Provide initialAdminPassword copied in previous step and complete the setup |
Note: You can follow the previous blog for post-installation configuration part.
Slave node configuration:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
###################### ## Slave Node Setup ## ###################### ## Login to the slave nodes (192.168.0.20, 192.168.0.30) ## Install Java JDK sudo yum install java-1.8.0-openjdk.x86_64 -y ## Set Java home path sudo cp /etc/profile /etc/profile_backup echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile source /etc/profile ## Check java version java -version ## Create a user named jenkins sudo useradd -d /var/lib/jenkins jenkins sudo passwd jenkins ## Generate SSH key on the master node (192.168.0.10) and copy to slave nodes (192.168.0.20, 192.168.0.30) sudo su -s /bin/bash jenkins ssh-keygen -t rsa ssh-copy-id jenkins@192.168.0.20 ssh-copy-id jenkins@192.168.0.30 ## Test ssh connection ssh system2 exit ssh system3 exit exit |
Add nodes in your Jenkins master configuration:
Login to your Jenkins UI and click ‘Manage Jenkins’.
Click ‘Manage Nodes’.
Click ‘New Node’ to add a new node.
Give a name to the node and select ‘Permanent Agent’. Click ‘OK’.
Provide slave node configuration details
Perform the same steps to add the other slave.
We have successfully added to slave nodes in our master Jenkins configuration.
You can test your target nodes by providing the label for your target node.
Hope you have enjoyed this article. To get more details on Jenkins, please refer to Jenkins documentation.