Continuous Integration/Delivery for Operations… Whaaaa?

Continues Integration (and yes I mean Jenkins) is awesome, plain and simple. CI provides a way for developers to continuously integrate, test and give a degree of confidence that it will work with in other environment(s) to ensure stability. If this tool is so great why aren’t sys admins using it as well? Shouldn’t sys admins test their code, measure code coverage, integrate with other sys admins configs and finally ensuring it will work across environments?

If your a sys admin I know what you’re thinking.. I do test my code and I do ensure it works in our environments. And you’re probably also thinking, Why use a CI tool? What is the point if we’re just writing scripts and doing system configurations?

That’s the point, it isn’t just some script that is written once and forgotten about. Generally the scripts sys admins write are the backbone of any organization and are tucked away in a file system somewhere. It is possible some testing was performed when the script was written, but things change and unless those tests are continuously be run when the script is applied again there is no sure way of knowing something is broken.

Step in CI server. What does this get us? Well, let me make the assumption, silly as it may be, that all of the IT operations scripts are under source control (I know, I know, not a safe assumption, but I want this as another blog post down the road). With that assumption in place, now lets assume the Ops team has also built some basic unit testing around what their script will do. After that we set up a job that will constantly poll our scripts for any changes and should a change occur our tests will run against the script verifying everything still works as expected.

What does this get the Ops team? Now anytime anybody updates the script, known tests are run to verify everything still works as expected and the entire team is notified the status of the script. There is now stability, repeatability, and an overall confidence in a companies infrastructure.

For those of us who hate writing documentation. Having this CI/CD process for scripts also acts as a living document. IT organizations can leverage it also as repository for intellectual capital. Yes, not all the code will live on the Jenkins server, but in an ideal scenario all scripts will be tested there having 1 place for everybody to review the health and assets for the IT Ops team.

Short story long. There is a movement to get more automation, rigor and confidence around doing IT Ops and the only way to get there is by writing good code, which implies good testing. This will help with less rework, unnecessary troubleshooting, loss of intellectual artifacts and allow the team to focus on more interesting things rather than trying to figure out what you did a year ago and why the script is no longer working.

Get in the habit of using CI/CD you and your IT organization will not regret it.


Jenkins and the Ansible plugin

I have been slacking on posting anything as of late and it hasn’t been from lack of topics, but my latest client has been me keeping quite engaged. The project has been really cool. I’ve been leading an installation of Red Hats PaaS environment, Openshift. Part of our engagement has been to demonstrate how DevOps and PaaS can increase developer productivity, quality of code, decrease time to market, and confidence the application will do what is supposed to.

During any installation of Openshift the recommendation is to always stand up DNS server specific for Openshift. The reason is that DNS dynamically adds and removes records as containers are added/removed from the PaaS and most network admins are not keen on allowing these dynamic features in the enterprise DNS servers (and rightfully so). This causes problems though. This means that any computer that logs into the Openshift console and creates a container won’t be able to access the container, because their computer won’t be able to resolve the new DNS entry. The fix is to basically forward on traffic to this server from the enterprise servers. However, since making such a change to a company’s DNS server(s) is not something to take lightly and takes time to evaluate risks and such, you may need to create a workaround like adding the Openshift DNS server to your computers list of DNS resolvers.

So nice workaround, right? Well, yes and no. We have worked around the issue of having the developers computer be able to resolve the correct FQDN, cool. What happens when as part of our DevOps process we leverage enterprise Jenkins server to push code directly to Openshift? You got it, the FQDN won’t resolve. The FQDN will not resolve without the correct DNS server.

What do you do? I mean the current set up is only temporary while the new enterprise DNS servers are set up to forward to the Openshift DNS. We really can’t blindly update the DNS servers for Jenkins, because there are a number of other development teams and quite possibly break their builds.

So what do we do… Yes, you got it. Ansible.

jenkins openshift bridge

We will use Ansible to bridge the Jenkins server to our server running the correct DNS to then remotely push an application to Openshift.You may¬† be asking yourself why not just use ssh? There are a few other little tidbits I didn’t mention and that is Openshift requires a special set of client tools that are not installed on the Jenkins server. There is a Openshift plugin which works, but only with ssh keys. The Jenkins server doesn’t use SSH keys, so no Openshift Plugin. However, even if we could use SSH keys the jenkins server wouldn’t be able to resolve the hostname. Which brings us back to using Ansible to execute commands on the bridge VM and then uses the Openshift client tools and the correct DNS server resolve and push code.

We ran into other issues with using the Ansible plugin that I’ll talk about in another post.