Puppet and Ansible… you’re being weighed, Part 2

Okay this is a follow up on my other post.

Wow, okay both configuration management tools do roughly the same type of thing, but I think they are geared for types users.


I’m sure you being a crawler of technology as I am know how important it is to have good documentation. I think one of the first turn offs to a tool is bad documentation, because it begs the question of does more than one person understand it or complex and there is no other way to represent it.

As I pointed out in my prior post, I had worked with puppet before so I know the documentation was decent, but what I quickly recalled was that the tutorials suck and actually there aren’t any unless you download their VM, which is for VirtualBox and I use KVM. I didn’t really need it because I have awesome books, Pro Puppet by James Turnbull and Jeffrey McCune as well as Puppet Cookbook by Thomas Uphill and John Arundel and http://www.puppetcookbook.com/ . With these tools I was able to become fairly proficient quickly.

When I started looking at Anisble I was taken aback at just the amount of information they had, albeit it was organized well. That part felt kind of intimidating. However, HOWEVER, wow what a pleasure to read! I really can’t reiterate that again. A PLEASURE. The Author(s) wrote a story that slowly built you up from knowing nothing through practically intermediate user of their tool in literally no time. The instructions were funny and had character that made me laugh as I was reading it and had little anecdotes in just the right places where I started to ask myself a question and boom they answer it. Really great stuff and kudos to those guys.


Starting with Pupppet, I stood up the standard Master, Agent topology using the instructions on puppetlabs. With Puppet, it is recommended for the Master to be a CA as well so that it can manage the agents that can communicate with it. And this part was not difficult, but I did not like the fact that I had to review and approve any requests (i think this goes away if I used a tool like Foreman), Any who, next I starting tinkering with the puppet.conf file, but that is elusive in its own right. I mean they only have a handful of options in the conf file, but are not really explained very well in the file itself, so I had some difficulties get it tweaked just right. After I set up the master the agent was much more of a breeze.

Ansible…. easy, easy, easy. Really the only thing you have to do is run

$ yum install ansible

and boom your done. Now you can use it as is and work with nodes, however, it is much nicer to leverage ssh-agent so that Ansible can communicate with nodes without prompting login. This is also very straight forward, however kind of annoying to have to manage ssh keys. I liken this process to the CA process above.

Ease of use:

ehhh. I’m really trying not to sound like a Ansible fan boy here…. As I said, going through the documentation I got a fairly good understanding of the mechanics involved with using it. I would say was up and running and testing stuff in an hour. Puppet on the other hand is a bit more complicated. You have to ensure that your master daemon is running and then on your agent machine you have to make sure it is also running and can connect to the master. I was running it in onetime mode so that I could test each time I made a change on the agent (I could have tested on my master first, but whats the fun in that). Each time I made a change to my puppet script I’d hop over to my agent and run a cli command.

Now the output from the command for Puppet is quite cryptic and what I would expect from a typical complier. The difference with Ansible is that once I finished and script change I could run it right then and there. I also felt the errors were better captured and explained. I think this is really a function of Ansibles decision to execute top-down approach vice dynamic (it may also be a function of python vice ruby exec pref), because it is easier to capture where things fail and consistency. When running Puppet there is no guarantee that the execution ordering will be the same, so you may get an error one pass of the code attempt to fix run it again and get another error fix that error only to run again and find out the original error you thought you fixed isn’t… which leads to grrrrr.


Time to code:

This kind plays in ease of use, but I thought we should have a separate section for it. So, here was my approach before starting this journey of configuration management. I first stood up a separate server and documented all the steps I needed to run my configuration across a set of nodes as if I was going to do it via bash or python. Then I took those steps and translated them into a Ansible and Puppet speak, easy right? hmmm…

Fan boy alert…

Well I’d like to say my experience with both tools was awesome and quick, since really I had done the hard part of figuring how everything should be installed and configured, but it wasn’t. Ansible probably took me a full day of working with the tool and understanding how to debug and the appropriate ways to leverage its tooling, Puppet on the other hand probably took me a minimum of 3 days. Mind you I did Anisble first, the tool I had never used before, so clearly you’d think I would have had an edge on it rather than puppet. I didn’t. It turns out there is a serious issues with dynamic execution when your configuration relies on a particular ordering of execution. Since Puppet is dynamic there are ways you can chain execution commands (resources) together, but it is actually difficult to do and more importantly troubleshoot. Even if you set print (notify) statements before and after each resource command and there is something required before it can run, the execution will drop into that required resource and so on and so forth and all then while you’re still getting other messages as Puppet attempts to run all the attributes of each resource and so even if you have print statements before and after each resource (you have to chain them together!) the output is not clean and difficult to trace. With Ansible being top down execution it was very easy to see what errors existed and what I needed to adjust to get working.

Another thing I found daunting and worth a mention is the idea of environments. So with Anisble you write a playbook and each playbook would be composed of a block of configurations you wanted to happen on a group of nodes and to run a different play book you simply reference it from the command line. This is awesome, because I want to setup and teardown my vm with my configurations so I can fully test everything is working correctly… Well that isn’t so easy with Puppet. Since puppet has a master/agent relationship we can’t have both setup and teardown as part of the communication, otherwise nothing would be configured and couldn’t guarantee which was executed when, so I had to setup what they call is environments. This is analogous to dev, test, prod. The problem is that I have to make this declaration in my puppet.conf file, which got annoying real quick when trying to build this project. I won’t begin to mention my discovery process to figure this out after my perspective from Ansible was skewed to easy 🙂 .


I think there is a place for both of these tools to cohabitate, but I’m not sure my uses cases would necessitate Puppet over Ansible other than if I was forced to use it by a Red Hat product. Puppet is the big kid on the block (for now at least) and is the defacto where as Ansible is the new kid and gaining popularity. I won’t recommend one over the other blanketly, because it honestly depends on use case. What I will say if you are migrating scripts that do some type of configuration management you will be wise to use Ansible, because you WILL have issues with execution order that you never really gave much thought to (unless of course you are the puppet master 🙂 ).

Happy configuration managements!!!!


Puppet and Ansible… you’re being weighed, Part 1

I have been hearing a lot of buzz around Puppet and Ansible for some time. Many many moons ago I tinkered with Puppet and did the little tutorials online but never really had a reason to go much deeper. I figured it was an obligation, since most of the products I use on day to day revolve around Red Hat and Red Hat has more or less adopted Puppet (for better or worse). During the same time Ansible was starting to traction. I really didn’t really see the advantages of using Ansible over Puppet other than the whole agent vice agent less approach.

Well as it turns out my client is in desperate need of a configuration management tool and I truly mean desperate. My experience with configuration management tools is that they are really a nice to have, but most shops have their own way of doing configuration management and when they weigh all the assets they have invested in (software, people, processes, etc) against migrating to a new way of doing things it is just too expensive for them to undertake (for now!).

My client doesn’t have this luxury. They are under staffed and under trained. Off the bat they are behind. If they were trained, maybe they’d be able to just, just keep up with operational demand, but any snag with throw everything behind. So this is the case we all hear about, but possibly never experience in real life where the work people do is not done well so they have to spend additional time fixing what the did putting them further behind on the next task.

So, when a team can’t write/manage complex scripts and they need to troubleshoot every configuration and environment because everything has been done ad-hoc and not to mention deadlines, what do they do? They have no choice but to use a configuration management tool to alleviate them from having to troubleshoot work they’ve already done.

In rolls Puppet and Ansible. My project [1] attempts to highlight the differences between Ansible and Puppet doing the same configurations. My hope is that it will give a good idea of the level of rigor involved in using the tools and to pick the one that is most appropriate for my client.

My next post will talk about some things I found annoying and useful about both tools and what I will inevitably recommend to my client.

[1] https://github.com/jmarley/anisble-puppet-workshop

automation with purpose…

Every engineer, architect, system administrator and project manager all has to make decisions on automation. Albeit, those decisions will be from different perspectives, but we all have to figure out what makes sense. The main reason I’m writing this blog, is that I have been seeing a lot chatter recently about posteriori (dynamic) configurations and grumblings about performance and I know doing more a priori will improve performance.

When does it make sense to do configurations a priori vice posteriori? Obviously it depends on what you’re trying to do. Some things to consider are,

  • Does the configuration(s) need to be dynamic?

When configuring the environment is there anything that needs to be dynamic? For services that have dynamic configurations, how much of the service configuration is actually dynamic vice static? In other words, if I have a dynamic configuration, but only 2 commands out of 50 need to be done dynamically, I may consider setting up static configurations for 48 commands and only doing 2 commands dynamically. There are diminishing returns for this type of approach so if the example was the other way around, then you could have 48 dynamic configurations and 2 static configurations. In this case depending on the scale will dictate whether to spend time doing both static and dynamic configurations; huge number of machines will make dynamic and static approach more reasonable, because even a tenths of a second for each server start up time gets expensive when the number gets high.

  • How many machines will this affect?

Regardless someone will have to design and create the architecture of the environment, but if there are only a few machines spending time creating a dynamic environment can inhibit productivity and will not be worth while.

  • What are the resource requirements for spinning up the environment?

When you’re creating your test machine how long does it take to configure the environment; this will be approximately how much time it will take each environment to dynamically be created.

  • Does this need to be repeatable?

What is life expediency of the machine and/or services? If the life expediency isn’t long then why spend the additional effort making the entire process automated?

The basic take away is always try to do things a priori and when you can’t, only do the configurations that need to happen dynamically, dynamically. Trying to do everything dynamically will be expensive implementing a repeatable solution and computationally expensive. If you aim to leverage automation with a priori configurations I guarantee that you will improve performance and lower costs.

Federate data sources via JBoss Data Virtualization

JBoss Data Virtualization, this product is probably the best hidden secret on the Market.

Anyone who has done anything with data (this includes technical and nontechnical folks) knows how much of a pain in the neck it is to:

  1. locate the data
  2. understand how to access the data
  3. understand the schema of the data
  4. define the transformations of the data
  5. define integration with other data/processes

It is really unfortunate that many businesses have unintentionally made their businesses less productive by creating silo’s of data. Generally this is a byproduct of using different tooling for creating and working data. Sometimes those tooling do not integrate well with others and/or is difficult to use, which is what creates barriers and data silo’s.

I could go on and on, but i know you’re all familiar with this oh so common problem. Wouldn’t it be great if there was a tool which would abstract these difficult API’s and allow me to easily build abstract transformations on top of it. Wouldn’t it be great if I didn’t have to schlep each data source’s data? Wouldn’t it be great to be able to expose this transformed data in an easy to use API that is familiar to almost everybody, i.e. SQL standard. Wouldn’t it also be totally awesome if we could easily expose these transformations in other easily consumable protocols?

Well… there is… JBoss Data Virtualization, this product is probably the best hidden secret on the Market. JDV is the productized version of JBoss Teiid, both are excellent and have vibrant communities.

I have a simply demo that I’ve put together for your viewing pleasure.


I have also have a git project that you’re more than welcome to clone and test it out and any feedback is always welcome.


How to Use Java Expressions in JBoss EAP System Properties

On my current client, they are using JBoss EAP 6.1.1 regardless of my incessant  pressure for them to upgrade to the latest and greatest. Since they are using an older version of EAP they can not leverage Java Expressions in EAP system properties. To some this may not cause a problem, however when your application is trying abstract, say the name of attributes that are vaulted across different environments and/or leverage the system property in more than place within the app servers configuration; there are other use cases as well.

I applied a patch that provides this functionality, but there was no clear direction on how to actually add Java expressions to the configuration file and after much chagrin I was able to decipher how to do it and am going to share it for others.

How to use system properties with vaulted attributes

Create keystore

keytool -genkeypair -v -alias alias -keyalg RSA  -keysize 2048 -dname "cn=blah,ou=device, ou=service" -keypass password -keystore myserver.keystore -storepass password

Add servers keystore to vault as an attribute

 $JBOSS_HOME/bin/vault.sh --keystore vault.keystore --keystore-password 'vault-password' --alias vault --enc-dir  --salt 11111111 --iteration 10 -a keystore_name -x 'myserver.keystore'

Vault Block:vb
Attribute Name:keystore_name
Configuration should be done as follows:
Vault Configuration in AS7 config file:
  <vault-option name="KEYSTORE_URL" value="vault.keystore"/>
  <vault-option name="KEYSTORE_PASSWORD" value="MASK-1dmPKRStsI..BzvEbFkZi"/>
  <vault-option name="KEYSTORE_ALIAS" value="vault"/>
  <vault-option name="SALT" value="11111111"/>
  <vault-option name="ITERATION_COUNT" value="10"/>
  <vault-option name="ENC_FILE_DIR" value=""/>
</vault><management> ...

Add keystore password to vault

$JBOSS_HOME/bin/vault.sh --keystore vault.keystore --keystore-password 'vault-password' --alias vault --enc-dir  --salt 11111111  --iteration 10 -a keypass -x 'password'

Vault Block:vb
Attribute Name:keypass
Configuration should be done as follows:
Vault Configuration in AS7 config file:
  <vault-option name="KEYSTORE_URL" value="vault.keystore"/>
  <vault-option name="KEYSTORE_PASSWORD" value="MASK-1dmPKRStsI..BzvEbFkZi"/>
  <vault-option name="KEYSTORE_ALIAS" value="vault"/>
  <vault-option name="SALT" value="11111111"/>
  <vault-option name="ITERATION_COUNT" value="10"/>
  <vault-option name="ENC_FILE_DIR" value=""/>
</vault><management> ...

Add user

$JBOSS_HOME/bin/add-user.sh --silent admin admin.2015

Start server


Add system property <kstore-name>

$JBOSS_HOME/bin/jboss-cli.sh --user=admin --password=admin.2015 -c controller=${HOSTNAME}:9999 --command='/system-property=kstore-name:add(value=$\\{VAULT::vb::keystore_name::1\})'

Add ssl

$JBOSS_HOME/bin/jboss-cli.sh --user=admin --password=admin.2015 -c controller=${HOSTNAME}:9999 --command="${add_keystore_cmd}"

tail logs

tail -f $JBOSS_HOME/standalone/log/server.log &

reload server

$JBOSS_HOME/bin/jboss-cli.sh --user=admin --password=admin.2015 -c controller=${HOSTNAME}:9999 --command="reload"

How to secure bash command line history

Lately I have become of aware of an unsafe practice when maintaining and implementing Jboss EAP servers (or any servers really). The idea is that whatever commands that are executed as a particular user are persisted to the ~/.bash_history file. Of course history can be disabled for particular users, but not all system administrators take this into consideration and hence the concern.

Why is this dangerous

If there is a security breach into a system, say with sudo privileges, the attacker will not only have access into the current system, but also bash_history files. The bash_history wrt JBoss EAP, is that the server uses several cli tooling to create server certificates, management users, vaults, and vaulted attributes and if care isn’t taken can lead to information leaks.


As a user I create a keystore, vault and add secure vault attributes

Login as good guy

[root@mybox ~]# sudo su - JbossAdm

Create keystore

[JbossAdm@mybox jboss-eap-6.3]$ keytool -genseckey -v -alias vaultAlias -keyalg AES -keysize 128 -storetype jceks -dname "CN=something" -keypass vaultKeyPass -keystore vault.jks -storepass vaultKeyPass

Create vault

[JbossAdm@mybox jboss-eap-6.3]$ ./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 -c

Create vault attribute

[JbossAdm@mybox jboss-eap-6.3]$ ./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 --attribute certKeystorePass --sec-attr hostKeyPass

Add management user

[JbossAdm@mybox jboss-eap-6.3]$ ./bin/add-user.sh --silent -u admin -p admin.2015

Nefarious user Login

It is fair to say, having root access alone does not guarantee visability into the passwords of vault, keystore or attributes. Look how easy it is to get passwords for JBoss EAP and server certificate store. With this information they can do untold damange.

Interogate bash history for keywords

[root@mybox ~]# grep '.*vault.*\|.*keytool.*\|.*add-user.*' /home/JbossAdm/.bash_history
./bin/add-user.sh --silent -u admin -p admin.2015
keytool -genseckey -v -alias vaultAlias -keyalg AES -keysize 128 -storetype jceks -dname "CN=something" -keypass vaultKeyPass -keystore vault.jks -storepass vaultKeyPass
./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 -c
./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 --attribute certKeystorePass --sec-attr hostKeyPass

How to prevent (at least slow down nefarious acts)

Since this scenario is in a data center that has been around for some time and who knows what has been run prior to this box, first state of affairs to clear history.

Remove only suspect values

This looks complicated, but it cats the history file, reverses order then greps for key values and the cuts the number associated with, then loops through and deletes each of the records found.

[JbossAdm@mybox ~]$ histnum=$(history | tac | grep '.*vault.*\|.*keytool.*\|.*add-user.*|.*jboss-cli.*' | sed 's/^[ ]*//;s/[ ].*//;p')
[JbossAdm@mybox ~]$ for del in $histnum; do  history -d $del; done
[JbossAdm@mybox jboss-eap-6.3]$ history -w
tac is needed other wise the ordering is inaccurate after the first removal and will extend past the total number of elements (in some cases), but with reverse order always starts with the last value and works to smallest ensuring they are always there.

Alternately, we could clear current history

[JbossAdm@mybox jboss-eap-6.3]$ history -c
[JbossAdm@mybox jboss-eap-6.3]$ history -w


Now, we can do this each time we have a server configured, but the ideal solution is to have it built as part of the VM profile so it is handled automatically. For more information on HISTIGNORE visit reference [1].

[JbossAdm@mybox jboss-eap-6.3]$ HISTIGNORE='keytool[ ]*':'*vault*':'*add-user*':'*jboss-cli*'
[JbossAdm@mybox jboss-eap-6.3]$ export HISTIGNORE

Execute commands again

I am supressing the output, because it really isn’t important to this example.

[JBossAdm@mybox jboss-eap-6.3]$ keytool -genseckey -v -alias vaultAlias -keyalg AES -keysize 128 -storetype jceks -dname "CN=something" -keypass vaultKeyPass -keystore vault.jks -storepass vaultKeyPass
[Storing vault.jks]
[JbossAdm@mybox jboss-eap-6.3]$ ./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 -c
[JbossAdm@mybox jboss-eap-6.3]$ ./bin/vault.sh --keystore vault.jks --keystore-password vaultKeyPass --alias vaultAlias --salt 12457898 --iteration 15 --attribute certKeystorePass --sec-attr hostKeyPass
[JbossAdm@mybox jboss-eap-6.3]$ ./bin/add-user.sh --silent -u admin -p admin.2015

list history

As we expect we see no values with vault, keytool or add-user

[root@mybox ~]# grep '.*vault.*\|.*keytool.*\|.*add-user.*' /home/JbossAdm/.bash_history


For the experienced and dedicated it is merly a road bump, but that is okay. This is merly meant to slow down and deter any nefarious activity.

Docker: Dockerfile oddities

I have been itching for a reason to create a project using docker. I wanted to get more hands on experience than the [well] documented tutorials from docker, because stuff always, always is left off. A few weeks ago I was asked to give a demo of Jboss Data Virtualization to some system integrators. Anybody familiar with JDV knows it is an extremely powerful tool with many use cases and almost immediately I’m thinking leveraging docker containers to isolate each use case. This would allow me to spin up these demos in a matter of seconds and don’t really have to worry about issues that always happen when running demos on host box.

Reason to use Docker…Check

Know how of Docker…ehhh Check

Use Cases… Check . It turns out Teiid, our community project of JDV has a slew of quickstarts that hit more or less all the use cases I would want to show and quite thoroughly documented. I start here trying to dockerize these quickstarts for the community.

Great! Things are going well and I’m working with our technical marketing team leveraging all their knowledge and minimizing the amount of rework.

Then, there is an oddity. Part of our instructions calls a script to configure the server, however the server needs to be running first before it can be configured. Clearly this is easy start the server then execute the script against the running server…

snippet of my project

RUN $DV_HOME/jboss-eap-6.1/bin/standalone.sh -c standalone.xml -b -bmanagement && \
$DV_HOME/jboss-eap-6.1/bin/jboss-cli.sh --connect --file=$DV_HOME/jboss-eap-6.1/teiidfiles/scripts/setup.cli


AFAIK, this will not work, Dockerfile layers (execution commands) are run in isolation and can’t have concurrent process running. Which kind of makes sense if there is unneeded complexity and whomever is pulling and building the image may be time and resource hog. But really who cares. Once the image is built its not like it has to be be built again and again once it is in your repository. Basically, due to this limitation I first need to run the server externally to create my custom server configurations then take that file and simply replace the current configurations.

Now it isn’t a big deal, but now instead of putting the burden on the container to do I now need to do an additional step of preparation, maintenance, documentation for this docker image.

With that said, once the image is built it is super nice to spin up a container on the fly and fast.