Couchbase and Teiid Bang!

Over the last few months I’ve been creating communities within our consulting teams. There are a lot of consultants that are very interested in JBoss Data Virtualization (a.k.a. Teiid in the JBoss Community)  and over the last few months I’ve been putting together a team of consultants with varying interests to get them familiar with this project/product.

So, for those of you who may not be familiar with Teiid(JDV), it has many powerful features and one of the most powerful is the ability to extend the product. Teiid comes with many translators that can bridge ANSI sql queries to NoSQL, traditional SQL, flat files and more types of data sources, however, some translators don’t exist for all data sources. In comes the community: massive exaggerated CHEERING. queue sound byte.  My team and I have been focused on extending Teiid to be able to make ANSI SQL translations between Couchbase, which itself is a layer on top of Couchdb. There is a thread as to why I ended up choosing Couch.

Also we choose Couchbase primarily from its ease of use. Couchbase comes with NQL, which is a very SQLish like api, so doing the translations from SQL to NQL is much simpler rather than working with a who new api with lots of switches and exceptions and most important having to learn it all before we start. My main goal was/is to build a foundation where we can branch out into more complicated data sources.

Here we are. It has taken a bit of time to get the project stood up in a repeatable fashion, such that I can add new team members without much time for set up. We have docker images set up for JDV, and Couchbase so it is a matter of just issuing a few docker commands to get the servers up. Currently, we have the resource adapter working and will soon be actually doing the translations with NQL. To get started with this project (please help 🙂 ) you can pull from the main teiid branch.

$ mkdir teiid-couchbase

$ cd teiid-couchbase

$ git init

$ git remote add upstream

$ git checkout -b couchbase upstream/8.12.x

Now you have the source for couchbase translator, now lets set up your environment

$ git clone

You should be able to follow the instructions on standing it up. If you see any issues let me know and do a pull request. That’s it! I’ll update when I have some news about this project.


Docker and Systemd Don’t Jive

What is the hubbub with this, Docker is awesomesauce you may say. Docker is awesomesauce…. for most things and reasonable expectations. Generally I don’t write about all my blunders because I would never have time to get anything done :). This one took the cake though.

I was working through a new security hacking book called Black Hat Python (which I highly recommend) and was happily hacking away when I got to a section where I was testing a basic proxy server. The idea was to use a basic proxy server to redirect traffic from a known service through the proxy to my remote target destination. In order for me to test out the code I wanted to set up a test ftp server. I thought, hey, this is a good opportunity for me to use docker. I’m familiar with docker, but by means an expert. I could have much easier just stood up a vm and been on my way. But nooooo.

I also decided I’d use the latest release from Fedora, Fedora 23 as my base. Again, it crossed my mind to just spin up a cloud image install the package and be on my… Okay, simple tell my DockerFile to install vsftp.


MAINTAINER Jason Marley <>
RUN dnf -y update && dnf clean all
RUN dnf install vsftpd -y

USER root

RUN systemctl enable vsftpd && systemctl start vsftpd

# Expose Ports

CMD ["/bin/bash"]


Super simple. Theoretically, once I start my docker container after the build I should have an ftp service I can set up my proxy to.

$ sudo docker build -t jmarley/fedora_vsftp DockerFile-fvsftp .</pre>


Nope. You’ll get an error like below.


$ sudo docker run -d -t jmarley/fedora_vsftp
Error response from daemon: Cannot start container 2f75e4425e7be7b49183e67e39bfc065c8fbe0523afb214e14c17abef19e6b6b: [8] System error: exec: "/usr/bin/systemctl start vsftpd": stat /usr/bin/systemctl start vsftpd: no such file or directory


And you’re saying, wait a second when I ran my build everything built as I expected with no errors and I know that the service vsftpd installed and is enabled. Here is the deal, since the switch from System V to System D there is the D-Bus service managing the service and if your docker container is doesn’t install and active the necessary services for D-Bus then you’ll see this error. Dan Walsh did a good write up on it as well .

On to fixing, so what do I do. Well you just use a cloud-init image, which probably makes more sense, but if you’re stubborn you’ll keep pressing. The solution is to add another layer that adds all the systemd dependencies so they are available to the container. You can see examples here .

Going through all the effort and searching for the issues I encountered above I should have simply stood up a cloud image and added the necessary services I needed. In my next blog or two I will dive into Fedora’s cloud images and why I like them. Also one of my current community projects is starting to get some steam and wanted to do a series on it as it deals with quite a lot of the stack

Happy Hacking

PaaS, Feelin the Waters

You don’t know what you don’t know. period.

What is the best way to find out what you don’t know? Doing it!

I have been on several engagements implementing OpenShift and most companies completely underestimate the amount of commitment needed to implement it successfully.

What is/should be the #1 goal of setting up a PaaS in your data center? The people and system(s) that need/want to use your services. If you don’t have customers that are interesting in using your services a PaaS solution probably isn’t a wise investment for your team. Intuitive right? Well most of us would like to think so, but more often than not leaders get caught up in the shiny new thing. Heck, I’m guilty of it. Whenever a new software tool comes out I’m all over it. It’s cool and exciting.

What is wrong with cool and exciting? Noting, absolutely nothing. Actually it is the grease that keeps our technical innovation moving. There is a problem though when someone blindly says, hey i just saw this new shiny thing that may work for us, so lets go ahead and fully implement and invest a lot of time and energy into this new thingy. While you can clearly see where I’m going and this is intentional, this sort of silly thing happens all the time.

Now, we talk PaaS. So, given my basic example from above, for talented IT teams there shouldn’t be an issue to standing up a new service or providing a new tool to the business. The problem is PaaS is not just “a” service . It is a multitude of services all working together in concert. And those services are all layered on top of different tooling and technologies. To configure, manage and maintain said services the administrators need to be fluent in each of those technologies and unless your team has some exceptionally clever people expecting them to pick up and run with this is foolhardy.

Okay, suppose we have a team of exceptional people, current with the latest tools/technologies what next? Good question. Well we would ask what are we (PaaS) doing here? What is PaaS making more efficient or services it will be replacing or creating? Suppose PaaS will be doing all of the above. Sweet! Are there plans for transition/decommission/migration plans for existing services?

It all boils down to expectations. The goal with something like this should be to test the waters and understand the commitment that is needed for running a successful PaaS. With that said, set expectations low. Not that you should expect the tool to not work as intended, but don’t jump in the water until you know what your jumping into.

Here are the steps I would follow if I were consider a PaaS environment for my data center (if I owned/lead one):

  1. Justify need; establish use cases. This may be directly related to current and recurring needs/issues/complaints.
  2. Define stakeholders
  3. Review personal resources (developers,administrators,testers, security, network, etc); technical specialties, workloads and adaptability.
  4. Pick a simple use case to test the waters
  5. Require all requirements needed for use case
  6. Perform PaaS bakeoff and pick product
  7. Hire consultants:
    1. Roadmap
    2. Installation
    3. Best Practices
  8. Evaluate lessons learned and plan forward

With this approach there is limited risk to over committing resources (financial, personal and otherwise) and allows time to set appropriate expectations as far as roll out, adoption and sustainability.



Hashi Corps Packer… and my QEMU snafu

How cool is Hashi Corps Packer product? I think it is pretty sweet. I primarily work with fedora and RHEL via Qemu, but it is neat that it can use one configuration file to apply the same settings to each platform if I wanted.

Anyways, a colleague brought this up to my attention last week and I thought as I’m going through the install for OpenShift v3, I might as well create a reusable configuration file for each system. And here we are 🙂 .

I wanted to bring this issue I was facing to light in case you’re like me and don’t like to do anything on your host machine, but rather spin something up in a VM server. I was hitting this issue.

2015/11/02 15:52:16 packer-builder-qemu: 2015/11/02 15:52:16 Qemu stderr: Could not initialize SDL(x11 not available) - exiting
2015/11/02 15:52:16 ui error: ==> qemu: Error launching VM: Qemu failed to start. Please run with logs to get more info.
==> qemu: Error launching VM: Qemu failed to start. Please run with logs to get more info.

What does the error ^^^^ mean? When Packer tries to start up Qemu, it tries to spin up a graphic image and since my server is headless I was getting the error above.  How to fix?

"builders": [{
     "qemuargs": [
        [ "-m", "4444M" ],
        [ "-nographic", "" ]

Simply add the code ^^^ to your JSON file and everything will start swimmingly..


Continuous Integration/Delivery for Operations… Whaaaa?

Continues Integration (and yes I mean Jenkins) is awesome, plain and simple. CI provides a way for developers to continuously integrate, test and give a degree of confidence that it will work with in other environment(s) to ensure stability. If this tool is so great why aren’t sys admins using it as well? Shouldn’t sys admins test their code, measure code coverage, integrate with other sys admins configs and finally ensuring it will work across environments?

If your a sys admin I know what you’re thinking.. I do test my code and I do ensure it works in our environments. And you’re probably also thinking, Why use a CI tool? What is the point if we’re just writing scripts and doing system configurations?

That’s the point, it isn’t just some script that is written once and forgotten about. Generally the scripts sys admins write are the backbone of any organization and are tucked away in a file system somewhere. It is possible some testing was performed when the script was written, but things change and unless those tests are continuously be run when the script is applied again there is no sure way of knowing something is broken.

Step in CI server. What does this get us? Well, let me make the assumption, silly as it may be, that all of the IT operations scripts are under source control (I know, I know, not a safe assumption, but I want this as another blog post down the road). With that assumption in place, now lets assume the Ops team has also built some basic unit testing around what their script will do. After that we set up a job that will constantly poll our scripts for any changes and should a change occur our tests will run against the script verifying everything still works as expected.

What does this get the Ops team? Now anytime anybody updates the script, known tests are run to verify everything still works as expected and the entire team is notified the status of the script. There is now stability, repeatability, and an overall confidence in a companies infrastructure.

For those of us who hate writing documentation. Having this CI/CD process for scripts also acts as a living document. IT organizations can leverage it also as repository for intellectual capital. Yes, not all the code will live on the Jenkins server, but in an ideal scenario all scripts will be tested there having 1 place for everybody to review the health and assets for the IT Ops team.

Short story long. There is a movement to get more automation, rigor and confidence around doing IT Ops and the only way to get there is by writing good code, which implies good testing. This will help with less rework, unnecessary troubleshooting, loss of intellectual artifacts and allow the team to focus on more interesting things rather than trying to figure out what you did a year ago and why the script is no longer working.

Get in the habit of using CI/CD you and your IT organization will not regret it.

Jenkins and the Ansible plugin

I have been slacking on posting anything as of late and it hasn’t been from lack of topics, but my latest client has been me keeping quite engaged. The project has been really cool. I’ve been leading an installation of Red Hats PaaS environment, Openshift. Part of our engagement has been to demonstrate how DevOps and PaaS can increase developer productivity, quality of code, decrease time to market, and confidence the application will do what is supposed to.

During any installation of Openshift the recommendation is to always stand up DNS server specific for Openshift. The reason is that DNS dynamically adds and removes records as containers are added/removed from the PaaS and most network admins are not keen on allowing these dynamic features in the enterprise DNS servers (and rightfully so). This causes problems though. This means that any computer that logs into the Openshift console and creates a container won’t be able to access the container, because their computer won’t be able to resolve the new DNS entry. The fix is to basically forward on traffic to this server from the enterprise servers. However, since making such a change to a company’s DNS server(s) is not something to take lightly and takes time to evaluate risks and such, you may need to create a workaround like adding the Openshift DNS server to your computers list of DNS resolvers.

So nice workaround, right? Well, yes and no. We have worked around the issue of having the developers computer be able to resolve the correct FQDN, cool. What happens when as part of our DevOps process we leverage enterprise Jenkins server to push code directly to Openshift? You got it, the FQDN won’t resolve. The FQDN will not resolve without the correct DNS server.

What do you do? I mean the current set up is only temporary while the new enterprise DNS servers are set up to forward to the Openshift DNS. We really can’t blindly update the DNS servers for Jenkins, because there are a number of other development teams and quite possibly break their builds.

So what do we do… Yes, you got it. Ansible.

jenkins openshift bridge

We will use Ansible to bridge the Jenkins server to our server running the correct DNS to then remotely push an application to Openshift.You may  be asking yourself why not just use ssh? There are a few other little tidbits I didn’t mention and that is Openshift requires a special set of client tools that are not installed on the Jenkins server. There is a Openshift plugin which works, but only with ssh keys. The Jenkins server doesn’t use SSH keys, so no Openshift Plugin. However, even if we could use SSH keys the jenkins server wouldn’t be able to resolve the hostname. Which brings us back to using Ansible to execute commands on the bridge VM and then uses the Openshift client tools and the correct DNS server resolve and push code.

We ran into other issues with using the Ansible plugin that I’ll talk about in another post.