CQ Unix Toolkit

12 July 2013
Arkadiusz Kita


During the last two decades we have been observing an explosion of web-browser & GUI tools which help us to perform complex tasks in an easier manner. In fact, there is one problem with GUI tools - particular actions cannot be queued or scheduled.

Furthermore, we cannot automate or batch several different operations unless a dedicated application supports this functionality. However, this is a rather complex and advanced feature not supported in simple applications.

But the most important fact is that you cannot link different applications together to build your highly customized tool that will improve your performance and resolve your current task or problem. Of course you can always try to do it manually: click & save, click, open the file in another tool & click, click, click and save your result to another file, click, click .... etc.

Yes - I "like" these clicks very much because if you have to do some complex task which is out of the scope of one particular application it takes a lot of click-time to perform "the whole processing chain" and obviously (as a developer or just a normal user) it is a pretty boring thing even if there is a beautiful and polished application on your desktop that might look amazing.

As you probably know, our predecessors in the 1970's/1980's had similar problems and tasks and they found solutions, but... Hold on! There was no GUI yet, there was no polished interface, so how could programmers and administrators perform complex tasks easily?

They had a simple CLI (Eng. Command-Line Interface) which became 'shells' nowadays and an alternative way of doing the same thing in a fast and automated way. One of the popular examples is the family of UNIX shells that allows us to perform different commands.

However, it's not as simple as it seems. Command-line interface is not only the place where you can write your commands and execute them and see what happens. It allows you to do many different things, especially:

  • pass generated results from one tool to another (pipelining)
  • divide complex tasks into simple actions (utilities)
  • create new commands using a combination of existing ones (aliases/scripts)
Imagine that you have a set of simple tools and each of them does only one very specific simple action and only that. This works exactly the same way in Math - we have a limited set of digits and operations and you can combine them into more complex forms... Possibilities are infinite...

...but it's time to get real!

I had very limited time to migrate content between many CQ instances. I had to do this concurrently due to a strict deadline and lack of free time and sufficient processing resources. I wanted to save content from the CQ repository only once and be able to use it for a few new instances at once. Additionally, I had to check whether the currently existing repositories were consistent before content extraction.

I had two ways of doing this. I knew that there was/existed the webtool CRX package manager (included in Adobe CQ) that allowed me to create, build, download, upload back and install packages with content. Therefore, the most obvious way to perform the task was to click, click and click once again to create a few packages on a few instances, do this a few times more and then get those packages and install them on a few new instances... ...and then I saw in my mind a big headache and mistakes during clicks, and some forgotten steps I should've performed. I thought that this method was very risky and required a lot of clicks in web tools and I had to do a few tasks concurrently, so I had to figure out how to automate some repetitive operations.

Of course, I could have used some "GUI macro recording tool" and use it later, but this was very tricky and time consuming too.

The alternative way I found was to use some documented HTTP requests to manage packages on Adobe CQ instances. I recalled my old friend, the cURL command-line tool, that can perform HTTP requests exactly as specified by me to match the Adobe CRX PackageManager API. For each operation I've created a separate script (command) that I could reuse without remembering HTTP details. So that's how I've created cqbld (cq build), cqdel (cq delete), cqget (cq download), cqput (cq upload), cqrun (cq install) commands that allow me to specify some dynamic paramters like CQ host and port, package name, etc.

Guess what?! I've combined these commands into scripts to perform more complex tasks easier, so I could execute them many times for different instances. Finally, this has saved me from a huge project delay and I ended it with successful content migration.

...but that's not the whole story!

A few weeks later it turned out that I had to make additional backups with a limited number of pages, but in a regular manner, so wow... my commands turned out to be helpful for this too.

Currently, I use them in a regular manner and other developers use them too because they don't need to know how it works. You can use it easily in CLI. It's very user friendly. Here is an example of how to get an Adobe packages number on my CQ instance:

 $ cqls -1 -u admin | grep -c adobe

Grep is well-known UNIX command for searching text pattern. cqls is one of my command I've written. We can just invoke it without parameters and see what it is:

 Usage: cqls [OPTION...]
List all packages in CQ Package Manager using instance URL.
cqls -u admin # List packages for localinstance
cqls -i http://localhost:5510 # List packages for localhost instance on tcp
-p secret # port 5510 with password provided: secret

Finally, I decided to add regular help messages/tips to the commands and to improve them to be less error-prone. It's nothing special, but the strength of the toolkit is that you can use it without having to look for API or anything else. Moreover, as mentioned earlier, I can do anything I want to with the output, or I can build more complex use-cases for these commands, i.e.:

  • transferring data between instances
  • synchronizing different environments (i.e. production and pre-production)
  • making backups
  • checking the CQ state based on different URLs
  • checking repository consistency and performing different maintenance tasks

Additionally, I've discovered that especially big packages are installed or built much quicker using the cURL tool instead of a web browser. This is caused by a lot of output generated by JavaScript in the Activity tab so your browser grows in memory and become bigger and bigger which slows down the whole process.

These tools are currently pushed to GitHub so you can visit it for the latest toolkit improvements or news.

I hope these commands will help you to operate on CQ automatically and it won't be difficult for you. Otherwise you can submit a new issue on GitHub or you can fix it yourself - it's an open source toolkit!