Ansible Tower Job Control with the awx cli

Ansible, Ansible Tower, API
Ansible Tower Job Control with the awx cli

Overview

Whilst I’m a big fan of Ansible Tower’s UI and the overall User Experience I generally try to avoid spending time in it these days. If you spend a lot of time clicking in an automation tool you’re probably "doing it wrong" IMHO!

Up until Ansible Tower 3.6 we have been using a mix of tower-cli, Python based API calls, and Ansible Playbooks to automate our use of Tower and in particular job launching. In fact it’s completely possible for us to deploy a Tower Cluster, use it on a daily basis, and then tear it down without ever logging into the console.

So in this article I want to explore how to use the new, as of Tower 3.6 awx command line utility for Ansible Tower API access with a focus on Job Control. We’ll cover:

  • Installation

  • Configuration and Authentication

  • Job Management

    • Listing past jobs

    • Retrieving job logs

    • Running jobs

    • Monitoring jobs

  • Resources and Documentation

Note
 Just to be clear and avoid confusion between awx the command line tool and AWX the upstream Ansible Tower Open Source project I’ll always be referring the cli tool when using the lowercase awx and uppercase AWX when referring to the Upstream Project.

Installation

The Installation Docs can be found here

  1. Installing via pip I strongly recommend the usage of yum when using Red Hat Enterprise Linux. However for the purpose of this article we’ll use pip as this is the most flexible across multiple different platforms.

    pip3 install --user \
  https://releases.ansible.com/ansible-tower/cli/ansible-tower-cli-latest.tar.gz
    Note
    		 If you exclusively use Python 3 like me (you should, Python 2 EOL is coming soon!) then you can change the pip3 to pip. You may also want to change the above if you want to make it available system-wide (drop the --user) or are deploying to a virtualenv. I mainly work in virtualenvs so I would: first install system-wide and then if I needed specific versions use pip to install in the relevant virtualenv. For a quick start guide to virtualenvs on RHEL look here. 
    pip install  https://releases.ansible.com/ansible-tower/cli/ansible-tower-cli-latest.tar.gz

Configuration and Authentication

This section is just a brief introduction to configuration and authentication and more details can be found here including Oauth2 and Personal Access Tokens. Different users use patterns will vary but in our team we use multiple different Tower clusters daily and the following pattern works well. If you have a more static environment, with perhaps just one Tower server or cluster then perhaps consider putting the environmental variables below in your .bashrc or equivalent.

  • Create 1 or more tower credential environment settings files

  • Source the appropriate file prior to interacting with a particular tower or tower cluster.

  • Use awx without having to specify credentials and hosts each time

    1. Create the environmental file(s) in an appropriate directory

      $ vim ~/secrets/my-production-tower.env
export TOWER_HOST=https://tower.example.com
export TOWER_USERNAME=<YOUR-USERNAME-HERE>
export TOWER_PASSWORD=<YOUR-PASSWORD-HERE>
export TOWER_VERIFY_SSL=false
      Note
      		 The last variable TOWER_VERIFY_SSL=false is only needed if you are using unsigned certificates. You can achieve the equivalent on the command line with awx -k.

    2. source the appropriate file

      $ source ~/secrets/my-production-tower.env

    3. Work with awx without having to specify user, host and password etc e.g.

      $ awx job list -f human
id name
== ===========================================================
1  deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
2  destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
3  deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-02
4  destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-02
5  deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-03
6  deploy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-01
7  deploy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-02
8  deploy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-03
9  destroy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-01
      Tip
      		 If you do work with multiple tower configurations then consider adding 1 more more line to each configuration file setting the name of the current tower in your shell prompt. Something like this should work: ` export PS1="prod-tower":$PS1` giving a shell prompt like this on my machine: 
      prod-tower:➜  ~ git:(article-job-control-with-awx) ✗

Resources and Actions

awx works on the concepts of resources and actions. Examples of resources include: user, job, project etc. A full list can be generated by just running awx with no arguments. Actions include: list, get, delete etc and are resource sensitive , for example awx user stdout makes no sense and will fail.

Tip
 awx often supports both singular and plural for key resources and the following commands are equivalent awx job list awx jobs list
Tip
 awx includes context sensitive help so if you can’t remember the possible options to pass to a command like awx job list then just add -h or --help.

Job Management

Now we have awx setup and configured it is time to start using it. In my team we generally use Ansible itself to create Tower Resources such as users, projects, job templates etc and use API calls primarily for Job Management e.g.

  • Listing past jobs

  • Retrieving job logs

  • Running new jobs

  • Monitoring jobs

Before going further it is worth noting that awx has 4 output modes:

  • json (default)

  • human

  • yaml

  • jq

These can be applied with the -f flag as shown in the example above (awx job list -f human)

Listing past jobs

The simplest way to list jobs that have been run previously is simply: awx job list and even with just a few jobs that output can be considerable, plus as mentioned above it defaults to json. (A single job might generate ~160 lines of json)

  1. List jobs

    $ awx job list
{
     count: 102,
     next: /api/v2/jobs/?page=2,
     previous: null,
     results: [
          {
               id: 4,
               type: job,
               url: /api/v2/jobs/4/,
               related: {
                    created_by: /api/v2/users/3/,
                    labels: /api/v2/jobs/4/labels/,

    So perhaps useful f you are piping it to another command expecting json but somewhat difficult for humans to parse. If you speak json like a pro then time to learn the -f jq option which will be the subject of a future article.

    Let’s make the output more human friendly with the -f human option:

    awx job list -f human
id name
== ===========================================================
1  deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
2  destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
3  deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-02
4  destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-02

    More manageable but now perhaps a little too terse, try filtering with the very useful --filter option:

    awx jobs list --all -f human --filter 'id,name,status'
id  name                                                        status
=== =========================================================== ==========
1   deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-01     successful
2   destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01    successful
3   destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01    successful
4   destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01    successful
5   destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01    successful
6   deploy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-01 failed
7   provision-agnosticd-development-4xxp8                       failed
8   destroy-agnosticd-development-4xxp8                         successful
...

  2. Retrieving the ansible logs

    I’d recommend shipping all logs to some form of external logging such as ELK which Tower has in-built support for. However what if you want to see why job 6 above failed? In other words to see the ansible output in much the same way as if you’d executed ansible-playbook? Use the stdout option:

    $ awx job stdout 6

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'


PLAY [Step 0000 - Setup output_dir] ********************************************

TASK [debug] *******************************************************************
Wednesday 20 November 2019  23:03:11 +0000 (0:00:00.029)       0:00:00.029 ****
skipping: [localhost]
...

    1. Retrieving job status by name (--name)

      If you know the name of a particular job, whether only run once or many times you can retrieve the status, and more, via the --name argument. For example here we have a job-template called destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 and to retrieve status we can pass this via --name destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01:

      awx jobs list --name destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 \
    -f human --filter 'id, name,status'
id name                                                     status
== ======================================================== ==========
9  destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 successful
13 destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 successful
16 destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 successful
19 destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01 successful
      Tip
      		 Can’t remember your job-templates name, try this and remember it awx resources use underscores and not dashes so job_template not job-template: 
      $ awx job_templates list -f human --filter 'id, name'
id name
== ===========================================================
17 deploy-agnosticd-three-tier-app-prod-1.14-3tier-sat-test-01
15 deploy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
35 destroy-agnosticd-development-jzz66
24 destroy-agnosticd-development-nx5lr
16 destroy-agnosticd-three-tier-app-prod-1.14-3tier-test-01
33 destroy-agnosticd-three-tier-app-prod-1.14-5wht5
31 destroy-agnosticd-three-tier-app-prod-1.14-6fn7r

Job Control

So far we’ve looked at using awx to examine the status and logs of jobs that have already been run. However one of the most powerful features of the API and awx is to launch new jobs. Reminder, a job is the launch of a job-template with or without passing variables.

Often examples of job launches are shown using job-template ids but it is far superior, IMO, to launch job templates via their name and the prior command shows how to look names up dynamically.

  1. Simple launch of a job-template called foo

    $ awx job_template launch foo -f human
id  name
=== ==========
120 foo

  2. Launching the same foo job-template with an extra var guid

    This is somewhat different from awx predecessor tower-cli and cannot simply be done in a single atomic command. Rather you to first modify the job_template with the new extra-vars and then execute the command.

    Note
    		 The job_template will retain your new vars so a subsequent launch will inherit those variables - so use carefully. 
    $ awx job_template modify foo --extra_vars guid=1234
$ awx job_template launch foo -f human
id  name
=== ==========
122 foo

  3. Launching the same foo job_template with an extra var file If you want to pass more than one var or just prefer the Infrastructure as Code properties and other advantages you can pass an entire var file using the @ symbol to proceed the path.

    $ awx  job_template modify foo --extra_vars @./vars/my-config.yml
$ awx  job_template launch foo -f human
id  name
=== ==========
124 foo

  4. Finally if you want to follow along with the log output as your job runs use the --monitor option

    $ awx job_template launch foo --monitor
id  name
=== ==========
126 foo

Resources

Ansible and Ansible Tower’s documents are, generally, excellent and I strongly recommend starting there. There is a dedicated documentation set for awx cli. At this point there are few blog articles out there given the relative newness of awx into mainstream Ansible Tower.

NB as awx calls the Ansible Tower V2 API it should work fine with earlier versions of Tower that support V2.