Welcome to today’s guide on how to install Ansible AWX on Ubuntu 20.04|18.04 with Nginx Reverse Proxy and optional Let’s Encrypt SSL Certificate. Ansible AWX is an open source tool which provides a web-based user interface, REST API, and task engine for easy and collaborative management of Ansible Playbooks and Inventories.

AWX allows you to centrally manage Ansible playbooks, inventories, Secrets, and scheduled jobs from a web interface. It is easy to install AWX on Ubuntu 20.04|18.04 (Bionic Beaver) Linux system. Use the steps shared below to install and configure Ansible AWX on Ubuntu 20.04|18.04 Linux server.

Starting in version 18.0 of AWX, the recommended installation method is via AWX Operator. As the operator installation method requires a Kubernetes Cluster, we will perform a single node Kubernetes installation on Ubuntu Linux using k3s.

Setup minimum requirements

  • Ubuntu 20.04|18.04 LTS Server
  • At least 8GB of RAM – More is better
  • 4vcpus – Pump more if you have
  • 10GB free disk storage
  • root or user with sudo for ssh

Step 1: Update Ubuntu system

Update and upgrade your system

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2: Install Single Node k3s Kubernetes

We will deploy a single node kubernetes using k3s lightweight tool. K3s is a certified Kubernetes distribution designed for production workloads in unattended, resource-constrained environments. The good thing with k3s is that you can add more Worker nodes at later stage if need arises.

K3s provides an installation script that is a convenient way to install it as a service on systemd or openrc based systems

Let’s run the following command to install K3s on our Ubuntu system:

curl -sfL https://get.k3s.io | sudo bash -
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

Installation process output:

[INFO]  Finding release for channel stable
[INFO]  Using v1.21.3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Validate K3s installation:

The next step is to validate our installation of K3s using kubectl command which was installed and configured by installer script.

$ kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
ubuntu-01   Ready    control-plane,master   33s   v1.22.5+k3s1

You can also confirm Kubernetes version deployed using the following command:

$ kubectl version --short
Client Version: v1.22.5+k3s1
Server Version: v1.22.5+k3s1

The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.

Step 3: Deploy AWX Operator on Kubernetes

This Kubernetes Operator has to be deployed in your Kubernetes cluster, which in our case is powered by K3s. The operator we’ll deploy can manage one or more AWX instances in any namespace.

Install git and make tools:

sudo apt update
sudo apt install git build-essential

Clone operator deployment code:

$ git clone https://github.com/ansible/awx-operator.git
Cloning into 'awx-operator'...
remote: Enumerating objects: 5626, done.
remote: Counting objects: 100% (2840/2840), done.
remote: Compressing objects: 100% (1025/1025), done.
remote: Total 5626 (delta 1910), reused 2434 (delta 1716), pack-reused 2786
Receiving objects: 100% (5626/5626), 1.38 MiB | 12.01 MiB/s, done.
Resolving deltas: 100% (3191/3191), done.

Create namespace where operator will be deployed. I’ll name mine awx:

export NAMESPACE=awx
kubectl create ns ${NAMESPACE}

Set current context to value set in NAMESPACE variable:

# kubectl config set-context --current --namespace=$NAMESPACE 
Context "default" modified.

Switch to awx-operator directory:

cd awx-operator/

Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch using git.

sudo apt update
git checkout 0.17.0

Deploy AWX Operator into your cluster:

# export NAMESPACE=awx
# make deploy

Command output:

cd config/manager && /root/awx-operator/bin/kustomize edit set image controller=quay.io/ansible/awx-operator:0.14.0
cd config/default && /root/awx-operator/bin/kustomize edit set namespace awx
/root/awx-operator/bin/kustomize build config/default | kubectl apply -f -
Warning: resource namespaces/awx is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
role.rbac.authorization.k8s.io/awx-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Wait a few minutes and awx-operator should be running:

# kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-68d787cfbd-z75n4   2/2     Running   0          40s

How To Uninstalling AWX Operator (Don’t run this unless you’re sure it uninstalls!“

You can always remove the operator and all associated CRDs by running the command below:

# export NAMESPACE=awx
# make undeploy
/root/awx-operator/bin/kustomize build config/default | kubectl delete -f -
namespace "awx" deleted
customresourcedefinition.apiextensions.k8s.io "awxbackups.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxrestores.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted
serviceaccount "awx-operator-controller-manager" deleted
role.rbac.authorization.k8s.io "awx-operator-leader-election-role" deleted
role.rbac.authorization.k8s.io "awx-operator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-proxy-role" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-leader-election-rolebinding" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "awx-operator-proxy-rolebinding" deleted
configmap "awx-operator-manager-config" deleted
service "awx-operator-controller-manager-metrics-service" deleted
deployment.apps "awx-operator-controller-manager" deleted

Step 4: Install Ansible AWX on Ubuntu 20.04/18.04 using Operator

Create Static data PVC – Ref AWX data persistence:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi
EOF

PVC won’t be bound until the pod that uses it is created.

# kubectl get pvc -n awx
NAME                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
static-data-pvc          Pending                                      local-path     43s

Let’s now create AWX deployment file with basic information about what is installed:

vim awx-deploy.yml

Paste below contents into the file:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName: static-data-pvc

We have defined resource name as awx and service type as nodeport to enable us access AWX from the Node IP address and given port. We also added extra PV mount on the web server pod.

Apply configuration manifest file:

$ kubectl apply -f awx-deploy.yml
awx.awx.ansible.com/awx created

Wait a few minutes then check AWX instance deployed:

$ watch kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                   READY   STATUS    RESTARTS   AGE
awx-postgres-0         1/1     Running   0          75s
awx-7c5d846c88-mjlvm   4/4     Running   0          64s

Fixing Postgres Pod in CrashLoopBackOff state

Check status of PostgreSQL Pod:

$ kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                   READY   STATUS             RESTARTS   AGE
awx-postgres-0         0/1     CrashLoopBackOff   4          2m38s
awx-66c64f8d67-qvfk9   3/4     ImagePullBackOff   0          2m27s

Check logs of the Pod:

$ kubectl logs awx-postgres-0
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

It means the Postgres pod cannot write to the persistent volume directory inside /var/lib/rancher/k3s/storage/:

# ls -lh /var/lib/rancher/k3s/storage/ | grep awx-postgres-0
total 0
drwx------ 3 root root 4.0K Aug 24 09:18 pvc-b2acb2d0-6d2f-457b-a40c-390cff5ec6d2_default_postgres-awx-postgres-0

Try setting the directory mode to 777

# chmod -R 777  /var/lib/rancher/k3s/storage/*
# kubectl delete pods -l "app.kubernetes.io/managed-by=awx-operator"
pod "awx-66c64f8d67-qvfk9" deleted
pod "awx-postgres-0" deleted

The Postgres container pod should come up in few seconds:

# kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                   READY   STATUS             RESTARTS   AGE
awx-postgres-0         1/1     Running            0          42s
awx-66c64f8d67-4qmh5   4/4     Running            0          42s

If you experience any issues with the Pods starting check deployment logs:

$ kubectl logs -f deployments/awx-operator

The database data will be persistent as they are stored in a persistent volume:

$ kubectl get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-awx-postgres-0   Bound    pvc-c0149545-8631-4aa1-a03f-2134a7f42aa6   8Gi        RWO            local-path     84s
static-data-pvc           Bound    pvc-6b6005de-0888-4634-b0a2-d5cc92eb85cc   1Gi        RWO            local-path     2m10s
awx-projects-claim        Bound    pvc-91e751e9-0e8e-40c8-9953-f8d9db5f612b   8Gi        RWO            local-path     77s

Volumes are created using local-path-provisioner and host path

$ ls /var/lib/rancher/k3s/storage/
pvc-edb29795-7dae-4a00-805f-2d989694fe3d_default_postgres-awx-postgres-0

Checking AWX Container’s logs

The awx-xxx-yyy pod will have four containers, namely:

  • redis
  • awx-web
  • awx-task
  • awx-ee

As can be seen from below command output:

# kubectl -n awx  logs awx-75698588d6-r7bxl
error: a container name must be specified for pod awx-75698588d6-r7bxl, choose one of: [redis awx-web awx-task awx-ee]

You’ll need to provide container name after the pod:

kubectl -n awx  logs awx-75698588d6-r7bxl -c redis
kubectl -n awx  logs awx-75698588d6-r7bxl -c awx-web
kubectl -n awx  logs awx-75698588d6-r7bxl -c awx-task
kubectl -n awx  logs awx-75698588d6-r7bxl -c awx-ee

Step 5: Access Ansible AWX Dashboard

List all available services and check awx-service Nodeport

$ kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
awx-postgres   ClusterIP   None           <none>        5432/TCP       2m5s
awx-service    NodePort    10.43.182.53   <none>        80:32708/TCP   116s

For the service from output above, it will be accessible on port32708.

http://hostip_or_hostname:32708

You can edit the Node Port and set to figure of your preference, but has to be >=32000

$ kubectl edit svc awx-service
....
ports:
  - name: http
    nodePort: 32000
    port: 80
    protocol: TCP
    targetPort: 8052

Confirm if the change was updated successfully:

$ kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
awx-postgres   ClusterIP   None           <none>        5432/TCP       8m58s
awx-service    NodePort    10.43.182.53   <none>        80:32000/TCP   8m49s

Ansible AWX web portal is now accessible on http://hostip_or_hostname:32000.

You are presented with the welcome dashboard similar to one below.





The login username is admin

Obtain admin user password by decoding the secret with the password value:

kubectl get secret awx-admin-password -o jsonpath="{.data.password}" | base64 --decode

Better output format:

kubectl get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Login with admin username and decoded password:


The new AWX interface is beautifully designed with Love. It is a serious redesign with good polishing.

Review Kubernetes Node resources to ensure they are enough to run AWX:

$ kubectl top nodes --use-protocol-buffers
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ubuntu   102m         1%     2534Mi          15%