Skip to main content

Managed Updates for Teleport Agents (v1)

warning

This document describes Managed Updates for Agents (v1), which is currently supported but will be deprecated after Managed Updates for Agents (v2) is generally available.

Managed Updates v1 uses a script called teleport-upgrade that is provided by the teleport-ent-updater package and configured by the cluster_maintenance_config Teleport resource. Managed Updates v2 uses a binary called teleport-update that is provided by the teleport package and configured by the autoupdate_version and autoupdate_config. The original updater and resource are described in this document.

Only Enterprise versions of Teleport can use Managed Updates v1.

Please consider using Managed Updates for Agents (v2), as it provides a safer, simpler, more flexible, compatible, and reliable update experience compared to Managed Updates v1.

Compatibility between Managed Updates v1 and v2

Managed Updates v2 is backwards compatible with the cluster_maintenance_config resource. The Managed Updates v1 teleport-upgrade script is forwards compatible with the autoupdate_config and autoupdate_version resources. Teleport Agents connected to the same cluster may use either resource.

If the autoupdate_* resources are configured, they take precedence over cluster_maintenance_config. This allows for a safe, non-breaking, incremental migration between Managed Updates v1 and v2.

Users of cloud-hosted Teleport Enterprise will be migrated to Managed Updates v2 in the first half of 2025 and should plan to migrate their agents to teleport-update.

On cloud-hosted Teleport Enterprise accounts, users must set up Managed Updates for Teleport Agents to ensure that the version of Teleport running on agents remains compatible with the version running on the Auth Service and Proxy Service. If an agent does not maintain version compatibility with your Teleport cluster, connections to those agents will become degraded or lost.

Cloud-hosted Teleport clusters are updated on a weekly basis. Major version updates are performed every 4 months. You can monitor and subscribe to the Teleport Status page to be notified of scheduled updates.

Teleport supports managed agent updates for SystemD-based Linux distributions using apt, yum, and zypper package managers, as well as Kubernetes clusters.

This guide explains how to enable Managed Updates v1 for Teleport Agents on Teleport Enterprise clusters, including both self-hosted and cloud-hosted clusters.

How it works​

When Managed Updates are enabled, a Teleport updater is installed alongside each Teleport Agent. The updater communicates with the Teleport Proxy Service to determine when an update is available. When an update is available, the updater will update the Teleport Agent during the next maintenance window. However, if a critical update is available, the Teleport Agent will be updated outside the regular maintenance window.

Prerequisites​

  • A Teleport Enterprise cluster. If you do not have one, sign up for a free trial or consult the Update Reference to read about manual updates.
  • Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components in your cluster.
  • Teleport Agents that are not yet enrolled in Managed Updates.
  • The tctl and tsh client tools version >= 17.3.3. Read Installation for how to install these.
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example: 
    tsh login --proxy=teleport.example.com --user=email@example.com
    tctl status
    Cluster  teleport.example.com
    Version  17.3.3
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/4. Enable Managed Agent Updates​

If you are running a cloud-hosted Teleport Enterprise cluster, skip to Step 2.

Before enabling managed upgrades in your self-hosted Teleport cluster, you must enable a version server. This section shows you how to enable a version server in your cluster.

Configure a maintenance schedule​

To enable Managed Updates in your cluster, you must create a cluster maintenance configuration. This configures a maintenance schedule for the Teleport cluster that agents use to determine when to check for upgrades.

  1. Create a Teleport role that can manage cluster maintenance configurations through the cluster_maintenance_config dynamic resource. No preset Teleport roles provide this ability, so you will need to create one.

    Create a file called cmc-editor.yaml with the following content:

    kind: role
version: v7
metadata:
  name: cmc-editor
spec:
  allow:
    rules:
    - resources: ['cluster_maintenance_config']
      verbs: ['create', 'read', 'update', 'delete']

    Create the role resource:

    tctl create cmc-editor.yaml
    tip

    You can also create and edit roles using the Web UI. Go to Access -> Roles and click Create New Role or pick an existing role to edit.

  2. Add the role to your Teleport user:

Assign the cmc-editor role to your Teleport user by running the appropriate commands for your authentication provider:

  1. Retrieve your local user's roles as a comma-separated list:

    ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')

  2. Edit your local user to add the new role:

    tctl users update $(tsh status -f json | jq -r '.active.username') \  --set-roles "${ROLES?},cmc-editor"

  3. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Create a cluster maintenance config in a file called cmc.yaml. The following example allows maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC:

    kind: cluster_maintenance_config
spec:
  agent_upgrades:
    # Maintenance window start hour in UTC.
    # The maintenance window lasts 1 hour.
    utc_start_hour: 2
    # Week days when maintenance is allowed
    # Possible values are:
    # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat
    # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
    weekdays:
      - Mon
      - Wed
      - Fri

  2. Apply the manifest using tctl:

    tctl create cmc.yaml
    maintenance window has been updated

[Optional] Assign the version served by the version server​

By default, the version server has a single default channel, serving the version of the Teleport Proxy Service. If you want to override the default version or add other channels you can use the automatic_upgrades_channels field in the Proxy Service configuration file:

proxy_service:
  enabled: "yes"
  automatic_upgrades_channels:
    # Override the default version channel reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version
    default:
      static_version: v14.2.1
    # Define a new version channel with a static version reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/m-static-channel/version
    my-static-channel:
      static_version: v14.2.0
    # Define a new version channel forwarding requests to an upstream version server
    my-remote-channel:
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud

You must ensure that all Proxy Service instances share the same automatic_upgrades_channels configuration. If some Proxy Service instances are configured differently, you will experience agents flickering between versions as the version served is not consistent across instances.

If your Proxy Service public address is teleport.example.com:443, you can query the version server with the following command:

curl "https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version"
17.3.3

Step 2/4. Find agents to enroll in Managed Updates​

Use the tctl inventory ls command to list connected agents along with their current version. Use the --upgrader=none flag to list agents that are not enrolled in Managed Updates.

tctl inventory ls --upgrader=none
Server ID                            Hostname      Services Version Upgrader------------------------------------ ------------- -------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node     v14.4.5 none...

Step 3/4. Enroll agents on Linux servers in Managed Updates​

  1. For each agent ID returned by the tctl inventory ls command, copy the ID and run the following tctl command to access the host via tsh:

    HOST=00000000-0000-0000-0000-000000000000
    USER=root
    tsh ssh "${USER?}@${HOST?}"

  2. Determine the Teleport version to install by querying the Teleport Proxy Service. This way, the Teleport installation has the same major version as the automatic updater.

    Replace stable/cloud with the name of your automatic update channel. For cloud-hosted Teleport Enterprise accounts, this is always stable/cloud:

    TELEPORT_VERSION="$(curl https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/stable/cloud/version | sed 's/v//')"

  3. Ensure that the Teleport repository is properly configured to use the stable/cloud channel, and install the teleport-ent-updater package. You must install teleport-ent-updater on each agent you would like to enroll into Managed Updates:

    curl https://cdn.teleport.dev/install-v17.3.3.sh | bash -s ${TELEPORT_VERSION?} cloud

    The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the installation guide.

  4. Confirm that the version of the teleport binary is the one you expect:

    teleport version
Running the agent as a non-root user

If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user:

sudo mkdir -p /etc/teleport-upgrade.d/
sudo touch /etc/teleport-upgrade.d/schedule
sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule

  1. Verify that the upgrader can see your version endpoint by checking for upgrades:

    sudo teleport-upgrade dry-run

  2. You should see one of the following messages, depending on the target version you are currently serving:

    no upgrades available (1.2.3 == 1.2.3)
an upgrade is available (1.2.3 -> 2.3.4)

    teleport-upgrade may display warnings about not having a valid upgrade schedule. This is expected immediately after install as the maintenance schedule might not be exported yet.

Step 4/4. Enroll Kubernetes agents in Managed Updates​

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

  1. Confirm that you are using the Teleport Enterprise edition of the teleport-kube-agent chart. You should see the following when you query your teleport-kube-agent release:

    helm -n `teleport` get values `teleport-agent` -o json | jq '.enterprise'
    true

    If another value such as null is returned, update your existing agent values.yaml to use the Enterprise version:

    enterprise: true

  2. Add the following chart values to the values file for the teleport-kube-agent chart:

    updater:
  enabled: true

  3. Update the Teleport Helm repository to include any new versions of the teleport-kube-agent chart:

    helm repo update teleport

  4. Update the Helm chart release with the new values:

    helm -n teleport  upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version="17.2.7"

  5. You can validate the updater is running properly by checking if its pod is ready:

    kubectl -n teleport-agent get pods
    NAME                               READY   STATUS    RESTARTS   AGE<your-agent-release>-0                         1/1     Running   0          14m<your-agent-release>-1                         1/1     Running   0          14m<your-agent-release>-2                         1/1     Running   0          14m<your-agent-release>-updater-d9f97f5dd-v57g9   1/1     Running   0          16m

  6. Check for any deployment issues by checking the updater logs:

    kubectl -n teleport logs deployment/teleport-agent-updater
    2023-04-28T13:13:30Z	INFO	StatefulSet is already up-to-date, not updating.	{"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting​

Teleport Agents are not updated immediately when a new version of Teleport is released, and agent updates can lag behind the cluster by a few days.

If the Teleport Agent has not been automatically updating for several weeks, you can consult the updater logs to help troubleshoot the problem:

journalctl -u teleport-upgrade

Troubleshooting managed agent upgrades on Kubernetes​

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

  1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:

    kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'

  2. To suspend Managed Updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Troubleshooting Managed Updates on Linux​

  1. If an agent is not automatically upgraded, you can invoke the upgrader manually and look at its logs:

    sudo teleport-upgrade run

  2. To suspend Managed Updates for an agent, disable the systemd timer:

    sudo systemctl disable --now teleport-upgrade.timer

  3. To enable and start the systemd timer after suspending:

    sudo systemctl enable --now teleport-upgrade.timer