Configuring and running a Kubernetes agent#

This guide is applicable to Dagster Cloud.

In this guide, you'll configure and run a Kubernetes agent. Kubernetes agents are used to launch your code in Kubernetes Jobs and Services.


Prerequisites#

To complete the steps in this guide, you'll need:


Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster Cloud instance.
  2. Click the user menu (your icon) > Cloud settings.
  3. In the Cloud settings page, click the Configure tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.


Step 2: Configure the agent#

In this step, you'll create a Kubernetes namespace for your Dagster Cloud resources. You'll also add the agent token to the Kubernetes cluster.

  1. Create a Kubernetes namespace for your Dagster Cloud resources:

    kubectl create namespace <namespace>
    

    For example, if the namespace is dagster-cloud:

    kubectl create namespace dagster-cloud
    
  2. Add the agent token you created in Step 1 as a secret in the Kubernetes cluster:

    kubectl create secret generic dagster-cloud-agent-token \
        --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<your_agent_token> \
        --namespace dagster-cloud
    

Step 3: Start the agent#

In this step, you'll spin up the agent with Helm.

  1. Add the agent chart repository:

    helm repo add dagster-cloud https://dagster-io.github.io/helm-user-cloud
    helm repo update
    
  2. Install the agent chart, specifying:

    • The namespace into which to install the chart
    • The Dagster Cloud deployment the agent will be responsible for. Note: When your Dagster Cloud organization is first created, the default deployment is prod:
    helm upgrade \
       --install user-cloud dagster-cloud/dagster-cloud-agent \
       --namespace dagster-cloud \
       --set dagsterCloud.deployment=prod
    

After the Helm chart is installed into the Kubernetes cluster, you can view the agent in Dagster Cloud.

In Dagster Cloud, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Instance Status

Next steps#

Now that you've got your agent running, what's next?

You can also further configure the jobs the agent runs, including adding secrets, mounting volumes, and setting resource limits.