Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP

Beta

BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or Redpanda Sales.
Redpanda Connect is in beta for BYOVPC clusters on GCP.

To enable Redpanda Connect on an existing BYOVPC cluster, you must update your configuration. You can also create a new BYOVPC cluster with Redpanda Connect already enabled.

Replace all <placeholders> with your own values.

Create two new service accounts with the necessary permissions and roles.

Show commands

# Account used to check for and read secrets, which are required to create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect-api \
  --display-name="Redpanda Connect API Service Account"

cat << EOT > redpanda-connect-api.role
{
  "name": "redpanda_connect_api_role",
  "title": "Redpanda Connect API Role",
  "description": "Redpanda Connect API Role",
  "includedPermissions": [
    "resourcemanager.projects.get",
    "secretmanager.secrets.get",
    "secretmanager.versions.access"
  ],
}
EOT

gcloud iam roles create redpanda_connect_api_role --project=<service-project-id> --file redpanda-connect-api.role

gcloud projects add-iam-policy-binding <service-project-id> \
  --member="serviceAccount:redpanda-connect-api@<service-project-id>.iam.gserviceaccount.com" \
  --role="projects/<service-project-id>/roles/redpanda_connect_api_role"

# Account used to retrieve secrets and create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect \
  --display-name="Redpanda Connect Service Account"

cat << EOT > redpanda-connect.role
{
  "name": "redpanda_connect_role",
  "title": "Redpanda Connect Role",
  "description": "Redpanda Connect Role",
  "includedPermissions": [
    "resourcemanager.projects.get",
    "secretmanager.versions.access"
  ],
}
EOT

gcloud iam roles create redpanda_connect_role --project=<service-project-id> --file redpanda-connect.role

gcloud projects add-iam-policy-binding <service-project-id> \
  --member="serviceAccount:redpanda-connect@<service-project-id>.iam.gserviceaccount.com" \
  --role="projects/<service-project-id>/roles/redpanda_connect_role"

Bind the service accounts.

The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is my-gcp-sa@my-project.iam.gserviceaccount.com, then the account ID is my-gcp-sa.

Show commands

gcloud iam service-accounts add-iam-policy-binding <redpanda_connect_api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect_api-gcp-sa-account-id>]"

gcloud iam service-accounts add-iam-policy-binding <redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect-gcp-sa-account-id>]"

Make a PATCH /v1/clusters/{cluster-id} request to update the cluster configuration.

Show request

export CLUSTER_PATCH_BODY=`cat << EOF
{
    "customer_managed_resources": {
        "gcp": {
            "redpanda_connect_api_service_account": {
              "email": "<redpanda_connect-api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
            },
            "redpanda_connect_service_account": {
              "email": "<redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
            }
        }
    }
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/<cluster-id>

Check Redpanda Connect is available in the Cloud UI.
1. Log in to Redpanda Cloud.
2. Go to the Connect page and you should see the Redpanda Connect tab alongside Kafka Connect.