Multinode setup using helm charts where access node and data node on different VM

TimescaleDB and PostgreSQL
,
1

Question
Is it possible to have a multinode setup using helm charts, where the Access node is on one virtual machine and data nodes on different nodes?

To Reproduce
With existing commands, all pods are created on the same node\virtual machine.

Expected behavior
On installation, access node should get deployed on the virtual machine \ Node and data nodes on other virtual machines?

Deployment

  • helm default value.yaml file
  • pg12.5-ts2.0.0-p0

Deployment
Linux Kubernetes

2

Hi @Mohammed_Iliyas_pate

I checked in with our engineering team, and here’s their advice.

The default values for the multinode recipe have a podAntiAffinity for pods of a same release that should theoretically avoid that situation. It is set to preferred though, not required, so the scheduler can ignore it if there are not enough eligible nodes in the cluster.

values.yaml

affinityTemplate: |
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        topologyKey: "kubernetes.io/hostname"
        labelSelector:
          matchLabels:
            app:  {{ template "timescaledb.name" . }}
            release: {{ .Release.Name | quote }}

preferred as a default makes sense, to be as flexible as possible. We’d recommend ensuring the cluster has enough capacity, then changing that default to requiredDuringSchedulingIgnoredDuringExecution

Additional thoughts: that affinityTemplate could also be used to ensure data nodes are in different AZs, if the AZ is available as a node label. Or, it could be used the other way around, to ensure they are in the same AZ to reduce paid cross-zone traffic, if cost is more important than availability.

3

Thank you @LorraineP for your quick support. I think this is what I was looking for. I will explore the affinity template and try to implement it.

I will update you on this forum.

1 Like
4

I too am struggling with this. I have installed several times on a k3s cluster with 3 worker nodes and the installation works but each time I see all pods being created on the same node, which is clearly undesirable. I would expect each pod to be created on a different node in the cluster as it makes no sense to me to have a distributed database which is only running on one node.

I tried the suggestion from Lorraine but it made no difference: I changed the podAntiAffinity as below. Maybe this is wrong, I’m not sure.

   podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app:  {{ template "timescaledb.name" . }}
            release: {{ .Release.Name | quote }}
        topologyKey: "kubernetes.io/hostname"

One other thing, the post installation instructions for obtaining the passwords for multi node are incorrect. Given a db name of “dbserver”, it tells you to get the postgres password using the command:

PGPASSWORD_SUPERUSER=$(kubectl get secret --namespace default dbserver-timescaledb -o jsonpath="{.data.password-superuser}" | base64 --decode)

It should be:

PGPASSWORD_SUPERUSER=$(kubectl get secret --namespace default dbserver-timescaledb-access -o jsonpath="{.data.password-superuser}" | base64 --decode)
1 Like
5

Hi @Steve sorry for the delay in response and thank you for that correction. I’ll get this to the team, just want to check you mean this file

Also, I’ve asked the Timescale team about targeting pods to specific nodes, and hope to have a response for you.

6

Hi Lorraine,

Yes, that’s the file.

BR. Steve

7

Hello again @Steve

I don’t think we have examples of this in our projects, but I have suggestions from the team that you could explore perhaps? Here they are:

Scheduling a pod to a specific node is possible using taints mechanism in Kubernetes. If a user wants to prevent workloads from being scheduled to the same node for HA purposes, it’s also possible to use topology spread constraints.

Hope this helps!

8

The way I have done it is, I have used the node selector option in the chart values file:

nodeSelector: { type: storage }

And I’ve tainted my storage nodes like so:

kubectl taint nodes my-storagenode type=storage:NoExecute

1 Like