Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for node priority #492

Open
volfco opened this issue May 27, 2018 · 1 comment
Open

Add support for node priority #492

volfco opened this issue May 27, 2018 · 1 comment

Comments

@volfco
Copy link

volfco commented May 27, 2018

Submission type

  • Request for enhancement (RFE)

Enhancement Description

I think it would be a valuable feature to be able to give preference to a node (or nodes) for master election. This could be via a flag or a numeric value. This could be applied at a per keeper setting, or a cluster setting that says "try and keep master at this node."

My reasoning for this is that it could be cost efficient (this is my use case) to setup a cluster where one node is better specked than the others in order to better handle incoming writes. It would make sense for Stolon to try and make sure this node is always master. If it goes down, then Stolon would move master to another node, but when it comes back up, Stolon would move master back to this node.

arssher added a commit to postgrespro/stolon that referenced this issue Mar 11, 2019
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Implements sorintlab#492
arssher added a commit to postgrespro/stolon that referenced this issue Jun 30, 2019
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
arssher added a commit to postgrespro/stolon that referenced this issue Jun 30, 2019
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
arssher added a commit to postgrespro/stolon that referenced this issue Jul 1, 2019
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
arssher added a commit to postgrespro/stolon that referenced this issue Jul 1, 2019
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
ololobus pushed a commit to postgrespro/stolon that referenced this issue Jul 9, 2020
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
ololobus pushed a commit to postgrespro/stolon that referenced this issue Jul 9, 2020
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
ololobus pushed a commit to postgrespro/stolon that referenced this issue Jul 10, 2020
Sentinel will promote keeper with higher priority than the current one if this
is possible. In async mode this is a bit non-deterministic because we always
elect node with highest LSN, and under heavy load prioritized node might never
report LSN higher than its stronger competitors. However, if nodes are equal
this should happen at some moment. In sync mode, we can just elect any of
synchronous standbies.

Priority can be set during keeper start (--priority) or later with new command
'stolonctl set keeperpriority'. The latter allows to update priority without
restarting the keeper (and its Postgres instance), which can be used for
controlled failover.

Implements sorintlab#492
@nvtkaszpir
Copy link

nvtkaszpir commented Jun 28, 2021

I tried this in different places and tools and at first it feels okay, but in edge situations it makes things worse:

  • if specific node is 'better' then it will force master node flapping - it's better to actually avoid failovers if possible
  • if specific nodes are 'worse' then it's even more possible that given node will be overloaded if better node is not capable of the workload.

I short:

  • do not force leader election if better node comes online
  • better avoid single node with higher priority, or just get better hardware to have at least 2 nodes with higher priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants