stolon-keeper execute pg_rewind and pg_basebackup error

[smkingsoft]

hi, @sgotti
I tested the master and slave switch, and found some errors.

1. One master and two standby operating normally
2. Kill the stolon-keeper process on the original master node, another standby node becomes the new master node.
3. Under the postgres user, use the same parameters to restart the stolon-keeper process on the original master node
4. stolon-keeper print error:

[I] 2017-06-04T03:29:39Z keeper.go:937: resyncing the database cluster
[I] 2017-06-04T03:29:39Z keeper.go:616: syncing using pg_rewind followedDB=551f4d5c keeper=postgres_03
[I] 2017-06-04T03:29:39Z postgresql.go:513: running pg_rewind
......
[E] 2017-06-04T03:30:55Z keeper.go:619: error syncing with pg_rewind error=error: exit status 1, output: could not find a "pg_rewind" to execute
fetched file "global/pg_control", length 8192
fetched file "pg_xlog/00000003.history", length 85
......

could not find a "pg_rewind" to execute
could not find a "pg_rewind" to execute
The program "initdb" is needed by pg_rewind but was
not found in the same directory as "pg_rewind".
Check your installation.
Failure, exiting

but pg_rewind could find in the directory.

[E] 2017-06-04T03:31:00Z keeper.go:994: failed to resync from followed instance error=sync error: error: exit status 1, output: could not find a "pg_basebackup" to execute
pg_basebackup: directory "/mydata/pgdata/my_data" exists but is not empty

because i define my custom tablespace at directory "/mydata/pgdata/my_data", stolon-keeper only delete the postgres directory "/mydata/pgdata/postgres".

so pg_basebackup can not be successful.

How do I define a script that can also delete a custom tablespace directory at the same time

[sgotti]

could not find a "pg_rewind" to execute
The program "initdb" is needed by pg_rewind but was
not found in the same directory as "pg_rewind".
Check your installation.
Failure, exiting

This is something I also saw in the output of the integration tests, need to dig into it, since it doesn't happen when run manually.

because i define my custom tablespace at directory "/mydata/pgdata/my_data", stolon-keeper only delete the postgres directory "/mydata/pgdata/postgres".

Now stolon doesn't work with tablespaces. Created #290. Let's continue there for the tablespaces support.

[sgotti]

could not find a "pg_rewind" to execute
The program "initdb" is needed by pg_rewind but was
not found in the same directory as "pg_rewind".
Check your installation.
Failure, exiting

This is something I also saw in the output of the integration tests, need to dig into it, since it doesn't happen when run manually.

Fixed in #294

[smkingsoft]

@sgotti hi, another question about pg_rewind
The machine power failure caused the system to crash, so postgres had crashed.

[I] 2017-06-13T09:13:19Z postgresql.go:513: running pg_rewind
[E] 2017-06-13T09:13:19Z keeper.go:619: error syncing with pg_rewind error=error: exit status 1, output: fetched file "global/pg_control", length 8192

target server must be shut down cleanly
Failure, exiting
[I] 2017-06-13T09:13:20Z postgresql.go:552: running pg_basebackup

maybe you can restart it, and shut it down right after recovery has finished, then running pg_rewind.

[sgotti]

maybe you can restart it, and shut it down right after recovery has finished, then running pg_rewind.

Seems possible, can you please file a new issue so we can track it better?

[smkingsoft]

New issue #295

[sgotti]

Closing since the main issue has been fixed in #294 and other issues are tracked in #294 and #295