Refactor metadata cache and async updating mechanism in client side

[wuchong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, the implementation of metadata cache (Cluster) and updating mechanism (MetadataUpdater) on the client side is not very sophisticated. It lacks the general retry mechanism (#480), the time out for UpdateMetadata request is too short to cause job failover (#311), cause job failover when a new partition created (#288).

Solution

This issue aims to introduce a new design of metadata cache and updating mechanism, to resolve above problems.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!