BYO PrivateDNSZone from a different ResourceGroup

[snehala27]

/kind bug

[Before submitting an issue, have you checked the Troubleshooting Guide self-managed & managed?]

What steps did you take and what happened:
CAPZ only takes DNSPrivateZone name as input and uses the cluster ResourceGroup for PrivateDNSZone(ref)
If the zone is not present in cluster resource group, CAPZ creates the zone and manages lifecycle

Scenario-1

• In a ResourceGroup(say RG1), create a private cluster with existing VNET from a different ResourceGroup(say RG-vnet)
• Specify PrivateDNSZone name(zone1) in NetworkSpec
  Here, CAPZ tries to create the PrivateDNSZone named zone1 in RG1 and links it to VNET in RG-vnet

Scenario-2

• Create another cluster in a new ResourceGroup(say RG2) with same configuration as above using the existing VNET from RG-vnet and specify PrivateDNSZone name as zone1
  Here, CAPZ will again create a new PrivateDNSZone zone1 in RG2 and will try to link it to VNET in RG-vnet causing name conflicts

What did you expect to happen:
CAPZ should support taking PrivateDNSZone ResourceGroup as input or support a flag UseVNETResourceGroup for PrivateDNSZone

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[kreeuwijk]

There is a bit more nuance to this issue. Yes, as shown in the example, it is easy for the user to get themselves into duplicate DNS zone name conflicts, when using the same private dns zone name in different cluster resource groups, but with the same network resource group.

However another issue is the proliferation of private dns zones with the current design. Since you can't re-use a private dns zone unless you only have 1 cluster resource group, the current design requires a (unique) private dns zone to be created for every cluster resource group that can be targeted. This is both unnecessarily complex and error prone. We have a customer that has many cluster & network resource groups and the prospect of zone proliferation and future naming conflicts is not acceptable to them.

If instead, we give the user the (optional) ability to select a resource group for the private dns zone, it centralizes the private dns functionality. Users can use a single private dns zone across all cluster deployments, regardless of cluster/network resource groups. As this RG selection is optional, it would be backwards compatible with existing clusters as well.

[kreeuwijk]

Additional comment from one of our customers that is affected by this gap today:

A centralized DNS zone offers the significant advantage of simplifying the integration with a central DNS resolver in Azure. Currently, each private DNS zone must be linked to the VNET of the companies central DNS server, which becomes increasingly difficult to manage as the number of clusters grows. Although a Private DNS Resolver could partially address this issue on a smaller scale, it is not a feasible solution for a very large number of private DNS zones due to inherent limitations.
By consolidating into a single private DNS zone for all clusters, the issues above could be elegantly resolved. This approach would streamline the management of DNS zones, reduce the risk of configuration errors, and enhance the scalability of DNS management in large-scale environments.