[Bug] 使用triple协议， 概率性的偶发 Invoke remote method timeout.

[hosen-lhs]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

Env： Dubbo + Nacos
Dubbo： 3.3.2
Nacos： 2.4.0

我们采用采用Nacos + dubbo的服务框架，由于我们原先采用的是REST服务进行内部调用， 升级Nacos + Dubbo以后， 为了保证能够全局使用Dubbo服务治理， 我们采用uri到类名的映射关系进行泛化调用。 在生产环境， 偶发性出现Invoke remote method timeout。 我们目前采用k8s进行部署， 出现问题的时候， 不会全部有问题， 会集中在某几个pod里面。

有下面几种场景都遇到过：

1. 多个consumer应用的pod 调用同一个provider大批量的接口出现Invoke remote method timeout。
2. 单个consumer调用所有provider应用， 大批量接口有问题。
3. 单个consumer调用单个provider应用有问题， 其他没问题。

具体堆栈：
org.apache.dubbo.rpc.RpcException: Invoke remote method timeout. method: $invoke, provider: DefaultServiceInstance{serviceName='da', host='172.22.2.83', port=50051, enabled=true, healthy=true, metadata={dubbo.metadata-service.url-params={"serialization":"hessian2","prefer.serialization":"hessian2,fastjson2","version":"2.0.0","dubbo":"2.0.2","release":"3.3.2","side":"provider","port":"50051","protocol":"tri"}, sentinel-transport-port=8719, dubbo.endpoints=[{"port":50051,"protocol":"tri"}], dubbo.metadata.revision=a7d4f6cb5616b15d3a88f548a88f181b, dubbo.metadata.storage-type=local, meta-v=2.0.0, timestamp=1738810956783}}, service{name='com.jooan.qacloud.da.controller.api.InnerDaService',group='',version='',protocol='tri',port='50051',params={check.serializable=false, side=provider, release=3.3.2, methods=alilvRegister,batchMarkDevice,clearDeviceMark,deviceCheck,deviceMark,deviceRegister,getBatchSizeByBatchNo,getDeviceInfoByDnPk,getDeviceInfoBySn,logout,queryDeviceInfo,queryDeviceInfoList,queryDeviceProductInfo,queryMqttOnline,query_pc_url,query_product_info,set_device_properties,set_device_region,statisticalUserBehavior,updateDeviceCsGiveStatus, logger=slf4j, deprecated=false, dubbo=2.0.2, interface=com.jooan.qacloud.da.controller.api.InnerDaService, service-name-mapping=true, generic=false, revision=3.0.0, serialize.check.status=WARN, serialization=hessian2, application=da, prefer.serialization=hessian2,fastjson2, dynamic=true, dispatcher=message, REGISTRY_CLUSTER=default:6a3d2225-a2e4-4fb0-bacc-e218dd3fe7ad},}, cause: Timeout after 30000ms waiting for result.
at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:331)
at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:195)
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71)
at org.apache.dubbo.rpc.filter.GenericImplFilter.invoke(GenericImplFilter.java:139)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.validation.filter.ValidationFilter.invoke(ValidationFilter.java:102)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.RpcExceptionFilter.invoke(RpcExceptionFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at io.opentelemetry.javaagent.shaded.instrumentation.apachedubbo.v2_7.TracingFilter.invoke(TracingFilter.java:56)
at io.opentelemetry.javaagent.instrumentation.apachedubbo.v2_7.OpenTelemetryFilter.invoke(OpenTelemetryFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:106)
at org.apache.dubbo.registry.client.ServiceDiscoveryRegistryDirectory$InstanceWrappedInvoker.invoke(ServiceDiscoveryRegistryDirectory.java:800)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:412)
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:366)
at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboConsumerFilter.syncInvoke(SentinelDubboConsumerFilter.java:83)
at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboConsumerFilter.invoke(SentinelDubboConsumerFilter.java:67)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:109)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at com.alibaba.csp.sentinel.adapter.dubbo3.DubboAppContextFilter.invoke(DubboAppContextFilter.java:47)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.tracing.filter.ObservationSenderFilter.invoke(ObservationSenderFilter.java:60)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:119)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:101)
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:107)
at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:171)
at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:294)
at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:64)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:81)
at com.alibaba.dubbo.rpc.service.GenericServiceDubboProxy0.$invoke(GenericServiceDubboProxy0.java)
at com.jooan.qacloud.commons.utils.CseHttpUtil.postByDubbo(CseHttpUtil.java:255)
at com.jooan.qacloud.commons.utils.CseHttpUtil.postForEntity(CseHttpUtil.java:225)
at com.jooan.qacloud.commons.utils.CseHttpUtil.lambda$postForEntity$2(CseHttpUtil.java:164)
at com.jooan.qacloud.commons.utils.CseHttpUtil.exceptionToNullWrapper(CseHttpUtil.java:177)
at com.jooan.qacloud.commons.utils.CseHttpUtil.postForEntity(CseHttpUtil.java:164)
at com.jooan.qacloud.iot.IotClient.request(IotClient.java:41)
at com.jooan.qacloud.iot.IotClient.getDeviceInfo(IotClient.java:117)
at com.jooan.qacloud.vas.service.VasProvisionService.queryDeviceInfo(VasProvisionService.java:53)
at com.jooan.qacloud.vas.action.VasProvisionAction.vasProvision(VasProvisionAction.java:162)
at com.jooan.qacloud.vas.controller.PayController.vasProvision(PayController.java:75)
at com.jooan.qacloud.vas.controller.PayControllerDubboWrap5.invokeMethod(PayControllerDubboWrap5.java)
at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:89)
at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:100)
at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:55)
at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at org.apache.dubbo.rpc.filter.ClassLoaderCallbackFilter.invoke(ClassLoaderCallbackFilter.java:38)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.validation.filter.ValidationFilter.invoke(ValidationFilter.java:102)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.tri.rest.filter.RestFilterAdapter.invoke(RestFilterAdapter.java:38)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.tri.h12.HttpContextCallbackFilter.invoke(HttpContextCallbackFilter.java:37)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:80)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:45)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at com.alibaba.csp.sentinel.adapter.dubbo3.SentinelDubboProviderFilter.invoke(SentinelDubboProviderFilter.java:82)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at io.opentelemetry.javaagent.shaded.instrumentation.apachedubbo.v2_7.TracingFilter.invoke(TracingFilter.java:56)
at io.opentelemetry.javaagent.instrumentation.apachedubbo.v2_7.OpenTelemetryFilter.invoke(OpenTelemetryFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:109)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:55)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.AccessLogFilter.invoke(AccessLogFilter.java:120)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:222)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at com.jooan.qacloud.dubbo.rpc.generic.rest.filter.RestGenericFilter.invoke(RestGenericFilter.java:187)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.protocol.tri.h12.HttpContextFilter.invoke(HttpContextFilter.java:50)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:54)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:41)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
at org.apache.dubbo.metrics.filter.MetricsProviderFilter.invoke(MetricsProviderFilter.java:37)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.tracing.filter.ObservationReceiverFilter.invoke(ObservationReceiverFilter.java:59)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ProfilerServerFilter.invoke(ProfilerServerFilter.java:66)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:191)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerCallListener.invoke(AbstractServerCallListener.java:69)
at org.apache.dubbo.rpc.protocol.tri.h12.UnaryServerCallListener.onComplete(UnaryServerCallListener.java:50)
at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$Http2StreamingDecodeListener.onClose(GenericHttp2ServerTransportListener.java:206)
at org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.close(DefaultListeningDecoder.java:47)
at org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onClose(StreamingDecoder.java:60)
at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:135)
at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.close(LengthFieldStreamingDecoder.java:81)
at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener.onDataCompletion(GenericHttp2ServerTransportListener.java:148)
at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener.onDataCompletion(GenericHttp2ServerTransportListener.java:50)
at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:188)
at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:172)
at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:105)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.TimeoutException: Timeout after 30000ms waiting for result.
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:223)
at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:293)
... 123 more

Steps to reproduce this issue

概率性出现， 出现以后就一直有问题。 大概率无法自动恢复。

What you expected to happen

给出问题的解决方案。

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct