Custom packet between velocity & paper

[PedroMPagani]

Is your feature request related to a problem?

Yes, Currently of course most people should not be noticing this as much, but a lot of packets such as block updates, ping/pongs, entity movement, particle packets, level packets, such packets tend to be smaller than most of the network-compression-threshold, I have noticed this after implementing metrics on pps on DonutSMP, and due to our gamemode players have huge farms which do thousands of block updates a second, including particles etc due to water streams and more, this used to be a huge issue for us, since when implementing the metrics I came to realize most of our backend -> proxy bandwidth was actually not being compressed. and a minority of the packets were, for this example here, 98% of our packet rate was smaller than 256 bytes. This would generate also millions of writes as well due to the high amount of packet rate with small byte size.

Describe the solution you'd like.

A "sort of" pack of a buffer, this buffer would contain multiples packets within, and get compressed at once only, allowing this buffer to have a way higher chance of getting a much better compression ratio, instead of not compressing it at all, I have implemented this and the results are quite great for me, my current implementation is sort of a shortcut, but I am sure there is probably a better design for this implementation.

Describe alternatives you've considered.

Well, the alternative is sort of don't have too many players or too much interactions XD

Other

Here are some metrics of the results:

[masmc05]

So basically, for the server to catch packets and then send a huge bundle packet?

[PedroMPagani]

So basically, for the server to catch packets and then send a huge bundle packet?

Sort of, it's not a bundle but a new packet, bundle doesn't have a single buffer

[vadcx]

How does turning off TCP_NODELAY compare to your implementation? In PPS, Avg/Median packet size and CPU time (probably sys time). Only on Paper's end in the scheme (Paper - localhost Velocity - Player)

To counteract the issue with latency I assume the send buffer of the socket must be reduced.

If the traffic to Velocity proxy is local, is uncompressed traffic really an issue? At what level do you measure the bandwidth? Does it include Ethernet encapsulation overhead or not?

I have no idea what the networking architecture here looks like, but some packets would benefit from immediate sends (at the expense of another syscall) like entity movement (the reason NODELAY was enabled after all) while others could be kept in a per type buffer (i.e. all particles together) purely for improved compression ratios and sent whenever, without an explicit flush or NODELAY. I think it's too optimistic to think such an overhaul will happen for a niche case of a busy server network. Maybe you can sponsor this? ;)

[PedroMPagani]

IF the proxy is local, there's no benefit mostly really due to the kernel handling it all, it doesn't even go through public on the first place between both. but on the scenairo of a lot of Minecraft Servers this might not be the case. No it does not include ethernet encapsulation. It's all just from the JVM and that's it, just CPU time mostly. I don't think sending packets like entity movement instantly. it doens't really optimize or benefit on anything from what I know. I have been running this for quite a while, with a lot of players and I have not seen anyone complain about anything like this.

[vadcx]

IF the proxy is local, there's no benefit mostly really due to the kernel handling it all,

Not for the sending side, but the receiving side should be hit with fewer packets. It's probably not noticable if you have idle CPU cores and memory bandwidth. Otherwise the Unix sockets for local connections were mentioned here before for efficiency and performance.

[PedroMPagani]

The receiving side does not receive less packets? if the proxy is local there's no reason to have compression. I don't think local proxies should be in this discussion, if you have a local proxy what you have is only the overhead of maybe the kernel which is minimal and the possible docker intras.

[electronicboy]

I have pondered stuff like this in the past but the architecture behind it sounds like a big headache of priorities vs my own slow progression back into working and such, as well as lower hanging fruit of the network manager itself being a larger issue in terms of day to day issues people face.

at the top, you have the general issue that hopping over into the netty pipeline can become a huge problem for these areas, it would be more more efficient if we could just batch calls over to the execute() method rather than having to pop over one by one, the buffer call at the end has never really been a huge issue;

it used to be that we could just tell people to disable compression, but, that is no longer a viable option given mojangs implementation around packet limits in which some people have hit the packet limit due to disabling compression on their backends, so it's not exactly an unlikely scenario, especially as this stuff gets more powerful and becomes a core part of servers.

being able to compress a packet on the proxy in a sizable manner does sound good, the only issue is the layer at which you do it and ensuring that you get the correct signaling in play, issue is though is, however, plugins are always a headache here in terms of how they expect the pipeline to work.

[PedroMPagani]

Yeah, in terms of plugins I have no idea exactly what would be fully affecting this, I can't see why a plugin would be touching something after the compression handler. My current patch seems to work within protocollib and packet events both in as well, so I don't know what kind of dogdy thing the plugin would be doing, but I bet if there's any breakage with this it would be such a small use-case