Server-side positional audio

[tuokri]

I'm currently working on implementing server-side positional audio via RPCs. The game server would control all PA parameters via Murmur RPCs for all users, and Murmur would then relay them to the player Mumble clients. I'm aware of the latency implications if the game server and Murmur are physically located far away from each other. The motivation is to prevent clients from sending forged PA data and to keep most of the logic on the server(s) similarly to how built in VOIP works in most games. I started working on this to offer an alternative to the horribly buggy Vivox implementation in Rising Storm 2: Vietnam.

As of now I've implemented getPosition, setPosition and removePosition ICE RPCs. I'm simply setting the position for audioData on the Server before processMsg. I was thinking of defining new Protobuf messages for the rest of the PA values (orientation, PA settings, etc.) and sending those to the clients in the RPCs.

I wanted to ask for any pointers, whether my approach is bad, is there something obvious I missed, etc. or any tips in general if this has been considered before.

This is my working branch: https://github.com/tuokri/mumble/tree/positional-rpc

[Krzmbrzl]

If I understood you correctly, you want to create the positional audio effect on the serverside.
While currently possible, this is something that we would not merge in main Mumble as it would conflict with a possible end-to-end encryption of audio packets. Except if that encryption was homomorphic, which is somewhat unlikely (and even then it is questionable whether things would work out)

[Krzmbrzl]

Ah no. You only set the position payload on the server. That doesn't conflict with E2E.
However, if you are worried about folks forging their positional data you also have to take into account that users could simply disable positional audio effects locally on their client or use a modified Mumble version that forges the data anyway before it is processed.
Aka: I'm not entirely sure what you gain by the serverside solution 🤔

As far as the Ice interface goes, I think you should bulk-update player positions rather than updating each player individually in order to minimize the overhead of each individual Ive call.
New Protobuf messages will probably make things more complicated than you would gain...

[tuokri]

If I understand it correctly, currently each client is responsible of telling the server their position. I'm not worrying about clients disabling PA or modifying data that is sent to them by the server, I'm worried about the clients sending garbage/fake/etc. positional data to the server, which is then sent to every other client. Maybe the server side approach is overkill, but I'd rather not trust anything the clients send. With client side approach, in theory, a client would be able to listen to locations that the client's in-game avatar is not located at, as a form of cheating. Also I would probably make a bare bones custom Mumble client with everything except the essential functionality stripped out.

Good point about bulk-updating the positions.

I've drawn a couple rough diagrams to better illustrate what I'm trying to achieve:

Deployment diagram:

Messaging sequences:

In order to emulate how the game's built-in (broken) VOIP works, I would also have to control much of the PA options on server side. I.e. the game has multiple channels for team, squad and proxy (positional) voice chat. The server would have to control who is able to hear whom (with the help of ACLs?). Proxy voice should only be sent to clients that are within hearing distance of other clients (based on in-game rules). Clients should only be allowed to speak on channels that they have access to in-game, on the game server. Clients would not be able to jump between channels manually, only when the server moves them (via RPCs).

This might seem overambitious or bloated in scope, but I have all the time in the world to work on this.

[Krzmbrzl]

The way positional audio is currently handled is that all clients receive all audio packets (as long as the context is the same) and then the client does the fade-off/mute effect based on the attached positional data. Thus, if a player wants to cheat, they can, regardless of what positional data was sent with the audio packet.

In order to circumvent that, you'd have to do the mixing on entirely the server side, which is currently not supported at all and probably also has latency implications.

Control over where a user can move to etc. can be done via ACL, but then bypassing these rules via RPC might only work under certain circumstances (the issuing user must have the privilege to move other users into the respective channels).

I think you would be better off with a client side plugin that moves users back if they move to a channel they are not supposed to go. This would also give you access to the client's settings, which the server has no access to at all.
To make sure that the plugin remains enabled, you could e.g. make the plugin move the Iser into a password protected channel (where only the plugin knows the password) and move them back out if the plugin is disabled.
You'd still have issues with people being able to use a custom build of the client to intercept that password, but I would guess that chances of that happening are somewhat low (unless you are planning on entering big e-sports or something). But then again, this would be a kind of cheating that is almost impossible to prevent as long as you do audio processing locally (as is the Mumble philosophy).