extend simple peer (not peerjs, oops) to handle buffered/packet transmission; add raw dependency w/MIT license

[disarticulate]

Closes #20

• Extends the simple peer class to override the send and _onChannelMessage functions to handle chunked data packets using custom defined packets encodePacket and decodePacket
• Adds bundled & minified (maybe you want to include it somewhere else) dependency to use integers for header details (8 bytes) as discussed in the issue, numbering packets.

The custom packet setup could probably be simplfied as we're counting each full data (txOrd), each packet sent (index), how many packets (length), the size of the packet's data (chunkSize) and the total size of the data (totalSize)

Ran the tests and get some typescript errors that come from the SimplePeerExtended extends Peer where we're using something from the parent class that isnt available. Also, the minified dependency would either need to be brought in officially, or ignored.

[dmonad]

Hey @disarticulate, thanks so much for this!

I like the approach that you have taken. But I'm a bit hesitant to release a new major release of y-webrtc with these breaking changes. However, I think it should be possible for us to make this change non-breaking by making it optional whether we adapt your polyfill (or another one which might fix other issues).

[dmonad]

I guess the reason why you don't use BigUint64Array is that it is not yet supported in Safari.

I developed an efficient encoder exactly for this problem: https://github.com/dmonad/lib0/blob/main/encoding.js

import * as encoding from 'lib0/encoding'
import * as decoding from 'lib0/decoding'

// example of encoding unsigned integers and strings efficiently

const encoder = new encoding.createEncoder()
encoding.writeVarUint(encoder, 256) 
encoding.writeVarString(encoder, 'Hello world!')
const buf = encoding.toUint8Array(encoder)

const decoder = new decoding.createDecoder(buf) 
decoding.readVarUint(decoder) // => 256 
decoding.readVarString(decoder) // => 'Hello world!' 
decoding.hasContent(decoder) // => false - all data is read

The documentation for other encoding techniques is here: https://github.com/dmonad/lib0

But I realize that this is already working in the current state. But maybe we can avoid pulling in more dependencies that are superflous.

[disarticulate]

For a header, we need to pad the values. Note the // 8 bytes comments I've mocked up a replacement class like:

class Int64 {
  constuctor (int64) {
    this.encoder = new encoding.createEncoder()
    encoding.writeVarUint(encoder, int64) 
  }
  toArrayBuffer () {
    return encoding.toUint8Array(this.encoder)
  }
}

However, when I test the recommended dependency, we get:

...
encoding.writeVarUint(encoder, 1)
encoding.toUint8Array(encoder)
> Uint8Array(1) [ 1 ]

its not obvious to me how to do pad/unpad safely, so the whole decode/encode would need a rewrite

[dmonad]

This is interesting. So you implemented a layer around simple-peer that handles this issue.

Would it be possible that you publish a separate package that we can include as a polyfill for the default implementation? This is already done in y-websocket where we define the WebSocket as an argument.

new WebsocketProvider(URL, room, { WebSocket: MyCustomWebsocketPolyfill })

We could do something similar to y-webrtc without breaking the existing API.

new WebrtcProvider(room, yjs, { SimplePeer: SimplePeerExtended })

I'd prefer that approach because it allows us to test out your implementation before breaking everyone else's existing deployments.

[disarticulate]

i can publish a module and put something up on github, but it seems like you'd need move SimplePeer to a peerDependency to avoid duplicating code, otherwise your library is pulling in Peer while your user is also pulling in a modified peer.

Also, it looks like WebrtcConn is what's using the Peer and that's being called from SignalingConn...etc...

I can't figure out how you'd thread that option into the proper slot because it call comes from a global import Peer from 'simple-peer/simplepeer.min.js'

[datakurre]

@disarticulate Is this patch the latest version you've been applying? I see errors when clients disconnect. Those seem to go away, if I include super.destroy() from you main branch.

Beyond that, a supported way to apply customized SimplePeer would be great 🙏.

[disarticulate]

@disarticulate Is this patch the latest version you've been applying? I see errors when clients disconnect. Those seem to go away, if I include super.destroy() from you main branch.

Beyond that, a supported way to apply customized SimplePeer would be great 🙏.

Yes. I haven't been developing with it recently, and haven't followed any yjs updates since this was proposed.

check https://github.com/disarticulate/y-webrtc for when it was developed. it looks like simple-peer would need be moved to a peer dependency. I think the behavior is stable, but how it should be integrated takes some considerations that I haven't returned to.

[disarticulate]

The boring answer is the encoder selected is encoding 64 bit integers. The arbitrary answer is it made the header simple to decode and encode. Someone could rightsize it all, some of those values are purely redundant. I considered a system that did packet routing so wanted space.

…

On Thu, Jul 28, 2022, 17:14 Vitor de Mello Freitas ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/SimplePeerExtended.js <#25 (comment)>: > +} + +class SimplePeerExtended extends Peer { + constructor (opts) { + super(opts) + this._opts = opts + this._txOrdinal = 0 + this._rxPackets = [] + this._txPause = false + this.webRTCMessageQueue = [] + this.webRTCPaused = false + } + + encodePacket ({ chunk, txOrd, index, length, totalSize, chunkSize }) { + const encoded = concatenate(Uint8Array, [ + new Uint8Array(new Int64BE(txOrd).toArrayBuffer()), // 8 bytes @disarticulate <https://github.com/disarticulate> Why do the values need to be padded to 8 bytes in the header? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFHWPLDSXMKA2FL3GAULI3VWMA2TANCNFSM45JCHYTA> . You are receiving this because you were mentioned.Message ID: ***@***.***>