Recursive preload in js-ipfs is fetching MBs of refs data

[olizilla]

The first data-set listed on the IPLD Explorer is the "Project Apollo Archives" which is ~57GB... when we fetch the directory listing, everything works as expected, but in the background js-ipfs hits the preload nodes to recursively fetch the entire dag.

That's not great, as ipld explorer is built around the idea that you can fetch arbitrarily huge datasets by only fetch thing subset that you browse... as it stands multiple preload nodes fetch the entire data set and just the refs response alone means ~13MB of json listing out all the refs in the dag is streamed to th browser. For larger datasets the problem would get worse.

It's a nice optimisation for js-ipfs to recursively preload the entire dag, but its expensive for larger datasets.

TODO

• set recursive to false on the refs call while preloading. That way we'd let the preload nodes find the blocks for us without pre-emptively fetching a huge dataset.

[olizilla]

The refs response for the apollo archive is 10MB compressed, 15MB uncompressed...

[olizilla]

@achingbrain am I right in thinking there is not currently a way to non-recursively pre-load things? I'm looking at https://github.com/ipfs/js-ipfs/blob/afcf142a7da8559500e4db0eaf739e9bfbf3a8b8/packages/ipfs/src/core/preload.js#L49 which is hardcoded to always pass r=true (aside, that opt seems to be missing from the docs https://docs.ipfs.io/reference/http/api/#api-v0-refs)

The explorer app would like to be able to let ipld do a get on a cid without triggering a recursive fetch of a abitrarily large dateset on the preload nodes. Would it make sense to provide a config option to overide that flag, or another arguement option to the api methods that trigger preloading?

cc @lidel

[olizilla]

or should i just disable pre-loading altogether?

[achingbrain]

I think maybe just disable it entirely - preloading is supposed to increase the availability of newly added content, which isn't something ipld-explorer is for.

[olizilla]

Would we also lose some discovery power too? I assume preloading acts as a rough way to delegate dht look ups to the preload nodes too...

[achingbrain]

Not AFAIK, the delegate routers are there to make DHT queries on behalf of js-IPFS nodes, but preloading is separate from that.

[lidel]

Related: https://github.com/ipfs/in-web-browsers/issues/173
TLDR: we may save a lot of bandwidth if preload is aware of unixfs structure and only preloads dir + roots of its items, and not entire dag.

[lidel]

This depends on ipfs/js-ipfs#3510

[lidel]

we switched to helia + preload nodes are gone. this is no longer happening :)