Proposal: Nxext P2P task building

[Jordan-Hall]

The goal here is to see what the view are for a distributed team-based tasking building and allow people to contribute computing rather than code to open source projects.

Motivation

Working recently on Distributed/grid computing recently and got me thinking we could speed up the build process similar to the NX cloud approach but you can build using your team's computer.

From Altan Nrwl cloud core team:

The agents used for Nx Cloud are configured and spun up by users of an individual workspace, normally with something that makes it easy to setup and tear down machines quickly, like GH Action runners or their equivalents for other CI providers.
There’s no VMs or hardware provisioned by Nrwl for distributed execution. As a concept, though, distributing tasks across your colleagues machines automatically would be kind of cool. I don’t think that’s something we have or are currently exploring.

How would it work?

In essence, we will be wrapping NX CLI to listen to commands. We use NX graph to build a tree of tasks that be run in isolation and feedback to the main tasks.

During install, we will install a small local node server that will listen to requests coming into your machine. The script reads the runKey and send it to our "signal server" and connect to available agents that had your "runKey" inside the "contributeTo" array.

Nx.json

{
  nxext-p2p: {
     runKey: "12345",
     computeBy: "anyone",
     contributeTo: [
      "nxext"
    ] 
  },
}

Locally:

If you are only set to computeBy "internal" then only your team will contribute to computing the task. We do this by cloning the existing repo locally and pnpm style sys link to node_modules (this is to reduce un need strain and file space). Otherwise, the script will run git clone on the repository and install node modules

Docker:
We can also run Docker containers on the user's machine to perform the tasks and then teardown after

Once the task is completed we zip the finished artefact up and set it back to the "original" job.

Nxext would then offer an API to allow projects to provide "credit" to people that's contributed their computer power to open source.

Main benefits of this proposal

• Decentralise task building
• Contribute to open source by "lending" your machine to help build projects

[santoshyadavdev]

This is a great feature, some questions though

1. This can be exploited by crypto bros to mine bitcoins, what can be done to avoid it
2. Why this when we have unlimited build time with GitHub Actions
3. Can we have read and read/write token as Nx does, where read token can only read from cache

[Jordan-Hall]

1. This can be exploited by crypto bros to mine bitcoins, what can be done to avoid it

This is something to bare in mind. Be interesting to see what others have done to prevent this.

2. Why this when we have unlimited build time with GitHub Actions

The main idea here is for local development but at the same time, we can help make CICD less painful, if we setup the build pipelines

3. Can we have read and read/write token as Nx does, where read token can only read from cache

Yes that sounds like a good idea.

Thank you @santoshyadavdev. This is why I love feedback to know if its something worth developing.