Correct way to cache node_modules + yarn caches in CI to minimise rebuilds?

[jdb8]

We're using the node_modules linker with yarn v4 in a monorepo, and we want to cache as much as possible across builds in CI to minimise unnecessary work. Our current strategy is to cache and restore the following:

• the entire top-level node_modules directory
• the contents of .yarn/cache
• .yarn/install-state.gz (after reading this recent comment)

Prior to adding .yarn/install-state.gz to the cache, we were getting slow link times + YN0007 errors since yarn couldn't tell that some binaries had already been built. Now that we cache that file, our YN0007's have turned into YN0008 warnings, i.e. "must be rebuilt because its dependency tree changed" warnings for all of our build scripts.

Beyond seeing YN0008, it's hard to tell exactly which "dependencies" have changed. Possible culprits based on my current research:

• the fact that we don't cache restore nested node_modules?
• some yarn state file that is using absolute paths, which reduces portability?
• some yarn state file that relies on mtimes, which are being wiped when we cache restore?

There's probably more detail I can go into, but at its core: what is the optimal way to safely cache and restore yarn's state in CI, assuming:

• our architecture is the same during save and restore (i.e. we're fine to cache platform-specific binaries)
• this is all in a fully controlled CI environment (i.e. no worries about lockfile poisoning)

Any help or pointers towards the algorithm yarn uses to invalidate installs would be greatly useful here - thanks!

[aspiers]

I probably have this question with yarn v3 too - admittedly I haven't tried adding .yarn/install-state.gz (for which the docs are here) to the cache yet, but I assume most likely I'll also see YN0007 turned into YN0008. BTW this question was also asked in actions/setup-node#785.

One would have hoped and expected that there would be a single maintained GitHub Action which takes care of all these details without individual developers having to waste time on it. I guess the intention used to be that https://github.com/marketplace/actions/yarn-install-cache would be that action, but then it got deprecated in favour of action/setup-node whose yarn support doesn't even cache node_modules/!

Furthermore, https://yarnpkg.com/features/caching#github-actions says:

We're still investigating the exact set of defaults that make GH Action caching more efficient. It's likely that we'll provide an official yarn-cache action mid-term for this purpose.

Surely this affects an extremely large group of devs who use yarn and GitHub Actions, so I really hope this can be addressed sooner rather than later.

[aspiers]

I added .yarn/install-state.gz and it worked even without any YN0008 errors. I'm guessing that was because we were already caching and restoring all node_modules/ hierarchies. The only final gotcha was that when this file is restored from cache, running yarn install will skip yarn postinstall, and we happened to have a key step in that which is required for CI, so I had to add that step to the CI after yarn install. Now it's working great, but it's still depressing that so much effort was required to reinvent wheels here.