GH2099: Add assembly gen and caching

[codeimpossible]

This updates the Roslyn script generator to emit the generated code to a DLL. The DLL is stored in .cache/ underneath the directory the Cake.dll resides in.

This work is heavily influenced by the ideas in #2099 as inspiration as well as the posted RFC: https://github.com/cake-build/rfcs/pull/2/files#diff-9b01d3179a039b60e4fcdb5e273d48c2.

If the cached assembly exists it will be loaded and executed, unless the --recompile flag is sent as an arg into cake.

For a bit more context, you can check out the original PR on our fork: hudl#1.

Results

I ran the new cake version against an internal project that we have that uses cake (one of many) and recorded the execution times of a fresh run and a cached run. FYI the paths still show cake 0.33.0 but that is just because I replaced the cake files there with the built ones from this branch (see "How do I test this?" below).

Fresh run

$> dotnet ./.marvel/dev/tasks/tools/cake0.33.0/cake.coreclr/0.33.0/Cake.dll build.cake --target=help --verbosity=Verbose

Task                           Duration
---------------------------------------------------
Hudl Command Line Tools - Help 00:00:00.0358184
Teardown                       00:00:00.6338775
---------------------------------------------------
Total:                         00:00:00.6696959


$> took
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 8
Milliseconds      : 716
Ticks             : 87164761
TotalDays         : 0.000100885140046296
TotalHours        : 0.00242124336111111
TotalMinutes      : 0.145274601666667
TotalSeconds      : 8.7164761
TotalMilliseconds : 8716.4761

Cached run

$> dotnet ./.marvel/dev/tasks/tools/cake0.33.0/cake.coreclr/0.33.0/Cake.dll build.cake --target=help --verbosity=Verbose

Task                           Duration
---------------------------------------------------
Hudl Command Line Tools - Help 00:00:00.0690991
Teardown                       00:00:00.6805018
---------------------------------------------------
Total:                         00:00:00.7496009

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 548
Ticks             : 15487039
TotalDays         : 1.79248136574074E-05
TotalHours        : 0.000430195527777778
TotalMinutes      : 0.0258117316666667
TotalSeconds      : 1.5487039
TotalMilliseconds : 1548.7039

How do I test this?

• Clone the repo locally, then run ./build.ps1 from the root
• Take the contents of the ./artifacts/v0.35.0-HUDL-SpikeAddAss0001/Cake-bin-coreclr-v0.35.0-HUDL-SpikeAddAss0001.zip file and replace the contents of cake.coreclr in your marvel projects tools directory. Then you can run dotnet <path/to/cake.dll> <path/to/build.cake> --target=<TARGET> --verbosity=Diagnostic --cache_enabled=true to run the new cake version with the verbose logs turned on for more context.

How the cache is invalidated

The cache is invalidated by any cake script changes. This adds ~2s to the execution time in my testing (because the analyzer has to run and hash all the script files). Still, reducing pre-script "build time" from 10s to 3s is still pretty impressive.

I updated the code so that the cache can be enabled via the built-in cake configuration, either with CAKE_CACHE_ENABLED=true and CAKE_CACHE_FORCE_RECOMPILE=true environment variables, --cache_enabled=true --cache_force_recompile=true args, or by adding the following to the cake.config:

[Cache]
Enabled = true
Force_Recompile = true

Default values for both of these are false.

[dnfclas]

All CLA requirements met.

[codeimpossible]

@patriksvensson any updates on this?

[patriksvensson]

@codeimpossible I think @mholo65 is the one who should review this 😄

[gep13]

@patriksvensson said...
@codeimpossible I think @mholo65 is the one who should review this 😄

We should likely also review and finalise this RFC before progressing any further with this PR.

[bjorkstromm]

@patriksvensson said...
@codeimpossible I think @mholo65 is the one who should review this 😄

We should likely also review and finalise this RFC before progressing any further with this PR.

Yes, my comment on the RFC is one of the unresolved issues (I think) https://github.com/cake-build/rfcs/pull/2/files#r229862533

[pascalberger]

@codeimpossible Thank you very much for your pull request 👍

We received several pull requests for this feature. We decided to merge #2650, and the feature will be available in upcoming Cake 2.2.

Because of the above reason I'll close this PR. Thank you again for your work on this