Cache compiled script on disk #2099

patriksvensson · 2018-03-23T16:42:49Z

Would be nice if we could cache compiled script to disk if it haven't changed.

guillaume86 · 2018-09-20T17:34:53Z

FYI I experimented a bit with caching the compiled script. I'd like to use Cake as a task runner and the 4 sec penalty seemed a bit much for running tasks that take under 1 sec on their own.

I just used Emit(path) to generate an assembly and figured out how to call it by reflection, and it seems encouraging:

normal run

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0114282
--------------------------------------------------
Total:                        00:00:00.0114282


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 5
Milliseconds      : 448
Ticks             : 54489236
TotalDays         : 6,30662453703704E-05
TotalHours        : 0,00151358988888889
TotalMinutes      : 0,0908153933333333
TotalSeconds      : 5,4489236
TotalMilliseconds : 5448,9236

compile run (build cache and run)

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0146321
--------------------------------------------------
Total:                        00:00:00.0146321


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 5
Milliseconds      : 347
Ticks             : 53478949
TotalDays         : 6,1896931712963E-05
TotalHours        : 0,00148552636111111
TotalMinutes      : 0,0891315816666667
TotalSeconds      : 5,3478949
TotalMilliseconds : 5347,8949

cached run

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0095663
--------------------------------------------------
Total:                        00:00:00.0095663


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 689
Ticks             : 6890002
TotalDays         : 7,97453935185185E-06
TotalHours        : 0,000191388944444444
TotalMinutes      : 0,0114833366666667
TotalSeconds      : 0,6890002
TotalMilliseconds : 689,0002

It shaved off 4.5 secs for cached runs and is now acceptable to run quick tasks, it will probably help with #2179.

I attached the changed file if anyone is interested (lots of things missing for a proper pull request).
RoslynScriptSession.cs.txt

alberto-chiesa · 2019-02-19T09:35:08Z

Hi, I would like to try to tackle this task and I'm looking for some suggestion on how to proceed.
I have some questions:

Where should I store the precompiled assembly? I see two options: (a) beside the build.cake file as a build.cake.dll and (b) inside a subdirectory under the cake.exe runner (like ./tools/cake/cache)
Do you think the last modified date is a good enough heuristic to check if a new compilation is due? I mean, before running the cache is checked and, if a compiled dll is found, the compilation date is compared against the one of the cakefile. If the compilation is after the last modified date of the cakefile, the precached version is used. An alternative is storing the dll as build.cake.md5ofthecakefile.dll and check the hash.
Do you think the precaching should be enabled by default, opt-in or opt-out? In case some command line switch should be added, can I have some pointers on where to look for example code?

Thanks!

gep13 · 2019-02-19T10:02:41Z

@alberto-chiesa there is an open RFC here: cake-build/rfcs#2 which discusses this, and we would need to get sign off there on the approach before proceeding with any work on this feature.

alberto-chiesa · 2019-02-19T10:24:14Z

Thanks @gep13 ! I didn't knew about the RFC repo. It seems the linked proposal answers my exact same questions. Nice to see I wasn't too off-target. Continuing there about the points to discuss.

alberto-chiesa · 2019-02-19T10:53:56Z

Thanks @gep13 ! I didn't knew about the RFC repo. It seems the linked proposal answers my exact same questions. Nice to see I wasn't too off-target. Continuing there about the points to discuss.

jamiegs · 2019-05-21T15:51:57Z

Is there any updates on this? the slowness of the startup of cake has been fairly annoying for me and I'm about to look into switching to Nuke for this reason only.

patriksvensson · 2019-05-21T16:01:11Z

@jamiegs No update on this since no one have submitted a PR for it yet.

devlead · 2019-05-21T16:02:20Z

@jamiegs happy to take contributions on this, it's not that we don't want this Core team currently have a few other things with higher priority atm.

If all you want is a is a pre compiled .NET Core app you could take a look at Frosting or Cake.Bridge

https://github.com/cake-build/frosting

https://github.com/devlead/Cake.Bridge/tree/master/src/CakeConsoleRunner

Frosting is a more Asp.Net core style and Cake.Bridge is more like 1:1 mapping of Cake script to console app.

Differences from RFC-02 1) The cache argument was removed. A new section was added to the cake.config file [Cache] with 2 possible properties Enabled and Path. Enabled defaults to false and Path defaults cache under the tools folder. 2) I originally used a time stamp to determine if the cache was valid or not. I have changed this to use a hash like pull request 2584 used. I did change it to use SHA1CryptoServiceProvider instead of MD5 since MD5 is not FIPS compliant.

GeertvanHorrik · 2020-12-02T09:26:05Z

Our situation:

We currently use (quite large) shared build scripts in all our repositories. Each stage (initialize, build, test, package, deploy, finalize) is a separate stage and requires recompilation of the cake scripts. Compiling the build scripts takes about 20 seconds, so it would be very sweet if we could decrease this to 0.

Since we have different agents (although the build server tries to run everything for a specific build on the same agent), we would like to cache this. I see 2 options:

In the %temp% based on the commit id (a huge improvement for us would be if we could cache based on commit id (saves us compiling 5 times (so 1 instead of 6)).
In the %temp% based on the combined hash of the script files (I am pretty sure that calculating the hash is faster than compiling). Then we could re-use the cache for the same build scripts for different repros
In the repository itself, but then we would have to be aware of recompilation every time. In our case that's not really an issue since we use RepositorySync so we only have to do it once.

My preferred option is 2. If there is anything I can do to help, let me know.

gep13 · 2020-12-02T09:41:47Z

@GeertvanHorrik thanks for your thoughts. Not sure if you have seen it, but there is an open RFC regarding the addition of caching into Cake. Before moving forward with any implementation, we need to get agreement regarding the way forward. This RFC is here:

cake-build/rfcs#2

If you wanted to read through that (if you haven't already) and add some thoughts to the PR, we can continue to refine the RFC, and then begin on the implementation.

devlead · 2020-12-02T09:43:14Z

There's an open PR #2650 to be reviewed after 1.0 right?

gep13 · 2020-12-02T09:58:21Z

@devlead yes, I believe that there is a PR for providing some of the functionality, however, we haven't agreed via the RFC exactly what we are hoping to achieve with this functionality, so until that is addressed, I don't think we can move forward with any PR.

devlead · 2020-12-02T10:04:23Z

Yip just wanted to make sure we didn't duplicate any effort

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

* Enable * Config: Settings.EnableScriptCache=true * Arg: --settings_enablescriptcache=true * Env: CAKE_SETTINGS_ENABLESCRIPTCACHE=true * Cache Location * Config: Paths.Cache = /path/to/cache * Arg: --paths_cache=/path/to/cache * Env: CAKE_PATHS_CACHE=/path/to/cache * fixes cake-build#2099 Co-authored-by: [email protected]

* relates to cake-build#2099

GH2099: Fix and add script cache integration test

cake-build-bot · 2022-04-14T22:54:07Z

🎉 This issue has been resolved in version v2.2.0 🎉

The release is available on:

Your GitReleaseManager bot 📦🚀

RehanSaeed · 2022-04-18T12:19:44Z

Any reason this is not turned on by default? Is it just off for the initial release? I'd prefer to avoid adding yet another config file to the root of my projects enable this feature if possible.

augustoproiete · 2022-04-18T13:24:03Z

@RehanSaeed Yes, it's a complex feature... We'll let it stabilize first and consider turning on by default in a future release

devlead · 2022-04-18T13:25:25Z

New major features generally opt-in to break as few builds as possible and enable iron out any kinks before enabling it for all.

Config file isn't required you can also configure via environment variable or command line arg

https://cakebuild.net/docs/running-builds/configuration/default-configuration-values#cache-compiled-script-on-disk

patriksvensson self-assigned this Mar 23, 2018

patriksvensson added the Feature label Mar 23, 2018

patriksvensson removed their assignment Mar 23, 2018

This was referenced Jul 19, 2019

HUDL - SPIKE - Adds output caching to cake runtime hudl/cake#1

Closed

GH2099: Add assembly gen and caching #2584

Closed

pascalberger added this to the v2.0.0 Candidate milestone Dec 2, 2020

pascalberger linked a pull request Dec 2, 2020 that will close this issue

GH2099: Cache compiled script on disk #2650

Merged

tstewart65 added a commit to tstewart65/cake that referenced this issue May 29, 2021

cake-buildGH-2099: Also takes care of RFC-02.

c81ce95

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

tstewart65 added a commit to tstewart65/cake that referenced this issue Oct 13, 2021

cake-buildGH-2099: Also takes care of RFC-02.

85d947c

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

pascalberger pushed a commit to tstewart65/cake that referenced this issue Feb 6, 2022

cake-buildGH-2099: Also takes care of RFC-02.

b4d5851

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

pascalberger modified the milestones: Next Major Candidate, Next Minor Candidate, v2.2.0 Feb 6, 2022

devlead pushed a commit to tstewart65/cake that referenced this issue Feb 15, 2022

cake-buildGH-2099: Also takes care of RFC-02.

3f89b58

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Mar 18, 2022

cake-buildGH-2099: Also takes care of RFC-02.

fdc51e8

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Mar 20, 2022

cake-buildGH-2099: Also takes care of RFC-02.

dd54f3c

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Mar 22, 2022

cake-buildGH-2099: Also takes care of RFC-02.

8d1f41f

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Mar 22, 2022

cake-buildGH-2099: Also takes care of RFC-02.

8bad918

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Mar 22, 2022

cake-buildGH-2099: Also takes care of RFC-02.

746d765

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead pushed a commit to tstewart65/cake that referenced this issue Apr 2, 2022

cake-buildGH-2099: Also takes care of RFC-02.

abba8ad

This is the same as pull request 2650 but has been updated to match the change made in Cake version 1.0.0.

devlead closed this as completed in #2650 Apr 3, 2022

devlead added a commit to devlead/cake that referenced this issue Apr 6, 2022

(cake-buildGH-2099) Fix and add script cache integration test

4cc8c5e

* relates to cake-build#2099

devlead mentioned this issue Apr 6, 2022

GH2099: Fix and add script cache integration test #3863

Merged

devlead added a commit to devlead/cake that referenced this issue Apr 8, 2022

(cake-buildGH-2099) Fix and add script cache integration test

c9cb034

* relates to cake-build#2099

devlead added a commit that referenced this issue Apr 8, 2022

Merge pull request #3863 from devlead/feature/GH-2099-integrationtest

fdb906a

GH2099: Fix and add script cache integration test

pascalberger mentioned this issue Apr 18, 2022

Add documentation for caching of scripts cake-build/website#2431

Open

gep13 mentioned this issue May 6, 2022

Adds RFC proposal for caching generated assembly cake-build/rfcs#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache compiled script on disk #2099

Cache compiled script on disk #2099

patriksvensson commented Mar 23, 2018

guillaume86 commented Sep 20, 2018 •

edited

Loading

alberto-chiesa commented Feb 19, 2019 •

edited

Loading

gep13 commented Feb 19, 2019

alberto-chiesa commented Feb 19, 2019

alberto-chiesa commented Feb 19, 2019

jamiegs commented May 21, 2019

patriksvensson commented May 21, 2019

devlead commented May 21, 2019 •

edited

Loading

GeertvanHorrik commented Dec 2, 2020 •

edited

Loading

gep13 commented Dec 2, 2020

devlead commented Dec 2, 2020

gep13 commented Dec 2, 2020

devlead commented Dec 2, 2020

cake-build-bot commented Apr 14, 2022

RehanSaeed commented Apr 18, 2022

augustoproiete commented Apr 18, 2022

devlead commented Apr 18, 2022

Cache compiled script on disk #2099

Cache compiled script on disk #2099

Comments

patriksvensson commented Mar 23, 2018

guillaume86 commented Sep 20, 2018 • edited Loading

normal run

compile run (build cache and run)

cached run

alberto-chiesa commented Feb 19, 2019 • edited Loading

gep13 commented Feb 19, 2019

alberto-chiesa commented Feb 19, 2019

alberto-chiesa commented Feb 19, 2019

jamiegs commented May 21, 2019

patriksvensson commented May 21, 2019

devlead commented May 21, 2019 • edited Loading

GeertvanHorrik commented Dec 2, 2020 • edited Loading

gep13 commented Dec 2, 2020

devlead commented Dec 2, 2020

gep13 commented Dec 2, 2020

devlead commented Dec 2, 2020

cake-build-bot commented Apr 14, 2022

RehanSaeed commented Apr 18, 2022

augustoproiete commented Apr 18, 2022

devlead commented Apr 18, 2022

guillaume86 commented Sep 20, 2018 •

edited

Loading

alberto-chiesa commented Feb 19, 2019 •

edited

Loading

devlead commented May 21, 2019 •

edited

Loading

GeertvanHorrik commented Dec 2, 2020 •

edited

Loading