v1.7.9 - Introduces Rollup__mdt.IsDisabled__c flag

• Update API versions to 63
• Adds IsDisabled__c flag on Rollup Metadata records to allow for easier on/off switch (as opposed to updating the rollup control to a disabled control record)

v1.7.8 - RollupCalcItemReplacer bugfix

Fixes #660 by ensuring calls to split don't accidentally greedily split when detecting whether or not RollupCalcItemReplacer needs to run

v1.7.7 - Additional Multi-Currency Bugfixes

• Fixes an issue with the max date range being retrieved for dated currency rates, added more tests
• Updated the way custom field mappings work so that multiple "jumps" between object level relationships are officially supported; previously, only one "hop" was supported

v1.7.6 - Better INCLUDES Support

• Fixes #656 by adding better INCLUDES support in where clauses
• Updates RollupDateLiteral code path with suggestion from @surajp
• Hard delete RollupState__c records at the end of bulk full recalcs thanks to community suggestions

v1.7.5 - Advanced Currency Bugfixes

• started tweaking PMD rules to allow graceful cutover to new scanner when the time comes
• Fixes #653 by ensuring MSP values are parsed correctly when using INCLUDES
• Possible fix for #654 - was not able to repro this locally, but I'm guessing the update will fix the problem reported
• Fixed a weird date literal bug with Australia time zones due to oddities with toStartOfMonth().addMonth(1) logic
• Added a possible fix for #651, as well

v1.7.4 - Allow Cursor Usage From Batch Apex

• Fixes issue reported by internal colleague with RollupCalcItemReplacer when two rollups with parent-level fields use mutually exclusive where clauses on the same parent-level field
• Fixes #646 by providing escape hatch for cursor usage when batch apex is the originating point for Apex Rollup

v1.7.3 - Fix Namespace Shadowing Issue

• Fixes #643 by properly referencing Schema namespace in RollupFieldInitializer

v1.7.2 - RollupCalcItemReplacer bugfix

• Fixes issue reported with RollupCalcItemReplacer incorrectly detecting Type__r fields as polymorphic

v1.7.1 - Scheduled Rollup Updates & Date literal comparisons

• Fixes #640 by properly ensuring RollupDateLiteral comparisons occur in local time since they are always based off of the running user and their time zone
• Fixes an issue brought up in #639 where scheduling a rollup can inadvertently exceed the max query rows

v1.7.0 - Improve Memory Management for LDV Full Recalculations

This is the first (potentially) non-breaking change medium version bump, but in reflecting on the scope of the changes being made, it feels appropriate to make the leap to version 1.7.0. Here's a breakdown of the changes:

Overview

• RollupFullBatchRecalculator is no longer an actual batch class, per se, though it still does batch processing. Instead, it takes advantage of the Apex Cursor beta to iterate through children records when doing full recalculations where the number of records exceeds the value for RollupControl__mdt.MaxLookupRowsBeforeBatching__c. Like its batch class predecessor, it will continue to default to calculating rollup values for 500 children at a time - still configurable using RollupControl__mdt.BatchChunkSize__c. Using Database.Cursor has several big advantages over using a batch class, and while the feature is still in beta, I expect that it will GA soon
• Previously, Apex Rollup took advantage of Database.Stateful, which is a marker interface that's part of the Database.Batchable framework to calculate what a parent record's prior value was when that parent record had children split between batches. This happens more often than you'd expect, even with very fine-tuned ordering of the children records. With large data volume Salesforce orgs, keeping track of which parents have been updated while iterating through all of the children records represents a serious problem, and at a certain point when relying on Database.Stateful to calculate millions of records and dozens of rollups for a single child type, the rollup calculation will inevitably fail as memory runs out.
• While managing everything in-memory is extremely convenient as the package author, I'd basically arrived at the end of the road when it came to optimizing the batch full recalc process to try to avoid running out of memory. There were no tricks left - just the cold, hard fact that 12 MB of space will eventually run out with a one-to-many relationship between each parent record and the rollups being applied to that record when there are millions of parent records involved. This was the problem reported in #615
• ⚠️ Salesforce orgs have a MAXIMUM number of active DataWeave scripts that can be used. ⚠️
  If you saw the phrase "(potentially) non-breaking change" above and went 🤔, v1.7.0 adds a DataWeave script to the package. While I view it as unlikely that there are orgs out there running up against that limit, it's worth calling out here. Contact your AE if you truly cannot accommodate additional active DataWeave scripts and want to update. See the section titled "Easily Parsing Data" below for more info on why this change is being made
• Because records are now created during "batch" full recalculations, if you abort a full recalculation using the Apex Jobs page in setup, or via some other means, there is the potential that RollupState__c records will be orphaned and won't be cleaned up. You will have to delete those records manually. More info is available in the "Easily Deleting Data" section.

Challenges

There's one natural conclusion when an in-memory data management strategy is no longer sufficient, and that's a record-based memory management solution. As a package author, however, it would be irresponsible (not to mention untidy) to implement a feature that contributes significantly to the data storage on a subscriber org. This means that the "easy" approach - a data record (henceforth referred to as "state", or "Rollup State", or the API name for the object) for each parent - was immediately disqualified. The solution would have to compress data such that the previously calculated values for a parent record could be:

• easily retrieved,
• easily parsed
• easily deleted

Easily Retrieving Data

In order to facilitate the easy retrieval of data, v1.7.0 uses a new custom object, RollupState__c, which has on it 11 different custom text fields ranging from RollupState__c.RelatedRecordKeys0__c all the way through RollupState__c.RelatedRecordKeys10__c. Because you can't filter on long text area fields within SOQL, and because the maximum length of a standard text field is 255 characters, creating many fields like this is a crucial piece of the data compression strategy that makes it possible to store state for many parents within a single record. Assuming that most rollups will be keyed to standard Salesforce ids, it follows then that each related record key field can hold (255/(18 + 1) = 13 parent ids per field (the + 1 accounts for the comma separating each key).

Despite using 11 custom fields for this (which might sound like a lot), that ends up allowing for a maximum of 13 x 11 = 143 parent records' data being compressed into a single RollupState__c. Now that the kinks have been ironed out with the logic that keeps track of which related record key field is being used at any given time, it will be possible to further expand the number of custom fields being used if there's any concern with the data storage use of Rollup State records for these calculations (which would allow for the data to be even further compressed). That being said, since relevant records are deleted within each cursor "chunk" that's processed, I am hopeful that the data storage concerns being discussed here are purely theoretical.

Easily Parsing Data

This proved to be an unexpected hurdle - initial benchmarking on deserializing the long text area on each Rollup State record was poor. As in, for some chunks that had only a dozen state records, the deserialization of those records back into Apex-defined types was causing the 60 second Apex CPU time limit to be exceeded 🤯.

This necessitated the next big change to Apex Rollup - the introduction of a DataWeave script to perform the deserialization. Using DataWeave, the deserialization time was cut from a process that at times was taking 30+ seconds to an operation that has always finished within a single second. That's enough of a justification for me to proceed with this piece of tech, though I do still have plans to circle back on the poor performance of the native deserializer.

Easily Deleting Data

At the end of each recalculation job, any remaining RollupState__c records are deleted. This is something to be mindful of. If you were to abort a currently running recalculation job, this would effectively "orphan" those records, and they'll need to be deleted separately (by you, presumably). The framework will not (at this time) do any periodic maintenance on removing older RollupState__c records that could be orphaned in this manner, though I am open to the idea of giving users a way to conveniently do so via the Apex Rollup app should the need for that arise.

Special Considerations

One of the... downstream effects... of moving away from using Batch Apex was that the majority of the Apex Rollup framework now relies on Queueables to perform async processing. In order to accommodate this, v1.7.0 takes (greater) advantage of Queueable Finalizers to coordinate work between different threads, as finalizers are the only place where it's possible to fully chain different jobs together without exceeding limits.

As I had many failing tests when replacing the Batchable version of RollupFullBatchRecalculator with a Queueable, and I've been performing integration testing to sanity check huge swathes of this work, I don't expect that this change should be visible to people in anything other than a possible increase in Debug Logs being generated (as Finalizers create their own, separate, log file). I've mentioned it here, though, on the off-chance that this change does somehow lead to deleterious effects.

---

A huge thanks to @jongpie for reviewing #636 for this, and for being an invaluable colleague and friend