feat: simulator generation mode improvements #682

georgi-l95 · 2025-02-20T10:06:54Z

Description

With this PR we aim to introduce new generator mode for the block stream, by introducing new implementation for the BlockStreamManager. The following changes are included in this pull request:

Addition of new CraftBlockStreamManager implementation with the purpose of crafting blocks on the fly and on demand. Currently we are adding random amount of items(transactions) in the block, according to the defined configuration.
4 new configuration properties for managing amount of block items in block
ADHOC mode in GenerationMode renamed to CRAFT, to better explain the purpose and function of this mode.
Handler for each type of introduced block item.
- BlockHeaderHandler responsible for crafting block header for each new block with needed fields.
- BlockProofHandler responsible for crafting block proof with adequte signature(for now) to pass verification.
- EventHeader and EventTransaction to represent the start of each new event and it's respective transctions.
- TransactionResult to represent the result of the transactions included in the block. For now we've included only HBAR TransferList and fungible TokenTransferList as result with randomly generated AccountAmounts.
Little change to how BlockStreamManager implementation is provided to Dagger.

Related Issue(s)

Fixes #502

Signed-off-by: georgi-l95 <[email protected]>

mattp-swirldslabs

Looks good

mattp-swirldslabs · 2025-02-20T16:27:25Z

simulator/src/main/java/com/hedera/block/simulator/generator/CraftBlockStreamManager.java

+import com.hedera.block.common.hasher.Hashes;
+import com.hedera.block.common.hasher.HashingUtilities;
+import com.hedera.block.common.hasher.NaiveStreamingTreeHasher;
+import com.hedera.block.common.hasher.StreamingTreeHasher;


Hey Georgi, we talked about this in a DM. We'll need to talk through whether the Hashing really belongs in the common module since it forces the Simulator to use PBJ types. I'm bringing it up for visibility. We can fix it with a future PR if the team wants to remove the dep here. Feel free to resolve this comment to merge.

Yes, thank you for bringing attetion to this, I was not aware that the PBJ types was dep non grata in common module at the time. we can discuss how to decouple the hasher from PBJ or maybe just create new hasher algorithm for simulator and return this one to server there are many alternatives to consider and at the time, I appreciate that you don't consider this to be a blocker 🙏 👍 💯

I support creating a different hasher for the simulator and moving the current hasher back to server.
It seemed a good choice, at the time, to share the code, but it's looking like we are better off keeping the two systems more separated in this specific case.
Thanks for the suggestions, @AlfredoG87.

mattp-swirldslabs · 2025-02-20T16:47:28Z

simulator/src/main/java/com/hedera/block/simulator/generator/itemhandler/ItemHandler.java

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: Apache-2.0
+package com.hedera.block.simulator.generator.itemhandler;


It looks like all the classes in this package are helpers to deal with the PBJ transitive dep. Hopefully we can figure out a way to remove that dep since you already have parsed the full object via protoc. We shouldn't have to use classes like *Unparsed

mattp-swirldslabs · 2025-02-20T16:55:36Z

simulator/src/main/java/com/hedera/block/simulator/config/data/BlockGeneratorConfig.java

        @ConfigProperty(defaultValue = "") String folderRootPath,
        @ConfigProperty(defaultValue = "BlockAsFileBlockStreamManager") String managerImplementation,
        @ConfigProperty(defaultValue = "36") int paddedLength,
        @ConfigProperty(defaultValue = ".blk.gz") String fileExtension,
        // Optional block number range for the BlockAsFileLargeDataSets manager
-        @ConfigProperty(defaultValue = "1") @Min(1) int startBlockNumber,
+        @ConfigProperty(defaultValue = "1") @Min(0) int startBlockNumber,


Good catch on this min value

Where did we land on this?
I recall that blocks were supposed to always start at 1 for genesis, and if a client requests a block range it needs to run from 1 to n (where if n is 0 then it means stream forever).
Did we find that the genesis block is numbered 0?

AlfredoG87

This is great work!! thank you @georgi-l95 💯

note: all comments are just comments and feel free to resolve them.

AlfredoG87 · 2025-02-20T20:29:25Z

simulator/src/main/java/com/hedera/block/simulator/generator/CraftBlockStreamManager.java

+import com.hedera.block.common.hasher.Hashes;
+import com.hedera.block.common.hasher.HashingUtilities;
+import com.hedera.block.common.hasher.NaiveStreamingTreeHasher;
+import com.hedera.block.common.hasher.StreamingTreeHasher;


Yes, thank you for bringing attetion to this, I was not aware that the PBJ types was dep non grata in common module at the time. we can discuss how to decouple the hasher from PBJ or maybe just create new hasher algorithm for simulator and return this one to server there are many alternatives to consider and at the time, I appreciate that you don't consider this to be a blocker 🙏 👍 💯

AlfredoG87 · 2025-02-20T20:33:30Z

simulator/src/main/java/com/hedera/block/simulator/generator/GeneratorInjectionModule.java

+                if ("BlockAsDirBlockStreamManager".equalsIgnoreCase(managerImpl)) {
+                    yield new BlockAsDirBlockStreamManager(config);


we might consider to get rid of this implementation in the future to reduce maintenance footprint, I mean, if is no longer needed.

The simulator may find benefit from having the dir option.
For instance, we might want to have an input folder with blocks (in block as dir) as a source and appreciate the ability to swap out individual block items without needing to rewrite entire blocks (or make changes on the fly). The option to use hard links for the parts that don't change is also valuable, at least to some extent.

That said, if it's costing a lot of maintenance, it's fine to remove now, and we can always restore the code or rewrite if it's worthwhile.

jsync-swirlds · 2025-02-20T21:17:57Z

...src/main/java/com/hedera/block/simulator/generator/itemhandler/TransactionResultHandler.java

+                        .setRealmNum(0)
+                        .setShardNum(0)


Could we add a story to support non-zero (and non-static) shard and realm in a future sprint?
Consensus is already adding this, and it would be nice to be able to test with the simulator.

jsync-swirlds · 2025-02-20T21:19:34Z

.../test/java/com/hedera/block/simulator/config/SimulatorMappedConfigSourceInitializerTest.java

+        new ConfigMapping("generator.minNumberOfEventsPerBlock", "GENERATOR_MIN_NUMBER_OF_EVENTS_PER_BLOCK"),
+        new ConfigMapping("generator.maxNumberOfEventsPerBlock", "GENERATOR_MAX_NUMBER_OF_EVENTS_PER_BLOCK"),
+        new ConfigMapping(
+                "generator.minNumberOfTransactionsPerEvent", "GENERATOR_MIN_NUMBER_OF_TRANSACTIONS_PER_EVENT"),
+        new ConfigMapping(
+                "generator.maxNumberOfTransactionsPerEvent", "GENERATOR_MAX_NUMBER_OF_TRANSACTIONS_PER_EVENT"),


Would it makes sense to elide the NUMBER_OF_ part of these names?

georgi-l95 added 6 commits February 17, 2025 12:08

add initial setup

1e3b8aa

Signed-off-by: georgi-l95 <[email protected]>

craft blocks with "valid" signature

ef16441

Signed-off-by: georgi-l95 <[email protected]>

Add additional items included in the blocks

1b946a2

Signed-off-by: georgi-l95 <[email protected]>

fix unit tests and build

28d6148

Signed-off-by: georgi-l95 <[email protected]>

add required javadoc

cb28efe

Signed-off-by: georgi-l95 <[email protected]>

add tests

12859e0

Signed-off-by: georgi-l95 <[email protected]>

georgi-l95 added New Feature A new feature, service, or documentation. Major changes that are not backwards compatible. Simulator Issue related to Block Stream Simulator labels Feb 20, 2025

georgi-l95 added this to the 0.5.0 milestone Feb 20, 2025

georgi-l95 self-assigned this Feb 20, 2025

georgi-l95 linked an issue Feb 20, 2025 that may be closed by this pull request

Simulator Generation Mode Improvements #502

Open

fix response code from successful transaction

4b0d57c

Signed-off-by: georgi-l95 <[email protected]>

georgi-l95 marked this pull request as ready for review February 20, 2025 12:11

georgi-l95 requested review from a team as code owners February 20, 2025 12:11

georgi-l95 requested review from Nana-EC, ata-nas, mattp-swirldslabs, jsync-swirlds and AlfredoG87 February 20, 2025 12:14

mattp-swirldslabs approved these changes Feb 20, 2025

View reviewed changes

mattp-swirldslabs mentioned this pull request Feb 20, 2025

chore: Refactor the hashing impl in common to be protobuf type agnostic #683

Open

AlfredoG87 approved these changes Feb 20, 2025

View reviewed changes

jsync-swirlds reviewed Feb 20, 2025

View reviewed changes

AlfredoG87 modified the milestones: 0.5.0, 0.6.0 Feb 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: simulator generation mode improvements #682

feat: simulator generation mode improvements #682

georgi-l95 commented Feb 20, 2025 •

edited

Loading

mattp-swirldslabs left a comment

mattp-swirldslabs Feb 20, 2025

AlfredoG87 Feb 20, 2025

jsync-swirlds Feb 20, 2025

mattp-swirldslabs Feb 20, 2025

mattp-swirldslabs Feb 20, 2025

jsync-swirlds Feb 20, 2025

AlfredoG87 left a comment

AlfredoG87 Feb 20, 2025

AlfredoG87 Feb 20, 2025

jsync-swirlds Feb 20, 2025

jsync-swirlds Feb 20, 2025

jsync-swirlds Feb 20, 2025

		@@ -0,0 +1,27 @@
		// SPDX-License-Identifier: Apache-2.0
		package com.hedera.block.simulator.generator.itemhandler;

		if ("BlockAsDirBlockStreamManager".equalsIgnoreCase(managerImpl)) {
		yield new BlockAsDirBlockStreamManager(config);

feat: simulator generation mode improvements #682

Are you sure you want to change the base?

feat: simulator generation mode improvements #682

Conversation

georgi-l95 commented Feb 20, 2025 • edited Loading

Description

Related Issue(s)

mattp-swirldslabs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlfredoG87 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgi-l95 commented Feb 20, 2025 •

edited

Loading