Skip to content

Commit

Permalink
chore: fix failing e2e tests (#2854)
Browse files Browse the repository at this point in the history
Fixes failing e2e tests from e.g.
[here](https://github.com/apify/crawlee/actions/runs/13426528432/job/37510389928)

In `kv-open-return-storage-object`, adds missing `await` (caused the
test to fail with Platform storage backend).

In `puppeteer-store-pagination-jquery-*`, this PR switches the network
wait from the `networkIdle0` logic to `networkIdle2`. This way, the
tests won't fail because of long-running requests. Note that the general
use of `networkIdle` is discouraged, as it brings exactly this kind of
issues. This risk is likely alright in our type of tests (and our
controlled environment), though.

In `cheerio-max-requests` we set the `maxRequestsPerCrawl` to `10`.
Expecting more than `10` requests in the test assert might be valid (due
to concurrent processing), but can cause the tests to fail when we
actually count it right.
  • Loading branch information
barjin authored Feb 20, 2025
1 parent e8b518f commit ca5397e
Show file tree
Hide file tree
Showing 4 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion test/e2e/cheerio-max-requests/test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ await initialize(testActorDirname);

const { stats, datasetItems } = await runActor(testActorDirname);

await expect(stats.requestsFinished > 10, 'All requests finished');
await expect(stats.requestsFinished >= 10, 'All requests finished');
await expect(datasetItems.length > 5 && datasetItems.length < 15, 'Number of dataset items');
await expect(
validateDataset(datasetItems, ['url', 'title', 'uniqueIdentifier', 'firstParagraph', 'modifiedDate']),
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/kv-open-return-storage-object/actor/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ const mainOptions = {

await Actor.main(async () => {
const kv = await KeyValueStore.open();
kv.setValue('storageObject', { storeObject: kv.storageObject });
await kv.setValue('storageObject', { storeObject: kv.storageObject });
}, mainOptions);
2 changes: 1 addition & 1 deletion test/e2e/puppeteer-store-pagination-jquery/actor/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ await Actor.main(async () => {
// enqueue product details from the first three pages of the store
for (let pageNo = 1; pageNo < 3; pageNo++) {
// Wait for network events to finish
await page.waitForNetworkIdle();
await page.waitForNetworkIdle({ concurrency: 2 });
// Enqueue all loaded links
await enqueueLinks({
selector: 'a.product-item__image-wrapper',
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/puppeteer-store-pagination/actor/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ crawler.router.addHandler('START', async ({ log, enqueueLinks, page }) => {
// enqueue product details from the first three pages of the store
for (let pageNo = 1; pageNo < 3; pageNo++) {
// Wait for network events to finish
await page.waitForNetworkIdle();
await page.waitForNetworkIdle({ concurrency: 2 });
// Enqueue all loaded links
await enqueueLinks({
selector: 'a.product-item__image-wrapper',
Expand Down

0 comments on commit ca5397e

Please sign in to comment.