[Bug] issues running ray transformations on Google colab

[sujee]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/universal/ededup

What happened + What you expected to happen

when running this notebook on google colab environment, ededupe step gets stuck with message

(autoscaler +2m35s) Warning: The following resource request cannot be scheduled right now: {'CPU': 0.7}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.

full output : https://gist.github.com/sujee/856b008657793642b08eabe464f3ad96

Reproduction script

https://colab.research.google.com/github/sujee/data-prep-kit-examples/blob/main/dpk-intro/dpk_intro_1.ipynb

Anything else

No response

OS

Other

Python

3.11.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[touma-I]

@sujee do you think the error is specific to ededup or does it occur for all ray transforms? thanks

[sujee]

@sujee do you think the error is specific to ededup or does it occur for all ray transforms? thanks

I think this is more RAY related.

Probably need to investigate

• if we are creating extra workers
• or not terminating them correctly

that is maxxing out the cluster resources