SNOW-1897976: Consider adding a new DataFrame method to allow applying transformations in the fluent-style

[padhia]

What is the current behavior?

Currently, to apply transformations to a dataframe, one needs to create a named variable to call a transformation function.

E.g. consider transformation functions such as

def dedup(df: DataFrame) -> DataFrame:
    ...

def add_audit_cols(df: DataFrame) -> DataFrame:
    ...

my_df: DataFrame = ...

t1 = dedeup(my_df)
transformed_df = add_audit_cols(t1)

What is the desired behavior?

Defined a new method that allows fluent style coding

class DataFrame:
   # This is a proposed method to allow fluent-style API
    def pipe(self, f: Callable[[DataFrame], DataFrame]) -> DataFrame:
        return f(self)

# revised code

transformed_df = my_df.pipe(dedup).pipe(add_audit_cols) 

How would this improve snowflake-snowpark-python?

Fluent-style code feels more ergonomic and matches the style of most built-in DataFrame methods

References, Other Background

[sfc-gh-sghosh]

Hi @padhia ,

Thanks for raising this issue, we are looking into it, will update.

Regards,
Sujan

[sfc-gh-sghosh]

Hello @padhia ,

We do support fluent style dataframe transformation, could you let us know what is you trying to achieve and what is not working.

Example without creating any temporary df variable we can do the following transformations

df = (
pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
.query("A > 1") # Filter
.assign(C=lambda x: x['B'] * 2)
.rename(columns={'A': 'Alpha'})
)

print(df)

[padhia]

Hello Sujan,

Thank you for getting back. I was referring to supporting custom transformations, in addition to, the built-in DataFrame class methods for fluent style APIs

In the example I mentioned above, I have abstracted complex logic into a couple of functions. It'd be nice if I could use the functions without having to call them explicitly.

transformed_df = my_df.pipe(dedup).pipe(add_audit_cols))

Note in the above code, the (proposed) DataFrame method pipe calls the functions as an argument (thus allowing fluent style), v/s the code below, where the function calls are nested and called with DataFrame as an argument.

transformation_df = add_audit_cols(dedup(my_df))