Add narwhals materializer for dataframe agnosticism. #189

matthewwardrop · 2024-08-06T23:50:20Z

This patch adds initial support for narwhals. It's... patchy.

You can try it out using:

import narwhals as nw
import pandas as pd
from formulaic import model_matrix

df = pd.DataFrame({"a": [1,2,None], "b": list("abc"), "c": pd.Categorical(list("abc"))})

model_matrix("a + b + c", df, materializer='narwhals', na_action="ignore")

import polars
model_matrix("a + b + c", polars.DataFrame._from_pandas(df), materializer='narwhals', na_action="ignore")

Note: The polars backend panics when na_action is not ignore.

There's a lot of hacks here, including fallbacks to pandas objects in places, and of course we still want sparse materialisation to work (which I don't think other backends support sufficiently to replace scipy sparse matrices).

matthewwardrop · 2024-08-06T23:53:59Z

As discussed in #187 , here's an initial PR for your review @MarcoGorelli. It's definitely not ready to replace the default backend, but it isn't toooo far either.

matthewwardrop · 2024-08-08T21:29:14Z

@MarcoGorelli Sorry, just in case it wasn't entirely clear from our conversation in #187 and my tagging you here, I was hoping you could take a look and suggest how to fill the gaps and/or improve things. No worries if you do not have time for now.

MarcoGorelli · 2024-08-08T22:06:53Z

Thanks for the ping! Yup, definitely taking a look, just had some holiday recently 🌴

MarcoGorelli

thanks for trying this out! left some comments, make some todos in Narwhals itself, but I really like the look of this and I'd like to invest more time into making sense of it and bringing it forwards! I'm away this weekend but I'll get back to this soon after

MarcoGorelli · 2024-08-09T14:14:28Z

formulaic/materializers/narwhals.py

+            if not isinstance(values.dtype, nwd.NumericType):
+                return True


do you also consider datetimes to be categorical?

MarcoGorelli · 2024-08-09T14:42:38Z

formulaic/utils/null_handling.py

+@drop_rows.register
+def _(values: narwhals.Series, indices: Sequence[int]) -> narwhals.Series:
+    # TODO: Is there a better narwhals native way to do this?
+    selector = numpy.zeros(len(values), dtype=bool)
+    selector[list(indices)] = True
+    return values.filter(~selector)


Interesting! I'd have thought there would be an easier way to do this, but it seems there's not in Polars either

I was also expecting that passing a numpy array to Series.filter would work, but that raises, as Polars expects a Series or a list:

In [15]: s = pl.Series([5, 2, 4, 1, 2]) In [16]: mask = np.array([True, True, True, False, False]) In [17]: s.filter(mask.tolist()) Out[17]: shape: (3,) Series: '' [i64] [ 5 2 4 ] In [18]: s.filter(mask) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[18], line 1 ----> 1 s.filter(mask) File ~/polars-api-compat-dev/.venv/lib/python3.11/site-packages/polars/series/series.py:2993, in Series.filter(self, predicate) 2991 if isinstance(predicate, list): 2992 predicate = Series("", predicate) -> 2993 return self._from_pyseries(self._s.filter(predicate._s)) AttributeError: 'numpy.ndarray' object has no attribute '_s'

I'll raise an issue about this in Polars and see what they say there.

Possible solutions could be:

for now, to do return values.filter((~selector).tolist()). This should be enough to address the error you're seeing when na_action isn't 'ignore'

in Narwhals, we make it easy to create a narwhals Series from a numpy array. Issue: Easy way to create nw.Series from numpy array narwhals-dev/narwhals#754

we support passing numpy arrays to filter (I'll report this to Polars and see what they think there)

We'll get back to you on this

i'm not sure there is an easier way to do this with the Polars API at the moment

MarcoGorelli · 2024-08-09T15:12:13Z

formulaic/materializers/narwhals.py

+        if isinstance(values, nw.Series):
+            values = values.to_pandas()


I think it should be possible to keep this Narwhals-native, as we have:

cat.get_categories: https://narwhals-dev.github.io/narwhals/api-reference/expr_cat/#narwhals.expr.ExprCatNamespace.get_categories

Series.to_dummies: https://narwhals-dev.github.io/narwhals/api-reference/series/#narwhals.series.Series.to_dummies

will try this out and let you know

MarcoGorelli · 2024-08-09T15:15:33Z

formulaic/utils/null_handling.py

+@find_nulls.register
+def _(values: narwhals.Series) -> Set[int]:
+    return set(values.is_null().arg_true().to_list())


MarcoGorelli · 2024-08-09T15:20:47Z

formulaic/materializers/narwhals.py

+        # TODO: Can we do better than this? Having to reconstitute raw data
+        # does not seem ideal.
+        combined = nw.from_dict(
+            {
+                name: (nw.to_native(col) if isinstance(col, nw.Series) else col)
+                for name, col in cols
+            },
+            native_namespace=nw.get_native_namespace(self.__narwhals_data),
+        )


You're right, this should be supported (and easier!) - I've opened an issue about it here narwhals-dev/narwhals#756

MarcoGorelli · 2024-09-09T08:17:15Z

hey - quick update, I've got most of it working in a branch, but am a bit busy with some conferences now - will get back to it soon. realistically I hope to have something review-ready in October 🤞 this has been quite interesting to work on!

matthewwardrop · 2024-12-02T09:42:54Z

Hi @MarcoGorelli ! We are nearing a 1.1 release, and I'm thinking to include this on something close to its current state as an experimental feature. I'll try to patch any obvious bugs. If you have a branch I'm also happy to cherry pick any fixes you might have come up with :).

MarcoGorelli · 2024-12-02T17:49:59Z

Hi @MarcoGorelli ! We are nearing a 1.1 release, and I'm thinking to include this on something close to its current state as an experimental feature. I'll try to patch any obvious bugs. If you have a branch I'm also happy to cherry pick any fixes you might have come up with :).

hey, sounds great thanks! sorry for the delay, I got a bit stuck into a PR we made to Plotly recently, and here I got a bit lost in all the dispatches 😳

s3alfisc · 2024-12-02T21:27:26Z

I would definitely try and volunteer testing the experimental feature on a pyfixest branch if this would make it into the next formulaic release!

stanmart · 2024-12-04T09:40:08Z

Same with tabmat, would be happy to test drive narwhals support.

MarcoGorelli · 2024-12-09T11:35:49Z

thanks both! i've given this a go in #226 if you fancy pulling it and trying it out

…implementation.

…s materializer features.

… for data-types.

matthewwardrop · 2025-01-09T18:03:08Z

@MarcoGorelli @s3alfisc @stanmart Thanks for you interest and support here. I'm merging this in now, and it will be part of the formulaic 1.2.0 release. It is still considered experimental at this point. I will try to increase testing coverage before the 1.2.0 release, but if you would like to put it through its paces ahead of that time with your libraries that would be greatly appreciated :).

MarcoGorelli · 2025-01-09T18:13:36Z

amazing, thank you for taking this forwards! we'll add this to our downstream tests

s3alfisc · 2025-01-09T21:06:54Z

Will spend some time to test it this weekend. Thanks @matthewwardrop & @MarcoGorelli !

MarcoGorelli reviewed Aug 9, 2024

View reviewed changes

s3alfisc mentioned this pull request Oct 23, 2024

narwhals = Pandas + polars + ... py-econometrics/pyfixest#533

Open

matthewwardrop marked this pull request as draft November 28, 2024 05:30

matthewwardrop force-pushed the main branch from 3d7bd48 to 8230d6e Compare November 30, 2024 07:18

MarcoGorelli mentioned this pull request Dec 8, 2024

wip: Explore narwhals #226

Merged

matthewwardrop force-pushed the main branch 4 times, most recently from dd375ec to e7250bc Compare December 16, 2024 08:08

matthewwardrop force-pushed the explore_narwhals branch 2 times, most recently from f96f2bf to 740ee4e Compare December 17, 2024 06:49

matthewwardrop added this to the 1.2.0 milestone Dec 17, 2024

matthewwardrop and others added 3 commits January 7, 2025 15:13

Move _check_for_nulls implementation to base FormulaMaterializer …

0d55094

…implementation.

Add initial implementation of a narwhals materializer.

64a8f43

Replace arrow materializer with narwhals, and complete/polish narwhal…

9e10f51

…s materializer features.

matthewwardrop force-pushed the explore_narwhals branch from 740ee4e to 9e10f51 Compare January 7, 2025 23:17

matthewwardrop added 2 commits January 9, 2025 09:46

Add support to FormulaMaterializer for registering fallback support…

dced1f8

… for data-types.

Skip coverage of narwhals experimental code for now.

fb10d57

matthewwardrop force-pushed the explore_narwhals branch from d7ba572 to fb10d57 Compare January 9, 2025 17:59

matthewwardrop marked this pull request as ready for review January 9, 2025 18:00

matthewwardrop merged commit 8552610 into main Jan 9, 2025
5 checks passed

matthewwardrop deleted the explore_narwhals branch January 9, 2025 18:03

MarcoGorelli mentioned this pull request Jan 9, 2025

tests: add formulaic to downstream tests narwhals-dev/narwhals#1778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add narwhals materializer for dataframe agnosticism. #189

Add narwhals materializer for dataframe agnosticism. #189

matthewwardrop commented Aug 6, 2024 •

edited

Loading

matthewwardrop commented Aug 6, 2024

matthewwardrop commented Aug 8, 2024

MarcoGorelli commented Aug 8, 2024

MarcoGorelli left a comment

MarcoGorelli Aug 9, 2024

MarcoGorelli Aug 9, 2024

MarcoGorelli Dec 2, 2024

MarcoGorelli Aug 9, 2024

MarcoGorelli Aug 9, 2024

MarcoGorelli Aug 9, 2024

MarcoGorelli commented Sep 9, 2024

matthewwardrop commented Dec 2, 2024

MarcoGorelli commented Dec 2, 2024

s3alfisc commented Dec 2, 2024

stanmart commented Dec 4, 2024

MarcoGorelli commented Dec 9, 2024

matthewwardrop commented Jan 9, 2025

MarcoGorelli commented Jan 9, 2025

s3alfisc commented Jan 9, 2025

		if not isinstance(values.dtype, nwd.NumericType):
		return True

		if isinstance(values, nw.Series):
		values = values.to_pandas()

Add narwhals materializer for dataframe agnosticism. #189

Add narwhals materializer for dataframe agnosticism. #189

Conversation

matthewwardrop commented Aug 6, 2024 • edited Loading

matthewwardrop commented Aug 6, 2024

matthewwardrop commented Aug 8, 2024

MarcoGorelli commented Aug 8, 2024

MarcoGorelli left a comment

Choose a reason for hiding this comment

MarcoGorelli Aug 9, 2024

Choose a reason for hiding this comment

MarcoGorelli Aug 9, 2024

Choose a reason for hiding this comment

MarcoGorelli Dec 2, 2024

Choose a reason for hiding this comment

MarcoGorelli Aug 9, 2024

Choose a reason for hiding this comment

MarcoGorelli Aug 9, 2024

Choose a reason for hiding this comment

MarcoGorelli Aug 9, 2024

Choose a reason for hiding this comment

MarcoGorelli commented Sep 9, 2024

matthewwardrop commented Dec 2, 2024

MarcoGorelli commented Dec 2, 2024

s3alfisc commented Dec 2, 2024

stanmart commented Dec 4, 2024

MarcoGorelli commented Dec 9, 2024

matthewwardrop commented Jan 9, 2025

MarcoGorelli commented Jan 9, 2025

s3alfisc commented Jan 9, 2025

matthewwardrop commented Aug 6, 2024 •

edited

Loading