Refactoring field interpolation and allow custom interpolation methods in Scipy mode #1816

VeckoTheGecko · 2025-01-08T09:32:57Z

This PR refactors many of the indexing methods and interpolation methods out of field.py, and moves indexing code to a separate file to make things more manageable and reduce the coupling with the Field class.

This PR also allows users to easily overwrite the behaviour of existing interpolation methods or (untested) define new interpolation methods in Scipy mode. This can be done via the register_2d_interpolator(...) and register_3d_interpolator(...) decorators. This behaviour is in beta and is subject to change.

This also makes the interpolation functions more easily testable by providing only the required data.

Fixes #1823

VeckoTheGecko

Thoughts on the below @erikvansebille ? Let me know if you want me to break this into separate PRs

parcels/_interpolation.py

parcels/tools/statuscodes.py

parcels/_interpolation.py

VeckoTheGecko · 2025-01-16T17:28:55Z

Note for review: de89311 includes testing that shows the refactor of the 3d interpolation is 100% equivalent to the previous version for all combinations of interp method, grid types, and cell locations.

VeckoTheGecko · 2025-01-16T17:47:34Z

I'm quite aware that this is getting to be a big PR. To be expected for such a big refactor, but I think it would be good to continue in other PRs (for vector interp and indexing) to keep this reviewable.

VeckoTheGecko · 2025-01-16T18:10:44Z

tests/test_interpolation.py

+    @pytest.mark.parametrize(
+        "func, eta, xsi, expected",
+        [
+            pytest.param(interpolation._nearest_2d, 0.49, 0.49, 3.0, id="nearest_2d-1"),
+            pytest.param(interpolation._nearest_2d, 0.49, 0.51, 4.0, id="nearest_2d-2"),
+            pytest.param(interpolation._nearest_2d, 0.51, 0.49, 5.0, id="nearest_2d-3"),
+            pytest.param(interpolation._nearest_2d, 0.51, 0.51, 6.0, id="nearest_2d-4"),
+            pytest.param(interpolation._tracer_2d, None, None, 6.0, id="tracer_2d"),
+            # pytest.param(interpolation._linear_2d, ...),
+            # pytest.param(interpolation._linear_invdist_land_tracer_2d, ...),
+        ],
+    )
+    def test_2d(self, data_2d, func, eta, xsi, expected):
+        ctx = interpolation.InterpolationContext2D(data_2d, eta, xsi, self.ti, self.yi, self.xi)
+        assert func(ctx) == expected
+
+    @pytest.mark.parametrize(
+        "func, eta, xsi, expected",
+        [
+            # pytest.param(interpolation._nearest_3d, ...),
+            # pytest.param(interpolation._cgrid_velocity_3d, ...),
+            # pytest.param(interpolation._linear_invdist_land_tracer_3d, ...),
+            # pytest.param(interpolation._linear_3d, ...),
+            # pytest.param(interpolation._tracer_3d, ...),
+        ],
+    )


Thoughts on test cases @erikvansebille ? I think it might be good to add some on the raw arrays

VeckoTheGecko · 2025-01-17T10:35:49Z

I added an extra commit in the history that shows 100% equivalence of the refactor. See updated #1816 (comment)

erikvansebille

First bit of reviews (of the three main new files). Rest to come after the weekend

erikvansebille · 2025-01-17T14:42:38Z

parcels/_index_search.py

+            + xsi * eta * grid.depth[:, yi + 1, xi + 1]
+            + (1 - xsi) * eta * grid.depth[:, yi + 1, xi]
+        )
+    z = np.float32(z)  # type: ignore # TODO: remove type ignore once we migrate to float64


I'm surprised by this statement here. I know it doesn't come from this PR, but why only cast to float32 in the s-case (and not the z-case)? In general, z can be either float32 or float64, depending on the lonlatdepth_dtype

OK, then I propose we remove it

Then I propose we remove it

Suggested change

z = np.float32(z) # type: ignore # TODO: remove type ignore once we migrate to float64

parcels/_interpolation.py

fix

…rpolate.py

…erpolation method

Try block should only have retrieving the function, not its execution

To help with refactoring

for more information, see https://pre-commit.ci

For some reason there is test pollution between the following 2 tests: tests/test_interpolation.py::test_interpolation_registry tests/test_interpolation.py::test_full_depth_provided_to_interpolators

Full context in Python Discord https://discord.com/channels/267624335836053506/1329136004459794483

To ensure refactoring is accurate

erikvansebille · 2025-01-17T16:15:02Z

parcels/_interpolation.py

+
+@dataclass
+class InterpolationContext2D:
+    """Information provided by Parcels during 2D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.


Suggested change

"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.

"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter and Van Sebille (2019), 10.5194/gmd-12-3571-2019 for more info.

Suggested change

"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.

"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.

erikvansebille · 2025-01-17T16:15:33Z

parcels/_interpolation.py

+
+@dataclass
+class InterpolationContext3D:
+    """Information provided by Parcels during 3D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.


Suggested change

"""Information provided by Parcels during 3D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.

"""Information provided by Parcels during 3D spatial interpolation. See Delandmeter and Van Sebille (2019), 10.5194/gmd-12-3571-2019 for more info.

erikvansebille · 2025-01-17T16:16:16Z

parcels/_index_search.py

+            + xsi * eta * grid.depth[:, yi + 1, xi + 1]
+            + (1 - xsi) * eta * grid.depth[:, yi + 1, xi]
+        )
+    z = np.float32(z)  # type: ignore # TODO: remove type ignore once we migrate to float64


OK, then I propose we remove it

erikvansebille

More comments, now on all files except for the tests/* files

parcels/_interpolation.py

erikvansebille · 2025-01-17T16:18:19Z

parcels/_index_search.py

+            + xsi * eta * grid.depth[:, yi + 1, xi + 1]
+            + (1 - xsi) * eta * grid.depth[:, yi + 1, xi]
+        )
+    z = np.float32(z)  # type: ignore # TODO: remove type ignore once we migrate to float64


Then I propose we remove it

Suggested change

z = np.float32(z) # type: ignore # TODO: remove type ignore once we migrate to float64

erikvansebille · 2025-01-17T16:23:15Z

parcels/field.py

Should we not also move the _search_indices(), _search_indices_curvilinear() and _search_indices_rectilinear() methods to the _index_search.py file? Why are they still in field.py?

In general, I see that even in this PR, field.py still contains a lot of methods/functions that don't specifically need to be here. That would really clean up the field.py file?

E.g. VectorField.dist, VectorField._is_land2D() and VectorField.jacobian can go to an interpolation_utils.py file (mimicking what is in include folder for C)?

And the spatial interpolation for VectorFields can also go to _interpolation.py?

erikvansebille · 2025-01-17T16:31:24Z

parcels/tools/statuscodes.py

+        if msg == "show_time":
+            message += " Try explicitly providing a 'show_time'."


We don't have show_time anymore (was part of the old plotting we got rid off years ago, so this is not needed

Suggested change

if msg == "show_time":

message += " Try explicitly providing a 'show_time'."

VeckoTheGecko commented Jan 8, 2025

View reviewed changes

parcels/_interpolation.py Outdated Show resolved Hide resolved

parcels/tools/statuscodes.py Outdated Show resolved Hide resolved

VeckoTheGecko commented Jan 9, 2025

View reviewed changes

parcels/_interpolation.py Outdated Show resolved Hide resolved

VeckoTheGecko requested a review from erikvansebille January 16, 2025 18:04

VeckoTheGecko changed the title ~~Refactoring indexing and interpolation~~ Refactoring field interpolation (and move indexing code) Jan 16, 2025

VeckoTheGecko commented Jan 16, 2025

View reviewed changes

VeckoTheGecko marked this pull request as ready for review January 16, 2025 18:18

VeckoTheGecko force-pushed the v/refactor-interp branch from 8abec45 to b38a17c Compare January 17, 2025 10:34

VeckoTheGecko changed the title ~~Refactoring field interpolation (and move indexing code)~~ Refactoring field interpolation and allow custom interpolation methods in Scipy mode Jan 17, 2025

erikvansebille reviewed Jan 17, 2025

View reviewed changes

VeckoTheGecko added 18 commits January 17, 2025 16:41

refactor error classes removing custom inits

4a7e25d

update to use _raise_out_of_bound_error

167681b

fix

update to use _raise_out_of_bound_error

41c5075

update to use _raise_out_of_bound_surface_error

1ff61f2

Rename parse_particletime

e2e2a61

update to use _raise_field_sampling_error

89a90f4

Renaming functions

9b6793b

Add exception helper

b5a02b8

Remove implementation for deprecated public interpolation methods

660c840

Move search_indices_vertical_z and search_indices_vertical_s to _inte…

fe46955

…rpolate.py

Lift field name during interpolation error handling into _spatial_int…

16275cd

…erpolation method

Refactoring

a8adf8a

Rename file

90cf266

Refactor 2D interpolators into separate file

20d6b0f

Refactor 3D interpolators into separate file

bf89077

Move interpolation call

fe2665b

Try block should only have retrieving the function, not its execution

Temporarily add interp_method to 3D interpolation context

792c3d5

To help with refactoring

Add test_interpolation.py

05cf2aa

VeckoTheGecko and others added 23 commits January 17, 2025 16:41

Add some tests for interpolation methods

a843c58

[pre-commit.ci] auto fixes from pre-commit.com hooks

0239ca1

for more information, see https://pre-commit.ci

Supporting code for 3d interpolation refactor

797fe35

Update fixture use method

230509d

Add test_full_depth_provided_to_interpolators

ab7974d

For some reason there is test pollution between the following 2 tests: tests/test_interpolation.py::test_interpolation_registry tests/test_interpolation.py::test_full_depth_provided_to_interpolators

Fix test pollution

3c5d3a0

Full context in Python Discord https://discord.com/channels/267624335836053506/1329136004459794483

Redefine zdim based on data array

6010650

Refactor using unit_square_to_target() function

d3d6102

Change order to eta, xsi

9d62028

refactor to get_3d_f0_f1

f56f9ad

Split up 3d interpolation functions

78cebae

Force kwargs

f85bda7

add z_layer_interp

dbe1784

Reduce code duplication

5fd0ede

Add interpolation testing for each grid cell

86cdf85

To ensure refactoring is accurate

Delete old 3d interpolator

df96f39

update comment

8afa581

Review feedback

4c34258

patch indexerror

c9e4edf

Rename file _indexing.py -> _index_search.py

46b534f

Refactor test

a898aed

Rename test data function

7fe211d

review feedback

14b93c9

VeckoTheGecko force-pushed the v/refactor-interp branch from bff8ca2 to 14b93c9 Compare January 17, 2025 15:41

VeckoTheGecko requested a review from erikvansebille January 17, 2025 16:11

erikvansebille reviewed Jan 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring field interpolation and allow custom interpolation methods in Scipy mode #1816

Refactoring field interpolation and allow custom interpolation methods in Scipy mode #1816

VeckoTheGecko commented Jan 8, 2025 •

edited

Loading

VeckoTheGecko left a comment

VeckoTheGecko commented Jan 16, 2025 •

edited

Loading

VeckoTheGecko commented Jan 16, 2025

VeckoTheGecko Jan 16, 2025 •

edited

Loading

VeckoTheGecko commented Jan 17, 2025

erikvansebille left a comment

erikvansebille Jan 17, 2025

VeckoTheGecko Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille left a comment

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

erikvansebille Jan 17, 2025

	"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter, P. and van Sebille, E (2019) for more info.
	"""Information provided by Parcels during 2D spatial interpolation. See Delandmeter and Van Sebille (2019), 10.5194/gmd-12-3571-2019 for more info.

		if msg == "show_time":
		message += " Try explicitly providing a 'show_time'."

Refactoring field interpolation and allow custom interpolation methods in Scipy mode #1816

Are you sure you want to change the base?

Refactoring field interpolation and allow custom interpolation methods in Scipy mode #1816

Conversation

VeckoTheGecko commented Jan 8, 2025 • edited Loading

VeckoTheGecko left a comment

Choose a reason for hiding this comment

VeckoTheGecko commented Jan 16, 2025 • edited Loading

VeckoTheGecko commented Jan 16, 2025

VeckoTheGecko Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

VeckoTheGecko commented Jan 17, 2025

erikvansebille left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikvansebille left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VeckoTheGecko commented Jan 8, 2025 •

edited

Loading

VeckoTheGecko commented Jan 16, 2025 •

edited

Loading

VeckoTheGecko Jan 16, 2025 •

edited

Loading