Background image for tracing

[ollimeier]

Playground draft PR for discussion.

[justvanrossum]

Quick feedback:

• instead of specifying all separate transformation parameters, use a fontTools.misc.transform.Transform object, which should serialize as a list of six items. We could also use DecomposedTransformation, like we do in components. I'm not 100% which would be better in this context
• Since I doubt the image will ever be part of interpolation, putting it on the StaticGlyph object may not be the best thing to do. Potentially it could be a new attr on the Layer object instead. But also here: I'm not 100% confident either way.

[ollimeier]

@justvanrossum I tried a lot, but sadly without success, yet. The main issue I have, is: how can we display an image stored locally. It's not allowed to access files locally from an http request. Do you have an idea how to solve this issue? Thanks a lot in advance.

[justvanrossum]

I didn’t realize you’d work on that today. That approach cannot work, as you’ve found out. We need to serve images as part of the font data.
I’m not yet sure what that’s concretely going to look like.

[ollimeier]

I didn’t realize you’d work on that today. That approach cannot work, as you’ve found out. We need to serve images as part of the font data. I’m not yet sure what that’s concretely going to look like.

I am not sure if this is something useful or not, but I figured out, that we can load an image via http://localhost:8000:
Please see my latest commit and the following screenshot:

The image is misplaced (it should actually be at the position of the rectangle), but it's there – yeah :)
No base64 used.

[ollimeier]

Correct position of the image:

Screen.Recording.2024-10-11.at.18.25.16.mp4

[justvanrossum]

The Image type should get an attribute named transformation of type DecomposedTransform.

Loading of the image should be done outside of the vis draw func, although loading could perhaps be triggered by draw (but only if it's cheap), then we should trigger a canvas redraw once the image has been loaded. It is vitally important that the draw func does not do any loading itself. The effect will be that if you put a glyph in edit mode, there may be a delay before the background image appears.

While it may be desirable to load images via regular http(s) instead of the websocket, this raises questions about how this data should then participate in undo and external changes. To do this correctly may add a lot of complexity that can perhaps be avoided if we allow image data to travel as base64 inside JSON messages. If we go that route, we may have to enforce a maximum image size.

I have some ideas that I'm trying to formulate, about potentially avoiding base64 encoding the data, but I'm not there yet.

Perhaps we can start with a simplistic approach, without worrying too much about efficiency.

(I'm considering to make the infrastructure general for any binary data, not limited to image data.)

ReadableFontBackend:

• getBinaryData(name: str) -> bytes

WritableFontBackend:

• putBinaryData(name: str, imageData: bytes) -> None
• deleteBinaryData(name: str) -> None

The FontHandler class will be responsible for converting to/from base64 (and can play a role in a different scheme, once we come up with one).

The Font class should then probably get a binaryData attribute, of type dict[str, bytes], but this is where things get complicated. I need to look into how FontController should deal with this data.

[ollimeier]

Current stage of my playground:

Screen.Recording.2024-11-01.at.12.44.13.mp4

[justvanrossum]

• Please start implementing getBinaryData() on the designspace backend, and get rid of the customData hack.
• Then add getBinaryData() on FontHandler as a "remotemethod", and do the base64 conversion there.
• Then add getBinaryData() on FontController. Convert the base64 to binary here.

This should allow you to get rid of the customData/ufoDir hack in designspace.py.

[ollimeier]

@justvanrossum Current stage without customData:

Screen.Recording.2024-11-06.at.08.08.32.mp4

[justvanrossum]

You ignored the method signature that I suggested:

getBinaryData(name: str) -> bytes

The method is intended to retrieve a single binary blob of data (say, for an image). Returning all binary data available in the entire font is insanely inefficient.

[justvanrossum]

No, "binaryData" is not a font attribute like "unitsPerEm" is. See general comment.

[justvanrossum]

So this would return all binary data for all layers. That's insanely inefficient: we always load "big things" on demand. But see the general comment, you ignored the method signature that I prescribed.

[justvanrossum]

The way to do this is to convert the decomposed transform to a "normal" transform with decomposedToTransform(image.transformation), and then you use the t.transformPoint() method for the four corners.

[justvanrossum]

For the next iterations, pls make sure that mypy doesn't complain (run locally before you commit).

[justvanrossum]

Please do not work on the Fontra backlend until we've actually designed how that data should be stored in there.

[justvanrossum]

What is this for? The base64 module comes with Python.

[justvanrossum]

I now understand why you set out retrieving all images, as we haven't defined a way to determine from which UFO to load an image. This can be part of the name string as passed to getBinaryData(), and we could use the associated font source identifier, but that's a little tricky to find from an arbitrary glyph layer. This needs more thinking and designing.

[justvanrossum]

Ok, I'm figuring out how to do this properly from the backend's perspective, and how Fontra + UFO can work. I will write this as a new PR, covering only the infrastructure changes.

[justvanrossum]

I'm closing this playground as it is largely superseded by #1775. I've opened a task issue for follow-up work: #1777.