FreeCAD stubs

[tritao]

Hello @ostr00000, thanks for your comment and review in the FreeCAD Pythons bindings PR.

I have been taking a deeper look at your work in this repo and it really is outstanding. 👏

As you're now aware I've also been trying to improve the FreeCAD Python typing situation, went through a slightly different route since I wasn't fully aware of how advanced your work on this repository was, but it's probably for the best as I think the direct typing stubs from the binding stubs has a few advantages.

One of the goals of my work has always been to try and introduce more Python typing into the FreeCAD project code and I believe that requires having a more direct integration in the core.

After the Python interface files are merged into the core, I was going to start looking for the other missing pieces, but you seem to have taken care of such things here already.

At this point I am wondering if when we introduce such Python stubs in the core, if you think that it should use such files directly at typing stubs or if we should input such files into your pre-existing generator and re-use the same process you have.

Or if you think it could be cleaner to refactor some things in the core to further ease the work of future maintenance and generation of complete and accurate typings.

So overall I think it could be very beneficial if we could collaborate on this as you have a lot of experience working on this problem and on the spirit of open-source, would be best to continue from all the great work you have done here.

[ostr00000]

your work in this repo and it really is outstanding. 👏

Thanks, but there are so many dirty regex 😃
(on the other hand, comparing to other extracting methods like fully parsing code, the regex are quite fast).

Actually, the repo started as a quick script to generate helper stubs for my personal usage. Then I read somewhere on the FreeCAD forum about .xml files, so I improved this code.
And at this moment I decided that I would be waste not to share my work with others, so I put it into a git repo. Then there was a request for easy install, so I uploaded it to PyPI. And later I have been gradually improving the code.

Recently, I started refactoring code to parse the original c++ code (to avoid regex/macros/etc.) and extract the python type using the same heuristics I developed in the regex version.
I am mainly Python developer, so I have to learn a lot about c++ (especially about clang internals).
The work is currently in a proof of concept stage, because I write it when I feel like it and when I have free time, so it is not pushed to this repo.

---

At this point I am wondering if when we introduce such Python stubs in the core, if you think that it should use such files directly at typing stubs or if we should input such files into your pre-existing generator and re-use the same process you have.

I believe the best place for stubs is to be together with the c++ code. There were even a few ideas to integrate this code with FreeCAD repo, but I decided to dismiss the decision in time and stabilize code a bit more.
The reasons are mainly:

• always up-to-date code (today there was exactly that kind of issue - makeLoft has wrong keyword arguments #34),
• possible errors are fixed together in cpp code (sometimes I reported issues about wrong code in FreeCAD repo, but it was not always fixed).

So in a long run, I would be ok, to make this repo archived.

---

I was going to start looking for the other missing pieces

I believe these pieces may be missing when the FreeCAD will start uses stub to generate cpp:

1. Dynamic property and getter/setter types - this may be very tricky, because (if I remember correctly), some properties are added dynamically in cpp code instead of declaration in xml. On the other hand, few years ago python descriptors (__get__/__set__) were not recognized in typing system, but now it should be easier to implement it as a descriptor helper class.
2. rich comparison/number protocols - The problem here is the following: cpp code requires a certain entry in a structure (Python docs), but the implementation frequently raises an exception (for example, you can call both __add__ and __mul__, but __mul__ may raise an exception). So probably this generator code should be more fragmented. This may be also tricky for many overloads, so maybe it should be written manually (example for vector that needs overload).
3. Some modules are not accessible for a normal Python import. I tried to hack it using private modules to not import it directly. But maybe this is a bug in FreeCAD cpp code? see this comment for more details.
4. Some module attributes are added dynamically in cpp/python code. For example FreeCAD.GuiUp. I do not know how stubs code will deal with it when generating code, but these module variables are very frequently used in python code.
5. As a part of stub distribution package, I provide a normal python package with constants. I almost never use direct string, but prefer to use it through a variable/const (this allows to avoid typos). This part of code may be easy to extract/move, because its whole content is almost in a single file. This is actually not related to subs, but I believe it is worth to give python dev all constant hidden somewhere in cpp in a single Python module.
6. The most difficult part is this handcrafted stub file. This is a Protocol class for FreeCAD Proxy object and FreeCADGui ViewProvider object. Despite my efforts, these subs are not 100% valid, because the python dev do not have to override every method. It is something like an optional protocol that currently does not exist in Python. I was recently involved in a discussion about optional protocols (and later the discussion moved to python forum). This is a proposition to a new PEP, so it has a long way to become a part of python, so I propose to just copy this handcrafted file. Maybe it would be nice if we can import this and use standard python inheritance/nominal typing, instead of requiring structural typing? (I remember when I meet a FreeCAD code the first time, I had a huge problem how to find all available methods for Proxy object).
7. Listing raised exceptions in docstrings - this is a very, very optional. Also, I think that recently the number of possible exception has been reduced. But sometimes a module from FreeCAD Ext uses a non-standard exception.

As you can see, there are many small issues that I tried to not bother about it the FreeCAD devs, but probably now it is a good time to fix it.

What do you think about each of these points/steps?
Are these infos useful, or you would like to know something more specific?

[tritao]

Recently, I started refactoring code to parse the original c++ code (to avoid regex/macros/etc.) and extract the python type using the same heuristics I developed in the regex version.
I am mainly Python developer, so I have to learn a lot about c++ (especially about clang internals).
The work is currently in a proof of concept stage, because I write it when I feel like it and when I have free time, so it is not pushed to this repo.

That's a great idea. I have a lot of experience with such an approach (developed https://github.com/mono/CppSharp) along with Clang internals, so let me know if I can provide any assistance.

About your other points, thanks very much, this was the kind of details I was looking for. I will investigate in more detail so I can provide more feedback.

Do you think it will still beneficial to have libclang-based approach even with the Python stubs being provided directly by FreeCAD? Is there any information that we need from the C++ files that is not in the bindings? I have not looked into everything in detail so I am not sure.

But maybe even that was the case maybe it could still be beneficial to have the libclang-based tools to make sure all the signatures are correct?

[ostr00000]

Do you think it will still beneficial to have libclang-based approach even with the Python stubs being provided directly by FreeCAD?

I think I would be better to focus on stub first. And later, if we notice we are missing something that cannot be generated, consider scanning cpp code.

Is there any information that we need from the C++ files that is not in the bindings?

Definitely some points from above:

• 1 - dynamic properties - but maybe this should be declared in stubs, and we generate some an initialization method in cpp for it? I'm not sure, but it is possible that this code must be defined in a core cpp class, and not in a python wrapper, so it may be challenging.
• 4 - variables assigned to the module in cpp - probably only a few variables (like Control, ActiveDocumment). If it will be possible to create .pyi file with arbitrary variable on module level, then this is not needed.
• 5 - searching for const values - but as I explained above this is one of the first code I have written - it is about 100 python lines.

make sure all the signatures are correct?

I think that cpp compiler should be good enough to detect most mistakes.
The only vulnerable point may be Python parsing functions (like PyArg_ParseTuple, PyArg_ParseTupleAndKeywords), because these functions take an arbitrary number of arguments.
From my experience, the mistakes are frequent in these functions.
Anyway, the generated code should be much safer than current code.