Rendering Engine Overhaul

[StarWitch]

Taisei runs on OpenGL 3.3, which is relatively fine for platforms like Linux, but other platforms have several problems with their OpenGL implementations that are going to bite us sooner or later:

1. OpenGL on Windows is a crapshoot with drivers, meaning it either runs great or doesn't run at all depending on the vendor, especially Intel (OpenGL renderer unplayable on Windows, Intel UHD 620 #164), and the situation will likely only get worse
2. OpenGL on macOS is deprecated, and will likely become less and less usable as time goes on

@Akaricchi mentioned wanting to investigate Sokol as a potential "universal replacement" for our rendering engine, since it can select the correct rendering API for any platform in a consistent way. It means that each platform could run their own native graphics API (with transpiled shaders), which would solve many problems for the future.

This likely won't make it into v1.4, but if anyone out there would like to assist us in this, please feel free to comment below.

[Akaricchi]

Alright, let me clarify my stance on this issue.

First of all, I reject the premise of OpenGL itself being problematic. In fact, I will go as far as to say that OpenGL, in 2021, is still the best 3D API choice for a small DYI cross-platform game engine such as ours — it's easy to work with, it's an open standard, and it's guaranteed to be supported at least to some extent on every relevant platform for a long time (even if that support comes from a third party). There is no direct competitor that satisfies all of those.

That being said, I don't think there aren't any problems. There absolutely are problems, and they lie within FOSS-hostile platforms like Windows and macOS undermining OpenGL support in favor of their own vendor-lock-in solutions (Direct3D and Metal, respectively). To that extent, these platforms can be considered broken.

Does that mean we get to ignore them? No, of course not. We don't hate our users after all. But I believe we shouldn't design around broken stuff unless absolutely necessary, either. Which is why I'm actually content with shipping workarounds such as translation layers for these platforms, even if the translation layer is a Lovecraftian google abomination.

That being said, I am not against implementing alternative rendering backends, quite the opposite actually. What I'm saying is, GL33 will likely remain the primary and reference backend for the foreseeable future, possibly even after a working alternative is implemented.

Now let's evaluate some of the options:

• Native Direct3D and/or Metal: most likely not going to happen, or at least, I'm not writing or maintaining this. But as long as someone else is willing to do it, I don't mind having the code in-tree.

• Vulkan: this is the one I might eventually want to do, for masocheducational purposes if nothing else.

  Advantages include: contemporary open standard; good multiplatform support (except apple, but moltenvk and gfx-rs exist); decent cross-platform tooling; we are already using Vulkan shader tooling; great if you enjoy pain.

  Disadvantages: extremely verbose and low-level, will take a lot of code and effort to get it on par with the GL33 backend; not usable in Emscripten.

• Sokol GFX: appealing due to its compact nature, pure C implementation, and relatively clean API — very unusual characteristics for graphics middleware. The hands-off approach to cross-API shader handling is a curse and a blessing: we bear the burden of supporting translation for each backend correctly, OTOH it gives us the freedom to implement a sane system that doesn't rely on "portability macros" and brittle text substitution, as is often the case with middleware.

  Downsides include the inherent overhead and reduced debuggability that comes with any middleware, and a bus factor of 1. I will not be able to adequately address problems with the D3D and Metal backends.

• WebGPU: it's cursed, I know, but may actually not be as crazy as it sounds. Unlike typical graphics middleware, this one actually has a formal open spec, and two independent implementations in development. It's also designed to run on top of Vulkan, D3D12, and Metal efficiently (at least it's supposed to be), and does not suffer from shader portability woes quite as much as typical middleware does (its shading language is beyond fucking stupid, but at least it's meant to be easy to generate from SPIR-V). It is also directly usable from Emscripten.

  Downsides: monster dependency written in C++ or Rust (pick your poison), and perhaps most importantly, the spec isn't even stable yet, it's all very much in development. Increased overhead and reduced debuggability also apply here.

• bgfx and similar middleware: I don't really see the point. It shares all the downsides with WebGPU, except that it's usable now, and has none of the advantages.

Regardless of what alternative rendering backend we decide to implement (if we ever do), getting it to live under the same abstraction layer with the existing renderer is going to be a challenge. The current API evolved from hardcoded OpenGL 2.1 gunk of pre-1.3 era, and it really shows, even though it's been refined over time. We will certainly have to fix some design flaws with it. But probably the hardest part will be reconciling various annoying conventional differences between different 3D APIs, e.g. left-handed vs. right-handed coordinate system, top-left vs. bottom-left image origin, vertex winding order, top-left of pixel vs middle of pixel on integer coordinates, etc. etc. etc. Some of these are configurable to a degree, some aren't, depends on the API. In general I have no idea how to deal with all of it robustly yet, but I guess we'll cross that bridge when we get there...

[Akaricchi]

Things like Emscripten are ever-moving goalposts, requiring special treatment to get working effectively

I don't see how this is related to rendering at all. Most of the problems with emscripten come from their apparent inability to maintain stable interfaces, or the kind of fuckups you really can only dream up high on Javascript. If anything, bolting a rendering middleware layer on top of it will probably have us end up patching the middleware as well.

[StarWitch]

That's fair, I clarified my initial post. The it there was vague and I meant to say other platforms since I do agree it's largely Microsoft and Apple's fault we're in this mess.

With the Emscripten stuff, I was mainly referring to the fact that several workarounds have had to be put in place directly in the rendering pipeline due to the way different browsers interact with it, but I suppose expecting even a library like Sokol to properly handle those edge-cases is a bit optimistic.