versioning and incompatibilities

[zenspider]

There are over 6700 lines of diff between 0.3 and 0.4 and it is not backwards compatible to 0.3 functionality. Please consider:

• Using a changelog to document important changes
• Using semantic versioning to highlight when incompatible changes can be expected. OR declaring that versioning strategy you're using so we don't have to guess if an upgrade is safe or not.
• Maintaining backwards compatibility across tiny bumps of ruby (eg best I can tell, upgrading to ruby 3.3.1 broke one of my gems because of net-imap changes).

[zenspider]

Tagging @nevans since it looks like most of the 0.4 work is from your work.

[nevans]

@zenspider Thanks for the feedback. The v0.4.0 release was a big one, but I did try to keep incompatibility to a minimum. In some cases incompatibilities were accidental or bugs, so please open tickets for any particular issues you run into. If it isn't documented, it's a bug (at least a documentation bug). If it is documented, maybe we need better documentation.

---

Re: your specific points

Using a changelog to document important changes

I'm currently using the generated GitHub release notes (and updating them by hand), and that's what's linked from the rubygems metadata (and rubygems.org). I would actually prefer to start from a CHANGELOG.md and use that to generate the GitHub release notes... but the convenience of generating GitHub release notes and the lack of any existing CHANGELOG.md has so far been enough friction that I haven't gotten around to it.

Using semantic versioning to highlight when incompatible changes can be expected.

(tongue-in-cheek response) Until version 1.0.0, we are following semver: "Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable." 😜

OR declaring that versioning strategy you're using so we don't have to guess if an upgrade is safe or not.

Seriously though, although I knew there would be some incompatibilities between 0.3 and 0.4, my intention was to have as few as possible, so long as you weren't monkey patching. All of the incompatibilities that I knew about at the time of release were documented on the v0.4.0 release page, but some backward compatibility bugs have been reported and fixed since then.

I'll see if I can summarize my "versioning strategy" in a future comment, but I don't have time right now and I'd rather reply sooner rather than delay until I'm ready with that. For what it's worth I've started to write a simple policy for the README on several occasions, but wound up throwing away those draft PRs for one reason or another.

Maintaining backwards compatibility across tiny bumps of ruby (eg best I can tell, upgrading to ruby 3.3.1 broke one of my gems because of net-imap changes).

That sounds like a bug to me. As far as I know, ruby 3.3.0 bundled net-imap 0.4.9 and ruby 3.3.1 bundled net-imap 0.4.10. And (ironically?) the only significant difference between those two versions was fixing a backwards incompatibility bug. Have you tested that the issue doesn't also exist in net-imap 0.4.9 with ruby 3.3.1?

It sounds like you may have spent a fair bit of time and energy on this already, but I'm interested in the incompatibilities you experienced. I'd like to either document or fix all of them.

---

For what it's worth, the main reason I haven't made any new releases recently is because I'm trying to figure out the best path forward for backwards compatibility. Incompatible changes need to be made to Net::IMAP#search for it to 1) properly support various extensions, and 2) avoid raw IMAP strings and the injection attack vectors those create. And getting that right, has taken up most of the time I have available to work on this project.

[nevans]

I took a quick look through the release notes and added the following breaking change to the v0.4.0 release notes:

• The #extension attribute on BODYSTRUCTURE structs no longer starts with location. The location is now parsed directly and available from the BodyStructure#location attribute.

...but there are other incompatibilities that were handled as bugs (and fixed in later releases).

I also created a PR for generating CHANGELOG.md from the GitHub Releases: #274. Although I would prefer to go in the other direction and generate the GitHub Release from CHANGELOG.md, this seems like the simplest change from our current release workflow. I'm considering adding a GitHub action to enforce CHANGELOG.md updates with each PR; something like https://raw.githubusercontent.com/ruby/syntax_suggest/4d9118f7eec06cf327a946a09a9ecb4e07a15955/.github/workflows/check_changelog.yml. What do you think?

[zenspider]

Known cheeky response, but I think when your 0.x is literally 2 decades old, you're not really allowed to be considered "unstable" at that point, no?

[zenspider]

As far as generating release vs changelog... I dunno... does it matter as long as (at least) one is automatically done? I generate my changelog entries from formatted commit messages and I don't bother with github releases at all. I tag (also generated) my releases so that diffs / partial logs are easy. It is a near-zero effort for me. I care about gem releases and I couldn't care less about github's.

[zenspider]

My bugs: I think that mostly I was violating internals. 😘

Some things don't matter like forcing authenticators public:

- Net::IMAP::Authenticators.send :public, :authenticators

but some things moved namespaces:

-        authenticators = Net::IMAP.authenticators
-        auth_types = authenticators.keys.sort.join ', '
+        auth_class = NEWER ? Net::IMAP::SASL : Net::IMAP
+
+        authenticators = auth_class.authenticators
+
+        auth_names = NEWER ? authenticators.names : authenticators.keys
+
+        auth_types = auth_names.sort.join ', '

so mostly it was on me and easy enough to figure out by reading your tests.... WHICH! I am totally stealing your fake_server in the hopes I can finally get some automated tests that don't rely on dovecot! Thank you for that!

[nevans]

Known cheeky response, but I think when your 0.x is literally 2 decades old, you're not really allowed to be considered "unstable" at that point, no?

Well, there is https://0ver.org/. 😜

Related and more seriously, when a project is decades old, like Net::IMAP, a lot of downstream users inevitably develop dependencies on internal implementation details.

As far as generating release vs changelog... I dunno... does it matter as long as (at least) one is automatically done?

Not really. I prefer CHANGELOG.md but some people and tools seem to prefer GitHub releases. So I do intend to automatically generate one from the other.

My bugs: I think that mostly I was violating internals. 😘

I still understand the frustration. One of my applications is still violating a lot of Net::IMAP internals, so I had many of my own v0.4 upgrade issues.

but some things moved namespaces:

I had intended to keep (deprecated) constants and methods for anything I moved that was public. But

so mostly it was on me and easy enough to figure out by reading your tests.... WHICH! I am totally stealing your fake_server in the hopes I can finally get some automated tests that don't rely on dovecot! Thank you for that!

Thanks! I'm glad you found that helpful. A few parts of its API still feel clunky to me. But it feels like a successful experiment, so I'll probably extract it into a separate gem at some point. In the meantime, you can just copy it into your own test suite. I'm curious to see what you do with it.