Handle RCS messages properly, and improve group MMS

[locofocos]

Fixes #1055 and #1027

This fixes RCS importing 🎉

It also improves how group MMS threads are imported. Basically it includes all the recipients in the "to" field, and puts everyone's name in the subject, so that gmail does a much better job mimicking the group threads on your phone.

I've been toying around with it on my Pixel 3a running Android 10. I've tested with a mix of SMS, group MMS chats, and RCS chats.

I'm completely open to the idea that I was a little heavy-handed with my refactor. I ran with what made sense. I'm fine if someone else wants to take this as as proof-of-concept and run in a slightly different direction to make this mergeable.

TODO

• See if there are any unit tests that have been broken
• Probably some new unit tests (to be honest..... I really don't want to build these out).
• Give this another look over for general code quality. The old version of Java (and me being a bit rusty with Java) made some things like sets/arrays/uniq/etc a bit clunky.

Testing

Tips for testing:

• If you want to try running this multiple times and tweaking how we generate ids/subjects/etc, make sure to delete your emails generated from testing AND empty gmail's trash. Otherwise gmail will cache things. I saw it caching the REFERENCES field when I re-used the same MESSAGE_ID.
• Once you've identified some messages on your phone for a certain test case, it can be helpful to just process those messages. You can hardcode timestamps in the queries in app/src/main/java/com/zegoggles/smssync/service/BackupQueryBuilder.java. https://www.epochconverter.com/ is your friend here.

Some test cases that I've gone through on my phone, and verified that the backup looks correct in Gmail:

• RCS message with 1 person. working as of bec0a00
• RCS message with multiple people. Working as of bec0a00
• MMS thread with lots of people going back and forth. working as of bec0a00
• Message with 1 person, both SMS and MMS (texting a non-RCS user, but sometimes sending images).
  • Less than 100 messages. working as of bec0a00
  • At least 100 sms messages before the MMS got backed up. Not working, but you won't see it break as long as you backup frequently
    • Unfortunately we'll probably have to live with this limitation. In theory it could be fixed, but it would require processing SMS and MMS messages interleaved, from oldest to newest. Those are pulled from two separate data sources and two queries, see BackupItemsFetcher.getItemsForDataType and Consts.MMS_PROVIDER / SMS_PROVIDER. So we would have to build some intermediate caching mechanism that gathers all the SMSs, gathers all the MMSs, interleaves them, then feeds them into the existing logic.
    • I tried using the "only backup 100 at a time" setting. Unfortunately it looks like this processes batches of SMSs first, then starts processing MMSs once the SMSs are exhausted. So it doesn't help.

Adopting this new version

Here is my 2 cents for how affected users could get their backups into a clean-ish state:

• The "I don't care" way: just start using the new version, and everything going forwards will be fixed.
• The lazy way: just reset the backup state (or reinstall the app), back up everything again, and deal with duplicates in gmail.
• The harder way: Go into Gmail and update all existing emails with label old-backups. Download the new app version. Reset the backup state (or reinstall the app) and backup everything again (to your favorite label, like SMS). For whatever time period you still have on your phone, delete emails from that time period with label old-backups. Verify that you have a nice continuity when looking at both old-backups and SMS. Then change all the old-backups emails over to the SMS label.

[kurahaupo]

@locofocos Thank you so much for doing this; it will close a large number of outstanding tickets.

I will review in detail shortly.

(It will still be up to @jberkel to publish to Google Play.)

[kurahaupo]

I'm really happy with the separation of records into sender and recipients.
This will fix not just RCS, but almost every issue in the "half missing" category.

Again, many thanks.

[locofocos]

Some context, from what I learned doing this: MESSAGE_ID is the unique message on gmail's end. REFERENCES is the field (on gmail's end) that determines which email an email is replying to.

For the last message grouping fix I built, I wasn't seeing it take effect, even after I deleted all my sms backups from gmail and ran the back again. I was able to determine that gmail was caching the REFERENCES field by clicking "show original" in gmail and looking at the message header. I had to force the app to generate new MESSAGE_IDs. Basically there's some string we pass into the hash function that generates MESSAGE_ID. You can just concatenate "v2", "v3", etc. to the hash input. It's almost like bumping a version number. Once I generated a new MESSAGE_ID for those messages I had already backed up before, then my last fix started working.

However, this has repercussions for releasing this fix. I tried backing up all the messages on my phone with a clean build of this current PR. I'm using a different gmail label (SMS-testing instead of the usual SMS) to keep things clean, but the app was still generating the same MESSAGE_ID as before. So when I checked on some older (already backed up) messages, they weren't grouped correctly 😢

I can vet this out, but I believe if we version/change the hash input (like we start concatenating "v2"), that will force gmail to recognize these as new messages. So then users will be able to reset the app state and re-backup their existing messages, and then older threads should get fixed 🎉 I expect the old messages would linger around in gmail as well, so there may some duplicates... but users will only get this if they choose to re-backup their existing messages.

This will also be a useful opportunity for me to check against more test messages. I'll check this out some day soon!

[locofocos]

Good news! Versioning the hash input ab80cc0 fixes this problem nicely. This will be helpful to force gmail to recognize these messages as new threads and group them correctly.

[locofocos]

@kurahaupo you're welcome! Thanks for the initial review. I'll get this cleaned up and tested further some time soon, I hope.

[locofocos]

@kurahaupo I've given this some solid testing on my phone, and I'm happy with the result.

Is the travis CI build running against this PR? I was looking for the build output so I could at least see which tests I need to update.

edit - I might need some some more time to look over the results and debug more. All of those test cases look good. But I did a another huge backup last night, and I'm seeing some ungrouped messages.

[kurahaupo]

I'll check on the CI when I get home to my PC

[hubono]

Hi
I tried to compile from source but it keeps failing. Can someone post an APK please

[locofocos]

@hubono I debated whether I should post a copy of my latest APK build, just in case life happens and we don't get this PR merged.

Caveats, if this wasn't obvious:

• This code hasn't been well reviewed
• It's code that some stranger on the internet wrote
• It's built in debug mode, with a self-signed certificate (something like that). So android won't allow you to upgrade your existing app with this. Android is rightly protecting you from installing an APK built by someone other than the author. So you'll have to uninstall your existing app first.

However, I've been using it for months. No promises, but I'm happy with the results. So here is the APK I built locally from ac71d55

SMS backup plus with RCS fix v2.apk.zip

[chadcancode]

@locofocos Question: I am currently running 1.5.11 from the app store. I want to try your improved version to see if it will fix the GMail threading issues I had. I am unclear what the upgrade path is. Since I am sideloading, do I need to uninstall 1.5.11 first? Also, you mention the "backup state" multiple times above but since this is technically a new app sideloaded, does that mean that by default, all my messages on my phone will get backed-up again into GMail?

[locofocos]

@chadcancode Since you're sideloading, yeah, Android will force you to uninstall 1.5.11 first.

I made some clarifications to the upgrade path under Adopting this new version above. The "backup state" is basically this app's knowledge of which SMS messages have been backed up so far. You're right - a new sideloaded app won't know which messages are backed up. IIRC this app will ask "Do you want to backup and skip all existing messages on this phone" the first time you launch it.

It's really up to you. If you want a perfect seamless state, do "the harder way". If you don't care about duplicates, use "the lazy way".

[kranix0]

@hubono I debated whether I should post a copy of my latest APK build, just in case life happens and we don't get this PR merged.

SMS backup plus with RCS fix v2.apk.zip

@locofocos may I humbly request that you (or some other kind soul) provide a version compiled in 64-bit, please?

On my Pixel 8 Pro I get the error "you can't install the app on your device" when installing. My understanding is that newer phones or Android versions no longer support 32-bit apps. I did uninstall the old app first.

[locofocos]

I made a few changes to get this compiling again - I think Google removed some library versions from their maven repo (thanks Google...).

I think we can compile it against 64 bit, but I'm not sure what all is required. I added ndk.abiFilters 'armeabi-v7a','arm64-v8a' to app/build.gradle following https://developer.android.com/google/play/requirements/64-bit#android-studio-gradle. Maybe that's sufficient to get it working. Or maybe someone needs to audit and upgrade every library/dependency to make sure those are using 64-bit-compatible versions.

@kranix0 See if this works on your Pixel 8: SMS.backup.RCS.maybe64bit.cb3a97f.apk.zip

[locofocos]

Actually I think 64 vs 32 bit might not be the issue. Stackoverflow says

If there is no directory lib, it supports all architectures.

and the APK doesn't have any lib folder. So I think it already supported 64 bit phones.

This looks much more promising: https://www.reddit.com/r/GooglePixel/comments/17bzbcy/install_failed_deprecated_sdk_version_any_way/ This app uses minSdkVersion 14 and targetSdkVersion 29. I'm curious if you can get the full error message by using adb to install it.

[kranix0]

@kranix0 See if this works on your Pixel 8: SMS.backup.RCS.maybe64bit.cb3a97f.apk.zip

@locofocos amazing… it works! I haven't tested the RCS backup, but it seems to work at least as well as I had on my Pixel 6.

Thank you!!!

[diinrel]

This works great on my Pixel 9 Pro with Android 15 QPR1, though first I had to select "allow restricted settings" for the app in Android in order to allow SMS permissions. Many thanks!

[eexit]

Hello,

Thank you for your work!
Do you plan to make a release with this fix?

Thanks!

[kurahaupo]

@eexit as a co-maintainer, I can apply pull requests and make other changes here on github, but I don't have access to make updates on Google Play or on F-Droid.

Perhaps other technical users can build and post an APK somewhere that you can download it.

[chadcancode]

I made a few changes to get this compiling again - I think Google removed some library versions from their maven repo (thanks Google...).

I think we can compile it against 64 bit, but I'm not sure what all is required. I added ndk.abiFilters 'armeabi-v7a','arm64-v8a' to app/build.gradle following https://developer.android.com/google/play/requirements/64-bit#android-studio-gradle. Maybe that's sufficient to get it working. Or maybe someone needs to audit and upgrade every library/dependency to make sure those are using 64-bit-compatible versions.

@kranix0 See if this works on your Pixel 8: SMS.backup.RCS.maybe64bit.cb3a97f.apk.zip

This worked for me as well. Thank you!

[al-right]

@eexit as a co-maintainer, I can apply pull requests and make other changes here on github, but I don't have access to make updates on Google Play or on F-Droid.

Perhaps other technical users can build and post an APK somewhere that you can download it.

Hmm. Would making a forked project like "SMS Backup++", "SMS Backup+ Rev" (revived), "SMS Backup+ TNG" or "SMS Backup+ BtL" (Back to Life) or something else make it easier. Giving names is not my strong side - as long as people know its name and origin it will work just fine for me.

If a developer account is needed I am quite sure we are several people that would like to sponsor the costs of the US$25: https://support.google.com/googleplay/android-developer/answer/6112435?hl=en#zippy=%2Cstep-pay-registration-fee

Please state your price :)

[chadcancode]

I swapped out the old version that I had been using for years from the Play Store with the compiled version posted by @locofocos . I then changed my SMS label in GMail to "SMS-old" and backed-up all my messages under a new SMS label.

I am having an interesting issue with the threading. Specifically between my wife and my messages. Most of them are broken out into separate messages and not threaded together even though it's just the two of us in the conversation on a decade long thread. I'm confused because the messages I sent show my Google user name in the FROM field while the messages she sent show my phone number in the from field in the format: +1########## +1##########@unknown.email (actual number is obfuscated).

What makes it even weirder is I do see some threads between us and it's the same thing as above. Messages in the thread are either FROM my username or To my phone number.

Any ideas where this could be going awry? I don't understand if this is on the Google side or the SMS Backup+ side.

[locofocos]

@chadcancode - If I had to guess, it sounds like you missed a step in "Adopting this new version" > "the harder way"

For whatever time period you still have on your phone, delete emails from that time period with label old-backups

Getting all types of messages lined up required changing the to/from addresses. It wasn't possible to make the new email format compatible with the old format. That's why you need to re-backup whatever time period you still have on your phone for the best continuity of messages. I hope it works out!

[chadcancode]

@chadcancode - If I had to guess, it sounds like you missed a step in "Adopting this new version" > "the harder way"

For whatever time period you still have on your phone, delete emails from that time period with label old-backups

Getting all types of messages lined up required changing the to/from addresses. It wasn't possible to make the new email format compatible with the old format. That's why you need to re-backup whatever time period you still have on your phone for the best continuity of messages. I hope it works out!

I didn't understand the last set of instructions:
"For whatever time period you still have on your phone, delete emails from that time period with label old-backups. Verify that you have a nice continuity when looking at both old-backups and SMS. Then change all the old-backups emails over to the SMS label."

I just renamed the label sms to sms-old and backed up all my messages again. I then in Gmail did a lot of lightweight comparison between some messages with the old label vs the new. I was planning on then deleting everything under the old label. Why would the instructions have me change all the old backup emails over to the sms label if I backed everything up from scratch?

Also, is there some header that SMS Backup+ is supposed to use to ensure messages are properly threaded? If it matters, I'm on Android and my wife is on iPhone but not sure why that should matter at all.

[kurahaupo]

Hmm. Would making a forked project like "SMS Backup++", "SMS Backup+ Rev" (revived), "SMS Backup+ TNG" or "SMS Backup+ BtL" (Back to Life) or something else make it easier. Giving names is not my strong side - as long as people know its name and origin it will work just fine for me.

There's a pinned issue titled "fork this project" that explains some of the issues with choosing a name, especially if we wish to regain "sensitive access" to read & write messages from a gmail account directly (rather than by trying IMAP).

If a developer account is needed I am quite sure we are several people that would like to sponsor the costs of the US$25: https://support.google.com/googleplay/android-developer/answer/6112435?hl=en#zippy=%2Cstep-pay-registration-fee

Please state your price :)

My main problem is that while I'm a very experienced developer in general, so I can easily read and modify code in many languages, I'm a newbie at setting up a build pipeline. I've run through various tutorials and they all assume that you're starting with a new (empty) code base, so that they conflict with what @jberkel wrote for for this package. And simply copying the existing build doesn't seem to work.