November 26, 2023

In early October I read a blog post by Toomas Vahter describing a change introduced in Xcode 15 beta 5 over the summer and that I didn’t notice in the release notes at the time (guilty, I don’t read them for every beta).

The post detailed the effects of the change, summarised in the following disclaimer Apple added to URL’s documentation:

For apps linked on or after iOS 17 and aligned OS versions, URL parsing has updated from the obsolete RFC 1738/1808 parsing to the same RFC 3986 parsing as URL. This unifies the parsing behaviors of the URL and URLComponents APIs. Now, URL automatically percent- and IDNA-encodes invalid characters to help create a valid URL.

After reading that post, I reviewed the official documentation, since I make a Mac app that allows Users to enter their IP camera URLs for streaming purposes; I also ran additional tests on Sonoma, even if I had been using the new macOS and Xcode betas on my main computer all summer, without noticing any issue. Since everything was fine, I concluded that my initial interpretation of the note – “For apps linked on or after iOS 17 and aligned OS versions” – was correct and that I would not need to deal with the change until I would have raised the deployment target of my app from macOS 10.14 to Sonoma or newer.

My assumption was quite the mistake: linked on does not mean targeting, turns out™ that Apple means running on. Most certainly a native English speaker would have had a better month than I had…

Next day update #1: As Jeff Johnson and Alexander Blach kindly and patiently explained, linked on refers to the SDK version: as soon as my app compiled with Xcode 15 (which includes the macOS 14 SDK) runs on macOS 14 Sonoma, the new URL behavior is used. I really appreciate their help in clarifying the meaning and implications of that note!

Here’s a quick recap of the long journey that reminded me, again, that words matter a lot and, most importantly, what you might need to know if you deal with URLs in an app running on Apple platforms.

A bit of context: my app does not have a large user-base, we’re talking very few thousands Users; on Sonoma launch day, I had a new version ready and for 99% of the Users who updated macOS, it was smooth sailing; I had a hiccup that required a quick update for Users still on 10.14, because Xcode 15 made a bit of a mess with linking, but everything went well after that, as I did expect after running the app on Sonoma all summer.

Then, over the following weeks, 5 different Users contacted me reporting a crash at launch; reinstalling a previous version of my app that I provided patched the problem, allowing them to immediately resume using the app, but any new beta I sent trying to investigate what was going on would crash and burn immediately at launch, and the crash reports they kindly provided pointed me to code that had been left unchanged for years.

There was no common hardware or software denominator among them, only that they ran Sonoma and had previously been using my app for a while. As for me, I could not reproduce the crash, and taking out every single line changed in last releases did not help: the same source code of the previous version they had installed and were using successfully immediately crashed after recompiling it. All of these Users have been so patient and supportive, I can’t thank them enough for helping me throughout very frustrating weeks.

Finally, I built a beta with logging on file for each single step the app performs at launch, and found this message logged:

 dataCorrupted(
            Swift.DecodingError.Context(
                codingPath: [_JSONKey(stringValue: "Index 3", intValue: 3), CodingKeys(stringValue: "webcamStreamURL", intValue: nil)],
                debugDescription: "Invalid URL string.",
                underlyingError: nil
            )
        )

One by one, all Users’ logs showed a similar message, so I immediately thought about that URL change that would not be relevant for me right now. But why only for these 5 people, and not hundreds of other Users who were using Sonoma without any issue?

I asked some of the 5 Users about their URLs, and found out they had special characters in their passwords. Uhm, interesting. Never been a problem, but I certainly could try using the new URL(string:encodingInvalidCharacters:) initialised with false and see if it helped.

It did not.

What certainly helped was discovering that just adding a @ inside a password would cause the last version of my app to crash, but not the previous ones. Oh, the joy of finally being able to reproduce a bug.

Manually escaping those special characters, or changing passwords to alphanumerical ones, also was a relevant discovery.

mmm.

Thanks to a precious suggestion from a friend who systematically saves my bacon, I then learned that a URL could be decoded as String; that piece of knowledge snowballed into learning very interesting – and to me unexpected – things.

So, let’s jump into what I did find out today, doing experiments with a Playground.

First, let’s review the URL behavior I have serenely enjoyed for over five years:

All three screenshots were taken on Ventura, both with Xcode 14 and 15.

You can see that the URL initialises successfully even when a “special character” is included in the password.

Now, look at what happens on Sonoma with Xcode 15 (daily I use 15.0, but the latest version available at this date – 15.1 beta 3 – behaves exactly the same):

Saw that on line 10? Just throw in a special character into the password and the URL is nil; fine, circa, at least now I know where my last weeks went (actually very annoying to have a breaking change like this one poorly documented).

But what about the new URL initialiser that has a parameter that enforces the previous behavior? Yeah, using that one – on line 11 – does not change the result, still nil. This is quite the surprise.

Next day update #2: Yeah, not so much of a surprise: the initialiser’s new parameter is called encodingInvalidCharacters:, and when I think about it, a space is an invalid character in an URL, but an @ is not, so it makes sense that it is not encoding it. This debugging experience feels like a minefield of onions, each one with layers and ready to explode if you make one wrong assumption…

Luckily, I (and you, if you found this on Google) can dance in and out of the problem by manually percent-encoding and decoding the string before using it to initialise the URL.

Next day update #3: Not so fast, kiddo. Manually adding the percent-encoding results in a non-nil URL, but having a URL that successfully resolves is a completely different story… in my case, “the dance” results in valid URLs that do not actually connect to the cameras. The solution for me will therefore likely be to manually handle specific special characters (so far, I’m sure about @ and /, but I’ll basically have to manually test most non-alphanumeric characters) strictly inside the credential components.

I still need to implement the change into my app (I will be very, very careful: the last thing I want and need in my life after the last month is to go from 5/X.000 users with a problem to X.000/X.000), but I wanted to blog about this because I cannot be the only person that has been bit by this change.

I wonder if there’s Feedback for Apple into this. Maybe about the docs? I now understand that I should have not stored and initialised URLs the way I did, but the fact that they just worked fine for years caught me by complete surprise when this change in URL behavior happened, and certainly this was not easy to debug.

I’ll think more about this, after I’ll have fixed my app for those extraordinary Users without breaking anything for the others; in the meantime I hope my fun fun fun journey at least helps some fellow developer.

March 15, 2021

GlanceCam 3 launched last week and, for such a niche product, I couldn’t be happier:

• Lots of Users reached out via email to let me know they’ve been waiting for multi-windows for a long time and loved the implementation;
• No dangerous bug or crash ruined the experience;
• Sales spiked more than I have expected (the ratio among lifetime unlocks and yearly subscriptions also surprised me: it’s currently 4 to 1); of course the rush is now almost over, but still, it was nice while it lasted;
• Cherry on top: iMore wrote about it!

So, I’ve decided to celebrate by reflecting on my mistakes in the 2 years that took me to launch GlanceCam 3, or I should say the mistakes I made causing me to take 2 years to ship it.

I’ll try to keep this of reasonable length and actionable to other indie developers:

1. Lack of focus: I started working on multi-windows in March 2019 and since then I’ve shipped 5 medium-size updates to GlanceCam, launched 3 small new apps (PhotosUpload, Link HUB and ClipBar), updated Walk More and Tame Time, and spent 5 months on an unfinished Catalyst project I still very much hope to complete some day (it’s almost there…).
   With a day job (I work in a completely unrelated field to programming) I can’t do that: if I’m lucky (and able to actually get up at 4.30 a.m.) I have 2.5 hours per morning during the week and a little more time on Saturdays and Sundays to program.
   I need to pick my battles carefully and I can’t give myself the illusion of being productive because I work on too many things at once.
2. You guessed it, as it goes hand in hand with the first mistake: lack of motivation. If you don’t make progress on something for a long time, you start procrastinating… maybe even launch 3 different apps instead of working on the only one that generates a little revenue.
   When you planned over and over to work on something and don’t make tangible progress, it’s a very dangerous place for that project to be in.
   Things need to keep moving or they die.
   I feel lucky to have been able to finally find the willpower to really put my head down and work for almost 300 hours on GlanceCam this year (since January, outside of my day job I haven’t done anything but coding, talking walks with Milla, eat, work and do very stupid things to my sleep-cycle), but I realise it was an unreasonable and unbalanced way to get both my and my app’s mojos back.
3. Branching proliferation: I wanted a big 3.0 with multi-windows, new Preferences and much more, so my idea was to keep updating GlanceCam while working on the other features. Then, when I got stuck on difficult problems connected to multi-windows development, I started from scratch again and again (branches: multiwindows_take2, _take3 and _take4, which eventually shipped) while also concurrently working on Preferences and even custom one-off features I sent to some Users who asked for tweaks. Merging was a nightmare, because in the meantime the shipping app progressed.
   If time is constrained, I must do ONE thing, finish it, and then move to the next one.
4. Aiming for the stars: a big 3.0 was a reasonable desire, and indeed I don’t think some press would have mentioned a camera viewer if it was a smaller update, but before this long release-cycle I was always smart enough to build small updates and ship them in a timely fashion.
   I’m returning to my better habits already, as GlanceCam 3.1 has already been submitted for review, but finding balance between frequent releases and their depth is very important, and probably will continue to be a struggle in the future, for instance when I’ll start thinking of GlanceCam 4 (or maybe of the much requested iOS version that’s more of a new app altogether… see, I am learning: I can foresee that, while promising in terms of market size, that’s also dangerous for focus and branching proliferation).
5. Managing project-related informations: during the weekend I asked other developers about their tools for keeping track of their work, feature requests, bugs and ideas and got very good answers. In that thread I’ve also shared where my journey on this ended (not far from where it started, OmniFocus, but with better structure).
   Working on different things at the same time, in different directions and on multiple apps, I’ve let my to-do lists slip a bit, then more than a bit, until they got stale and their periodical review became painful and stopped happening.
   I know better than that: my mind works best if I have a single stop between home and work, a single and up-to-date bucket for everything that’s related on planning and developing a project.

I sure have made plenty of other mistakes (letting a single feature request derail my plans, allowing a review to mess with my mood, starting work on something before doing the appropriate research and due diligence, leaving marketing to the last few days, even building a whole new website the day before launch…), but this is probably enough introspection for me to go through in one sitting, and for you to read… thank you for coming along, hopefully there was some lesson for you to extract from my mistakes!

June 21, 2020

Hello visitor from Google, I hope you are well, but most likely you are not, because you are fighting with syncing issues.

I posted this on Stack Overflow earlier today, but it actually also makes sense to have it here on my blog.

In a new app I’m working on, I had syncing issues (not upon launch, when it synced fine, but only during the app being active but idle) in a project that uses Core Data + CloudKit with NSPersistentCloudKitContainer.

My app was built using Xcode’s 11 Master-Detail template with Core Data + CloudKit from the start, so I had to do very little to have syncing work initially:

1. Enable Remote Notifications Background Mode in Signing & Capabilities for my target;
2. Add the iCloud capability for CloudKit;
3. Select the container iCloud.com.domain.AppName
4. Add viewContext.automaticallyMergesChangesFromParent = true

Basically, I followed Getting Started With NSPersistentCloudKitContainer by Andrew Bancroft and this was enough to have the MVP sync between devices (Catalina, iOS 13, iPadOS 13) not only upon launch, but also when the app was running and active (thanks to step 4 above) and another device edited/added/deleted an object.
Being the Xcode template, it did not have the additional customisations / advanced behaviours of WWDC 2019’s sample project, but it actually accomplished the goal pretty well and I was satisfied, so I moved on to other parts of this app’s development and stopped thinking about sync.

A few days ago, I noticed that the iOS/iPadOS app was now only syncing upon launch, and not while the app was active and idle on screen; on macOS the behaviour was slightly different, because a simple command-tab triggered sync when reactivating the app, but again, if the Mac app was frontmost, no syncing for changes coming from other devices.

I initially blamed a couple of modifications I did in the meantime:

• In order to have the sqlite accessible in a Share Extension, I moved the container in an app group by subclassing NSPersistentCloudKitContainer;
• I changed the capitalisation in the name of the app and, since I could not delete the CloudKit database, I created a new container named iCloud.com.domain.AppnameApp (CloudKit is case insensitive, apparently, and yes, I should really start to care less about such things).

While I was pretty sure that I saw syncing work as well as before after each one of these changes, having sync (apparently) suddenly break convinced me, for at least a few hours, that either one of those modification from the default path caused the notifications to stop being received while the app was active, and that then the merge would only happen upon launch as I was seeing because the running app was not made aware of changes.

I should mention, because this could help others in my situation, that I was sure notifications were triggered upon Core Data context saves because CloudKit Dashboard was showing the notifications being sent:

So, I tried a few times clearing Derived Data (one never knows), deleting the apps on all devices and resetting the Development Environment in CloudKit’s Dashboard (something I already did periodically during development), but I still had the issue of the notifications not being received.

Finally, I realised that resetting the CloudKit environment and deleting the apps was indeed useful (and I actually rebooted everything just to be safe ;) but I also needed to delete the app data from iCloud (from iCloud’s Settings screen on the last iOS device where the app was still installed, after deleting from the others) if I really wanted a clean slate; otherwise, my somewhat messed up database would sync back to the newly installed app.

And indeed, a truly clean slate with a fresh Development Environment, newly installed apps and rebooted devices resumed the notifications being detected from the devices also when the apps are frontmost.
So, if you feel your setup is correct and have already read enough times that viewContext.automaticallyMergesChangesFromParent = true is the only thing you need, but still can’t see changes come from other devices, don’t exclude that something could have been messed up beyond your control (don’t get me wrong: I’m 100% sure that it must have been something that I did!) and try to have a fresh start… it might seem obscure, but what isn’t with the syncing method we are choosing for our app?

2020.12.02 - Update for Users who set NSPersistentStoreDescription to interact with public databases and are noticing delays in the updates.

A few days ago, Ben Radler brought to my attention a behavior that those who use public databases with CloudKit (I was using the private database in the original post) need to take into account when testing:

If you are setting the NSPersistentStoreDescription to interact with the public database rather than the private database (previous default before iOS 14 I believe) via description.cloudKitContainerOptions?.databaseScope = .public, it will only poll for changes from cloudkit every 30 minutes by design (see 10:30 into the WWDC 2020 session video on this topic).

I want to thank Ben for pointing this out, since it might also be a cause for head-scratching when testing CloudKit sync!

April 6, 2018

Alternate title: TIL Xcode does not symbolicate Mac applications .crash files, that only works for iOS apps.

I made a pretty obscure mistake in GlanceCam 1.3 that lead to a crash during its app review.

I’m very sorry for wasting app review some time (they have been amazing, both understanding and allowing an hedge-case for sandboxing and giving GlanceCam lighting fast reviews, usually less than a day!) , but they have been so kind to attach a crash report and I had to symbolicate it (“Symbolication replaces memory addresses with human-readable function names and line numbers”).

Having done that a few times in the past directly in Xcode, I went straight there, but as I mentioned above, no luck. There’s plenty of resources online on how to do that “by hand”, but I want to point out a couple, very useful, solutions for other fellow developers:

Bob Matcuk’s gist perfectly describes all the steps to manually symbolicate a crash report, including checking the build number; this involves using the atos command in Terminal, which is not bad at all, but it can get a bit repetitive if you want to check more addresses (lines). Tomaz’s Symbolicator is a Swift Xcode project that compiled right out the box (sometimes, with Swift the language fast-paced evolution makes you work a bit before a project from the Internet actually builds) generating a very easy to use command line application that automatically fetches the DSYM files from the Xcode archives and overwrites the crash report with functions and line numbers that a simple human being like myself can actually understand.

July 11, 2017

My friend Becky recently wrote a post about the dilemma she’s facing with a new, interesting app she’s making.

She likes Core Data, and would like to use it for her project; she also wants to add sync capabilities, as that’s a requirement for most modern apps.

Here lies her dilemma. And my dilemma. And many developers’ dilemma… There’s no clear path towards a Core Data app with cloud sync.

Becky doesn’t want to use Core Data + iCloud since it’s deprecated. I might add, I don’t want a friend to use it, because I value her sanity… when I was setting up Tasktic’s sync mechanism, I spent 4 months and 3 complete rewrites before realizing my code wasn’t the problem: it was the actual API that was an almost un-debuggable black box that sometimes, very rarely but still too often, lost an object during the sync process with apparently no reason, and no way to get it back.

A few weeks ago, I suggested Ensembles to Becky, since it was the solution I adopted for Tasktic and given how well it behaved for me, restoring my sanity after those awful 4 months. But the free version of Ensembles still uses Core Data + iCloud under the hood***, so it’s not very future proof given the deprecation mentioned above, and on top of that there’s a more modern, faster solution provided by Apple that everybody want to use: CloudKit.

CloudKit is very tempting because everyone has experienced how fast and reliable it is with Notes.app and Photos.app for macOS and iOS; the most important thing for an app that sync is to never, ever lose users’ data, and CloudKit passes that test with flying colors. It is also very modern, with private/public data, web support via CloudKitJS, and most important, it is what Apple has chosen for the future, and following Apple’s lead is always a good idea.

The thing is, in order to get CloudKit to play nice with Core Data, you have to write most of the sync logic yourself, converting NSManagedObjects to CKRecords, handling updates, reacting to duplication, etc. It can be done, and many developers do it “by hand”.

As Becky notes, it can also be done via libraries like Seam3, which is currently the best open source implementation I found (and I looked really long and hard) of a Core Data - CloudKit bridge, albeit only if you don’t have many-to-many relationships in your schema…

The fact is, I really don’t want a dependency for my sync code anymore, especially when starting a new project. If I were willing to accept the risk of my sync code being abandoned someday in the future, I might as well look into Realm (not that such a loved mobile stack is going anywhere, but as we’ve seen in the past, companies get acquired, or sometimes move on to different projects…).

Where does this leave us? To Apple, of course.

Apple made such an amazing object graph and persistence framework with Core Data, and a fantastic syncing backend with CloudKit. They never connected those dots officially, though.

Becky’s post reminded me of something I really wanted (needed) for a really long time: an Apple sample project showing their idea of the best, most modern and Swifty implementation of Core Data for local persistence + CloudKit for sync implementation.

This is a step they really should take: while it’s great that they provide sample projects for ARKit, data persistence and sync is top priority for a lot of developers, and it would only be appropriate for Apple to show how they think a “great” implementation should look like.

So, if you have a friend who works at the best fruit company in the world, pass along the message: the indie developer community, and especially us beginners, would love some help in this area!

** This is true only if you want to stick with native cloud solutions, like I prefer, otherwise you can also pair Ensembles with your own backend or use Dropbox: the great thing about Ensembles is that it’s actually backend agnostic, but for CloudKit you need Ensembles 2, which isn’t free and so it’s not an option for most indie developers like myself.