Cloudkit - What to Do When a User Adds, Modifies or Deletes an Object While Offline

CloudKit: Delete local objects based on CKQueryNotification

If CloudKit doesn't give you any indication of the type of record that was deleted, it's kind of a pain to deal with. You can't delete objects in Core Data without knowing the entity type, so if CloudKit doesn't give you any clues then you need to check every entity that could have the recordId.

The delete process is the same as usual with Core Data. Do a fetch request with a predicate of something like `recordId = %@" to find a matching object. If you find one, delete it. Except that you have to repeat this for each potential entity.

One approach that might help is to store the recordId in a new, separate entity. Create a new entity called something obvious like CKRecordInfo and keep the recordId there. Every other entity that has CloudKit info would have a one-to-one relationship to this entity. With this setup you'd fetch an instance of the new CKRecordInfo entity, and delete whichever object it's related to.

At the same time though-- I haven't used CloudKit, and it's more than a little surprising that it would give you just the recordId with no information about the record type. Getting the info from the notification would be ideal, if it's possible.

CloudKit public database deleted record

Detecting changes in the public database is also handled via subscriptions, but there are different types of subscriptions for different aspects of cloudkit. As noted at https://developer.apple.com/library/content/qa/qa1917/_index.html (emphasis added)

Note: The initializers for creating a CKSubscription object with a
subscriptionID are deprecated, so use CKQuerySubscription,
CKRecordZoneSubscription, or CKDatabaseSubscription on iOS 10.0+,
macOS 10.12+, and tvOS 10.0+. Be aware that CKQuerySubscription is not
supported in the shared database, and CKDatabaseSubscription currently
only tracks the changes from custom zones in the private and shared
database.

So, you'll need to use a CKQuerySubscription to detect changes in the public database. With a CKQuerySubscription you will specify a record type, optional search parameters (via NSPredicate) and specify if the subscription should fire on creation, update, and/or deletion of the record.

The app will then receive a push notification when the trigger condition is met and is responsible to update the user's local data store as appropriate.

Cloudkit CKRecordZoneNotification how to know whether Add or modify happened

There is no info in the recordChangedBlock that tells you whether it was added or changed. Keep in mind, even if it did, you still have to check whether the record exists or not in the local store. A record can be added into CloudKit and then changed several times all while your app isn't running. When your app finally runs, it will get only the last change notification. But the record doesn't exist in your local cache yet. So you must always see if you have the record locally or not and add/update accordingly.

With deletion, all you get is the CloudKit record id. Nothing else. What I do is ensure the CloudKit record id is based on the local key. This way I can easily find and remove the local record when the data is removed from Cloudkit. It also means that the local copy of the CloudKit data on all of the user's devices ends up with the same keys.

Sync an iOS app that uses core data to cloud

CloudKit and CoreData do not automatically work together seamlessly, so you will need to write that logic yourself.

There are different types of iCloud storage options, one or two of which do seamlessly integrate with CoreData, but CloudKit is not one of them, and CloudKit is what you will need to use if you have aspirations of enabling your users to share data with others.

AKA: You will need to do the heavy lifting yourself, but if you use good design practices you can do the work once without having to rewrite much of your existing code.

So, here's something similar to what I did in one of my projects that used both frameworks:

• Create Core Data object model and NSManagedObject subclasses like you almost certainly already have.

• Turn on CloudKit in Xcode project capabilities and log in to CloudKit Dashboard

• Use CloudKit Dashboard to design your record model modeled after your Core Data entity model

• (Back in Xcode) Create methods somewhere (most conveniently as extensions to your NSManagedObject subclasses) that know how to create the given Core Data object from a CKRecord, and to create a CKRecord from a Core Data object.

• Create one or more Swift classes dedicated to dealing with your CloudKit records and synching them with Core Data. This class(es) would be responsible for performing all CloudKit operations at a high level, including fetching, adding, deleting, modifying, etc. You can design this public API however you want (it should be customized to your needs), but this class would most likely use the methods you created in the previous step to convert to and from Core Data types.

With this approach, your CloudKit specializing class (we'll call it CloudBrain) does all of the heavy lifting, and if you want you can make it do it all behind the scenes. For example, you could define another class, SyncBrain, that would automatically listen for changes in the Core Data managed object context, and call corresponding methods on CloudBrain to ensure all changes are being kept in sync with iCloud. It would also need to do the reverse, listening for changes in iCloud and applying them to Core Data. This will of course require fetching changes initially from CloudBrain and you'll also want to look into CKSubscription for real-time updates.

The beauty of this approach is that if you set up all of that correctly, you can keep all of your other code the same, because every time your other classes interact with Core Data, SyncBrain automatically ensures that all changes in Core Data are reflected in iCloud and vice versa.

As for sharing with other users, this feature is new in iOS 10 and it does not appear as though Apple has yet updated the CloudKit Quick Start. You should therefore watch What's New with CloudKit from WWDC this year.

Important Note: When designing your record model in CloudKit Dashboard, be sure to follow the iCloud Design Guide and not have parent record types with fields holding arrays of a child record type. This is not great performance. Instead, define the child record type to have a single CKReference field that points to its parent. That way if you need the children for a parent, you can just create a query that requests all objects with their parent set to the parent you want (as opposed to having to wait for all of the children to download when all you wanted was the parent).

Here are some WWDC sessions. Older sessions still contain very useful information but some of it is out of date.

• 2014 – Introducing CloudKit (A must-watch to get the concepts correct)
• 2014 – Advanced CloudKit
• 2015 – What's New in CloudKit
• 2015 – CloudKit Tips and Tricks
• 2016 – What's New with CloudKit
• 2016 – CloudKit Best Practices

What is the right way to perform persistent history purging, without affecting the correctness of CloudKit?

UPDATE: Particularly important for a CoreData + ClouKit setup

In this post from a WWDC22 Core Data Lab, an Apple Core Data framework engineer answers the question "Do I ever need to purge the persistent history tracking data?" as follows:

No. We don’t recommend it. NSPersistentCloudKitContainer uses the
persistent history token to track what to sync. If you delete history
the cloud sync is reset and has to upload everything from scratch. It
will recover but it’s not a good customer experience. It shouldn’t
normally be necessary to delete history. For example, the Apple Photos
app doesn’t trim its history, so unless you’re generating massive
amounts of history don’t do it.

tl;dr:

It seems that purging the persistent history after 7 days works in almost all cases.

It probably does not, if GBs of data have to be synced.

What I did:

I could reproduce the error:

If in Apple' demo app data are synced after the persistent history is purged, wrong data may be displayed. Apparently some info has been deleted that is essential for the demo app.

Below, I started testing with a clean setup:

I deleted the app from simulator and device, and cleared all CD_Post records in the iCloud private database, zone com.apple.coredata.cloudkit.zone, using the dashboard.

To check for info that might have been deleted unintentionally, I inserted in func processPersistentHistory() a print statement in the guard statement that filters the persistent history for transactions:

guard let transactions = result?.result as? [NSPersistentHistoryTransaction],
      !transactions.isEmpty
      else {
        print("**************** \(String(describing: result?.result))")
        return
      }  

If I run the app on the simulator under Xcode, no entries were shown as expected, and the log shows now many such entries:

**************** Optional(<__NSArray0 0x105a61900>(

)
)  

Apparently the persistent history contains iCloud mirroring housekeeping information that is deleted when the persistent history is purged. This indicates to me that the mirroring software needs "enough time" to finish its operation successfully, and thus only "old" history entries should be purged. But what is "old"? 7 days?

Next, on the simulator under Xcode, I installed and executed the app with immediate purging as in Test 1 of the question.

// Remove history before the last history token
let purgeHistoryRequest = NSPersistentHistoryChangeRequest.deleteHistory(before: lastHistoryToken)
do {
  try taskContext.execute(purgeHistoryRequest)
} catch {
  print("\(error)")
}

On the simulator, I added an entry. This entry was shown in the dashboard.

Then, on the device under Xcode, I also installed and executed the app with immediate purging. The entry was correctly shown, i.e. the iCloud record was mirrored to the persistent store of the device, the history was processed and immediately purged, although, maybe, the the mirroring software did not have "enough time" to finish its operation successfully.

On the simulator, I added a 2nd entry. This entry was also shown in the dashboard.

However, on the device the 1st entry disappeared, i.e. the table was now empty, but both entries were still shown in the dashboard, i.e. the iCloud data were not corrupted.

I then set a breakpoint at DispatchQueue.main.async of func processPersistentHistory(). This breakpoint is only reached when a remote change of the persistent store is processed. To reach the breakpoint in the device, I added a 3rd entry in the simulator. Thus the breakpoint was reached in the device, and in the debugger I entered

(lldb) po taskContext.fetch(Post.fetchRequest())  
▿ 3 elements
  - 0 : <Post: 0x281400910> (entity: Post; id: 0xbc533cc5eb8b892a <x-coredata://C9DEC274-B479-4AF5-9349-76C1BABB5016/Post/p3>; data: <fault>)
  - 1 : <Post: 0x281403d90> (entity: Post; id: 0xbc533cc5eb6b892a <x-coredata://C9DEC274-B479-4AF5-9349-76C1BABB5016/Post/p4>; data: <fault>)
  - 2 : <Post: 0x281403390> (entity: Post; id: 0xbc533cc5eb4b892a <x-coredata://C9DEC274-B479-4AF5-9349-76C1BABB5016/Post/p5>; data: <fault>)

This indicates to me that the persistent store in the device has correct data, and only the displayed table is wrong.

Next I investigated func update in the MainViewController. This function is called from func didFindRelevantTransactions, which is called when history is processed, and relevant transactions are posted. During my tests, transactions.count is always <= 10, so the transactions are processed in the block transactions.forEach.

I tried to find out what NSManagedObjectContext.mergeChanges does. Thus I modified the code as

transactions.forEach { transaction in
  guard let userInfo = transaction.objectIDNotification().userInfo else { return }
  let viewContext = dataProvider.persistentContainer.viewContext
  print("BEFORE: \(dataProvider.fetchedResultsController.fetchedObjects!)")
  print("================ mergeChanges: userInfo: \(userInfo)")
  NSManagedObjectContext.mergeChanges(fromRemoteContextSave: userInfo, into: [viewContext])
  print("AFTER: \(dataProvider.fetchedResultsController.fetchedObjects!)")
}  

To see, what happens to the viewContext, I implemented

@objc func managedObjectContextObjectsDidChange(notification: NSNotification) {
  guard let userInfo = notification.userInfo else { return }
  print(#function, userInfo)  
}

and to see how this influences the fetchedResultsController, I implemented also

func controller(_ controller: NSFetchedResultsController<NSFetchRequestResult>, 
                didChange anObject: Any, 
                at indexPath: IndexPath?, 
                for type: NSFetchedResultsChangeType, 
                newIndexPath: IndexPath?) {
  print("**************** ", #function, "\(type) ", anObject)
}  

To keep the logs relatively short, I deleted in the dashboard all CD_Post entries except the 1st one, and deleted the app from the simulator ans the device.

I then run, under Xcode, the app on the simulator and the device. Both show the 1st entry.

I then entered another entry in the simulator. As unfortunately expected, the table on the device was cleared. Here is the log of the device:

BEFORE: [<Post: 0x2802c2d50> (entity: Post; id: 0x9aac7c6d193c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p1>; data: {
    attachments =     (
    );
    content = nil;
    location = nil;
    tags =     (
    );
    title = "Untitled 3:40:24 PM";
}), <Post: 0x2802d2a80> (entity: Post; id: 0x9aac7c6d195c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p2>; data: <fault>)]
================ mergeChanges: userInfo: [AnyHashable("deleted_objectIDs"): {(
    0x9aac7c6d195c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p2>,
    0x9aac7c6d193c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p1>
)}]
managedObjectContextObjectsDidChange(notification:) [AnyHashable("managedObjectContext"): <_PFWeakReference: 0x2821a8100>, AnyHashable("deleted"): {(
    <Post: 0x2802d2a80> (entity: Post; id: 0x9aac7c6d195c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p2>; data: {
    attachments =     (
    );
    content = nil;
    location = nil;
    tags =     (
    );
    title = nil;
}),
    <Post: 0x2802c2d50> (entity: Post; id: 0x9aac7c6d193c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p1>; data: {
    attachments =     (
    );
    content = nil;
    location = nil;
    tags =     (
    );
    title = "Untitled 3:40:24 PM";
})
)}, AnyHashable("NSObjectsChangedByMergeChangesKey"): {(
)}]
****************  controller(_:didChange:at:for:newIndexPath:) NSFetchedResultsChangeType(rawValue: 2)  <Post: 0x2802d2a80> (entity: Post; id: 0x9aac7c6d195c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p2>; data: {
    attachments =     (
    );
    content = nil;
    location = nil;
    tags =     (
    );
    title = nil;
})
****************  controller(_:didChange:at:for:newIndexPath:) NSFetchedResultsChangeType(rawValue: 2)  <Post: 0x2802c2d50> (entity: Post; id: 0x9aac7c6d193c7772 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/Post/p1>; data: {
    attachments =     (
    );
    content = nil;
    location = nil;
    tags =     (
    );
    title = "Untitled 3:40:24 PM";
})
managedObjectContextObjectsDidChange(notification:) [AnyHashable("updated"): {(
    <NSCKRecordZoneMetadata: 0x2802ce9e0> (entity: NSCKRecordZoneMetadata; id: 0x9aac7c6d193c77d2 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/NSCKRecordZoneMetadata/p1>; data: {
    ckOwnerName = "__defaultOwner__";
    ckRecordZoneName = "com.apple.coredata.cloudkit.zone";
    currentChangeToken = "<CKServerChangeToken: 0x2823fcdc0; data=AQAAAAAAAACQf/////////+gT9nZvOBLv7hsIaI3NVdg>";
    database = "0x9aac7c6d193c77e2 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/NSCKDatabaseMetadata/p1>";
    encodedShareData = nil;
    hasRecordZoneNum = 1;
    hasSubscriptionNum = 0;
    lastFetchDate = "2022-06-15 13:55:25 +0000";
    mirroredRelationships = "<relationship fault: 0x2821a3c60 'mirroredRelationships'>";
    needsImport = 0;
    needsRecoveryFromIdentityLoss = 0;
    needsRecoveryFromUserPurge = 0;
    needsRecoveryFromZoneDelete = 0;
    needsShareDelete = 0;
    needsShareUpdate = 0;
    queries = "<relationship fault: 0x2821a2560 'queries'>";
    records =     (
    );
    supportsAtomicChanges = 1;
    supportsFetchChanges = 1;
    supportsRecordSharing = 1;
    supportsZoneSharing = 1;
})
)}, AnyHashable("managedObjectContext"): <_PFWeakReference: 0x2821a1900>, AnyHashable("deleted"): {(
    <NSCKRecordMetadata: 0x2802ce850> (entity: NSCKRecordMetadata; id: 0x9aac7c6d193c7762 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/NSCKRecordMetadata/p1>; data: {
    ckRecordName = "3FB952E5-6B30-472E-BC6E-0116FA507B88";
    ckRecordSystemFields = nil;
    ckShare = nil;
    encodedRecord = "{length = 50, bytes = 0x6276786e f7090000 52070000 e0116270 ... 61726368 69000ee0 }";
    entityId = 3;
    entityPK = 1;
    lastExportedTransactionNumber = nil;
    moveReceipts =     (
    );
    needsCloudDelete = 0;
    needsLocalDelete = 0;
    needsUpload = 0;
    pendingExportChangeTypeNumber = nil;
    pendingExportTransactionNumber = nil;
    recordZone = nil;
}),
    <NSCKRecordMetadata: 0x2802cdcc0> (entity: NSCKRecordMetadata; id: 0x9aac7c6d195c7762 <x-coredata://496D2B54-DDB9-47EF-945A-CC1DBA1E14E8/NSCKRecordMetadata/p2>; data: {
    ckRecordName = "0919480D-16CB-49F9-8351-9471371040AC";
    ckRecordSystemFields = nil;
    ckShare = nil;
    encodedRecord = "{length = 50, bytes = 0x6276786e f7090000 52070000 e0116270 ... 61726368 69000ee0 }";
    entityId = 3;
    entityPK = 2;
    lastExportedTransactionNumber = nil;
    moveReceipts =     (
    );
    needsCloudDelete = 0;
    needsLocalDelete = 0;
    needsUpload = 0;
    pendingExportChangeTypeNumber = nil;
    pendingExportTransactionNumber = nil;
    recordZone = nil;
})
)}]
managedObjectContextObjectsDidChange(notification:) [AnyHashable("managedObjectContext"): <_PFWeakReference: 0x2821a3060>, AnyHashable("invalidatedAll"): <__NSArrayM 0x282f75830>(

)
]
AFTER: []  

This indicates to me:

• Before NSManagedObjectContext.mergeChanges, the table was correct, i.e. it contained both posts p1 & p2.
• Merging was done again with both posts.
• In the viewContext, both posts were deleted (AnyHashable("deleted")).
• The fetchedResultsController responded by deleting both posts also (NSFetchedResultsChangeType(rawValue: 2)).
• Eventually it is logged that the fetchedResultsController has no objects, and thus the table is empty.

As a final check, I out commented in func processPersistentHistory() the code that purges the history, and as expected, the table was displayed correctly, also when I entered another entry in the simulator.

What are the conclusions?

• On both persistent stores (simulator & device), and in iCloud, all data were always correct.
• Merging of remote store changes to a context fails, if the mirroring software does not have enough time to process its entries in the persistent history.
• How long this takes depends probably on the amount of data that has to be synced. My experience is that some kb take some seconds, but this depends of course on many parameters. But if so, 7 days correspond to some GB to sync, which is rather unusual. In this respect, purging the persistent history after 7 days seems to be a good compromise between memory consumption and correct app operation.

Further hints to reproduce the tests (this may help others who try the same):

As suggested, I downloaded Apple's demo app and the core data stack modified by you.

It did compile for a simulator, but for the device I had to set 3 additional settings in the Signing & Capabilities tab of the target:

• Set the development team
• Set the bundle identifier to a reasonable value, e.g. com.<your company>.CoreDataCloudKitDemo.
• Select the right iCloud container, e.g. iCloud.com.<your company>.CoreDataCloudKitDemo.
• Additionally I had to ensure that the simulator and the device were logged in to the same iCloud account. Note that for the simulator, one has to re-log in about once a day. Mostly one is reminded to do so, but sometimes not.

Then, I could run the app on the simulator and the device.

I verified in the CloudKit Console that in the Private Database, zone com.apple.coredata.cloudkit.zone there are no records of type CD_Post. Since data are not shared, the iCloud Sharing database is not used.

---

Related Topics