Implementing Fast and Efficient Core Data Import on iOS 5

Implementing Fast and Efficient Core Data Import on iOS 5

You should probably save the master MOC in strides as well. No sense having that MOC wait until the end to save. It has its own thread, and it will help keep memory down as well.

You wrote:

Then at the end of the import process, I save on the master/parent
context which, ostensibly, pushes modifications out to the other child
contexts including the main context:

In your configuration, you have two children (the main MOC and the background MOC), both parented to the "master."

When you save on a child, it pushes the changes up into the parent. Other children of that MOC will see the data the next time they perform a fetch... they are not explicitly notified.

So, when BG saves, its data is pushed to MASTER. Note, however, that none of this data is on disk until MASTER saves. Furthermore, any new items will not get permanent IDs until the MASTER saves to disk.

In your scenario, you are pulling the data into the MAIN MOC by merging from the MASTER save during the DidSave notification.

That should work, so I'm curious as to where it is "hung." I will note, that you are not running on the main MOC thread in the canonical way (at least not for iOS 5).

Also, you probably only are interested in merging changes from the master MOC (though your registration looks like it is only for that anyway). If I were to use the update-on-did-save-notification, I'd do this...

- (void)contextChanged:(NSNotification*)notification {
    // Only interested in merging from master into main.
    if ([notification object] != masterManagedObjectContext) return;

    [mainManagedObjectContext performBlock:^{
        [mainManagedObjectContext mergeChangesFromContextDidSaveNotification:notification];

        // NOTE: our MOC should not be updated, but we need to reload the data as well
    }];
}

Now, for what may be your real issue regarding the hang... you show two different calls to save on the master. the first is well protected in its own performBlock, but the second is not (though you may be calling saveMasterContext in a performBlock...

However, I'd also change this code...

- (void)saveMasterContext {
    NSNotificationCenter *notificationCenter = [NSNotificationCenter defaultCenter];    
    [notificationCenter addObserver:self selector:@selector(contextChanged:) name:NSManagedObjectContextDidSaveNotification object:masterManagedObjectContext];

    // Make sure the master runs in it's own thread...
    [masterManagedObjectContext performBlock:^{
        NSError *error = nil;
        BOOL saveSuccess = [masterManagedObjectContext save:&error];
        // Handle error...
        [notificationCenter removeObserver:self name:NSManagedObjectContextDidSaveNotification object:masterManagedObjectContext];
    }];
}

However, note that the MAIN is a child of MASTER. So, it should not have to merge the changes. Instead, just watch for the DidSave on the master, and just refetch! The data is sitting in your parent already, just waiting for you to ask for it. That's one of the benefits of having the data in the parent in the first place.

Another alternative to consider (and I'd be interested to hear about your results -- that's a lot of data)...

Instead of making the background MOC a child of the MASTER, make it a child of the MAIN.

Get this. Every time the BG saves, it automatically gets pushed into the MAIN. Now, the MAIN has to call save, and then the master has to call save, but all those are doing is moving pointers... until the master saves to disk.

The beauty of that method is that the data goes from the background MOC straight into your applications MOC (then passes through to get saved).

There is some penalty for the pass-through, but all the heavy lifting gets done in the MASTER when it hits the disk. And if you kick those saves on the master with performBlock, then main thread just sends off the request, and returns immediately.

Please let me know how it goes!

How to improve large data import performance with core data in ios

I will raise the same issue every Core Data question gets, and then move on: Make sure that you really want to use Core Data for this and not just sqlite directly. Core Data is for persisting object graphs, not a general database. If you've already implemented a database for Android, then you may want to use the same schema and design on iOS.

OK, got that out of the way. Let's assume CD is really the best tool for the job here (or you can't change it at this point). My first thought here is to cheat. Granted, that is often one of my early thoughts....

See how fast you can insert these objects with no relationships. If that's fast enough, then here's how you cheat: don't store actual relationships at first. Store a string list of identifiers that describe the relationship. Then, once everything's loaded and the user can start working, over time convert the string relationships into real relationships in the background. Whenever you fetch a record, you need to check if it has a cheater-property still set and if so, you'll need to fetch its relationships by hand (and then clear the cheater-property).

This doesn't make the full import faster, but it gives the illusion that it's faster, and that's the goal in iOS 90% of the time. You might even have to prevent certain operations (like delete) until you've finished gluing everything, but that's probably still better than blocking the user entirely.

How to add many objects into core data?

Proceed in two stages.

1. In stage one, create the core data database and store the 5000 objects in it. Now remove that code (or arrange so that you never run it again), and copy that database into your app bundle.

2. So in stage two, your app launches, copies the database out to the documents folder (if it isn't there already), accesses it, and uses it.

Thus, when the end user actually comes to use your app, there's the database all prepared and ready to use instantly, because the user experiences only stage two.

Core Data background importing using UIManagedDocument and parent/child context in iOS 5

I'm still in the thick of all this kind of stuff (Core Data & UIManagedDocument working together) but I think this question might address your situation: Core Data managed object does not see related objects until restart Simulator

It involves forcing temporary ids to be permanent before the "normal" flow using:
[context obtainPermanentIDsForObjects:[inserts allObjects] error:&error]

Core Data Performance issue while Saving

1. You should do the import on a non-UI thread with a context bound directly to the persistent store coordinator, not a child context of the main contex
2. You should invoke [managedObjectContext save] once in every several hundreds of new objects inserted, depending on the object size and graph complexity. See this answer for details
3. You should wrap your batch from step 2 in an @autoreleasepool block and reset the context after save before the autorelease block is exited. See this answer

Core Data Import - Not releasing memory

Ok this is quite embarrassing... Zombies were enabled on the Scheme, on the Arguments they were turned off but on Diagnostics "Enable Zombie Objects" was checked...

Turning this off maintains the memory stable.

Thanks for the ones that read trough the question and tried to solve it!

Implementing CoreData using the parent/child approach with temporary objects

In a similar project I used this solution which was also favored by the users:

Keep a time stamp attribute in the downloaded items and delete them when the time stamp is older than a certain threshold and they are not marked as favorite.

---

Related Topics