Servicestack.Net Redis: Storing Related Objects VS. Related Object Ids

ServiceStack.Net Redis: Storing Related Objects vs. Related Object Ids

Rather than re-hash a lot of other documentation that's out there in the wild, I'll list a couple around for some background info around Redis + ServiceStack's Redis Client:

• What to think about when designing a NoSQL Redis application
• Designing a NoSQL Database using Redis
• General Overview of Redis and .NET
• Schemaless versioning and Data Migrations with C# Redis Client

There is no magic - Redis is a blank canvas

First I want to point out that using Redis as a data store just provides a blank canvas and doesn't have any concept of related entities by itself. i.e. it just provides access to distributed comp-sci data structures. How relationships get stored is ultimately up to the client driver (i.e. ServiceStack C# Redis Client) or the app developer, by using Redis's primitive data structure operations. Since all the major data structures are implemented in Redis, you basically have complete freedom on how you want to structure and store your data.

Think how you would structure relationships in code

So the best way to think about how to store stuff in Redis, is to completely disregard about how data is stored in an RDBMS table and think about how it is stored in your code, i.e. using the built-in C# collection classes in memory - which Redis mirrors in behavior with their server-side data-structures.

Despite not having a concept of related entities, Redis's built-in Set and SortedSet data structures provide the ideal way to store indexes. E.g. Redis's Set collection only stores a max of 1 occurrence of an element. This means you can safely add items/keys/ids to it and not care if the item exists already as the end result will be the same had you called it 1 or 100 times - i.e. it's idempotent, and ultimately only 1 element remains stored in the Set. So a common use-case is when storing an object graph (aggregate root) is to store the Child Entity Ids (aka Foreign Keys) into a Set every time you save the model.

Visualizing your data

For a good visualization of how Entities are stored in Redis I recommend installing the Redis Admin UI which works well with ServiceStack's C# Redis Client as it uses the key naming convention below to provide a nice hierarchical view, grouping your typed entities together (despite all keys existing in the same global keyspace).

To view and edit an Entity, click on the Edit link to see and modify the selected entity's internal JSON representation. Hopefully you'll be able to make better decisions about how to design your models once you can see how they're stored.

How POCO / Entities are stored

The C# Redis Client works with any POCOs that have a single primary key - which by default is expected to be Id (though this convention overridable with ModelConfig).
Essentially POCOs gets stored into Redis as serialized JSON with both the typeof(Poco).Name and the Id used to form a unique key for that instance. E.g:

urn:Poco:{Id} => '{"Id":1,"Foo":"Bar"}'

POCOs in the C# Client are conventionally serialized using ServiceStack's fast Json Serializer where only properties with public getters are serialized (and public setters to get de-serialized back).

Defaults are overrideable with [DataMember] attrs but not recommended since it uglifies your POCOs.

Entities are blobbed

So knowing that POCOs in Redis are just blobbed, you only want to keep non-aggregate root data on your POCOs as public properties (unless you purposely want to store redundant data). A good convention is to use methods to fetch the related data (since it wont get serialized) but also tells your app which methods make remote calls to read data.

So the question on whether the Feed should get stored with the User is whether or not it's non-aggregate root data, i.e. whether or not you want to access the users feeds outside the context of the user? If no, then leave the List<Feed> Feeds property on the User type.

Maintaining Custom Indexes

If however you would like to keep all feeds accessible independently, i.e. with redisFeeds.GetById(1) then you will want to store it outside of the user and maintain an index linking the 2 entities.

As you've noticed there are many ways to store relationships between entities and how you do so is largely a matter of preference. For the child entity in a parent>child relationship you would always want to store the ParentId with the child entity. For the Parent you can either choose to store a collection of ChildIds with the model and then do a single fetch for all child entities to re-hydrate the model.

Another way is to maintain the index outside of the parent dto in its own Set for each parent instance. Some good examples of this is in the C# Source code of the Redis StackOverflow demo where the relationship of Users > Questions and Users > Answers is stored in:

idx:user>q:{UserId} => [{QuestionId1},{QuestionId2},etc]
idx:user>a:{UserId} => [{AnswerId1},{AnswerId2},etc]

Although the C# RedisClient does include support for a default Parent/Child convention via its TParent.StoreRelatedEntities(), TParent.GetRelatedEntities<TChild>() and TParent.DeleteRelatedEntities() APIs where an index is maintained behind the scene that looks like:

ref:Question/Answer:{QuestionId} => [{answerIds},..]

Effectively these are just some of your possible options, where there are many different ways to achieve the same end and in which you also have the freedom to roll your own.

NoSQL's schema-less, loose-typing freedoms should be embraced and you shouldn't be worried about trying to follow a rigid, pre-defined structure you might be familiar with when using an RDBMS.

In conclusion, there's no real right way to store data in Redis, e.g. The C# Redis Client makes some assumptions in order to provide a high-level API around POCOs and it blobs the POCOs in Redis's binary-safe string values - though there are other clients will prefer to store an entities properties in Redis Hashes (Dictionaries) instead. Both will work.

AddItemToSet vs StoreRelatedEntities

The AddItemToSet API in ServiceStack.Redis is a 1:1 mapping that calls Redis' Server SADD Operation, i.e. adds an item to a Redis SET.

The StoreRelatedEntities is a higher-level operation that also maintains an index containing relationship between the entities described in detail in this Storing Related Entities in Redis answer.

ServiceStack.Redis: Query a subset of objects by object properties stored using redisClient.StoreAll()

It's not clear what you mean by querying since data in Redis is typically accessed by key and Redis doesn't have any explicit support for querying values which are effectively opaque to redis.

I recommend reading this previous answer on how you can store related objects in Redis using the ServiceStack.Redis client. Which shows how you can use Indexes to create relationships between types.

If you just want to search through keys you can use Redis Scan APIs, e.g:

var userKeyPattern = IdUtils.CreateUrn<User>("*"); //= urn:User:*
var scanUsers = Redis.ScanAllKeys(userKeyPattern);
//Stop after retrieving 10000 user keys 
var sampleUsers = scanUsers.Take(10000).ToList(); 

But you can't do custom adhoc server-side querying of Redis Values unless you create a custom LUA script to parse the JSON value payload. You would need to create custom indexes for all the relationships you want to maintain otherwise you will need to fetch the results on the client and query them in memory.

How does ServiceStack Redis function in retrieving data

The efficiency is less network calls vs more data. Data in Redis just gets blobbed, most of the time a single API call maps 1:1 with a redis server operation. Which means you can think about the perf implications as simply downloading a json dataset blob from a remote server's memory and deserializing it on the client - which is effectively all that happens.

In some APIs such as GetAll() it requires 2 calls, 1 to fetch all the ids in the Entity set, and the other to fetch all the records with those ids. The source code of the Redis Client is quite approachable so I recommend having a look to see exactly what's happening.

Because you've only got 3 categories, it's not that much extra data you're saving by trying to filter on the server.

So your options are basically:

• Download the entire entity dataset and filter on the client
• Maintain a custom index mapping from Category > Ids
• More Advanced: Use a server-side LUA operation to apply server side filtering (requires Redis 2.6)

Retrieve selection of servicestack redis session objects based on values of properties

Instead of SearchKeys you want to be using the newer Redis Scan API's which let you iterate over keys in a cursor.

Redis values are blobbed and not indexed so there's not much opportunity to optimize this. The most efficient way would be to inspect the values using a custom server-side LUA operation which will minimize the number of requests and payload sent to the client.

Redis embedded version of LUA has cjson which can be used for deserializing JSON values, the Lua guide for Redis users has some examples of using this.

Since Redis doesn't support server-side querying or indexes, the Redis way would be to premept the queries you need to query on and maintain custom indexes whenever a Session is saved, some info on how to maintain custom indexes in Redis are at:

• Storing Related Entities and Maintaining Custom Indexes
• Designing a NoSQL Database using Redis

In ServiceStack you can override OnSaveSession() in your AppHost which gets called whenever a Session is saved.

Redis key partitioning practices with linked items

The best practice in redis is to maintain an index of the relationship you want to query.

Manually maintaining an index in Redis

An index is just a redis SET containing the related Ids you want to maintain, given that you want to "retrieve all posts related to one group" I would maintain the following index:

const string GroupPostIndex = "idx:group>post:{0}";

So that everytime you store a post, you also want to update the index, e.g:

client.Store(post);
client.AddItemToSet(GroupPostIndex.Fmt(groupId), post.Id);

Note: Redis SET operations are idempotent in that adding an item/id multiple times to a SET will always result in there being only one occurrence of that item in the SET, so its always safe to add an item to the set whenever storing a POST without needing to check if it already exists.

Now when I want to retrieve all posts in a group I just need to get all the ids from the SET with:

var postIds = client.GetAllItemsFromSet(GroupPostIndex.Fmt(groupId));

Then fetch all the posts with those ids:

var posts = redis.As<Post>().GetByIds(postIds);

Using ServiceStack.Redis Related Entities API's

The above shows what's required to maintain an index in Redis yourself, but as this is a common use-case, ServiceStack.Redis also offers a high-level typed API that you can use instead.

Which lets you store related entities with:

client.As<Group>().StoreRelatedEntities(groupId, post);

Note: this also takes care of storing the Post

and retrieve them with:

var posts = client.As<Group>().GetRelatedEntities<Post>(groupId);

It also offers other convenience API's like quickly finding out how many posts there are within a given group:

var postsCount = client.As<Group>().GetRelatedEntitiesCount<Post>(groupId);

and deleting either 1 or all entities in the group:

client.As<Group>().DeleteRelatedEntity<Post>(groupId, postId);
client.As<Group>().DeleteRelatedEntities<Post>(groupId); //all group posts

ServiceStack Entities Id field name

Yes you can override the default using ModelConfig with:

ModelConfig<User>.Id(x => x.UserId);
ModelConfig<Item>.Id(x => x.ItemId);

This needs to be configured once on startup before you use the RedisClient.

---

Related Topics