How to Read from Redis Inside a Multi Block in Ruby

Getting multiple key values from Redis

Doing a loop on the items and synchronously accessing each element is not very efficient. With Redis 2.4, there are various ways to do what you want:

by using the sort command
by using pipelining
by using variadic parameter commands

With Redis 2.6, you can also use Lua scripting, but this is not really required here.

By the way, the data structure you described could be improved by using hashes. Instead of storing user data in separate keys, you could group them in a hash object.

Using the sort command

You can use the Redis sort command to retrieve the data in one roundtrip.

redis> set users:1:name "daniel"
OK
redis> set users:1:age 24
OK
redis> set users:2:name "user2"
OK
redis> set users:2:age 24
OK
redis> sadd events:1:attendees users:1 users:2
(integer) 2
redis> sort events:1:attendees by nosort get *:name get *:age
1) "user2"
2) "24"
3) "daniel"
4) "24"

Using pipelining

The Ruby client support pipelining (i.e. the capability to send several queries to Redis and wait for several replies).

keys = $redis.smembers("events:1:attendees")
res = $redis.pipelined do
   keys.each do |x|
      $redis.mget(x+":name",x+":age")
   end
end

The above code will retrieve the data in two roundtrips only.

Using variadic parameter command

The MGET command can be used to retrieve several data in one shot:

redis> smembers events:1:attendees
1) "users:2"
2) "users:1"
redis> mget users:1:name users:1:age users:2:name users:2:age
1) "daniel"
2) "24"
3) "user2"
4) "24"

The cost here is also two roundtrips. This works if you can guarantee that the number of keys to retrieve is limited. If not, pipelining is a much better solution.

What is the best way to use Redis in a Multi-threaded Rails environment? (Puma / Sidekiq)

You use a separate global connection pool for your application code. Put something like this in your redis.rb initializer:

require 'connection_pool'
REDIS = ConnectionPool.new(size: 10) { Redis.new }

Now in your application code anywhere, you can do this:

REDIS.with do |conn|
  # some redis operations
end

You'll have up to 10 connections to share amongst your puma/sidekiq workers. This will lead to better performance since, as you correctly note, you won't have all the threads fighting over a single Redis connection.

All of this is documented here: https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling

should I check error each step on a redis multi transaction?

To transact, you need to Send() each command, and only Do() the EXEC. Error checking should be done for the Do() only, like so:

conn.Send("MULTI")
conn.Send("SET", "foo", "bar")
...
reply, err := conn.Do("EXEC")
if err != nil {
    ...
}
...

Redis gem watch and unwatch confusion

I am not a Ruby nor Redis expert.

Though by reading the Redis documentation about transaction here https://redis.io/topics/transactions we learn that a watched key is unwatched in two ways : either after EXEC is used. (then the transactions completes) or either by UNWATCH.

So there is indeed a reason to use UNWATCH. If the multi block is not called then EXEC is not called either. Then the key is never flushed.

Also I think the conditionnal bit containing redis.unwatch can indeed happen to be called. As redis.get("key") == "some value" seems to be purely arbitrary and basically a code design.
It could well be redis.get("key") > "some value" for example.

(Also it seem having a block passed to watched does not UNWATCH the key automatically as per documentation, as the block is optionnal)

For your second question you are correct, you cannot unwatch a specific key it seems. Though redis.unwatch seems to work fine.

This does not seem to be a Ruby design. Watching a key is only useful in order to process an atomic transaction, there is no point in leaving watched keys behind us when transaction is either successful or aborted.

I don't see any example of using UNWATCH in the Redis documentation https://redis.io/topics/transactions but it feels natural to flush every key after the transaction block.

Redis Future + Rails

Future objects are typically returned when methods are called in a pipeline or a transaction.

The returned value is only available when the EXEC command has been applied to the Redis server. With redis-rb, it means you should exit a pipelined or multi block before.

If you want to select/read data, do it before the multi/exec block, and do only write in the multi/exec block.

By the way, it will be more efficient to use $redis.sunion() to generate the result on server-side.

Retrieving/Listing all key/value pairs in a Redis db

You can explore the Redis dataset using the redis-cli tool included in the Redis distribution.

Just start the tool without arguments, then type commands to explore the dataset.

For instance KEYS will list all the keys matching a glob-style pattern, for instance with: keys * you'll see all the keys available.

Then you can use the TYPE command to check what type is a given key, if it's a list you can retrieve the elements inside using LRANGE mykey 0 -1. If it is a Set you'll use instead SMEMBERS mykey and so forth. Check the Redis documentation for a list of all the available commands and how they work.