Marshal ruby hash with default proc - remove the default proc?
Just reset the default:
h.default = nil
More explicitly:
def dumpable_hash(h)
return h unless h.default_proc
copy = h.clone
copy.default = nil # clear the default_proc
copy
end
In Ruby 2.0, you can also write h.default_proc = nil
if you prefer. Available for all Rubies with require 'backports/2.0.0/hash/default_proc'
.
Marshal can't dump hash with default proc (TypeError)
Ruby doesn't have a Marshal
format for code, only for data. You cannot marshal Proc
s or lambdas.
Your apps
hash has a default_proc
, because
hsh = Hash.new { some_block }
is more or less the same as
hsh = {}
hsh.default_proc = ->{ some_block }
IOW: your apps
hash contains code, and code cannot marshalled.
Marshal serializing a Hash with a block constructor
Remove the default behavior before dumping the hash.
myHash = Hash.new {|h, k| h[k] = []}
myHash[5] << 1
myHash.default = nil
Marshal.dump(myHash)
Or, since you seem to be interested in conserving vertical space:
myHash = Hash.new {|h, k| h[k] = []}
myHash[5] << 1
Marshal.dump(myHash.tap {|h| h.default = nil })
However, this permanently changes the hash so that it no longer has a default. If that bothers you, but temporarily duplicating the top level of your hash does not, throw in a .dup
:
Marshal.dump(myHash.dup.tap {|h| h.default = nil })
and now the default behavior of the original hash remains unchanged.
Rails.cache error in Rails 3.1 - TypeError: can't dump hash with default proc
This might be a little verbose but I had to spend some time with the Rails source code to learn how the caching internals work. Writing things down aids my understanding and I figure that sharing some notes on how things work can't hurt. Skip to the end if you're in a hurry.
Why It Happens
This is the offending method inside ActiveSupport:
def should_compress?(value, options)
if options[:compress] && value
unless value.is_a?(Numeric)
compress_threshold = options[:compress_threshold] || DEFAULT_COMPRESS_LIMIT
serialized_value = value.is_a?(String) ? value : Marshal.dump(value)
return true if serialized_value.size >= compress_threshold
end
end
false
end
Note the assignment to serialized_value
. If you poke around inside cache.rb
, you'll see that it uses Marshal to serialize objects to byte strings before they go into the cache and then Marshal again to deserialize objects. The compression issue isn't important here, the important thing is the use of Marshal.
The problem is that:
Some objects cannot be dumped: if the objects to be dumped include bindings, procedure or method objects, instances of class IO, or singleton objects, a TypeError will be raised.
Some things have state (such as OS file descriptors or blocks) that can't be serialized by Marshal. The error you're noting is this:
can't dump hash with default proc
So someone in your model has an instance variable that is a Hash and that Hash uses a block to supply default values. The column_methods_hash
method uses such a Hash and even caches the Hash inside @dynamic_methods_hash
; column_methods_hash
will be called (indirectly) by public methods such as respond_to?
and method_missing
.
One of respond_to?
or method_missing
will probably get called on every AR model instance sooner or later and calling either method makes your object unserializable. So, AR model instances are essentially unserializable in Rails 3.
Interestingly enough, the respond_to?
and method_missing
implementations in 2.3.8 are also backed by a Hash that uses a block for default values. The 2.3.8 cache is "[...]is meant for caching strings." so you were getting lucky with a backend that could handle whole objects or it used Marshal before your objects had hash-with-procs in them; or perhaps you were using the MemoryStore
cache backend and that's little more than a big Hash.
Using multiple scope-with-lambdas might end up storing Procs in your AR objects; I'd expect the lambdas to be stored with the class (or singleton class) rather than the objects but I didn't bother with an analysis as the problem with respond_to?
and method_missing
makes the scope
issue irrelevant.
What You Can Do About It
I think you've been storing the wrong things in your cache and getting lucky. You can either start using the Rails cache properly (i.e. store simple generated data rather than whole models) or you can implement the marshal_dump
/marshal_load
or _dump
/_load
methods as outlined in Marshal. Alternatively, you can use one of the MemoryStore backends and limit yourself to one distinct cache per server process.
Executive Summary
You can't depend on storing ActiveRecord model objects in the Rails cache unless you're prepared to handle the marshalling yourself or you want to limit yourself to the MemoryStore cache backends.
The exact source of the problem has changed in more recent versions of Rails but there are still many instances of default_proc
s associated with Hashes.
How can I marshal a hash with arrays?
s = Hash.new
s.default = Array.new
s[0] << "Tigger"
s[7] << "Ruth"
s[7] << "Puuh"
This code changes the default 3 times (which is probably what showed up in the dump), but it does not store anything in the hash. Try "puts s[8]", it will return [["Tigger"], ["Ruth"], ["Puuh"]].
A Hash.default_proc will do what you want
s = Hash.new{|hash,key| hash[key]=[] }
But you can't marshall a proc. This will work:
s = Hash.new
s.default = Array.new
s[0] += ["Tigger"]
s[7] += ["Ruth"]
s[7] += ["Puuh"]
This works because []+=["Tigger"] creates a new array.
An alternative, creating less arrays:
s = Hash.new
(s[0] ||= []) << "Tigger"
(s[7] ||= []) << "Ruth"
(s[7] ||= []) << "Puuh"
Only creates a new array when the key is absent (nil).
Rails - 'can't dump hash with default proc' during custom validation
Without a stack trace (does it lead anywhere, or does it just not appear?) it is difficult to know what exactly is happening, but here's how you can reproduce this error in a clean environment:
# initialize a new hash using a block, so it has a default proc
h = Hash.new {|h,k| h[k] = k }
# attempt to serialize it:
Marshal.dump(h)
#=> TypeError: can't dump hash with default proc
Ruby can't serialize procs, so it wouldn't be able to properly reconstitute that serialized hash, hence the error.
If you're reasonably sure that line is the source of your trouble, try refactoring it to see if that solves the problem.
def already_wants? want_name
wants.any? {|want| want_name == want.name }
end
or
def already_wants? want_name
wants.where(name: want_name).count > 0
end
Actual Hash getting modified when copy is modified in ruby
If want to copy one hash to another you can do it just like this. Then you can manipulate the copied hash or even do it in the loop. And then manipulate the copied hash it for your task. In here it copies the key-value pair for the hash,
@original = {0=>{0=>[0, 4, 5, 6], 2=>[3, 7], 1=>[1, 2]}, 1=>{0=>[0, 4, 5, 6], 2=>[1], 1=>[2, 3, 7]}, 2=>{0=>[0, 4, 6], 1=>[1, 2, 5], 2=>[3, 7]}, 3=>{0=>[0, 4], 2=>[1, 2, 3, 6, 7], 1=>[5]}, 4=>{0=>[4], 2=>[1, 5], 1=>[2, 3, 6, 7, 0]}, 5=>{1=>[0, 1, 2, 5], 2=>[3, 6, 7], 0=>[4]}, 6=>{1=>[0, 1, 2, 5, 4], 2=>[3, 6, 7], 0=>[]}}
copy = Hash.new
@original.each do |k, v|
copy[k] = v.dup
end
p copy #prints the copied hash
Why does dumping and loading a Hash using Marshal in Ruby throw a FormatError?
The combination of the 2 answers from @Josh and @derp work for me. Here is the code (written to a file):
hashtime = Hash.new(Time.mktime('1970'))
hashtime[1] = Time.now
File.open('timehash','wb') do |f|
f.write Marshal.dump(hashtime)
end
newhash = Marshal.load (File.binread('timehash'))
p newhash
p newhash.default
Results in the following output:
c:\apps\ruby>ruby h.rb
{1=>2011-10-05 08:09:43 +0200}
1970-01-01 00:00:00 +0100
Related Topics
Error Installing "Kgio-2.9.2" Gem on Windows
Controller Method #Show Getting Called
Store Image in Database Using Rails Paperclip Plugin
Rails Gem to Break a Paragraph into Series of Sentences
Using Ruby and Mechanize to Fill in a Remote Login Form Mystery
Can't Setup Ruby Environment - Installing Fii Gem Error
Bundle Uses Wrong Ruby Version
Markdown to Plain Text in Ruby
Should Gemfile.Lock Be Committed to Source Control on Windows
How to Define a Method in Ruby Using Splat and an Optional Hash at the Same Time
Upgrade Ruby on Elastic Beanstalk
How to Create All Possible Combinations with Two Arrays
How to Remove a Row from a CSV with Ruby
How Would I Go About Converting This Time String to Epoch Time in Ruby
Rails Not Rolling Back Transaction After Failed Save()
Rake Aborted! Undefined Method 'Migration_Error=' for Activerecord::Base:Class