Why Are Metaclasses Created in Ruby

why are metaclasses created in ruby?

Just to be super duper clear.

Here is a quick ruby script that explains the question:

#!/usr/bin/env ruby
puts ObjectSpace.count_objects[:T_CLASS] #>> 471
class X
  def self.foo
  end
  def bar
  end
end
puts ObjectSpace.count_objects[:T_CLASS] #>> 473

This is what the OP meant by "ObjectSpace.count_objects[:T_CLASS] increments the count by 2." Let's call the extra class the singleton class of X, because that appears to be what Ruby calls it internally.

irb> X
=> X
irb> X.singleton_class
=> <Class: X>

Notice that the #foo method is an instance method of X.singleton_class, not X.

irb> X.instance_methods(false)
=> [:baz]
irb> X.singleton_class.instance_methods(false)
=> [:foo]

So why is :foo stored in X.singleton_class instead of X? Isn't there only ever going to be one X?

I believe the main reason is for consistency. Consider the following, simpler scenario concerning plain instance objects.

car = Car.new
def car.go_forth_and_conquer
end

As @mikej explained superbly, this new method is stored in car's singleton class.

irb> car.singleton_class.instance_methods(false)
=> [:go_forth_and_conquer]

Classes are Objects

Now, classes are objects too. Each class is an instance of Class. Thus, when a class (say, X) is defined, ruby is really creating an instance of Class, and then adding methods to the instance (similar to what we did to car above.) For example, here is an alternative way to create a new class

Car = Class.new do
  def go_forth_and_conquer
    puts "vroom"
  end
end
Car.new.go_forth_and_conquer

Therefore, it is much easier to just reuse the code and do it the same way (i.e. keep foo in X.singleton_class.) This probably requires less effort and will lead to fewer surprises, so no one will ever need to write code to handle Class instances differently from other instances.

Probably Doesn't Matter

You might be thinking that if Ruby did not have singleton classes for instances of Class, there could be some memory savings. However, it sounds to me that where bar is actually stored is an implementation detail that we probably shouldn't count on. Rubinius, MRI, and JRuby could all store methods and instances differently, as long as the behavior is consistent. For all we know, there could be a reasonable implementation of Ruby that doesn't eagerly create singleton classes for class objects, for the exact same reasons you outlined, as long as the overall behavior conforms to the ruby spec. (E.g. an actual singleton class doesn't exist until the #singleton_class method is first invoked.)

What is Ruby's analog to Python Metaclasses?

Ruby doesn't have metaclasses. There are some constructs in Ruby which some people sometimes wrongly call metaclasses but they aren't (which is a source of endless confusion).

However, there's a lot of ways to achieve the same results in Ruby that you would do with metaclasses. But without telling us what exactly you want to do, there's no telling what those mechanisms might be.

In short:

Ruby doesn't have metaclasses
Ruby doesn't have any one construct that corresponds to Python's metaclasses
Everything that Python can do with metaclasses can also be done in Ruby
But there is no single construct, you will use different constructs depending on what exactly you want to do
Any one of those constructs probably has other features as well that do not correspond to metaclasses (although they probably correspond to something else in Python)
While you can do anything in Ruby that you can do with metaclasses in Python, it might not necessarily be straightforward
Although often there will be a more Rubyish solution that is elegant
Last but not least: while you can do anything in Ruby that you can do with metaclasses in Python, doing it might not necessarily be The Ruby Way

So, what are metaclasses exactly? Well, they are classes of classes. So, let's take a step back: what are classes exactly?

Classes …

are factories for objects
define the behavior of objects
define on a metaphysical level what it means to be an instance of the class

For example, the Array class produces array objects, defines the behavior of arrays and defines what "array-ness" means.

Back to metaclasses.

Metaclasses …

are factories for classes
define the behavior of classes
define on a metaphysical level what it means to be a class

In Ruby, those three responsibilities are split across three different places:

the Class class creates classes and defines a little bit of the behavior
the individual class's eigenclass defines a little bit of the behavior of the class
the concept of "classness" is hardwired into the interpreter, which also implements the bulk of the behavior (for example, you cannot inherit from Class to create a new kind of class that looks up methods differently, or something like that – the method lookup algorithm is hardwired into the interpreter)

So, those three things together play the role of metaclasses, but neither one of those is a metaclass (each one only implements a small part of what a metaclass does), nor is the sum of those the metaclass (because they do much more than that).

Unfortunately, some people call eigenclasses of classes metaclasses. (Until recently, I was one of those misguided souls, until I finally saw the light.) Other people call all eigenclasses metaclasses. (Unfortunately, one of those people is the author of one the most popular tutorials on Ruby metaprogramming and the Ruby object model.) Some popular libraries add a metaclass method to Object that returns the object's eigenclass (e.g. ActiveSupport, Facets, metaid). Some people call all virtual classes (i.e. eigenclasses and include classes) metaclasses. Some people call Class the metaclass. Even within the Ruby source code itself, the word "metaclass" is used to refer to things that are not metaclasses.

Ruby metaclasses: why three when defined singleton methods?

Looking at MRI code, every time you create a Class which in Ruby is object of type Class, automatically, ruby creates "metaclass" class for that new class, which is another Class object of singleton type.

The C function calls (class.c) are:

rb_define_class
  rb_define_class_id
    rb_class_new(super);
    rb_make_metaclass(klass, RBASIC(super)->klass);

So, every time that you define a new class, Ruby will define another class with meta information.

When you define a class method, I mean, def self.method, internally, ruby calls rb_define_singleton_method. You can check it doing the follow step:

Create a ruby file test.rb:

class A
  def self.foo
  end
end

And run the following command:

ruby --dump insns test.rb

You will have the following output:

== disasm: <RubyVM::InstructionSequence:<main>@kcount.rb>===============
0000 trace            1                                               (  70)
0002 putspecialobject 3
0004 putnil
0005 defineclass      :A, <class:A>, 0
0009 leave
== disasm: <RubyVM::InstructionSequence:<class:A>@kcount.rb>============
0000 trace            2                                               (  70)
0002 trace            1                                               (  71)
0004 putspecialobject 1
0006 putself
0007 putobject        :foo
0009 putiseq          foo
0011 opt_send_simple  <callinfo!mid:core#define_singleton_method, argc:3, ARGS_SKIP>
0013 trace            4                                               (  73)
0015 leave                                                            (  71)
== disasm: <RubyVM::InstructionSequence:foo@kcount.rb>==================
0000 trace            8                                               (  71)
0002 putnil
0003 trace            16                                              (  72)
0005 leave

The define_singleton_method is mapped to the rb_obj_define_method C function (object.c), which do following calls:

 rb_obj_define_method
   rb_singleton_class(obj)
   rb_mod_define_method

The function rb_singleton_class exposes the metaclass created when the class was defined, but it also creates a new metaclass for this metaclass.

According the Ruby documentation for this function: "if a obj is a class, the returned singleton class also has its own singleton class in order to keep consistency of the inheritance structure of metaclasses".

That is the reason why the number of class increases by 1 when you define a class method.

The same effect happens if you change your code by:

class A
end
A.singleton_class

The singleton_class is mapped to rb_obj_singleton_class C function, which calls the rb_singleton_class.

Even if you create a class method and call the singleton_class method, the number of created classes will not change, because all classes necessary to handle meta information is already created. Example:

class A
  def self.foo
    nil
  end
end

A.singleton_class

The code above will keep returning 3.

Why does ruby create 3 objects after a class is created?

From https://bugs.ruby-lang.org/issues/16788:

Creating a class automatically creates a singleton class (which is not accessible to the user). Referencing the singleton class of a class automatically creates a singleton class of that singleton class. This is to keep consistency of the inheritance structure of metaclasses. Otherwise, class methods wouldn't inherit from the superclass's metaclass, which is necessary as the superclass's class methods should be available as the subclass's class methods.

Modifying the question code a bit:

$old_classes = []
def print_objects
  new_classes = []
  ObjectSpace.each_object(Class){|x| new_classes << x}
  puts "New classes: #{new_classes - $old_classes}" unless $old_classes.empty?
  puts "Counts: #{ ObjectSpace.count_objects[:T_CLASS] }"
  $old_classes = new_classes
end

print_objects

class Test
end
puts 'Test class created'
print_objects

class Test
  def self.foo
  end 
end
puts 'Test singleton class referenced'
print_objects

I get the following results:

Counts: 690
Test class created
New classes: [Test]
Counts: 692
Test singleton class referenced
New classes: [#<Class:Test>]
Counts: 693

I tried it with Ruby 2.6 and 2.0 both inside and outside a console (the numbers differ but the difference is the same) and @SajibHassan with 1.9.3 (version in which the method count_objects was introduced).
This means that the difference has always been 3 and that the first singleton class created is not accessible for the user.

The book Ruby Under a Microscope (written in 2012 after the release of Ruby 2.1) also describes the creation of only two metaclasses, which doesn't match the result we get.

Note that methods like Module#prepend (introduced in Ruby 2.0), which was mentioned by @JörgWMittag in the comments as the possible reason for this extra class, use T_ICLASS. Check the commit in which the method was introduced for details. I guess that T_ICLASS stands for internal class and consequently internal classes shouldn't be visible to the user (which makes sense). I am not sure though why some T_CLASS are accessible to the user and some others are not.

Ruby metaclass confusion

how do this work

Easy: it doesn't. Not in Ruby, anyway.

Just like in most other languages, there are some core entities that are simply assumed to exist. They fall from the sky, materialize out of thin air, magically appear.

In Ruby, some of those magic things are:

Object doesn't have a superclass, but you cannot define a class with no superclass, the implicit direct superclass is always Object. [Note: there may be implementation-defined superclasses of Object, but eventually, there will be one which doesn't have a superclass.]
Object is an instance of Class, which is a subclass of Object (which means that indirectly Object is an instance of Object itself)
Class is a subclass of Module, which is an instance of Class
Class is an instance of Class

None of these things can be explained in Ruby.

BasicObject, Object, Module and Class all need to spring into existence at the same time because they have circular dependencies.

Just because this relationship cannot be expressed in Ruby code, doesn't mean the Ruby Language Specification can't say it has to be so. It's up to the implementor to figure out a way to do this. After all, the Ruby implementation has a level of access to the objects that you as a programmer don't have.

For example, the Ruby implementation could first create BasicObject, setting both its superclass pointer and its class pointer to null.

Then, it creates Object, setting its superclass pointer to BasicObject and its class pointer to null.

Next, it creates Module, setting its superclass pointer to Object and its class pointer to null.

Lastly, it creates Class, setting its superclass pointer to Module and its class pointer to null.

Now, we can overwrite BasicObject's, Object's, Module's, and Class's class pointer to point to Class, and we're done.

This is easy to do from outside the system, it just looks weird from the inside.

Once they do exist, however, it is perfectly possible to implement most of their behavior in plain Ruby. You only need very barebones versions of those classes, thanks to Ruby's open classes, you can add any missing functionality at a later time.

In your example, the class Class is not creating a new class named Class, it is reopening the existing class Class, which was given to us by the runtime environment.

So, it is perfectly possible to explain the default behavior of Class#new in plain Ruby:

class Class
  def new(*args, &block)
    obj = allocate # another magic thing that cannot be explained in Ruby
    obj.initialize(*args, &block)
    return obj
  end
end

[Note: actually, initialize is private, so you need to use obj.send(:initialize, *args, &block) to circumvent the access restriction.]

BTW: Class#allocate is another one of those magic things. It allocates a new empty object in Ruby's object space, which is something that cannot be done in Ruby. So, Class#allocate is something that has to be provided by the runtime system as well.

class self idiom in Ruby

First, the class << foo syntax opens up foo's singleton class (eigenclass). This allows you to specialise the behaviour of methods called on that specific object.

a = 'foo'
class << a
  def inspect
    '"bar"'
  end
end
a.inspect   # => "bar"

a = 'foo'   # new object, new singleton class
a.inspect   # => "foo"

Now, to answer the question: class << self opens up self's singleton class, so that methods can be redefined for the current self object (which inside a class or module body is the class or module itself). Usually, this is used to define class/module ("static") methods:

class String
  class << self
    def value_of obj
      obj.to_s
    end
  end
end

String.value_of 42   # => "42"

This can also be written as a shorthand:

class String
  def self.value_of obj
    obj.to_s
  end
end

Or even shorter:

def String.value_of obj
  obj.to_s
end

When inside a function definition, self refers to the object the function is being called with. In this case, class << self opens the singleton class for that object; one use of that is to implement a poor man's state machine:

class StateMachineExample
  def process obj
    process_hook obj
  end

private
  def process_state_1 obj
    # ...
    class << self
      alias process_hook process_state_2
    end
  end

  def process_state_2 obj
    # ...
    class << self
      alias process_hook process_state_1
    end
  end

  # Set up initial state
  alias process_hook process_state_1
end

So, in the example above, each instance of StateMachineExample has process_hook aliased to process_state_1, but note how in the latter, it can redefine process_hook (for self only, not affecting other StateMachineExample instances) to process_state_2. So, each time a caller calls the process method (which calls the redefinable process_hook), the behaviour changes depending on what state it's in.

In Ruby, are the terms metaclass, eigenclass, and singleton class completely synonymous and fungible?

tl;dr: Yes.

In the past, nobody knew what to call them, so everybody called them something else. Here is just a small sample of the names different authors have used or proposed over time:

singleton class
eigenclass
metaclass
ghost class
own class
virtual class
shadow class
myclass
selfclass
overclass
underclass
anchorclass
embedded class
intrinsic class
innate class
nameless class
unit class
atom class
singular class
singularity
bongo class
inner class

Originally, matz didn't want to choose a name, rather he wanted the community to settle on one. Unfortunately, it didn't. When matz and David Flanagan wrote The Ruby Programming Language, they had to choose, and they chose eigenclass (but singleton method). Eigenclass was also used in the first drafts for the ISO Ruby Language Specification (but mainly because it was very easy to run a search&replace on that name once an "official" one had been found).

However, in the later drafts and the final version of the ISO Ruby Language Specification, singleton class is used. That name was also chosen, when the Object#singleton_class accessor method was finally formally introduced in Ruby 1.9.2 and Module#singleton_class? was introduced in Ruby 2.1.

Metaclass was the term used by _why the lucky stiff in Why's (Poignant) Guide to Ruby (and other writings and libraries), which was highly influential on a whole generation of non-Japanese-speaking Ruby developers, and for a long time the best introduction to Ruby metaprogramming (and the only English one).

Nowadays, the terms singleton class and eigenclass seem to predominantly used (with singleton class taking over, thanks to the method names in the core library), with the occasional mention of metaclass.

Personally, I never liked metaclass, because it actually already has a different meaning: the metaclass is the class of a class. In Ruby, classes have two classes that define their behavior: their singleton class and the Class class, either one of which could be considered their metaclass. (Although neither of those classes can do what metaclasses in other languages can do, e.g. change the rules the method lookup or inheritance.)

Virtual class is even more problematic, because not only does it have a different meaning in programming in general (a nested class that can be overridden in a subclass, as in e.g. Beta or Newspeak, and proposed but abandoned for Scala), it even already has a third different meaning within the Ruby community: virtual class is the name given inside the MRI and YARV source code to singleton classes and include classes.

Inner class also already has a meaning as synonym to nested class (a class which is a member of an outer class, just like a method).

Personally, I always liked eigenclass, in symmetry to already established technical terms such as eigenvalue, eigenvector, eigenspace, eigenfunction, eigenface, eigenwave, eigenplane, eigenstate, eigenproblem etc. However, I have now adopted the term singleton class, simply because it's easier to talk about the singleton_class method as returning the singleton class.

What is the relationship between the metaclass of Base and Derived class in Ruby?

There are four class objects in play here:

<Class>---class---><Class>
Base               #Base
   ^                  ^
   |                  |
   |                  |
 super              super
   |                  |
   |                  |
<Class>            <Class>
Derived---class--->#Derived

Nomenclature:

<...> is the class of each object.
The name of the class is on the second line.
If the name starts with #, it's the eigenclass (aka singleton class).
super points to a class's superclass
class points to the class's class.

When you call Derived.class_method, Ruby follows the "right one and then up" rule: First go to the object's class, then follow the superclass chain up, stopping when the method is found:

The receiver of the "class_method" call is Derived. So follow the chain right to Derived's class object, which is its eigenclass (#Derived).
Derived does not define the method, so Ruby follows the chain up the chain to #Derived's superclass, which is #Base.
The method is found there, so Ruby dispatches the message to #Base.class_method

You don't think I knew all this stuff off the top of my head, did you? Here's where my brain got all this meta juju: Metaprogramming Ruby.

Part 2. How to make an "eigenclass" (aka "singleton class") come out of hiding

class Object
  def eigenclass
    class << self
      self
    end
  end
end

This method will return the eigenclass of any object. Now, what about classes? Those are objects, too.

p Derived.eigenclass               # => #<Class:Derived>
p Derived.eigenclass.superclass    # => #<Class:Base>
p Base.eigenclass                  # => #<Class:Base>

Note: Above is from Ruby1.9. When run under Ruby 1.8, you get a surprise:

p Derived.eigenclass.superclass    # => #<Class:Class>

Ruby metaclasses class hierarchy

I’ve adapted the inheritance extension from the post linked above to Ruby 2.0:

inheritance.c

#include "ruby.h"

VALUE real_super(VALUE self)
{
  return RCLASS_SUPER(RBASIC(self)->klass);
}

VALUE real_klass(VALUE self)
{
  return RBASIC(self)->klass;
}

void Init_inheritance()
{
  rb_define_method(rb_cClass,"real_super",real_super,0);
  rb_define_method(rb_cClass,"real_klass",real_klass,0);
}

test.rb

require_relative 'inheritance'

puts "Object real class: #{Object.real_klass}"
puts "Object real superclass: #{Object.real_super}"

puts "Class real class: #{Class.real_klass}"
puts "Class real superclass: #{Class.real_super}"

puts "Class metaclass real class: #{Class.singleton_class.real_klass}"
puts "Class metaclass real superclass: #{Class.singleton_class.real_super}"

puts "Object metaclass real class: #{Object.singleton_class.real_klass}"
puts "Object metaclass real superclass: #{Object.singleton_class.real_super}"

puts "An object singleton class real class: #{Object.new.singleton_class.real_klass}"
puts "An object singleton class real superclass: #{Object.new.singleton_class.real_super}"

The output being:

Object real class: #<Class:Object>
Object real superclass: #<Class:BasicObject>
Class real class: #<Class:Class>
Class real superclass: #<Class:Module>
Class metaclass real class: #<Class:#<Class:Class>>
Class metaclass real superclass: #<Class:#<Class:Module>>
Object metaclass real class: #<Class:#<Class:Object>>
Object metaclass real superclass: #<Class:#<Class:BasicObject>>
An object singleton class real class: #<Class:Object>
An object singleton class real superclass: #<Class:BasicObject>

So that in the case of metaclasses, klass points to the metaclass itself (and that part of the diagram is correct, but that points at some inconsistencies in and ).

Given an instance of a Ruby object, how do I get its metaclass?

Yep.

metaclass = class << obj; self; end

Why Are Metaclasses Created in Ruby