Closures and for Loops in Ruby

Closures and for loops in Ruby

Okay, this is getting ridiculous. Every time I try to answer a question about how for loops work in Ruby, I get it wrong.

The reason for this is, of course, that I don't use for loops in Ruby, neither does anybody else, so it really doesn't matter to me :-)

Anyway, to settle the matter once and for all, I went straight to the ultimate source, the December 1, 2009 preliminary Draft of the IPA Ruby Language Specification (destined to become the ISO Ruby Language Specification):

§11.4.1.2.3 The for expression

Syntax

for-expression → for for-variable in expression do-clause end

for-variable → left-hand-side | multiple-left-hand-side

The expression of a for-expression shall not be a jump-expression.

Semantics

A for-expression is evaluated as follows:

Evaluate the expression. Let O be the resulting value.

Let E be the primary-method-invocation of the form primary-expression [no line-terminator here].each do | block-formal-argument-list | block-body end, where the value of the primary-expression is O,the block-formal-argument-list is the for-variable, the block-body is the compound-statement of the do-clause.

Evaluate E, but skip Step c of §11.2.2.

The value of the for-expression is the resulting value of the invocation.

Okay, so basically this means that

for for_variable in expression
  do_clause
end

gets translated to

O = expression
O.each do |for_variable|
  do_clause
end

Or, in your case:

for i in 1..5
  puts j if i > 1 #undefined local variable or method `j' (NameError)
  j = i
end

gets translated to

(1..5).each do |i|
  puts j if i > 1 #no excpetion here, works just fine ??!!??
  j = i
end

Aha! But we forgot something! There's this ominous "skip Step c of §11.2.2." thing! So, what does it say?

Push an empty set of local variable bindings onto ⟦local-variable-bindings⟧.

Note that Step b

Set the execution context to E_b.

is not skipped.

So, as far as I can see, a for loop gets its own execution context, which starts out as a copy of the current execution context, but it does not get its own set of local variable bindings. IOW: it gets its own dynamic execution context, but not its own lexical scope.

I must admit, I'm still not sure I fully understand it, but it doesn't get any more precise than this.

How can I use PHP 5.3 Closures like We use Blocks in Ruby

If you are looking at using lambdas to iterate over a PHP array, there are certain functions that you could use to accomplish that. Better illustrate it, I used a wrapper class enum:

class enum {
    public $arr;

    function __construct($array) {
        $this->arr = $array;
    }

    function each($lambda) {
        array_walk($this->arr, $lambda);
    }

    function find_all($lambda) {
        return array_filter($this->arr, $lambda);
    }

    function inject($lambda, $initial=null) {
        if ($initial == null) {
            $first = array_shift($this->arr);
            $result = array_reduce($this->arr, $lambda, $first);
            array_unshift($this->arr, $first);

            return $result;
        } else {
            return array_reduce($this->arr, $lambda, $initial);
        }
    }

}

$list = new enum(array(-1, 3, 4, 5, -7));
$list->each(function($a) { print $a . "\n";});

// in PHP you can also assign a closure to a variable 
$pos = function($a) { return ($a < 0) ? false : true;};
$positives = $list->find_all($pos);

// inject() examples
$list = new enum(range(5, 10));

$sum = $list->inject(function($sum, $n) { return $sum+$n; });
$product = $list->inject(function($acc, $n) { return $acc*$n; }, 1);

$list = new enum(array('cat', 'sheep', 'bear'));
$longest = $list->inject(function($memo, $word) {
        return (strlen($memo) > strlen($word)) ? $memo : $word; }
    );

That being said, closures in PHP aren't meant to replace the for loop nor do they behave like ruby blocks.

Implementing JavaScript-like Closure in Ruby

While Javascript has one type of lambda construct (an anonymous function), Ruby has several. The most common are blocks (passed to methods as in each in your example), Procs (which are like blocks that are bound to a execution context which they retain even when passed around), and Lambdas which are probably closest to anonymous functions. I won't go into the details of Procs vs Lambdas here.

Here are examples of these constructs in Ruby:

# Using a block + yield
def my_method(arg)
  puts arg + yield(3)
end

my_method(5) { |arg| arg * 2 }
# => 11

# Binding the block to a Proc then using #call
def print(&block)
  puts block.call
end

# Creating a Proc that can be passed around and then #called
def other_method(arg, other_args*)
  arg.call(other_args*)
end

var = 3
prc = Proc.new { puts var }
other_method(prc)

# Creating a Lambda using stabby syntax
num = 3
l = -> (arg) { arg + 2 + num }
other_method(l, 1)

In Ruby you could write your JS example as:

def outer(input)
  closure_var = "I'm here with "

  inner = -> { closure_var + input }

  inner.call
end

outer('outer input')

In IRB:

jbodah@Joshs-MacBook-Pro-2 2.1.3p242 ~ (none) $ irb
irb(main):001:0>     def outer(input)
irb(main):002:1>       closure_var = "I'm here with "
irb(main):003:1>
irb(main):004:1*       inner = -> { closure_var + input }
irb(main):005:1>
irb(main):006:1*       inner.call
irb(main):007:1>     end
=> :outer
irb(main):008:0>
irb(main):009:0*     outer('outer input')
=> "I'm here with outer input"

How to add scheduler with ruby loop?

Your problem here is you are using a for loop instead of an Enumerable method such as each.

Short answer, just change for x in 0..s3 to sume2.each_index do |x|

Longer answer, you can compare the following:

for x in 0..3
  Thread.new { loop { puts(x); sleep 1 } }
end

with:

(0..3).each do |x|
  Thread.new { loop { puts(x); sleep 1 } }
end

The first just prints 3 repeatedly, but the second prints 1,2,3 as intended

Why? It's because with the for loop. your x variable gets overwritten each loop. With each, your x variable is scoped to the block and you have a closure.

For more explanation, see Closures and for loops in Ruby, or https://scotch.io/tutorials/understanding-ruby-closures, or just google "closures in ruby"

Note, it's not idiomatic to use for loops in Ruby, for this particular reason

Execution flow of times iterator and for loop in ruby

The reason you are seeing this behavior is because 3.times is executing a block while the for .. in loop is executing in the same context it is called. This is where closures come into play.

In your first example, sum hasn't been defined anywhere that the block can access it. Therefore, sum is re-defined each time the block is executed. If you add a puts sum statement after the block, you will see that sum is undefined.

For the for loop example, you stay in the same context so the second time through the loop sum has already been defined in the context so it uses the existing value. You can verify this by adding a puts sum statement after the loop and you will see that it has the last value that it was set to in the loop.

When you execute your 3.times block after you have executed your loop, sum exists in the closure so the block uses the already defined variable instead of re-defining it each time through the loop.

How are variables bound to the body of a define_method?

Blocks in Ruby are closures: the block you pass to define_method captures the variable name itself–not its value—so that it remains in scope whenever that block is called. That's the first piece of the puzzle.

The second piece is that the method defined by define_method is the block itself. Basically, it converts a Proc object (the block passed to it) into a Method object, and binds it to the receiver.

So what you end up with is a method that has captured (is closed over) the variable name, which by the time your loop completes is set to :destroy.

Addition: The for ... in construction actually creates a new local variable, which the corresponding [ ... ].each {|name| ... } construction would not do. That is, your for ... in loop is equivalent to the following (in Ruby 1.8 anyway):

name = nil
[ :new, :create, :destroy ].each do |name|
  define_method("test_#{name}") do
    puts name
  end
end
name # => :destroy

For loop variable scope

That is expected. According to the documentation:

The for loop is similar to using each, but does not create a new variable scope.

Example with for:

for i in 1..3
end
i #=> 3

Example with each:

(1..3).each do |i|
end
i #=> NameError: undefined local variable or method `i'

If I'm not mistaken, methods (each, map, loop, upto) create variables scopes, whereas keywords (for, while, until) don't.

Weird closure behavior in Ruby

The problem in your code boils down to this:

results = [
  {a: [1, 2, 3]}, 
  {a: [4, 5, 6]},
]

funcs = []

while not results.empty?
  result = results.shift

  2.times do |i|
    val = result[:a] + [i]

    #funcs << lambda { p val }
    funcs << lambda { p result[:a] + [i] }
  end
end

funcs.each do |func|
  func.call
end

--output:--
[4, 5, 6, 0]
[4, 5, 6, 1]
[4, 5, 6, 0]
[4, 5, 6, 1]

A closure closes over a variable--not a value. Subsequently, the variable can be changed, and the closure will see the new value when it executes. Here is a very simple example of that:

val = "hello"
func = lambda { puts val }  #This will output 'hello', right?

val = "goodbye"
func.call

--output:--
goodbye

In the lambda line inside the loop here:

results = [
  {a: [1, 2, 3]}, 
  {a: [4, 5, 6]},
]

funcs = []

while not results.empty?
  result = results.shift
    ...
    ...

    funcs << lambda { p result[:a] + [i] }  #<==HERE
  end
end

...the lambda closes over the whole result variable--not just result[:a]. However, the result variable is the same variable every time through the while loop--a new variable is not created each time through the loop.

The same thing happens to the val variable in this code:

results = [
  {a: [1, 2, 3]},
  {a: [4, 5, 6]},
]

funcs = []

while not results.empty?
  result = results.shift
  val = result[:a] + [1]

  funcs << lambda { p val }
end

funcs.each do |func|
  func.call
end

--output:--
[4, 5, 6, 1]
[4, 5, 6, 1]

The val variable is assigned a newly created array each time through the loop, and the new array is completely independent of result and result[:a], yet all the lambdas see the same array. That's because all the lambdas close over the same val variable; then the val variable is subsequently changed.

But if you introduce a block:

while not results.empty?
  result = results.shift

  2.times do |i|
    val = result[:a] + [i]
    funcs << lambda { p val }
  end
end

--output:--
[1, 2, 3, 0]
[1, 2, 3, 1]
[4, 5, 6, 0]
[4, 5, 6, 1]

...every time the block executes, the val variable is created anew. As a result, each lambda closes over a different val variable. That should make some sense if you consider that a block is just a function that gets passed to the method, in this case the times() method. Then the method repeatedly calls the function--and when a function is called, the local variables, like val, are created; and when the function finishes executing, all the local variables are destroyed.

Now back to the original example:

while not results.empty?
  result = results.shift

  2.times do |i|
    val = result[:a] + [i]

    #funcs << lambda { p val }
    funcs << lambda { p result[:a] + [i] }
  end
end

The reason the two lambda lines produce different results should now be clear. The first lambda line closes over a new val variable every time the block executes. But the second lambda line closes over the same result variable every time the block executes, so all the lambdas will refer to the same result variable--and the last hash assigned to the result variable is the hash that all the lambdas see.

So the rule is: loops do not create new variables every time through the loop and blocks do.

Note that it would be better to declare all loop variables outside the loop, lest we forget that the variables inside the loop are not created anew every time through the loop.

Define a method that is a closure in Ruby

This works (tested in irb):

NOTE: This changes only str - not all instances of String. Read below for details as to why this works

another_str = "please don't change me!"
str =         "ha, try to change my to_s! hahaha!"
proc = Proc.new { "take that, Mr. str!" }

singleton_class = class << str; self; end

singleton_class.send(:define_method, :to_s) do
  proc.call
end

puts str.to_s         #=> "take that, Mr. str!"
puts another_str.to_s #=> "please don't change me!"

# What! We called String#define_method, right?

puts String           #=>  String
puts singleton_class  #=>  #<Class:#<String:0x3c788a0>>

# ... nope! singleton_class is *not* String
# Keep reading if you're curious :)

This works because you are opening str's singleton class and defining a method there. Because this, as well as the call to Module#define_method, have what some call a "flat scope", you're able to access variables that would be out of scope if you used def to_s; 'whatever'; end.

You may want to check out some of these other "metaprogramming spells" here:

media.pragprog.com/titles/ppmetr/spells.pdf

Why does it only change str?

Because Ruby has a couple interesting tricks up it's sleeves. In the Ruby object model, a method invocation results in the reciever searching not only it's class (and it's ancestors), but also it's singleton class (or as Matz would call it, it's eigenclass). This singleton class is what allows you to [re]define a method for a single object. These methods are called "singleton methods". In the example above, we are doing just that - defining a singleton method name to_s. It's functionaly identical to this:

def str.to_s
  ...
end

The only difference is that we get to use a closure when calling Module#define_method, whereas def is a keyword, which results in a change of scope.

Why can't it be simpler?

Well, the good news is that you're programming in Ruby, so feel free to go crazy:

class Object
  def define_method(name, &block)
    singleton = class << self; self; end
    singleton.send(:define_method, name) { |*args| block.call(*args) }
  end
end

str = 'test'
str.define_method(:to_s) { "hello" }
str.define_method(:bark) { "woof!" }
str.define_method(:yell) { "AAAH!" }

puts str.to_s #=> hello
puts str.bark #=> woof!
puts str.yell #=> AAAH!

And, if you're curious...

You know class methods? Or, in some languages, we'd call them static methods? Well, those don't really exist in Ruby. In Ruby, class methods are really just methods defined in the Class object's singleton class.

If that all sounds crazy, take a look at the links I provided above. A lot of Ruby's power can only be tapped into if you know how to metaprogram - in which case you'll really want to know about singleton classes/methods, and more generally, the Ruby object model.

HTH

-Charles

Scope of a local variable in a block

Look at the code below:

2.times do
  p defined? i
  i ||= 1
  p defined? i
  p "#{i} "
  i += 1
  p "#{i} "
end

Output:

nil 
"local-variable"
"1 "
"2 "
nil
"local-variable"
"1 "
"2 "

That means in each iteration a new scope is created,and i is known to that scope only; which is proved by nil and "local-variable".

Now i is created outside of block, and see the output(no nil comes):

i = nil
2.times do
  p defined? i
  i ||= 1
  p defined? i
  p "#{i} "
  i += 1
  p "#{i} "
end

Output:

"local-variable"
"local-variable"
"1 "
"2 "
"local-variable"
"local-variable"
"2 "
"3 "

To know more about ||= look What Ruby’s ||= (Double Pipe / Or Equals) Really Does

Closures and for Loops in Ruby