Why Is Autoload Failing to Load Files for Gems

`autoload` raises an error but `require` does not (ruby)

You'll need to add the 'crack' gem to your $LOAD_PATH by doing:

gem 'crack'

This is necessary because RubyGems replaces Kernel#require with a method that attempts to "activate" the gem before requiring it if necessary, but doesn't do the same thing for Kernel#load - and autoload calls load on the backend.

Ruby autoload conflicts between gmail and parse_resource gems

I had installed the gems with bundler.

To fix the issue with loading both libraries, I installed the gems with the gem command:
sudo gem install gmail parse_resource

With this done, I was able to require the libraries in any order, and to connect to gmail and parse without issue.

-Nick

Why the autoload method cannot find the file in my library?

I find the problem is I made a wrong gemspec.

spec.files = Dir["{lib/*,spec/*}"] + %w{README}

It didn't package the file under "lib/renren_api/".
To change it like this will solve this problem.

spec.files = Dir["{lib/**/*,spec/*}"] + %w{README}

Why Rails autoload failed with the file is there?

After debugging into Rails source code, I found the problem.

First, since the helper file is inside helpers folder, the file should be defined inside Helpers module:

module API
  module Helpers
    module BaseHelper
      def test
        "I am a test helper"
      end
    end
  end
end

Second, API::Helpers::BaseHelper has the path suffix api/helpers/base_helper, so make sure "#{Rails.root}/lib/" is inside your autoload path. ThenActiveSupport` will find it for you.

Rails: Auto-reload gem files used in dummy app

To reload the gem code on file-system changes, I needed to do three steps:

1. Unregister the zeitwerk loader defined in foo_gem.rb that handles loading the different gem files.
2. Define a new zeitwerk loader in the development.rb of the dummy app that is configured with enable_reloading.
3. Setup a file-system watcher and trigger a reload when a gem file changes.

I'm sure there is a simpler and cleaner solution. Feel free to post it as a separate answer. I'll post my solution here, but do consider it a workaround.

Zeitwerk::Loader in foo_gem.rb

# ~/rails/foo_gem/lib/foo_gem.rb

# require 'foo_gem/bar`  # Did not work. Instead:

# (a) use zeitwerk:
require "zeitwerk"
loader = Zeitwerk::Loader.new
loader.push_dir File.join(__dir__)
loader.tag = "foo_gem"
loader.setup

# or (b) use autoload:
module FooGem
  autoload :Bar, "foo_gem/bar"
end

Note:

• In the past, I've just loaded all ruby files of the gem with require from a kind of index file called just like the gem, here: foo_gem.rb. This does not work here, because zeitwerk appears to ignore files that have previously been loaded with require. Instead I needed to create a separate zeitwerk loader for the gem.
• This loader has no enable_reloading because otherwise, reloading would be enabled for this gem whenever using the gem, not just while developing the gem.
• I have given the loader a tag, which allows to find this loader later in the Zeitwerk::Registry in order to un-register it.
• Instead of using zeitwerk in foo_gem.rb, one could also use autoload there like the devise gem does. This is the best way if one wants to support rails versions earlier than 6 because zeitwerk requires rails 6+. Using autoload here also makes step 1 in the next section unnecessary.

Zeitwerk::Loader in development.rb of the dummy app

# ~/rails/foo_gem/spec/dummy_app/config/environments/development.rb

# 1. Unregister the zeitwerk loader defined in foo_gem.rb that handles loading
#    the different gem files.
#
Zeitwerk::Registry.loaders.detect { |l| l.tag == "foo_gem" }.unregister

# 2. Define a new zeitwerk loader in the development.rb of the dummy app
#    that is configured with enable_reloading.
#
gem_root_path = Pathname.new(File.expand_path(Rails.root.join("../..")))
gem_loader = Zeitwerk::Loader.new
gem_loader.push_dir gem_root_path.join("lib")
gem_loader.enable_reloading
gem_loader.setup

# 3. Setup a file-system watcher and trigger a reload when a gem file changes.
#
Listen.to gem_root_path.join("lib"), only: /\.rb$/ do
  gem_loader.reload
end.start

Note:

• Zeitwerk does not allow two loaders managing the same files. Therefore, I need to unregister the previously defined loader tagged "foo_gem".
• The new loader used in the dummy app has enable_reloading. Therefore, when using the dummy app with rails server, rails console, or when running the specs, the gem files can be reloaded.
• The gem files are not automatically reloaded by zeitwerk. One needs a file-system watcher to trigger the reload on file-system changes. I did not manage to get the ActiveSupport::FileUpdateChecker working. Instead, I've used the listen gem as file-system watcher.

With this setup, when using the rails server, the rails console of the dummy app or integration tests using the dummy app, gem files are reloaded after being edited, which means that one does no longer need to restart the rails server to pick up the changes.

References

• Zeitwerk::Loader
• Zeitwerk::Registry
• https://github.com/fxn/zeitwerk
• https://guides.rubyonrails.org/classic_to_zeitwerk_howto.html
• https://guides.rubyonrails.org/autoloading_and_reloading_constants.html

Zeitwerk: Add engine/gem directories to autoload path of parent Rails app

Rails sets up two loaders main and once:

Rails.autoloaders.main
Rails.autoloaders.once

These are just instances of Zeitwerk::Loader. Rails also gives you a config to add root directories to these loaders:

config.autoload_paths         # main
config.autoload_once_paths    # once

When gem's lib directory is added to autoload through one of these configs, lib becomes a root directory:

# config.autoload_paths += paths["lib"].to_a

>> Rails.autoloaders.main.root_dirs
=> 
...                    
 "/home/alex/code/stackoverflow/my_engine/lib"=>Object,
...

When a class from the gem is called, zeitwerk uses registered loaders to look up and to load the file corresponding to this class.

If the gem then sets up its own loader:

require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
loader.setup

another instance of Zeitwerk::Loader is created with its own root directories:

>> Zeitwerk::Registry.loaders.detect { |z| z.tag == "my_engine" }
=> 
#<Zeitwerk::GemLoader:0x00007fe5e53e0f80
...
 @root_dirs={"/home/alex/code/stackoverflow/my_engine/lib"=>Object},
...

# NOTE: these are the two loaders registered by rails
>> Zeitwerk::Registry.loaders.select { |z| z.tag =~ /rails/ }.count
=> 2

Zeitwerk doesn't allow two loaders to have a shared directory and raises an error showing two conflicting loaders.

Because the gem is a Rails::Engine the best option is to let rails manage zeitwerk loaders and remove Zeitwerk::Loader.for_gem setup.

# only use rails config
config.autoload_paths += paths["lib"].to_a

On the other hand, gem loader is already set up and config.autoload_paths is not needed.

# NOTE: without any loaders
>> MyEngine::Test
# (irb):1:in `<main>': uninitialized constant MyEngine::Test (NameError)                                            
# MyEngine::Test                                
#         ^^^^^^

# NOTE: with gem loader
#
#   require "zeitwerk"
#   loader = Zeitwerk::Loader.for_gem
#   loader.setup                           
#
>> MyEngine::Test
=> MyEngine::Test

# NOTE: with rails `main` loader
#
#   config.autoload_paths += paths["lib"].to_a
#
>> MyEngine::Test
=> MyEngine::Test

# NOTE: with gem loader and rails loader
$ bin/rails c
# /home/alex/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:480:in
# `block (3 levels) in raise_if_conflicting_directory':
# loader (Zeitwerk::Error)

Update

# Use rails loaders
# config.autoload_path          .->  Zeitwerk::Loader(@tag=rails.main)
# config.autoload_once_path     |->  Zeitwerk::Loader(@tag=rails.once)
#                               |
# Or create a new loader        |
# Zeitwerk::Loader.for_gem      |->  Zeitwerk::GemLoader(@tag=my_engine)
#                               |
# my_engine/lib can only be in one of these

Zeitwerk does the loading and reloading. Rails is just another gem here.

If you don't use rails config, Zeitwerk will find files through Zeitwerk::GemLoader(@tag=my_engine), that the gem has created.

If you use rails config, Zeitwerk will find files through Zeitwerk::Loader(@tag=rails.main), that rails has created (making GemLoader unnecessary).

If lib is a root directory in any of the existing loaders there is no need to have any requires or autoloads for files in lib directory. Except for things that are needed before Zeitwerk kicks in, like MyEngine::Engine from lib/my_engine/engine.rb.

---

Related Topics