Has Anyone Figured Out a Way to Run the Same Cucumber Scenario on Multiple Browsers/Web Drivers

Has anyone figured out a way to run the same cucumber scenario on multiple browsers/web drivers?

So, I wound up rolling my own solution to this. Not sure if it was the best or most elegant approach, but I actually just wound up:

1. Abstracting all common environment stuff into env.rb
2. Using Cucumber profiles which would require a specific environment file (such as firefox.rb) that required env.rb and then set the default driver for Capybara to the appropriate driver.
3. Wrote a big ol' thor class with tasks that bundle up a bunch of cucumber commands and call out to run the bad boy with the proper profile.
4. Wrote an 'all_browsers' task which bundles up the commands, then calls out to each specific driver task, so I can now have one task that runs any set of scenarios I supply on all the supported drivers.

Working like a charm and I think might have actually wound up better in the end than anything I was trying above, as within the Thor file I was able to add things like a benchmarking option, as well as whether or not to split the feature run up into multiple threads.
Still curious if anyone else came up with a solution for this though.

cucumber.yaml:

Here, the all_features file just does a glob of everything ending in .feature, because if I pulled in the entire features directory it would pull in everything beneath it, including all the profile files, etc, which isn't what I wanted since each profile file sets the default capybara driver to a different value. Once you specify -r as an option to cucumber, all autoloading of any file is halted.

default: --format pretty

chrome: --format pretty -r features/support/profiles/chrome.rb -r features/all_features -r features/step_definitions

firefox: --format pretty -r features/support/profiles/firefox.rb -r features/all_features -r features/step_definitions

celerity: --format pretty -r features/support/profiles/celerity.rb -r features/all_features -r features/step_definitions

firefox.rb (the 'profile' file):

require File.dirname(__FILE__) + "/../env.rb"

Capybara.configure do |config|
  config.default_driver = :selenium_firefox
end

selenium_firefox.rb (where I register the driver, and set some tag capability which I've wound up not needing now, as the @selenium_firefox tag was part of my original attempt at this posted in the question):

# Register a specific selenium driver for firefox
Capybara.register_driver :selenium_firefox do |app|
  Capybara::Driver::Selenium.new(app, :browser => :firefox)
end

# Allows the use of a tag @selenium_firefox before a scenario to run it in selenium with firefox
Before('@selenium_firefox') do
  Capybara.current_driver = :selenium_firefox
end

feature_runner.thor:

require 'benchmark'

class FeatureRunner < Thor
  APP_ROOT = File.expand_path(File.dirname(__FILE__) + "/../")

  # One place to keep all the common feature runner options, since every runner in here uses them.
  # Modify here, and all runners below will reflect the changes, as they all call this proc.
  feature_runner_options = lambda { 
    method_option :verbose, :type => :boolean, :default => true, :aliases => "-v"
    method_option :tags, :type => :string
    method_option :formatter, :type => :string
    method_option :other_cucumber_args, :type => :string
  }

  desc "all_drivers_runner", "Run features in all available browsers"
  method_option :benchmark, :type => :boolean, :default => false
  method_option :threaded, :type => :boolean, :default => true
  feature_runner_options.call # Set up common feature runner options defined above
  def all_drivers_runner
    if options[:threaded]
      feature_run = lambda { 
        thread_pool = []

        t = Thread.new do |n|
          invoke :firefox_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :chrome_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :celerity_runner
        end
        thread_pool << t

        thread_pool.each {|th| th.join}
      }
    else
      feature_run = lambda { 
        invoke "feature_runner:firefox_runner", options
        invoke "feature_runner:chrome_runner", options
        invoke "feature_runner:celerity_runner", options
      }
    end

    if options[:benchmark]
      puts "Benchmarking feature run"
      measure = Benchmark.measure { feature_run.call }
      puts "Benchmark Results (in seconds):"
      puts "CPU Time: #{measure.utime}"
      puts "System CPU TIME: #{measure.stime}"
      puts "Elasped Real Time: #{measure.real}"
    else
      feature_run.call
    end
  end

  desc "firefox_runner", "Run features on firefox"
  feature_runner_options.call # Set up common feature runner options defined above
  def firefox_runner
    command = build_cucumber_command("firefox", options)
    run_command(command, options[:verbose])
  end

  desc "chrome_runner", "Run features on chrome"
  feature_runner_options.call # Set up common feature runner options defined above
  def chrome_runner
    command = build_cucumber_command("chrome", options)
    run_command(command, options[:verbose])
  end

  desc "celerity_runner", "Run features on celerity"
  feature_runner_options.call # Set up common feature runner options defined above
  def celerity_runner
    command = build_cucumber_command("celerity", options)
    run_command(command, options[:verbose])
  end

  private
  def build_cucumber_command(profile, options)
    command = "cd #{APP_ROOT} && ./bin/cucumber -p #{profile}"
    command += " --tags=#{options[:tags]}" if options[:tags]
    command += " --formatter=#{options[:formatter]}" if options[:formatter]
    command += " #{options[:other_cucumber_args]}" if options[:other_cucumber_args]
    command
  end

  def run_command(command, verbose)
    puts "Running: #{command}" if verbose
    output = `#{command}`
    puts output if verbose
  end

end

Where everything wound up, in relation to the root directory:

.
|____cucumber.yml
|____features
| |____all_features.rb
| |____google_search.feature
| |____step_definitions
| | |____google_steps.rb
| | |____web_steps.rb
| |____support
| | |____custom_formatters
| | | |____blah.rb
| | |____env.rb
| | |____paths.rb
| | |____profiles
| | | |____celerity.rb
| | | |____chrome.rb
| | | |____firefox.rb
| | |____selenium_drivers
| | | |____selenium_chrome.rb
| | | |____selenium_firefox.rb
| | | |____selenium_ie.rb
| | | |____selenium_remote.rb
| | |____selenium_drivers.rb
|____tasks
| |____feature_runner.thor
| |____server_task.rb  

Output of thor -T

feature_runner
--------------
thor feature_runner:all_drivers_runner  # Run features in all available browsers
thor feature_runner:celerity_runner     # Run features on celerity
thor feature_runner:chrome_runner       # Run features on chrome
thor feature_runner:firefox_runner      # Run features on firefox  

Now I can run something like:

thor feature_runner:all_drivers_runner --benchmark
This would run all features on all capybara drivers in a thread for each driver, benchmnarking the results.

Or

thor feature_runner:celerity_runner
This would run all features only on celerity.

But I can now also supply some other options to the thor command which get passed onto cucumber such as:

--tags=@all_browsers
--formatter=hotpants
--other_cucumber_args="--dry-run --guess --etc"

What a feature file can now look like:

Feature: Start up browser
  @all_browsers
  Scenario: Search Google
   Given I am on the home page
   When I fill in the search bar with "Capybara"
   And I press "Search"
   Then I should see "Capybara"

Seems like a lot of setup, but now if I tag a feature with @all_browsers, I can build out a suite to test against all capybara drivers, in a multi-threaded environment, with one thor command:

thor feature_runner:all_drivers_runner --threaded --tags=@all_browsers

Or build out a smoke test suite that runs in celerity:

thor feature_runner:celerity_runner --tags=@smoke_test

Is it possible to execute cucumber scenario's in parallel on different browsers(chrome and firefox) at same time?

You are running a your tests against a matrix of browsers. Typically this matrix configured in CI and provided to the test execution via environment variables. For example using Gitlab CI Matrix:

test:
  stage: test
  script:
    - mvn test
  parallel:
    matrix:
      - OS: Windows
        OS_VERSION: 10
        BROWSER: [Chrome, Firefox, Edge]
      - OS: OS X
        OS_VERSION: Big Sur
        BROWSER: [Chrome, Firefox, Edge, Safari]

You then create the web driver in the before hook using the environment variables.

    @Before
    public void before(Scenario scenario){
        String os = System.getenv("OS");
        String osVersion = System.getenv("OS_VERSION");
        String browser = System.getenv("BROWSER");
        driver = createDriver(os, osVersion, browser);
    }

You could also use Maven Profiles or Gradle Tasks to define these different sets of environment variables.

However key is to let these jobs in parallel on your CI system by starting multiple JVMs rather then only in Cucumber by starting multiple threads.

Cucumber with selenium opens several browsers

I found the problem, it was because I had several files xxx_steps.rb in step_defintions folder and in each xxx_steps.rb I had "require 'watir-webdriver'"

So each time cucumber met require 'watir-webdriver' it opened a new browser.

Watir-webdriver keeps opening multiple browsers

The problem is that env.rb is being loaded twice:

• It is automatically included when running the cucumber command
• It is being included a second time in GoogleSearch.rb when calling the line require_relative "../support/env".

As a result, each of the hooks is registered twice. In other words, Cucumber is seeing the hooks to run before each scenario as:

Before do
  @browser = Watir::Browser.start app_host, :firefox
end

Before  do |scenario|
  @scenario_tag = scenario.source_tag_names
  @browser.cookies.clear
end

Before do
  @browser = Watir::Browser.start app_host, :firefox
end

Before  do |scenario|
  @scenario_tag = scenario.source_tag_names
  @browser.cookies.clear
end

As you can see, Watir::Browser.start is called twice resulting in the two browsers. The first one is not used since the second call uses the same variable.

To solve the problem, simply remove the require_relative "../support/env" line.

Note that this will only address the issue with opening two browsers for each scenario. You will notice that you will still get a new browser for each scenario and that only the last browser gets closed. If you only want one browser for all scenarios, you should look at the global hooks.

---

Related Topics