12 tips to boost RSpect and Cucumber

In our daily TDD/BDD work, we spent most of our time in test cases, we analyze requirements, write test case, run it and watch it fail, implement code to make the test pass, repeat the process "until money runs out". Thus the speed of running those test cases have significant impact on a team's performance, ! You can't tolerance writing a test case, waiting 1 minute finish; You can't tolerance CI finishes running all RSpec/Cucumber tests for half an hour! I can't! So I motivated to improve this!

The result

In the project I was working on there are 1566 RSpec examples for all models, after I applied most tips appropriate for out project, the result below is a comparison before and after boosting, the command is:

time bundle exec rspec spec/models

Before boost: After boost:

One minute and 8 seconds are saved after the boost!!! And it is only part of our all RSpect test, we definitely have tests for controllers, views, libs, etc., a 50.3% improvement!

To be honest from the beginning I admit I don't have too much confident I can make things better, until I saw this presentation: Grease your suite which surprised, inspired and encouraged me quite a lot! I did two whole days research and summarized 12 super useful tips for speeding up the RSpec/Cucumber tests!

Use Spork

Spork has two major advantages:

1. It hosts a DRB server and pre-load Rails environment into the server instance, after that all further running tests can communicate with the DRB server thus no need to reload Rails environment every time! Much faster than common way!
2. It forks (using Kernel.fork) a copy of server each time we run our tests, thus ensure a clean state between each other.

My real experience is it is awesome! I wrote a few line of code and I run the test, I can see the result in much shorter time than before! BDD development in daily work becomes more enjoyable, less frustration when waiting for slow tests times and times. Trust me, once you get used to it you'll never wanna fall back to the old slow way!

Spork resources

• Spork on Ruby Forge
• Spork repo@Github
• RailsCast video for Spork

GC tuning for Ruby Enterprise Edition

Before getting into the detailed tuning option I would like to review a few important points:

• Unlike many famous GC implementations like JVM, CLR and V8, Ruby's GC it not generational, which means it need scan ALL object to verify whether it is reachable.
• Every time GC decides to collect (used out pre-allocated memory typically), it will suspend all running thread and claim the resource, after that it resume all thread, this process takes 10% of CPU by average and cost a few hundred milliseconds (sometimes hundreds of)!
• Ruby's default GC settings are not optimized for Rails application, the settings can be modified for Ruby Enterprise Edition (REE).

By reading the official documentation, we got the following optimizable options:

• RUBY_HEAP_MIN_SLOTS
  This specifies the initial number of heap slots. The default is 10000.
• RUBY_HEAP_SLOTS_INCREMENT
  The number of additional heap slots to allocate when Ruby needs to allocate new heap slots for the first time. The default is 10000. For example, suppose that the default GC settings are in effect, and 10000 Ruby objects exist on the heap (= 10000 used heap slots). When the program creates another object, Ruby will allocate a new heap with 10000 heap slots in it. There are now 20000 heap slots in total, of which 10001 are used and 9999 are unused.
• RUBY_HEAP_SLOTS_GROWTH_FACTOR
  Multiplicator used for calculating the number of new heaps slots to allocate next time Ruby needs new heap slots. The default is 1.8. Take the program in the last example. Suppose that the program creates 10000 more objects. Upon creating the 10000th object, Ruby needs to allocate another heap. This heap will have 10000 * 1.8 = 18000 heap slots. There are now 20000 + 18000 = 38000 heap slots in total, of which 20001 are used and 17999 are unused. The next time Ruby needs to allocate a new heap, that heap will have 18000 * 1.8 = 32400 heap slots.
• RUBY_GC_MALLOC_LIMIT
  The amount of C data structures which can be allocated without triggering a garbage collection. If this is set too low, then the garbage collector will be started even if there are empty heap slots available. The default value is 8000000.
• RUBY_HEAP_FREE_MIN
  The number of heap slots that should be available after a garbage collector run. If fewer heap slots are available, then Ruby will allocate a new heap according to the RUBY_HEAP_SLOTS_INCREMENT and RUBY_HEAP_SLOTS_GROWTH_FACTOR parameters. The default value is 4096.

We can reference Twitter's GC tuning options below used in their production:

RUBY_HEAP_MIN_SLOTS=500000
RUBY_HEAP_SLOTS_INCREMENT=250000
RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
RUBY_GC_MALLOC_LIMIT=50000000

Twitter’s settings mean: Start with enough memory to hold the application (Ruby’s default is very low, lower than what a Rails application typically needs). Increase it linearly if you need more (Ruby’s default is exponential increase). Only garbage-collect every 50 million malloc calls (Ruby’s default is 6x smaller). The expert who did the PPT on Heroku I mentioned aboverecommended the following options:

export RUBY_HEAP_MIN_SLOTS=1000000
export RUBY_HEAP_SLOTS_INCREMENT=1000000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=1000000000
export RUBY_HEAP_FREE_MIN=500000

How to enable the GC options above? Personally I store those values in my .rvmrc file under project folder, you can export them in other places. And please notice the option values above are for your reference, might not quite appropriate for every concrete project, how to set those options suitable for your real project can be determined by on a simple rule: better settings makes less times of GC running. I wrote the following code in spec_helper.rb to display how many times GC runs and how much time it cost in total after running given tests examples.

RSpec.configure do |config|
 config.before(:suite) do
   GC.enable_stats
 end
 config.after(:suite) do
   puts "\n"
   puts "********GC STATUS SUMMARY AFTER ALL TESTS FINISHED********"
   puts "GC has done #{GC.collections} times collections!"
   puts "Total time spent in GC collections: #{GC.time}!"
   GC.disable_stats
 end
end

The sample output of GC stats after running 360 examples:

More resources on Ruby GC tuning:

• Garbage collector performance tuning - REE official documentation
• Twitter blog - Building a Faster Ruby Garbage Collector
• Ruby Programming/Reference/Objects/GC
• Ruby GC Tuning
• Ruby’s GC Configuration
• What makes Ruby slow?

Deferred GC

Other than adjusting GC tuning options we can also deffer GC runs! Create a file under spec/support/deferred_garbage_collection.rb and paste code below:

class DeferredGarbageCollection

 DEFERRED_GC_THRESHOLD = (ENV['DEFER_GC'] || 15.0).to_f

 @@last_gc_run = Time.now

 def self.start
   GC.disable if DEFERRED_GC_THRESHOLD > 0
 end

 def self.reconsider
   if DEFERRED_GC_THRESHOLD > 0 && Time.now - @@last_gc_run >= DEFERRED_GC_THRESHOLD
     GC.enable
     GC.start
     GC.disable
     @@last_gc_run = Time.now
   end
 end
end

Refer: RSpec speed-up (24.6%) by tweaking ruby garbage collection

Increase log level to reduce IO

By default in Rails development/test environment the logger's level is 0 which means :debug, however, most of the time we don't need verbose information flushed into log file, we can absolutely increase the log level to reduce IO! Setting it to 3 (:error) would be best IMHO, there is no redandunt information but still be able to trace failed tests.

Rails.logger.level = 3

Reduce Devise.streches

Reduce Devise.streches can improve Bcrypt performance, because we don't need strong encryption during testing, the default value is 10, we can set it to 1 in test environment.

Devise.stretches = 1

Mock real HTTP requests

Real HTTP requests can vastly slower down your tests, unless you explicitly want to "test" the request (usually in integration testing), you should mock it to isolate your testing "target" and focus on the behavior of your code.

WebMock stubs HTTP requests at low Net::HTTP level so there is no need to change any of your tests, by default it disables all real HTTP requests, you can register requests you want to test to cross the wall, or you can put WebMock.allow_net_connect!at your spec_helper.rb for RSpec or env.rb for Cucumber, personally I don't quite suggest allow all requests at a high level, because sometimes unobtrusive HTTP request happens and it is not easy to discover by just reviewing the code, for example:

• You call a property getter on module A, but the getter method was delegated to module B, and inside the real implementation in module B it submits request to remote.
• When you call ActiveRecord::Validations.valid?, all the validation callbacks will be invoked, and some of them probably have relationship with other modules, unfortunately "other modules" are retrieved from remote!

So if you disable all the real requests in testing (by default when using WebMock), those unobtrusive request can be easily found out.

Force ActiveRecord to use one shared connection instance

We are Capybara with JavaScript testing, when Capybara runs, it start two processes, one for hosting the server and another invokes underline web driver to communicate with real browser, this results in config.use_transactional_fixtures = true not robust since in RoR transaction works on one DB connection instance! So this enhancement forces using one shared connection when running Capybara, and this improves performance a little bit as well.

Put the following code in spec_helper.rb and env.rb:

class ActiveRecord::Base
mattr_accessor :shared_connection
@@shared_connection = nil

def self.connection
  @@shared_connection || retrieve_connection
end
end
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection

Parallel testing

Run tests in multi CPU cores in parallel is extremely powerful, on a four-core Intel i7 processor with hyper-threading enabled, parallel_tests can kick-off 8 processes to run test separately on each core, in the idea situation it could reduce time cost down to 50%, and the more case you run in parallel, the more dramatical differences you will get.

However there are couple of things we need aware:

1. It won't reduce the total time comparing to single process if there are hundreds of local DB operations in the test!
2. Process in OS level is very expansive, using 8 processes to deal with 50 examples might not be wise, time cost during processes' life cycle will neutralize the time saved in parallel manner! Considering use traditional way to save some electricity:)

Force ActiveRecord to use one shared connection instance

If you are Capybara with JavaScript testing, you should know when Capybara runs, it start two threads, one for hosting the server and another invokes underline web driver to communicate with real browser, this results in use_transactional_fixtures not robust since in RoR transaction works on one DB connection instance! So this enhancement forces using one shared connection when running Capybara, and this improves performance a little bit as well.

Put the following code in spec_helper.rb and env.rb:

class ActiveRecord::Base
 mattr_accessor :shared_connection
 @@shared_connection = nil

 def self.connection
   @@shared_connection || retrieve_connection
 end
end
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection

QuickerClip

Many Rails application uses the famous image processing library PaperClip, it uses libraries of ImageMagick, they are all great libraries, but the problem for us is we don't want to really invoke them during test in most of the time, testing functionalities in those third party library makes no sense for your project, we should isolate it and focus on our own business logics.

Add following line into test.rb, it will avoid invoking real ImageMagick command thus significantly improve tests related with PaperClip.

module Paperclip
 def self.run cmd, params = "", expected_outcodes = 0
   case cmd
   when "identify"
     return "100x100"
   when "convert"
     return
   else
     super
   end
 end
end

class Paperclip::Attachment
 def post_process
 end
end

Origin: Stubbing out Paperclip/ImageMagick in Tests

In-Memory DB or even no DB

We can use an in-memory DB for testing environment would be sweet, because there is no need to persistent. We do have a lot of choices of in-memory DB for example:

• MySQL
• SQLite
• PostgreSQL
• And many...

More resources on this
In memory SQlite database for testing Rspec, Cucumber: best speed database clean strategy

Actually in idea situation, we can consider adopting "No Database strategy" for our tests! Refer UnitRecordfor reference and see how fast the author did!

Finished in 19.302702 seconds.
4920 tests, 7878 assertions, 0 failures, 0 errors

Be Organized

This is the last tip for testing performance, unlike above tips/techniques: it is non-technical, however it is probably the most important one!

1. More unit tests, less integration tests. KISS principle also suitable for tests. Usually integrations test has dependencies on external resources (database, web service, message queue or IO), I'd prefer running unit tests locally and integration tests cloud CI (usually faster than local), keep your local development responsive!
2. Use tags to identify test's attribute (use RSpec as an example Cucumber is similar), for example:
   
   it "Display login page", :mandatory => true do; end
   it "Redirect to homepage", :speed => 'fast' do; end
   it "Show user profile page", :require_login => true do; end
   it "Retrieve user's twitts through Twitter API using OAuth", :speed => 'slow' do; end
   
   So that we can run filtered tests by:
   rspec . --tag speed:'fast'
rspec . --tag mandatory:true

Conclusion

Daily BDD/TDD with fast, "responsive" tests which can give you feedback in short time makes development life more enjoyable and happy, so does CI! The 12 tips above might not all suitable for your real project, but a number of them should do. If you didn't get a good result after boosting, run rspec with --profile/-p, RSpec will then track the top 10 slowest examples, for Cucumber pass "usage" as a output format will also reveal slowest step definitions.

Wish the tips help you!