Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Rails 6 built-in Parallel Testing #2104

Open
zoras opened this issue Apr 1, 2019 · 77 comments
Open

Add support for Rails 6 built-in Parallel Testing #2104

zoras opened this issue Apr 1, 2019 · 77 comments

Comments

@zoras
Copy link

zoras commented Apr 1, 2019

Use Rails 6 built-in parallelizer API for running parallel RSpec executors

https://edgeguides.rubyonrails.org/testing.html#parallel-testing

rails/rails#31900 (comment)

@JonRowe
Copy link
Member

JonRowe commented Apr 1, 2019

We'd love to support parallel tests with RSpec itself, its a rather large task however. If we get parallelisation support into rspec-core you can be sure we'll integrate with the rails helpers.

@jrochkind
Copy link
Contributor

Note that linked #1254 says:

There are bunch of gems that allow for parallel execution, but all of these are process based and not thread based.

I believe the parallel testing now built into Rails can be process-based or thread-based, and by default is process-based.

Is supporting Rails process-based parallel testing less of a challenge? It looks like one of the main things Rails does for you when supporting this is setting up a separate database for each test worker, as well as orchestrate getting each worker to run different tests and aggregating the results.

@JonRowe
Copy link
Member

JonRowe commented Apr 30, 2019

In the short term all we could offer is to help integrate 3rd party parallel testing with Rails tools.

Our own parallel journey requires changes to rspec-core which are mostly identical for process or thread based.

Please note there are parallel testing extensions for RSpec already!

@p8
Copy link
Contributor

p8 commented Aug 13, 2019

Discourse now has a solution based on parallel_tests discourse/discourse#7778

@iainbeeston
Copy link

Is there any update on supporting the activesupport parallelize helper in rspec-rails, now that rails 6 has been released?

@iainbeeston
Copy link

Sorry, I meant to include a link with more details:

rails/rails#31900

@JonRowe
Copy link
Member

JonRowe commented Aug 21, 2019

No, RSpec itself needs to support parallelisation in order for it to supported

@iainbeeston
Copy link

Sorry @JonRowe - can I please clarify something?

rspec-core needs updating even for multi-process parallelisation, right? (not just multi-threaded parallelisation as described in rspec/rspec-core#1254, which I would assume is more challenging)

@JonRowe
Copy link
Member

JonRowe commented Aug 21, 2019

The main rspec-core runner doesn't support parallelisation, so it has no means of collating results from workers, be they threads or processes. Process based parallelisation is simpler than thread based due to well, the GIL on MRI rubies and lack of 100% thread safety within the other gems. (I'm fairly certain for example that there are threaded bugs lurking within the mocks code).

My ideal plan for taking us to parallelisation is to use the "fork" model that the bisect runner uses, maybe with a fall back to a "shell" model (essentially processes) but I haven't had the time to work much on it.

@krtschmr
Copy link

to summarize: regardless if thread or processes: rspec-core isn't supporting it yet, so if we run a rails 6.0 application and use rspec, we won't benefit from it. correct?

@JonRowe
Copy link
Member

JonRowe commented Aug 25, 2019

The helpers are there to support parallelisation, rspec on its own doesn't support parallelisation so you'll need an extension gem to benefit from it.

@p8
Copy link
Contributor

p8 commented Sep 5, 2019

My ideal plan for taking us to parallelisation is to use the "fork" model that the bisect runner uses, maybe with a fall back to a "shell" model (essentially processes) but I haven't had the time to work much on it.

@JonRowe Would you copy all bisect code and renaming the namespace to parallel and make the required changes?

@JonRowe
Copy link
Member

JonRowe commented Sep 5, 2019

No I'm writing it from first principles using the bisect runner as a guide and some other external gems.

@wwahammy
Copy link

wwahammy commented Nov 8, 2019

@JonRowe thanks for taking this on! Is there anything we can do to help this along? Not sure I'm qualified to write any of the runner but if there's something, please let us know!

@courtsimas
Copy link

Hey guys! What's the latest on this?

Copy link
Member

JonRowe commented Jan 23, 2020

Still awaiting finishing the core work

@pirj
Copy link
Member

pirj commented Feb 5, 2020

Since Support Multithread Execution ticket on rspec-core doesn't seem to be a good fit, let me do a brain dump of tools that allow for parallelization of RSpec suite execution here:

All or most of those tools allow for parallelized execution in processes, arranging work with several databases via more or less common TEST_ENV_NUMBER env var.

With such a variety of tools, it seems that parallel testing of Rails applications with RSpec is not really an unsolved problem.
On my last project, we were running a dozen machines each running eight RSpec processes, that allowed to reduce spec suite wall clock run time 50x.
It's not by the way constrained by the usage of Rails 6, works equally in Rails 4.2+.

Basing on the efforts that had to be spent on building and maintaining such a tool while working full-time, I wouldn't really expect that a comparable tool to appear out of thin air in RSpec distribution and to immediately become better than everything else on the market.

@jtoy
Copy link

jtoy commented Mar 11, 2020

this will be awesome with rails 6+

@LandonSchropp
Copy link

LandonSchropp commented May 7, 2020

@pirj I recently attempted to improve our CI testing using some of the tools you mentioned above, and I didn't have much luck. I didn't test all of them, but I did test the ones that were the most popular, and none really seemed to meet what we're looking for.

In an ideal world, a solution for this would do the following:

  • If multiple cores are available, automatically parallelize the specs. This would have to be configurable.
  • Automatically aggregate the results into a single, readable output stream using the provided formatter.
  • Continue to work with coverage tools like SimpleCov.

My understanding of the Rails feature is that it gets us more of the way there. This still feels like an unsolved problem to me for Rspec. I believe Jest does something like this at the file level in JavaScript. It would be awesome to see.

@pirj
Copy link
Member

pirj commented May 7, 2020

@LandonSchropp I have bad news for you if you really need to aggregate the results into a single standard RSpec report. It's a hard task with a number of non-obvious edge cases. We've solved it on my previous project, at least for those cases that we've faced. The report is gathered from different workers, spread over different processes or even machines. Unfortunately, it's not open-sourced yet.
If you're using a JUnit-compatible XML-based reporter, combining the results in the end for CI to consume shouldn't be a big deal though.

As to my memory, we had no big issues with aggregating SimpleCov results, but that worked before I joined and I didn't work on this part.

Auto-scaling to multiple available cores is a trivial task. Once you (depending on your system flavour and CPU type) get the number of processors, you're able to run rake db:test:create and rspec by prefixing them with a TEST_ENV_NUMBER, and you're done. Some prefer leaving one core, but you may play around and run 2x more parallel processes than you have cores available. It depends on your tests what will yield the best results. Typically this is all done it a 20 LoC shell script.

Another non-trivial task is to evenly distribute your specs across those parallel processes. If you don't, you risk that one of the processes will be running for a significant amount of time, while other cores will finish already and will be idle. It is solved in Knapsack Pro and AFAIR in rrrspec.
The key is to keep the execution time of the specs, and feed specs to RSpec runner dynamically, one by one. Most time-consuming specs changed specs since the last run, and newly introduced specs are run first to reduce the unevenness of the distribution.
Frankly, I don't have any proof that this queuing system is faster than pre-divided execution on a pretty large suite.
Don't divide the spec into individual examples if you're using let_it_be, or some other form of before(:context) with a transaction rollback. The expensive setup will multiply in this case.

My understanding of the Rails feature is that it gets us more of the way there.

Honestly, I'm not so sure about that. A major flaw I could spot is that Rails doesn't detect if a worker died and did not finish all the workload it was supposed to execute. A better way is to have a timeout and consider worker as dead and spread over its remaining workload among alive workers. This is also solved in my previous project.

@LandonSchropp
Copy link

LandonSchropp commented May 8, 2020

@pirj Thanks for the detailed reply!

Sorry if my post sounded like I was trivializing the amount of work involved in this. That definitely wasn't my intention. My goal in my last message was more to enumerate my team's use case in the hopes that it could influence the future design of this feature. I'm sure that a refactor along these lines is a massive undertaking.

As to my memory, we had no big issues with aggregating SimpleCov results, but that worked before I joined and I didn't work on this part.

I'll have to take a second look at that. When I was investigating, it wasn't clear to me how I'd go about that. I found one plugin for integrating SimpleCov with parallel-tests, but it said it was only compatible with CircleCI, and we're using Codeship, so I didn't explore it further. If you have any hints you could share, I'd appreciate it.

Auto-scaling to multiple available cores is a trivial task. Once you (depending on your system flavour and CPU type) get the number of processors, you're able to run rake db:test:create and rspec by prefixing them with a TEST_ENV_NUMBER, and you're done.

Yep, that's what we're currently doing. In my laptop, I've got 8 cores, each with hyperthreading, so that spins up 15 processes to run on. With that many simultaneous tests running, you can see how managing output would be helpful. I hadn't looked into aggregating JUnit XML compatible results. Do you have an example of something like that with Rspec you could point me to? Maybe that would help solve some of our problems. Our CI processes have four cores available, so it's somewhat less of a concern there.

Frankly, I don't have any proof that this queuing system is faster than pre-divided execution on a pretty large suite.

That's roughly my experience as well. We have a few specs in our application are that are slower than others, but for the most part the costs of individual specs are amortized over our files. Based on my observations of using Jest, it looks like it breaks the spec files into a queue, and then runs each file as a worker becomes available. In our use case, pre-splitting the specs has worked fine.

When I was exploring Knapsack, I actually felt like it was trying to solve a different problem than what we were experiencing. As you said, for us, it was less about distribution and more about taking advantage of multiple cores locally and in our CI environment.

A major flaw I could spot is that Rails doesn't detect if a worker died and did not finish all the workload it was supposed to execute. A better way is to have a timeout and consider worker as dead and spread over its remaining workload among alive workers. This is also solved in my previous project.

I can't speak to that. I'm sure you're absolutely correct, and I agree with you that the behavior you're describing sounds better than what's being done by the Rails runner.

What I meant more from my comment is that what's advertised from the Rails runner is more what we're after. It advertises that it automatically detects the number of cores on the machine, creates the test databases for us, and then splits up our tests and runs them. We love Rspec and will continue to use it over Minitest, but it would be awesome if there were an analog to the parallel features with Rspec that was as easy to use as the Rails solution.

Thanks again for your detailed reply. I really appreciate it!

@rabidaudio
Copy link

I can't speak to the rest of it, but as for merging JUnit reports, I've done that before in a different context. The file format is pretty simple, it would just be a matter of a little Nokogiri code to open each report and output a bigger report with the contents of each. Just have each rspec process output in JUnit format and then handle the aggregation and printing of results yourself

@SamSaffron
Copy link

@JonRowe Discourse would be happy to fund some work here if you or any other contributors are looking to build it.

Some requirements from me :)

  • Dedicated gem, plug-and-play into your existing rspec rails app. Must be drop in replacement for Discourse.

  • Fork per testing process

  • Pull vs push model. (our bin/turbo_rspec is still the old push model, it means that some workers can be doing nothing while other workers have backlogs of work)... instead similar to @tmm1's test_queue master process has a queue of tests and child processes just ask it for the next test to run via an IO.pipe or some other cheap transport.

  • Non interleaved output like Discourse's bin/turbo_rspec

  • Long term also offers discourse bin/rake autospec which is basically a interruptible guard.

If anyone want to work on this and would like the work funded, DM me via Twitter. @SamSaffron

@JonRowe
Copy link
Member

JonRowe commented May 18, 2020

My plan is to build support for parallelism into rspec-core, which solves many of the problems with plug'n'play, both as our own option, and the ability to farm work out to other plugins. Fork is absolutely on those planes. I very much like the idea of the master rspec process controlling the queue like that, as it allows much easier non interleaved output, which was very much essential.

From there rspec-rails would than integrate the Rails system test helpers for their database support etc

I hadn't thought about the possibility of such a model being a long running process, and reloading / rerunning specs, thats got a set of complications all on its own (as I'm sure you know!)

@thbar
Copy link

thbar commented May 18, 2020

@JonRowe it must be noted too that parallelism in Rspec will also give a strong acceleration to all the "TDD devops" tools which happen to be based on RSpec (like InSpec and ServerSpec). Just a side-dish, but a very nice one, and definitely useful to the reputation of Ruby as a whole ^_^. Given that, maybe Chef could also be willing to provide some funding in that area (I'm not working there, but just making a wild guess!).

@wwahammy
Copy link

@JonRowe @SamSaffron My company, CommitChange, would be able to add a small amount to the parallel testing pot. It'd be well under $1000 (we just don't have the resources to spend a lot more) but it's something if it'd help. I totally get if this isn't enough to bother but there's likely many smaller users who might be able to put in a few hundred.

@SamSaffron
Copy link

thanks everyone, I have been chatting privately to @JonRowe and it looks like Jon may have some time for this project, stay tuned 🎊

@pirj
Copy link
Member

pirj commented Jan 21, 2022

this year

No promises. This depends on contributors mostly. RSpec core team has limited resources and is working on releasing support for Rails 7, trying to get RSpec 4 out of the door, and a number of housekeeping tasks.

I encourage you to take a stab at it, and report how far you went through, what went wrong. We can adjust RSpec Core/Mocks as needed to support in-process concurrency.

@bmulholland
Copy link

bmulholland commented Jan 21, 2022

I think where this was left off is that the result of "take a stab at it" is https://github.com/ioquatix/turbo_test/, and so next steps for anyone who wants parallel test support (me included :) ) is to implement that gem and report any issues. Is that right?

If turbo_test gets enough adoption and validation, would that approach then be folded into rspec?

@pirj
Copy link
Member

pirj commented Jan 21, 2022

Checking Rails testing guide:

forking processes is the default method

Process forking approach has been available for running RSpec in parallel in the local development for a long time already. Check #2104 (comment) for a list. There are numerous blog posts how to set it up, including for Rails.

This shifts the "inherent deficiency, obviously a downside" argument into "is not available out of the box, needs additional setup". And, from my perspective, shifts its priority a bit down.

Should we embrace and absorb, bundle or reinvent one of those tools?
It comes with a burden to support it.

Should we work on supporting in-process concurrency (threads, fibers, ractors, ... whatever comes next)?
Basically, the same answer as above.

Should we give our preference and advertise one of the tools in our documentation, optionally provide it as part of our rails generate rspec:install template?
As you've seen, turbo_test development was very active at some point, and then suddenly stopped. This happened to other tools, too. We have no control over that.

I would be happy to see a streamlined approach, but it's unlikely to happen without outstanding contributors' dedication to develop and spread the word. The tool should really stand out, as the currently existing ones work in most cases, too.

@supairish
Copy link

@griou We run our rspec/capybara test suite using https://github.com/grosser/parallel_tests It works well, you just have to make some different config adjustments for things that run outside of rails during the test suite run, i.e. capybara ports, database selection (per process), elasticsearch indices (per process), redis, etc... See the wiki for examples https://github.com/grosser/parallel_tests/wiki

@Unknown-Guy
Copy link

Unknown-Guy commented Jan 24, 2022

@pirj I think I have to add another perspective here. Been around Rails projects since 2.3 was latest and greatest. Up until now for every new project we went with RSpec. But now starting on a new project there is really big incentive to switch - it's included out of the box, automatic and will be supported.

So it is not only missing feature, but becoming kind of branding issue.

In-process concurrency would be great, but maybe first step is to just provide up to date list of existing tool and provide some options in install template. The template option might be bridge to other tool owners to iron out some integration issues and simply drive interest to them.

@ilyazub
Copy link

ilyazub commented Jan 24, 2022

https://github.com/serpapi/turbo_tests author here. Shameless plug, but it's available as a gem and works as a replacement for the parallel_tests. We use it for 1.5 years in CI and in development every day.

PS. All I did is copy-pasted part of the source code from Rubygems and Discourse, adjusted it a bit, and packaged it into a gem (#2104 (comment)).

@bmulholland
Copy link

FWIW, https://github.com/serpapi/turbo_tests worked out of the box (added gem, ran bundle exec turbo_tests) and halved my dev machine test suite runtime. Everything passed first time (admittedly a very simple test suite).

By contrast, https://github.com/ioquatix/turbo_test required a bunch of setup steps (ioquatix/turbo_test#6) and is now throwing an error when I try it again.

@pirj
Copy link
Member

pirj commented Jan 26, 2022

copy-pasted part of the source code from ... Discourse

MIT vs GPLv2 😱
@SamSaffron What do you think?

Worked perfectly fine as a drop-in replacement/wrapper for parallel_tests. Good job, @ilyazub !

Before:

bundle exec rspec spec/models/  646.73s user 251.24s system 86% cpu 17:14.30 total

After:

bundle exec turbo_tests spec/models  966.81s user 339.88s system 216% cpu 10:04.89 total

What was missing (and what Rails provides with its default testing framework):

Active Record automatically handles creating a database

Without createdb myproject2 etc it didn't work. I've seen rails db:test:prepare, but db:create wasn't apparently called. I blame this on parallel_tests.

Spreading out tests evenly, dynamically (with a queue), and considering individual specs' CPU/IO ratio is out of comparison and is on par with Rails.

@jkeen
Copy link

jkeen commented Jan 26, 2022

FWIW, https://github.com/serpapi/turbo_tests worked out of the box (added gem, ran bundle exec turbo_tests) and halved my dev machine test suite runtime.

Same! Been following this thread since 2019 and this week it paid off—thanks @ilyazub!

@ilyazub
Copy link

ilyazub commented Jan 28, 2022

@bmulholland @pirj @jkeen Thanks for your kind words!


copy-pasted part of the source code from ... Discourse

MIT vs GPLv2 scream @SamSaffron What do you think?

I guess I can change the license from MIT to GPLv2 and add copyright headers if needed. I've used the MIT license because that's what bundle gem turbo_tests provided.


What was missing (and what Rails provides with its default testing framework):

Active Record automatically handles creating a database

Without createdb myproject2 etc it didn't work. I've seen rails db:test:prepare, but db:create wasn't apparently called. I blame this on parallel_tests.

Yes, parallel_tests requires manual invocation of rake parallel:setup. But I've incorrectly written in the gem description about how the parallel_tests gem is used. Please don't blame parallel_tests for that :-)

@pirj Thank you for mentioning the automatic DB creation in Rails. I've opened an issue in the turbo_tests repo about an automatic setup of the database.


Btw, ActiveRecord::TestCase uses DRb for IPC. At some point, I'd like to try using DRb in turbo_tests.

@ArturT
Copy link

ArturT commented Jan 28, 2022

I guess I can change the license from MIT to GPLv2 if needed. I've used the MIT license because that's what bundle gem turbo_tests provided.

Most likely you have to use GPLv2. Only users of your gem might have some problem if they don't check the license is GPLv2.
This reminds me of last year mess in the Rails community https://news.ycombinator.com/item?id=26566925

@pirj
Copy link
Member

pirj commented Jan 29, 2022

users of your gem might have some problem if they don't check the license is GPLv2

Only if they decide to supply a modified version of turbo_tests as an integral part of their proprietary software to another party. To my best knowledge, GPL does not apply any restrictions on using GPL-licensed software for CI or local development, commercial or otherwise, modified or not.

The Rails community incident is the reason why I mention GPL vs MIT above:

GPL header removed.
The license that you're shipping mimemagic under (MIT) isn't compatible with shared-mime-info's.

@SamSaffron
Copy link

SamSaffron commented Jan 29, 2022 via email

@pirj
Copy link
Member

pirj commented Jan 29, 2022

Thanks a lot, @SamSaffron ! ❤️

@ilyazub I suppose you can consider this written permission to use parts of GPL-licensed Discourse code in your MIT-licensed turbo_tests.

10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.

@ioquatix
Copy link

Great discussion, I'm really excited to see people passionate about testing!

Just reading through some of the comments here, it seems like we need a better model for parallel testing. From what I see, I think turbo_test (my hack/implementation) has the right approach by separating out the worker setup into an explicit step, given that serpapi/turbo_tests#18 seems to be exactly the same issue. What we put into that worker setup depends on your app, db setup, and Rails.

Regarding parallelism, I think there are enough issues in Rails test harness that some improvements there would be useful: better handling of TEST_ENV_NUMBER would be an obvious one, along with better db migrations using a proper interface rather than shelling out to rake. RSpec also has a ton of global state which makes it hard to work with (think hacks to reset global state after each worker runs a test), so there is also work to be done here.

To this end, I decided to try implementing a green-field parallel test runner and ended up with https://github.com/ioquatix/sus which I'm now using in a handful of projects. It's designed from the ground up for multi-thread parallelism by ensuring all tests can be run in total isolation (i.e. the semantic model is built from the ground up for isolation). Interestingly enough, I've been toying with an idea to host other test frameworks within sus, i.e. hosting Minitest/RSpec tests within the sus parallel runner.

Maybe rspec-core can follow a similar model to sus but a large amount of rspec would need to be rewritten, essentially the entire output/notification system (it's unsafe/hard to parallelise/global state) and the test statistics capture (currently also seems like global state). @JonRowe correct me if I'm wrong please. Separation between running a set of tests and the test model itself would also be really useful. sus exposes tests as an hierarchy of callable objects and you can execute them any way you want, so this gives a lot of power for different execution models - multi-thread - multi-process - multi-host - etc. Tests have (guaranteed) consistent unique identifiers which you can transit across those boundaries (i.e. hosts) and reload in a target process which ensures that these models all work as expected (doesn't depend on load order etc).

After playing around in this space for a while, I think what we want is this:

  1. A way to load a callable hierarchy of tests with unique identifiers from a specific test framework e.g. sus, minitest, rspec, etc.
  2. A command to execute this callable hierarchy and report failures.

sus unfortunately combines both of these things into a single gem, but it's been on my todo list to explore having a clean separation between these two, and write an adapter for rspec. If we adopt this approach generally, it should be far easier to define a clean model for multi-thread/process/host parallelism that works generally across a wide range of test definitions.

@Unknown-Guy
Copy link

Shelling out on MRI is not that bad. Did a quick test in our project with turbo_test - jRuby and Open3.popen3 is not the best friends in this case:

bin/turbo_tests spec/models  2460.21s user 748.40s system 356% cpu 15:00.20 total

vs

bin/rspec spec/models  661.43s user 33.36s system 70% cpu 16:31.66 total

@ilyazub
Copy link

ilyazub commented Jul 4, 2022

@Unknown-Guy Yes, a one-minute speed-up could be better. How many *_spec.rb files do you have?

@Unknown-Guy
Copy link

Unknown-Guy commented Jul 5, 2022

@ilyazub there's 277 files there, 4425 examples

Edit: think the startup is what eats the time here, shelling out and starting another jruby process is very expensive. Maybe this could work together with something like Theine.

Just wanted to add that that the approach with shelling out has pretty big drawback on jRuby.

@ryanong
Copy link

ryanong commented Dec 3, 2022

Looks like a new tool for rspec parallel testing https://github.com/briandunn/flatware

@ilyazub
Copy link

ilyazub commented Dec 5, 2022

Looks like a new tool for rspec parallel testing https://github.com/briandunn/flatware

@ryanong Thank you for suggesting it! It uses drb and I love the flatware's code structure and quality.

Unfortunately, it's slower than turbo_tests (and now I'm not sure about re-writing it using drb).

$ time bundle exec flatware rspec -w 8 ./spec/search/google/
real    6m7.343s
user    0m10.354s
sys     0m1.710s

$ time bundle exec turbo_tests ./spec/search/google/
real    4m56.069s
user    23m9.252s
sys     0m11.208s

Tested with Ruby 2.7.2 on 8 core i7-6700HQ CPU @ 2.60GHz.

$ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

There're 196 spec files (2,1K examples) in the directory I've used to compare the speed.

find ./spec/search/google/ -name '*_spec.rb' -type f | wc -l
196

find ./spec/search/google/ -name '*_spec.rb' -type f -print0 | xargs -0 grep -o 'it ' | wc -l
2149

P.S. Maybe it's possible to parallel execution of examples instead of spec files using flatware 🤔 💡

@mvz
Copy link
Contributor

mvz commented Oct 16, 2023

For me, flatware was faster than turbo_tests, so best to benchmark this I think.

@jdelStrother
Copy link
Contributor

jdelStrother commented Nov 13, 2023

Not very rigorous, but FWIW on my M1 MBP (8-core, 32GB), running our Rails project's 4260 examples across 526 spec files:

tool times
non-parallel rspec 194.53 real 137.02 user 27.78 sys
parallel_tests 60.51 real 209.87 user 47.46 sys
turbo_test 60.26 real 201.67 user 44.91 sys
flatware 55.53 real 183.73 user 35.53 sys

I ran each a handful of times and reported the best time (which is admittedly unfairly biased in favour of parallel_tests, which would often have by far the worst of the parallel runners' times due to poorly balanced specs)

flatware does seem consistently faster for me, although it has a few rough edges with things like not being able to interrupt the tests with ctrl-c, the extra complications of a flatware-sink background process, and an unexplained single test failure I haven't figured out the cause of yet.

@briandunn
Copy link

@jdelStrother Just released a new version - would love to address these concerns could you open an issue?

@jdelStrother
Copy link
Contributor

@briandunn seems pretty good now! It behaves properly with ctrl-c, and I don't seem to be seeing the random test failure from last time I tried.

@aronwolf90
Copy link

My concern is, that with all the tools one need to execute a different command than bundle exec rspec, this is not very intuitive for new commers to a project that uses rspec. Is there a solution to hidde the parallelization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests