Writing Tests for Plugins

[angus]

Rspec goal: 80% Coverage

I’d like to keep tackling rspec tests until we can reach at least 80% coverage on each of our respective plugins. On the coverage % front, Discourse has a useful gem called “simplecov” which will calculate the percentage of your code that is covered by tests based on what lines are touched in the course of a test suite. I’ve found that you can restrict SimpleCov to a plugin by adding the following to your spec/plugin_helper.rb file:

SimpleCov.configure do
  add_filter do |src|
    src.filename !~ /discourse-custom-wizard/ ||
    src.filename =~ /spec/
  end
end

Once this is added you run your plugin test suite like this to get a coverage report for your plugin:

bundle exec rake plugin:spec[plugin_name] COVERAGE=1

You’ll then see an additional line of output at the end of the tests:

Coverage report generated for RSpec to /Users/angusmcleod/discourse/discourse/coverage. 589 / 1213 LOC (48.56%) covered.

If you look at the coverage report, it’ll tell you exactly what lines of code are not being tested.

I got a head start on this today by writing up rspec and qunit tests for the multilingual plugin I’m building for Wikimedia.

Add tests and hardening

Some tips

1. You should require the spec/rails_helper at the top of each spec file

   require 'rails_helper'
   

   This will load all the necessary stuff for rspec testing, including whatever you put in plugin/spec/plugin_helper.rb.

   This means plugin_helper.rb is a good spot to put rspec global config. For example I wrapped all tests in a code block to ensure that non-local language data was properly loaded for each test, see: discourse-multilingual/plugin_helper.rb at main · paviliondev/discourse-multilingual · GitHub

2. It’s easier to run qunit tests in the browser directly, than via the command line, e.g.

   http://localhost:3000/qunit?qunit_skip_core=1&qunit_single_plugin=discourse-multilingual
   

3. For those who already have a .travis.yml file, e.g. events and tlp, this probably needs to be updated in line with the instructions here: (Superseded) Set up plugin continuous integration tests on Travis CI - developers - Discourse Meta.

4. You may find that data persists between tests, particularly data in the PluginStore. To ensure this doesn’t happen you can add this to your plugin_helper.rb file

RSpec.configure do |config|
  config.around(:each) do |example|
    ActiveRecord::Base.transaction do
      example.run
      raise ActiveRecord::Rollback
    end
  end
end

[merefield]

Why might I be getting 0/0 for my coverage report?

merefield@scw-40efa1:~/discourse$ RAILS_ENV=test bundle exec rake plugin:spec[discourse-follow] COVERAGE=1
LOAD_PLUGINS=1 /home/merefield/.rbenv/versions/2.6.3/bin/ruby -S rspec ./plugins/discourse-follow/spec/lib/updater_spec.rb --profile

Randomized with seed 35452
...

Top 3 slowest examples (0.32995 seconds, 34.2% of total time):
  Follow::Updater expect users followers to include follower
    0.13899 seconds ./plugins/discourse-follow/spec/lib/updater_spec.rb:8
  Follow::Updater expect users followers to include multiple followers
    0.1113 seconds ./plugins/discourse-follow/spec/lib/updater_spec.rb:14
  Follow::Updater sent a notification
    0.07965 seconds ./plugins/discourse-follow/spec/lib/updater_spec.rb:22

Finished in 0.96455 seconds (files took 9.02 seconds to load)
3 examples, 0 failures

Randomized with seed 35452

Coverage report generated for RSpec to /home/merefield/discourse/coverage. 0 / 0 LOC (100.0%) covered.

Branch here:

https://github.com/paviliondev/discourse-follow/blob/add_tests/spec/lib/updater_spec.rb

[angus]

You shouldn’t be running it in the test env. Use

bundle exec rake "plugin:spec[discourse-follow]" COVERAGE=1

[merefield]

Bizarrely that makes no difference, still getting:

Coverage report generated for RSpec to /home/merefield/discourse/coverage. 0 / 0 LOC (100.0%) covered.

[angus]

The issue seems to be that simplecov isn’t detecting your tests (i.e. 0 Lines)

Perhaps you’ve added a filter in your plugin_helper.rb that’s filtering out all the plugin paths?

add_filter do |src|
    src.filename !~ /discourse-custom-wizard/ ||
    src.filename =~ /spec/ ||
    src.filename =~ /db/ ||
    src.filename =~ /api/ ## API features are currently experimental
end

[merefield]

Something fishy with that pattern.

If instead I use:

SimpleCov.configure do
  add_filter do |src|
    src.filename =~ /discourse-follow/ &&
    src.filename =~ /spec/
  end
end

I get:

Coverage report generated for RSpec to /home/merefield/discourse/coverage. 341 / 1371 LOC (24.87%) covered

Which seems about right to me?

[angus]

This

src.filename =~ /discourse-follow/ &&
src.filename =~ /spec/

means filter out any file that includes BOTH discourse-follow and spec in its path. So it’s a bit broad, as nothing really will be filtered, apart from the follow specs. It will include files outside of the plugin. If you then look at the coverage report itself (it will appear as an index.html in a “coverage” folder in the root discourse foloder), we can see the actual files it’s covering

Your previous version

add_filter do |src|
  src.filename !~ /discourse-follow/ ||
  src.filename =~ /spec/
end

Actually works for me (I pulled your tests branch)

I’m getting a 64% figure.

Now if we look at the coverage report index.html it’s only covering the plugin files

[merefield]

Super odd. I’ll take another look.

On the plus side: getting close

[merefield]

OK, i’m getting the same problem with two other plugins (topic-previews and custom-wizard) so it’s something to do with my dev setup …

[merefield]

@angus This is driving me nuts.

• I’ve rolled back to match your version of Ruby (2.6.1).
• I’m on latest master

Still I get 0/0 for any plugin coverage.

[angus]

Just for posterity, Rob and I had a 1.5hr call to try and debug this, but still no joy. If anyone has any experience with a similar simplecov issue, any tips would be most appreciated.

@merefield As we discussed, let’s move on for now, but it might be worth posting on meta too.