How do you test your plugins?

Discussion in 'BungeeCord Plugin Development' started by BeastsMC, Jun 21, 2015.

  1. I've never been happy with how I test my plugins. It usually involves actually logging in with a client and interacting with the plugin, but this is slow and can't be automatically done. I've always wanted to be able to write unit tests and do regression testing without having to manually go through a ton of steps in game. The main issue I've faced is that the plugins do operations on data structures of Minecraft, which in turn depend on even more Minecraft data structures to the point where an entire real server needs to be running for everything to be setup to run a test, and I've never found a good way to do this.

    So, how do you all do your testing? Do any of your write actual test cases, or do you do manual tests in game?
     
  2. clip

    Benefactor

    I usually test in game also. I own a few accounts so when the plugin requires multiple players I just open multiple clients. After my tests are successful locally I usually deploy the update to testplugins.com and wait for bug reports from users that login there. I also have a select few people that are usually available to test on their own if I need to get a second opinion or extra testing done.
     
    • Like Like x 1
  3. Sounds like a pretty good method as far as in game testing goes. Still has some short comings, such as making it difficult to do regression testing, but better than what I currently do.
     
  4. I suppose I'm lucky in that I have a 20 man staff team for my server as well as a development server to test all my stuff on. I do all my tests there.
     
  5. I'm big on unit testing, if you'd like I can help you get started.

    Here's an example abstract class I've written to help test a BungeeCord plugin, this plugin has 74% code coverage. :)

    There are very few times I have to go in game to test something, even with world modification plugins I maybe go ingame 5x max over the course of developing the plugin. Unit testing is actually helpful for productivity reasons too, I find myself far less distracted.

    If you need to use NMS classes regularly and they depend on other NMS classes then mocking them is surely the way to go, though if you mock any class then it's worth noting you don't have to mock every method. :)
     
    #5 smrkn, Jun 23, 2015
    Last edited: Jun 23, 2015
    • Like Like x 3
    • Informative Informative x 1
    • Useful Useful x 1
  6. I log into a test server and extensively test everything that could've been affected by what I've changed recently.
    Most plugins aren't complex enough to justify unit testing.
     
  7. Unit testing needs to be justified? It won't hurt and it can be more efficient than logging in unless you need some kind of visual output, nothing wrong with it even for simple event listeners.
     
    • Agree Agree x 4
  8. It needs to be justified otherwise you could spend more time writing the tests than actually writing the plugin. You don't need a complex set of unit tests for a plugin that only listens to a few events and handles a few commands.
    If you refactor things, you have to update the unit tests as well, slowing down your development time as your code-base grows larger.
    So unit tests aren't free - they require time. Time to create, and time to maintain.
     
    • Agree Agree x 1
  9. It is also very wrong to believe that: Lots of unit test -> Strong software. Lots of unit tests can serve to confuse if they aren't consistently refactored and cleaned. A couple of unit tests to attend to core functionality is fine and all but any more in terms of a minecraft plugin is probably OTT. Most development is strictly error driven anyway.
     
  10. Code isn't free - it requires time. Time to create, and time to maintain. Plugins are coded, plugins are not free.

    Nobody said the tests had to be complex, you can write simple "is it working" tests for the scenario you described in minutes.

    Definitely a mistake if you believe the first thing, however if you have high code coverage and not just lots of tests then it's clear that your code does what you want it to do.

    Refactoring unit tests isn't really hard and doesn't take much time at all. I don't know about you but I'd be far more confident in a plugin if it had 80% code coverage because without reading any kind of reviews or comments about it I already know that it's stable, I wouldn't find that OTT.

    Test driven development is definitely something worth looking into.
     
    • Agree Agree x 3
  11. Apologies for the late post. To answer the initial question: we try to unit test our plugins as much as possible; though its easier said than done. As with SmirkingNinja, we use Mockito and PowerMock to quickly "provide" certain parts of CraftBukkit/Spigot. Once code is commited to the main repository, Jenkins & SonarQube take over to run final tests, quality checks, etc. Passing builds are deployed to a local or remote test server for actual in-game testing. Once blessed, updated plugin(s) is queued for installation on production servers. This could also be automated, but we prefer a "hands on" approach when it comes to those servers.

    And to answer the subtext of some of the other posts: Anyone that thinks they need justification for unit testing clearly hasn't worked on a project that involved more than themselves. If you are entering or in the field of software engineering, I strongly suggest you keep your hesitancy to unit testing as your dirty little secret - certainly, it wouldn't be a good opinion to bring up at an interview or design kick off meeting.

    That would be called Test-driven Development.
     
    #11 Frelling, Oct 17, 2015
    Last edited: Oct 17, 2015
    • Informative Informative x 1
    • Useful Useful x 1
  12. If you refactor things unit tests should work without you touch them because you tests the functionality. If you expand your Plugin then you have to change the Unit Tests because you have probably changed the way the code works.

    The positiv Factor of Unit Tests is, that you can tests small parts if they work properly. If you write a Plugin (whether it is big or small is not significant) and you do only integration tests you do know that it works or not, but if a Plugin does not work you have to find out which part is not working correctly. For this purpose you use Unit Tests.

    And the most important part is, that you can redo the Tests everytime you want without changes. So if you update your plugin, but you want to keep the functionality small code extensions can break the code. For this you use Unit Tests, too. Like i explained you can always tests the code with the same tests and same method you did previously.

    I do Unit Tests with all my Plugins. It is not important if it is only a Listener or a complex Plugin.
     
  13. I don't test plugins, I just put them randomly on spigot or send them to my customers. But shhh, don't tell anyone ;)
     
    • Funny Funny x 2
    • Winner Winner x 1
    • Friendly Friendly x 1
    • Optimistic Optimistic x 1
  14. i am just running an small mc server with 512MB on my VPS or one on my laptop for to debug them and testing
     
  15. I would be interested in seeing this example class as basis for setting up some integration testing. However it seems the link does not exist in Github anymore. Would it be possible to send an updated link to the example?
     
    • Agree Agree x 1
  16. Asphyxite

    Supporter

    I do manual tests on a localhost server (if spigot) or a localhost bungeecord instance connected to external spigot servers (to ensure I can still run all other programs as well as they need to). I prefer manual testing if I'm honest, and I enjoy it somewhat, It's really rewarding when you test and find no issues whatsoever, or if there is and you fix, then there's still you having overcome another obstacle towards plugin completion.

    I haven't personally tried automated testing so I can't really say which is better or worse, but I can say it's my personal preference to be hands-on when it comes to bug testing, as I do enjoy it.
     
    • Like Like x 1