Strange issue with our server for around two months

Discussion in 'Server & Community Management' started by Keebs, Feb 11, 2013.

  1. I recently switched to Spigot around a month ago because we got out of memory errors TONS, on 2GB of ram server, the issue started out of the blue we always ran just fine before its just now its choking and crashing every other hour.

    We recently upgraded to an 8GB dedicated CentOS box, and after about 1 hour of game play it will kick all players for end of stream, and all the IP's lose connection and server is inaccessible for around 10-15 minutes, what I'm asking is why? Could it be a DDoS, and if so how can I block it?

    I start it and it automatically eats up 3GB of RAM, then slowly but surely eats the rest till players get disconnected, no errors from console, just severe lag then crash. I can reboot it after about 10 minutes, but if I don't it comes back in about 15-20 minutes by itself, no reboot just completely normal from before aside from around 15-30 /###.###.###.### lost connection in console, and each user disconnected for end of stream.

    We run 4 worlds and these plugins, and all are mostly recent worlds aside from our main which has been the same for around 8 months with diameter of 14000 blocks. (Current build of Spigot, 1.4.7 r1.1)
    Ships, TuxTwoLib, PVPGun, Multiinv, Worldedit, Essentials (spawn, core, chat), PEX, Factions, Citzens, SnoGro, AutoAnnouncer, LogBlock, LWC, Worldguard, Craftbook, AutoCraft, Vault, AdvIPLog, MCMMO, Dynmap, Chestshop, WorldBorder, Multihome, Multiverse-Core, Stargate, Votifier, Voxel Sniper and Spacebukkit (The issues were before SpaceBukkit, and continued after a change test)

    Any help is much appreciated, we have run out of options and may just shut down due to the stress and loss of player base. Thanks!
     
  2. To my (extremely) limited knowledge, sounds like a memory leak. You might want to post the actual spigot build you're using, not just server version, though I'd be much more inclined to suspect a plugin.

    Side note: Argh, PEX D:
     
  3. butbut pex is awesomesauce! Thanks for quick reply.
    Build is #514 (latest)
     
  4. [offtopic]I loved PEX too, till it broke things ;-; Didn't realize what was the matter until someone linked me this lovely doc.[/offtopic]

    I'd recommend disabling plugins methodically and see how the server performs, not sure what else to say D:
     
    • Disagree Disagree x 1
    • Informative Informative x 1
  5. I have :s. We removed plugins slowly, it just sorta prolonged the crash. Strange thing is when we got the new box, a world edit that would crash the old server immediately, with through seamlessly. With this new box it lasts 2 1/2 hours TOPS, then crash. Old box was around 30 minutes-1 hour. If this is a memory leak and not a DDoS like I assume (tons of threats, can't keep track), is there any logs for this?

    *EDIT* I read that doc, don't rustle my jimmies I like pex D: life as I know it is going away *EDIT*
     
  6. Reason I suspect a memory leak rather than a DDoS is that (again, from my limited knowledge) a DDoS shouldn't necessarily directly affect your server performance (you'd probably see loads of connections spamming console if it was, right?) and instead should prevent your players from joining, making the server appear offline

    Edit: I know there's a way to get a lovely performance readout from Spigot, but, for the life of me, I can't remember how to do it. Back in a couple minutes with an answer, time to research D:

    Edit: Righto. According to my best buddy Google, you'll want to get a report of timings.
    To do that, set "settings.plugin-profiling" in bukkit.yml to true. That allows you to use the /timings command and figure out how your plugins are performing.
     
    #6 Jikoo, Feb 11, 2013
    Last edited: Feb 11, 2013
  7. I do have connections spamming console, on the soft crash it does, and the server does appear offline and prevents players from joining, but then it comes back after around 15 minutes of I don't know, catching up with it self?

    Im looking into more about memory leaks right now and I want to see what some other people might know about this issue so I can get this fixed and we can stay open. Were going on a year and a half soon, so our players would be crushed. I have posted other places but this is my main since I am using Spigot currently and I love what it does with our server. Thanks for your help.
     
  8. YoFuzzy3

    Supporter

  9. md_5

    Administrator Developer

    I dont think that leak ever made it to release, only dev builds.
     
  10. We are using v1.70 of logblock, so I don't think thats it. Everything is updated around every two to three days, keeping the old jars just in case the new one doesn't get along with the current build.
     
  11. How do I use /timings, like what node would I use after I have choices from /timings reset|merged|seporate

    Sorry for double post, didnt see his edit.
     
  12. You might try running the Craftbukkit 1.4.7 RB and see how it compares to Spigot if you're pretty sure it isn't a plugin.
    /me banned for blasphemy
    It's just for reference, I swear.


    With that thought, I must sleep, so.. best of luck!

    Edit: Ah,
     
    • Optimistic Optimistic x 1
  13. I have also tried that :[ I see no change really, but I prefer Spigot because no errors in console about can't keep up, system time change even though its running on 8GB ram. Good night, thanks for the help!!

    Can anyone tell me how to use /timings? Its enabled currently on the server just not sure entirely on the functionality.
    What does this mean exactly? (The timings report)

    This is the names0.txt
    http://pastebin.com/4yGLycjv

    This is the timings.txt
    http://pastebin.com/f2SSzzEY
     
    #14 Keebs, Feb 11, 2013
    Last edited: Feb 11, 2013
  14. I really need to stop editing in while you're replying, but this post >.<
     
  15. YoFuzzy3

    Supporter

    I guess that explains why LogBlock was once causing my server to use 16GB of ram with 30 players on, lol.
     
  16. Do these readings seem okay? This was when no lag was on the server, when about 10-20 people get on after work and school I can post back with another and compare.

    http://aikar.co/timings.php?url=1629483
     
  17. Update your Multiverse core :)