Ram leaking, LagSpikes, Crashes 1.4.5

Discussion in 'Performance Tweaking' started by Rojoss, Dec 13, 2012.

  1. I'm having a lot of issues with my server for almost 2 months now.
    Normally i never ask for help for running the server but i'm really getting crazy from it now.
    I will first explain what's going on etc.

    First of all it started after 1.4 i think.
    There where a lot lag spikes and really huge lag spikes that completely froze the server.
    After weeks of debugging and removing plugins and switching maps etc etc
    We found out that it was save-interval what created the lag spikes.
    We set it to 0 and the lag spikes where gone.
    then i added a script that runs /save-all every 2 minutes.
    Also tried more delay but that didn't improve any performance.

    But then a new problem started.
    Huge ram leaking and crashes like every 3<>8 hours.
    This is still happening now and i think it's only getting worse some times it's not even 2 hours on.
    I have already disbled each plugin 1 by 1 and switched the world to a default world.
    None of this solved the ram leaking.
    I have been using different dev builds from normal cb.
    Yesterday i have installed spigot with the garbage collector.
    But it didn't solve anything either.
    Still a lot crashes and ram leaking away what also cause tps lag.

    We started with 50+ players on average before the issues started.
    Now we have lost a bunch and have around 10 online.
    It's making me go crazy i don't really care about the player amount but i do care about their playing experience on our server.
    I have tried a lot and have been debugging for weeks/months now without any good results.

    I hope someone is having this same issues or had the same issues.
    And that someone can tell me what i can do to solve the issues.
    And i'm not sure if all plugins i use are updated to new api so i prefer only builds that where before the api change.


    Some server info:
    ram: 3gb (Tried with 8gb but that didn't make it any better)
    players average: 10 now
    os: Linux 3.2.0-29-generic (amd64)
    java: 1.7.0_07
    average tps: (after start 18<>20, After a hour it drops and getting lower and lower.)
    worlds: 3
    - Main world (6k border radius)
    - Shop/Events world (1k border radius, Players can't build etc here.)
    - Collecting world ( 2,5k border radius, Players can just collect materials here.)
    Plugins: 61
    FWAbility : 1.0
    WorldBorder : 1.6.1
    PlugMan : 1.7.2
    FWfix : 0.1
    Vault : 1.2.21-b271
    LSoup : 1.2
    Multiverse-Core : 2.5-b641
    AutoMessage : 2.0.3
    PointShop : 1.9.9
    VariableTriggers : 1.2.1
    PlayerPoints : 1.522
    ProtocolLib : 1.8.0
    FWFaction : 0.5
    FWDeath : 0.1
    Votifier : 1.9
    FWSignRank : 1.0.1
    LagMeter : 1.11.1
    Citizens : 2.0.4-SNAPSHOT (build 591)
    Jobs : 2.8.5
    HideStream : 2.5
    GoldIsMoney : 2.0.2
    WorldEdit : 1301-d8636ec
    FWDonate : 1.0
    NoCheatPlus : 3.8.4-RC-b271
    FWBank : 1.0
    TagAPI : 1.8
    LogBlockQuestioner : 0.02
    PermissionsEx : 1.19.3
    TerrainControl : 2.3.4
    LogBlock : 1.60
    FWGlitch : 1.0
    Quester : 0.6.2
    HelperBot : 2.0
    FWCore : 1.0
    WorldGuard : ${project.version}
    FWEmpire : 1.0
    ImgMap : 0.1A
    ReportRTS : 1.0.0-b110
    BuyRegion : 1.1.5
    CraftBookCommon : 1406-3b8168b
    BKCommonLib : 1.38
    FWPunish : 1.0
    FWChar : 1.0
    FWChat : 1.0
    Essentials : Pre2.9.6.1
    Factions : 1.7.9_beta
    Trade : 0.4.2
    CraftBookMechanisms : 1406-3b8168b
    Multiverse-Portals : 2.5-b658
    Enjin Minecraft Plugin : 2.4.0-bukkit
    MagicSpells : 2.11
    Denizen : 0.7
    DisguiseCraft : 4.1
    Sentry : 1.2.3
    FWInv : 1.0
    CraftBookCircuits : 1406-3b8168b
    MobArena : 0.94.4.84
    XBP : 1.9.2
    Blacksmith : 1.1
    EssentialsSpawn : Pre2.9.6.1
    MagicSpellsCastModifiers: 1.2

    If you need any other info just ask for it please.
    I really hope someone can help me try resolve the issues.
     
    • Agree Agree x 1
  2. Try to enable timings, and keep an eye out. You can do this by editing bukkit.yml to set plugin-profiling to true. After some time (preferably after you start noticing things take a dump), do "/timings merged" without the quotes in game. It will tell you how much CPU time each plugin (and specific hooks) are using. This might not help much, as you mentioned about ram leaking, but give it a try to see if anything unusual pops up.

    Beyond that, 3gb _might_ be too little for 50 concurrent players, but since you've tried 8gb, I suppose we can temporarily rule that one out for now.
     
  3. PhanaticD

    Patron

    thats a ton of plugins remove all the ones you can do without, update the rest and see how it works

    also: post bukkit.yml server.properties and a worldguard report

    and exact system specs, and is it a VPS or dedicated
     
  4. Hi
    This isn´t really good solution because /save-all does NOTHING.
    The reason of crashes is that your server without /save-on never unloads chunks..

    Try to get the number of loaded chunks after a few hours after restart.
    And after XX minutes get another report of loaded chunks and I think that you´ll see that the number of chunks is only increasing.

    I am experiencing the same problem. My server ´s performace is really bad with /save-on (even on server running in RAMdisk). I tried /save-off and run /save-all every XX minutes and then I found out that /save-all with /save-off does really nothing.


    If there is anyone who can solve that problem I´ll be glad.
     
  5. Try what I said, enable timings, and look at your plugins.
     
  6. I dont have problems with memory leaking etc...
    I am experiencing 1s lag each time server is saved with /save-on
     
  7. Try using newest Spigot build #325. It works really well on my server.
     
  8. But there are those changes that will break my plugins, right?
     
  9. You can find builds with that change reverted on my page: www.murocraft.fi/tiedostot/Spigot
    There's also pom.xml so you can compile plugin friendly Spigot with it if you want to or do not trust me. Just clone Spigot repo and replace pom.xml with that one, commit it and build. Then you're good to go.
     
    • Like Like x 1
  10. It´s all about these line?
     
  11. If you are running into lag, there is something causing it to slow down. If you do not think plugins are involved, then the only other real likely source of problem I can think of is insufficient RAM and running into SWAP. What setup are you on, game service providers (i.e.: "I buy a 10 slot server from you"), VPS (i.e.: "I'm getting root access for 2GB guaranteed ram"), dedicated server (i.e.: "I'd like to have root access on this dual Xeon 5450, 16G ram, with 2x500G hd"), or something else all together?
     
  12. Tonigh I´m gonna try Spigot and do some larger update so I´ll report later.
     
    1. Chop useless plugins.
    2. Only use plugins coded by devs with PERFORMANCE in mind.
    3. Upgrade hardware.
    4. Manage expectations.
     
    • Agree Agree x 2
  13. Get NoLagg, go into your server, and type in /nolagg monitor.
    Give me a snapshot of that monitor in-game after it running for 1-5 seconds, just so it can fully start picking up data.
     
  14. /save-all causes lag.
     
  15. for like a micro second, its unnoticeable.
     
  16. Depending on how many changes were made actually.
     
  17. Anyway, on my server, we save-all every 25m, and it creates no lag at all, from 20-200 online, or from 5000 chunks to 25000 chunks loaded. (Xeon CPU)