Memory Leak [Payment Reward]

Discussion in 'Systems Administration' started by wydgabriel, Jul 20, 2018.

  1. Hey Spigot,

    I have a game server that gets 250 - 350 online daily. The server works well, without lag (TPS stands between 19.3 - 20.0) buuut i think I'm going through a memory leak.
    I give for the server 2g in start flag and 60g for the maximum.
    When the server starts he get 5 - 8 g and every min the ram increase a bit. After 3hour the allocated ram are in 45-50g and after 3-4hours the ram gets the max and the servers crashs, so, Ive put an automatically restart for every 3hrs to the server don't crashs but I need find my Memory Leak problem.
    I already have head dumped it but i really have no idea how to analyze it.. if someone can analyze help me.

    Currently timings: https://timings.aikar.co/?id=54ab064f43a547f187d3651f4a8d34f0
    Info Dedicated: 64g I7 7700K OC

    Thanks for all.
     
    #1 wydgabriel, Jul 20, 2018
    Last edited: Jul 20, 2018
  2. Is it just one server or multiple connected via a network? If it's a single server, 60 GB is WAY TOO MUCH to be allocating to a single minecraft server. Lower it to around 8 - 10 GB and see if your issues persist.
     
  3. Yes, this dedicated is only for this server, for the bungee and others I use other dedicateds.
    I already have tested with 2g min and 8g max but server have still crashed. Basically, I just put the max so highest cuz I get more time until server crashs.
     
  4. Holy that's a rapid increase...
    Any custom plugins? That'd be the first place to look.
    Does the issue also occur when running the same files without players? If so, copy them over and try removing plugins one by one until you know which one it is.

    Other things which you might want to try would be to update any plugins as well as your server jar.
     
  5. Yes, I have some custom plugins hahah ;-;. If I remove one by one I'll take decades..
    As I sayed, my timings don't accuse some things, I need know if its possible to find the exactly thing that is causing the issue in the heap dump analyze.
     
  6. Timings dont give a fuck about memory, so it makes sense that that wont get you anywhere.
    Dont know how hard heap dumps are to analyze, but as you only have ~30 plugins loaded by the looks of it, I'd just go with the tried and tested method of disabling one by one.
    I have yet to see anything that analyzes memory usage per plugin, so I dont know if that data is even present anywhere or in a dump, might waste a lot of time trying to make sense of something that holds nothing of value.

    EDIT:
    Apparently there are a lot of tools that analyze/parse heap dumps, give them a try I guess, even oracle has one.
    https://docs.oracle.com/javase/7/docs/technotes/tools/share/jhat.html
     
  7. Then don't remove them one by one. Split them in half, if the problem persists, then you've narrowed the possibilities by a lot. If it doesn't persist, then it's the other half. Keep splitting until you find the culprit. Of course, you might need to keep dependencies such as Vault or ProtocolLib.
     
    • Like Like x 1
    • Agree Agree x 1
  8. Firstly, don't allocate a server a maximum of 60Gb of Ram..that's a terrible idea, lower that to around 6-12gb depending on usage. The server will never need that much but by allowing it to take that much is going to let it build up much more rapidly than it should, lower that and your primary problem should go away... aside from that, start with custom plugins then go to premium plugins from spigot, etc.
     
  9. You also might want to try using a profiler such as VisualVM.
     
  10. Thanks, I'll try it!
    Thanks I'll try!
    I already have tried to use visualvm, the problem is, I don't have any ideia how to use it correctly :/
     
  11. Thats a thread dump, that just tells you what all the threads were doing at the time of the crash, 99% sure thats useless.
     
  12. Weaves

    Resource Staff

    I use VisualVM. Select your server in the VM tree on the left side. Then go to the profiler tab. Select memory and watch for classes that keep going up and up as you play but then dont occasionally drop back down.
     
  13. Ill try it, thank u!
     
  14. How has no one mentioned this?

    It's RocketPvP causing the majority of the lag.

    Remove the plugin and I am more than sure that lag will disappear.

    If it's a custom plugin, ask the developer you hired to fix it. If you have the source and the developer won't/can't fix it, hire someone from here to fix it.
     
    • Like Like x 1
  15. Shopguiplus is causing more lag, and even then, rocketpvp is using just 3-6% of cpu time
    He is looking for a memory leak
    . The high cpu time could well have a link with cpu usage due to trying to write and read a lot in memory, or looking up a lot of things in giant ever expanding maps/lists, but it does not have to be.
    The memory leak could just as well be any other plugin with less usage just writing and writing away data in other ways, like for example a big linked lists, which takes next to no time to expand if you have the head/tail saved. Or a stack. Or nested objects.

    Unless youve found a magic way to read ram consumption per plugin from timings? In that case Id love to know how and swallow my words :)
     
  16. These guys are scammers, don't do business with them.

    Server is rocketmc.com.br

    I spent 2 days straight helping them fix their issue, staying up til 4am one night. Figured the problem out, walked them through how to fix it, they kept adding things on that I had to do before I got paid. Pulled them up on it, got them to pay me for fixing the original issue, and now because I wasn't at my computer when they messaged me some vague error message, they're demanding their money back saying I'm not willing to keep helping them.

    Honestly I've been doing this for a number of years and have never met a group of people so immature running a 300 player network.
     
    • Like Like x 1
  17. That really dose suck to hear. Make sure you keep the money even if they charge back. You did your job.

    Would you mind posting what the issue was for this post? So others can see it if they run into the same thing, for the future?
     
  18. Unclosed database connections in their custom plugins.
     
  19. Thanks for posting the fix. Sounds like a rookie mistake. Good job! :)