Server freeze after reached full memory

Discussion in 'Performance Tweaking' started by Anderson66, Sep 28, 2019.

  1. Now, after did /spark heapdump I have a bigger problem. Very big. When I start the server with xms set to 4GB the server on idle with 0 players take 12GB RAM! I noticed that the ram grows very fast from 4 to 12GB when the worlds are loading. I deleted plugin Multiverse-Core and problem dissapeared but this plugin is most important for me. I dont know why this happened. Before /spark heapdump everything was OK and server on idle take only XMS value (4GB) (I checked it by htop on ubuntu). Now server takes 12GB on idle and turns on about 20secs! Before spark heapdump it was take to turn on about 9 - 11 secs..
     
  2. That's not at all what it is saying. You don't want the two values to be equal to each other as it causes issues with Minecraft Garbage collecting. Having the ram equal to each other causes more problems then if they aren't equal. Your server's XMS will automatically Dynamically assign itself based on the amount of ram your server NEEDS at a given time.
     
  3. In that case you ARE agreeing with me!
    But, once a server exceeds the Xmx value the Garbage collector will go nuts!
     
  4. Can you please just stop arguing about whether Xms should be Xmx or if it should be lower? Thanks.

    I suggest you start your server for testing with -Xms1G -Xmx4G with all plugins and players jpin and worlds loading and when your server gets close to 4 GB ram, use /spark heapdump. (If you have ssh access, you can zip the heap dump on your vserver)

    I could try analysing the heap dump for you.
    Put it heapdump into a ZIP (if not already done; the zip should be much smaller than 4GB), and then upload it for example to google drive and send the link.
     
  5. We are NOT arguing.
    Don't be off-put by my current signature picture, it is a default provided by the image supplier to say that they want me to re-register my chosen image.
    upload_2019-10-13_3-32-14.png
     
    #25 Goldentoenail, Oct 7, 2019
    Last edited: Oct 13, 2019
  6. Thanks, the problem that server on idle takes 12GB RAM dissapeared after a lot of restart. Now its OK - it takes only the xms value (4GB). But I still want to analyze my 11GB heap dump. Could someone do this for me? I know I am asking for a lot. But I'm sure there is an answer what causes memory leak. Maybe someone is interested which plugin takes so much memory...
     
  7. The problem is that I only have 8 gb ram and no (v)server with more.

    But if you say that the memory leak is gone now without any plugin change, the lag could just be that you had too much ram, causing java to let it run full and then having to do huge garbage collection phases (ram cleaning) which can cause much lag
     
  8. But the problem is that, I used to have a VPS (2 weeks ago) with only 16GB RAM and xmx set to 14GB. Then I had many long lags (10 - 20 secs without respond) when the memory reached 14GB (20 players average). I decided to buy a 32GB VPS and problem is gone but the memory is still about 15 - 20 GB with 20 players. I'm worried that when I will have a 40 players the memory again reach 27 GB (my xmx) and lags appear again.. So I think to upgrade it to 64GB but you are saying that isn't a good idea.. .

    Perhaps there is a relationship between VPS memory and XMX? Maybe it will work better with XMX set to 14GB on 32GB VPS than XMX set to 14GB on 16GB VPS?
     
  9. I think just zip and upload that heapdump and some people might be more willing to analyze it for you.
     
  10. Just set your xmx to a sane value like 4GB and when it fills up and starts to freeze/lag take a heapdump. Then analyze that smaller heapdump.
    You shouldn't need that much memory for 30 players, that's why everyone is telling you to use less. There is no point in just increasing memory over and over while ignoring the problem that is causing so much use.
     
    • Agree Agree x 1
  11. I will upload this big heap dump later. I will decrease the memory too. But now I have a different problem. The memory is OK - still below 10GB but the CPU usage is still 100% on one core. A few days ago the CPU Usage wasnt a problem - now it is.

    My timings (21 players):

    https://timings.aikar.co/?id=115c74e260c549f9941d65380ca5aa98

    Maybe my VPS provider do to much overselling? Can I check it?
     
  12. Not sure if you've tried this yet, but make a backup of the server and try using PaperSpigot instead (unofficial fork of Spigot - not affiliated with Spigot or Bukkit/CurseForge). Paper claims to increase performance and keep the TPS maintained, and should help memory.

    EDIT: yes, Paper is literally a drag-and-drop replacement for the server jar
     
  13. Of course I am using it. Just look at the timings in last post
     
  14. A timings report longer than 2 minutes would be much more acceptable.

    And 60% of your cpu is being used for chunks, chunk ticks, and chuck entities.
     
  15. I decided that I buy a dedicated server. Here is specs:

    128GB SSD
    Intel Xeon E3-1265L v3 4c/8t 2.50-3.70GHz
    RAM: 4 x 8GB Hynix DDR3 UDIMM 1600MHz ECC

    So far, everything is OK. Today, I have 30 players and no lags.
     
  16. In summary :-
    You have reduced -Xms and -Xmx
    You have removed a defective plugin
    You have upgraded to a more performant Linux server

    All of which have provided improvements to your performance along the way.

    To further fine tune your Linux server base, you may wish to try using a plugin which reports the server usage of Memory for each 'Game' server and the CPU usage.
    This will help you to analyze, and advise you on more optimal Memory settings for the each 'Game' server.

    Your new server has 32GB of RAM. Lets say that Linux uses 2GB of that for the OS, you have 30GB of available memory left.
    Therefore, you should be able to host 6 'Game' servers on there, each using an -Xmx of 5GB (Each holding 50 to 80 players).
    (Provided that the CPU and Internet Connection can handle the throughput of data)
     
    #36 Goldentoenail, Oct 9, 2019
    Last edited: Oct 9, 2019
  17. Whaaaat? XMX set to 5GB and 50 - 80 players on 1.14.4? I do not believe. I thought it was possible in 2012/2013 but not on 1.14.4.. Now my server on IDLE takes 1 - 3GB RAM (with one plugin - Multiverse) and when I have 30 players the server takes 10 - 20GB (with few plugins). I think I can't fix this so I came to terms with it
     
  18. OK, Two days ago I set XMX to 16GB (previous was 27GB). I decided to do the test. I wanted to generate a little more terrain so I did /wb fill (plugin WorldBoard). While the terrain was being generated, I watched the memory (the physical memory in htop on Ubuntu). The memory still growing and growing. When it reached 16GB, the memory was released to 4 - 5 GB and start growing again without any lags! It's OK for me. But today, when I was normally playing with 20 - 30 players, after 2 - 3h the memory reached 16GB and... the server started crashing and freezing. I had an errors like "The server has not responding for 20 - 30 secs". The memory wasnt released... still 16GB... Why?
     
  19. I thought we had explained this, but you were not listening very close.
    Set your Xms to 5G
    Your Xmx is currently set to 16GB, right, when the garbage collector comes along, he has a HUGE refuse bin to empty (16GB - 5GB = 11GB).

    Lets say that garbage is only emptied when the bin is full. (For simplicity)
    This HUGE refuse bin takes time to empty (This is done on the main CPU thread), meaning that almost everything stops whilst the garbage is taken away!

    The smaller the bin the faster it can be emptied, and the sooner things can return back to normal.
    Just to pluck a random number out of the air which will have less affect upon the server, lets say your Xms is 5GB, and you add 25% and round it up to the next GB, this will give you an Xmx of 7GB
    Set your Xmx to 7GB

    That will allow the refuse men to use a 2GB refuse bin. (Xmx - Xms = garbage size)
    Meaning that the garbage will be emptied over 5 times more often, and more than 5 times faster, when compared to an 11GB bin.
     
  20. I changed my XMX to 10GB. Tomorrow I will see what will happen. I'm very curious that "Allocated Memory" can decrease at all.. Becouse, so far when Allocated Memory reached XMX, then just stops - no decreasing, no increasing (but only freezing server)