[GUIDE] Optimizing Spigot - Remove lag, Fix tps & Improve performance!

Discussion in 'Performance Tweaking' started by frash23, Jun 22, 2014.

Thread Status:
Not open for further replies.
  1. For the love of god, please stop altering the GC parameters if you have no idea what they do, java can do a better job at that than you will ever be able to do.
     
  2. Schaumnificent

    Supporter

    Yeash srry mr hypixel dev expert. I'll stop doing everything and take your incredibly useful advice. Thanks
     
    • Funny Funny x 1
    • Like Like x 1
  3. Untrue. Java is designed to be tuned for each application that runs under it. I actually have done the research and I'm the main one supplying the java parameters.

    If you can provide scientific data with evidence on my research being wrong, I would love to see it, but my servers running with optimal GC activity proves that my research has done a lot of good.

    Not to mention that Java's default garbage collector is COMPLETELY WRONG FOR MINECRAFT

    Minecraft = Latency based need (CMS, G1GC)
    Default = Throughput based (Parallel)

    Meaning your ONLY options are CMS and G1 for GC. Never use default.
     
    #184 Aikar, Oct 9, 2015
    Last edited: Oct 9, 2015
    • Like Like x 1
    • Agree Agree x 1
  4. @frash23 I haven't gotten to the blog post yet, but here is my new recommendation on flags:

    Code (Text):
    java -Xms6G -Xmx6G -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=50 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:InitiatingHeapOccupancyPercent=10 -XX:G1MixedGCLiveThresholdPercent=50 -XX:+AggressiveOpts -jar paperspigot.jar
    If you are running with 10GB or less memory for MC, you should not adjust these parameters. (I use 10GB myself)

    -- brace for controversy! --
    1. -Xms matching -Xmx - Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx! You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you can't support if java uses it all.

      Now, that means if -Xms is lower than -Xmx - YOU HAVE UNUSED MEMORY! Unused memory is wasted memory. G1 (and probably even CMS to a certain threshhold, but I'm only stating what I'm sure about) operates better with the more memory its given. G1 adaptively chooses how much memory to give to each region to optimize pause time. If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1)

      The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted. With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.

      G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently. If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps.
    2. UnlockExperimentalVMOptions - needed for some of the others specified
    3. TargetSurvivorRatio: I'm sure your all use to seeing this one suggested. Good news! It's actually a good flag to use :D This setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen. The reason behind this is to be able to handle memory allocation spikes.

      However, MC allocation rate for most part is pretty steady (steadily high.....), and when its steady its safe to raise this value to avoid premature promotions.
    4. G1NewSize Percent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.

      With G1, things are better! You now can specify percentages of an overall desired range for the new generation.

      With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least!

      Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (BlockPosition)

      now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second!!!

      This is bad! You will have so many pauses that TPS has risk of suffering, and Spigot might be unable to catch up TPS with the cost of GC's.

      Then combine the fact that objects will now promote faster, resulting in your Old Gen growing faster.... This is bad and needs to be avoided.

      Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more effecient GC behavior.

      if you run with larger heaps (15GB+), you may want to lower the minimum to say 30%, but don't go lower than 30%. This will let G1 have more power in its own assumptions.
    5. InitiatingHeapOccupancyPercent/G1MixedGCLiveThresholdPercent: Controls when to include Mixed GC's in the Young GC collection, keeping OldGen tidy without doing a normal OldGen GC collection.

      On larger heaps(10GB+), you can raise InitiatingHeap to around 20 to reduce CPU usage, but I wouldn't go higher than that. And you also need to REDUCE the Maximum New Percentage to around 60. If you use 80% NewGen Max, you must keep this at 10 then. It doesn't hurt to leave it at 10, but "effeciency wise" you can improve it to 20 if you reduce your new gen size, but if you start seeing Old Gen GC, lower it back.

    Also for Large Pages - IT's even more important to use -Xms = -Xmx! Large Pages needs to have all of the memory specified for it or you could end up without the gains. That memory CAN NOT be used by the OS anyways, so let something use it!
    Additionally use these flags (Metaspace is Java8 Only, don't use it for Java7):
    Code (Text):
    -XX:+AlwaysPreTouch -XX:+UseLargePagesInMetaspace
    AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more.
     
    #185 Aikar, Oct 9, 2015
    Last edited: Oct 24, 2015
    • Like Like x 6
    • Useful Useful x 4
    • Informative Informative x 3
  5. Oh, i'm well aware that you totally know what you're doing, but it's more or less aimed at the general public, things like "-XX:+UseConcMarkSweepGC" can actually make the application run a lot slower.

    edit @Aikar "-Xms matching -Xmx"

    -Xms is allocated at startup and it will grow up to -Xmx, setting ms == mx effectively turns off this behavior. While this used to be a good idea in older JVMs, it is no longer the case. Growing and shrinking the heap allows the JVM to adapt to increases in pressure on memory yet reduce pause time by shrinking the heap when memory pressure is reduced. Sometimes this behavior doesn't give you the performance benefits you'd expect and in those cases it's best to set mx == ms. OOME is thrown when heap is more than 98% of time is spent collecting and the collections cannot recover more than 2% of that. If you are not at max heaps size then the JVM will simply grow so that you're beyond that boundaries. You cannot have an OutOfMemoryError on startup unless your heap hits the max heap size and meets the other conditions that define an OutOfMemoryError.
     
    #186 Chunkr, Oct 9, 2015
    Last edited: Oct 9, 2015
  6. The +UseConcMarkSweepGC is REQUIRED unless you use UseG1GC. it will not make things run slower.

    But, I personally suggest and prefer G1 over CMS with the flags I just provided.

    As for Xms and Xmx. Yes, for Other GC's thats what I said in my post. G1 operates completely different. G1 was DESIGNED for large heaps such as 64GB.

    If the memory isn't needed, it won't be touched. G1 is much smarter about that. That's why its able to handle 64GB+ heaps.

    note that a majority of java documentation is still pre G1GC, and G1 information is mostly found on G1 specific pages.
     
  7. Interesting, i'm going to give it a shot! :)
     
  8. IT's also important to note the 'behavior' change of G1 over CMS.

    On CMS, you will be accustomed to seeing your 'memory usage' stable. THIS WILL NOT BE THE CASE ON G1

    Memory usage will spike up and down constantly, and this is perfectly normal!!!

    See my memory:
    [​IMG]
    You'll see that my heap used is saw blade like. your goal is to keep the old generation consistent and stable.

    Made a video showing it:
     
    #189 Aikar, Oct 9, 2015
    Last edited: Oct 9, 2015
    • Like Like x 1
  9. Feeling special now, you just used my quote on your blog :D <3
    But btw. what do you think personally is better?
    G1GC or the CMS-GC?

    // Edit: Just saw your postings... xD
     
    #190 Wolfezz, Oct 9, 2015
    Last edited: Oct 9, 2015
  10. @Aikar I tested the latest GC parameters, they work pretty nice, still room for some extra testing but it's better than before, also what would you suggest changing about the tick limiter? Since it's a feature i sponsored.
     
  11. Absolute removal =P It's just a broken concept unless Mojang changes Entities/TileEntities to support the idea of skipping ticks. My article mentions numerous bugs on it. A first step would be to check EAR immunities, never skip those, and for anything that is skipped, to least call .inactiveTick() on them.

    But for TE's, no solution even exists yet. For example, there is a method tileentity.x() that is a flag for "Should be removed".
    Currently the tick limiter is going to skip ticking a tile entity that might be marked for removal, leaving it in the list longer than it was suppose to be... Also as mentioned in the article, many elements of tile entities do tick % someRate == 0. Skip that tick and now that TE skipped an entire interval.

    The system does nothing to improve performance, it just reduces how low the TPS can go based on how overloaded you are. But as mentioned, you should work on ensuring your server runs with a load it can handle, not replacing lag with inconsistent/buggy (Tile) Entity state that doesn't follow Vanilla semantics.
     
  12. It has worked for me in the past, since i haven't really found a good and proper way to limit the amount of entities on lets say a skyblock, or a survival server without severely limiting the spawning ratios, or hindering players, any suggestions?
     
  13. The best way is to limit Activation Range settings to the lowest possible that doesn't affect gameplay.

    Misc for example can be almost 0 even. Which makes items and arrows stuck in the ground almost free.
    Monsters can even reliably go lower than 16 IF you're ok with up to 1 second extra time for them to become aggressive.

    Animals, can target around 8 too. The Percentage on Average Entities on Timings represents your Activation Rate. It goes orange/red if your rate is high, meaning EAR is not making your entities go inactive as much.

    for ex this persons timings, http://timings.aikar.co/?url=12758325 while my color shows it orange, 64% isn't too bad. I need to re-adjust those color scales.

    they are using:
    Bringing monsters down could help more.

    These settings dont hinder spawning. just active entities.

    Tile Entities are the next big user. Hoppers being the main one. I have a change I use that makes hoppers a lot less laggy, I'll see if PaperSpigot has included it or not.

    Then with my removeAll fix, tile entities in general become much cheaper.

    Then you have reliable entities and much less lag.
     
  14. After all the discussions, can you update the flags?
     
  15. Finally unbanned, will get to updating the guide when I'm home from vacation (Friday).

    Thanks for the extremely informative input, @Aikar!
     
  16. Hey, is there any specific reason to why you dropped these flags:
    Code (Text):

    -XX:+UseConcMarkSweepGC
    -XX:+CMSIncrementalMode
    -XX:+CMSIncrementalPacing
    -XX:+CMSParallelRemarkEnabled
     
    You mentioned before that those are better for Minecraft but now you no longer recommend them?
     
  17. Because he used the G1GC instead of the CMS, and the last 3 flags are for the CMS, not for the G1GC (What means, that they won't work with it)
     
    #198 Wolfezz, Oct 15, 2015
    Last edited: Oct 15, 2015
    • Agree Agree x 1
  18. anyone know how to get mcmyadmin to run paperspigot ?
     
  19. Just rename the paperspigot.jar? :D
     
Thread Status:
Not open for further replies.