STABLE and OPTIMIZED for speed Startup Script for 100 -> 200+ server

Discussion in 'Performance Tweaking' started by Meow, Dec 25, 2012.

  1. This topic will be used for trying to find a stable and optimized startup script for minecraft. Please copy and paste the startup script that you think fits the purpose of the topic and explain why it would work out well. Please also provide the following information for reference and comparison purposes:
    _ server specification & Operating system
    _ Shared or dedicated server
    _ number of players online at the same time
    _ plugins
    _ TPS.
    _ Which would this script favor? CPU or RAM?
    _ What's the advantage and disadvantage.

    Happy Holiday!
     
  2. There is really no one-size-fit-all scripts, as the java parameters needs to be tailored to servers. I'll share a few things I know off hand from my script, but I'd like to preface it with the fact that JVM does a very good job at profiling what your system can and cannot do, and will take advantage of things it can figure out very well by itself already. Whether or not adding additional java parameters will help is another issue up for debate.

    This here is my launch script:
    Code (Text):
    #! /bin/bash
    # Configuration Variables
    #server_jar="minecraft_server.jar"
    #server_jar="craftbukkit.jar"
    server_jar="spigot.jar"
     
    # Launch server
    # Removed parameters:
    #        -Xincgc \                      # Does not work (should not work) with -XX:+UseConcMarkSweepGC
    #        -Xnoclassgc \                  # Bukkit uses this and leaves dead RAM if disabled
    #        -XX:+UseCompressedStrings \    # Support removed in Java 7
    #        -XX:+UseAdaptiveSizePolicy    # it is incompatible with UseConcMarkSweepGC
     
     
    #        -Xmn1G -Xms2G -Xmx4G \
     
    rm server.log
    logfile="logs/"`eval date +%Y%m%d%H%M%S`".log"
    touch $logfile
    ln --symbolic $logfile server.log
     
    java \
            -d64 \
            -Xms4G -Xmx4G \
            -XX:PermSize=128m -XX:MaxPermSize=256m \
            -XX:NewRatio=3 -XX:+UseThreadPriorities \
            -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:SoftRefLRUPolicyMSPerMB=2048 \
            -XX:CMSInitiatingOccupancyFraction=90 \
            -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled \
            -XX:MaxGCPauseMillis=50 -XX:ParallelGCThreads=4 \
            -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking \
            -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 \
            -oss4M -ss4M -XX:UseSSE=4 -XX:+UseLargePages \
            -XX:+UseStringCache \
            -XX:+UseCompressedOops -XX:+OptimizeStringConcat \
            -XX:+UseFastAccessorMethods -XX:+AggressiveOpts \
            -jar $server_jar \
            nogui
    Fair bit of the things I've gotten from this post, so they may or may not be the most optimized.

    For my server, from what I fully understand:
    -d64 - forces Java to run in 64bit mode instead of 32 bit mode. This should only be used on 64bit systems, as there is no way for a 32bit system to run in 64bit mode. Typically, you wouldn't need this anyways, because if JVM detects you are on a 64bit system, it will run in 64bit mode.

    -Xms#G -Xmx#G -Xmn#G - sets the starting, maximum, and "young generation" heap size. I only use starting and maximum, and they are set the same, because I know that is enough for the slot counts I've set (30; theoretically, it should be workable up to 40 with 4G), and I'd rather JVM not try to allocate more memory on the fly. To understand these three parameters further, consider giving this page on JVM tuning a read.

    I don't fully understand the garbage collection details, but they work at getting rid of discarded data to free up resource for other stuff. The settings I use now are mixture from that thread I linked earlier, and another post somewhere else. I don't really know if it is fully optimized, but I'm not running into problems with them yet, so I'm happy for now.

    -XX:UseSSE=4 - Takes advantage of SSE form CPU. Check in /proc/cpuinfo for the highest level of SSE your cpu can support, and put it there. JVM _should_ be able to figure this out automatically as well, so you don't usually have to bother with setting it unless you have real explicit need for this to be set manually.

    -XX:UseLargePages - Like reading from harddrives, it is typically a little bit faster to read continuous data. UseLargePages tells Java to try to make large continuous pages so it reads faster. Your kernel must support this feature, or it will not work and you will get error message on launch. With this parameter, when launching, Java will try to get a section of memory at least the size of your Xms. However, be warned that on older kernels (i.e.: Debian 6.0.6 latest stable), there is no AnonLargePages support, so tweaking nr_hugepages in sysctl according to HuagePageSize in /proc/meminfo will result in OS pre-allocate the memory on boot, and reserve it for the specified group. This means nothing else on your server can use those memory. On newer kernels, AnonLargePages option is available so it is not pre-allocated.

    nogui - Actually a bukkit parameter. Tells bukkit to not spawn a gui window for managing server. This is not needed if you are running it on a linux headless server. I don't even know why I left it in there... probably because I'm too lazy to remove it...


    Edit: Forgot to mention about CPU vs RAM.
    In general, if you are not doing weird things, there is not much you can do in terms of making script favor one or another. They have very different purposes, and so there is not as much things you could do to tell it to go one way or another. The amount of difference you'll see from garbage collection is so minimal that for the most part, you should not really notice the difference. The biggest potential of "lag" for most server is either CPU too slow to keep up (too much entities, too much red stone, bad plugin, etc. etc.), causing control thread to slow down, OR harddisk too old and cannot handle all the write requests for saving chunk data. The only thing that you could use to help an old platter drive out is RAM disk. But that comes at expense of RAM as the name suggests. RAM disk is also a lot more volatile, so if you lose power or forget to copy the contents out, you lose your server forever. Be sure to take regular backups if you opt that route.
     
    #2 andyhuang, Dec 26, 2012
    Last edited: Dec 26, 2012
    • Informative Informative x 1
  3. PhanaticD

    Patron

    all those flags arent need at all. the only ones I use are the huge pages and ram allocation. the best with the ram allocation xms and xmx is to make them equal to each other.

    java -Xms16G -Xmx16G -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -jar craftbukkit.jar

    the real performance gains are in plugin configuration and spigot configurations and hardware
     
  4. Yep, that's what I prefaced the whole wall of text with. JVM will do a very good job on its own, and whether or not the options will do any good is up for debate. Since he asked about the parameters, I figured I'd share what I know :)
     
  5. Do you really have to specify the LargePageSizeInBytes?
     
  6. PhanaticD

    Patron

    probably not, i think 2 mb is default
     
  7. Do I need -server flag and could you explain what it does?
     
  8. Also, I'm using CentOS with 32GB Ram, how many GB should I leave for the system and for java?
     
  9. Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (errno = 12).
    Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (errno = 12).
    Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (errno = 12).
    This is what I got when I used flag -XX:+UseLargePages
     
  10. PhanaticD

    Patron

    im pretty sure -server is done automatically on 64 bit java

    id use somewhere from 20-28 if you arent using it for anything else

    that means its not supported by your system, which version of centos are you using?
     
    • Disagree Disagree x 1
  11. I recall reading somewhere that -server is done automatically for 64bit as well. My understanding, which I'm not particularly familiar with, is that the -server flag just tells Java to allocate higher percentage of heap for young/new generation. These are data Java just load into memory, and haven't had garbage collection executed on them yet. So, think newly loaded chunk, for example. The percentage takes away from old generation, which is the space Java puts things to after they survive garbage collection (i.e.: there's still player on this chunk, don't unload it). The idea is that things will change more frequently than not, so having more space available for the young generation heap will help things perform better. Though, I'd imagine Java doesn't explicitly need to follow this, and will handle things just fine for the most part by itself.

    PhanaticD made the exact same recommendation I'd give. Roughly 20 to 28G for java, on top of that, have 2GB set aside for MySQL (my.cnf settings, there are a lot of different settings to sum things up, use my-huge.cnf example as reference and tweak as needed), and leave the rest for your OS and other things (i.e.: map render is another big one people might have).

    Errno 12 means the hugepage is not setup properly, or there isn't enough space. For example, if you told your system to set aside 16GB for huegpage, but you are trying to allocate a Xmx of 24G. Run these on a sudo'er account with sudo or as root:

    Code (Text):

    sysctl -A | grep huge
    cat /proc/meminfo | grep Huage
     
    For the first one, check your vm.nr_hugepages, and for the second one, check for Hugepagesize. If vm.nr_hugepages * Huagepagesize is smaller than your Xmx, then you will want to increase your vm.nr_hugepages to match that. Keep in mind, for most computing science intends, we use MiB/GiB instead of MB/GB that harddrive vendors use. So, you'll need to divide by 1024 instead of 1000 for the unit conversion. You're starting with KiB (even though it says KB) since Hugepagesize is in KiB. You can change it by either editing it directly in /etc/sysctl.conf or a separate file in /etc/sysctl.d/. I usually put my in /etc/sysctl.d/hugepage.conf so I can find it quickly, but as long as it is ending in .conf, system should pick it up no problem.
     
  12. I'm using CentOS 6.3 64bit. And here is the output of those command for HugePages:

    sysctl -A | grep huge
    vm.nr_hugepages = 0
    vm.nr_hugepages_mempolicy = 0
    vm.hugetlb_shm_group = 0
    vm.hugepages_treat_as_movable = 0
    vm.nr_overcommit_hugepages = 0
    No output for cat /proc/meminfo | grep Huage
    I hope you can help go from here
     
  13. Blah, sorry, I tpyo'ed (narcissisticly used to typing my lastname, so instead of hugepage, I've typoed multiple times as huagepage).

    Need to check for Huge in meminfo: `cat /proc/meminfo | grep Huge`

    Check out my post here for how to setup hugepage, and the ramifications: http://www.spigotmc.org/threads/server-launch-sh.35/page-4#post-2407
     
  14. think you need to set a proper value for shmall as well. not only shmmax

    shmall would be atleast shmmax/page size
     
    #14 sirmyllox, Jan 12, 2013
    Last edited: Jan 12, 2013
  15. Yep, sorry I missed that. shmall is the total you want, so be sure to set it to at least the same for your shmmax. It is calculated with result from `getconf PAGE_SIZE` instead of `cat /proc/meminfo | grep Huge`.
     
  16. Thanks. I got it:

    AnonHugePages: 2883584 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0
    HugePages_Surp: 0
    Hugepagesize: 2048 kB

    What can I do now to properly set up HugePage? Thanks