1.4 Lag - Reasons/Analysis/Discussion

Discussion in 'Spigot Discussion' started by wow4201, Nov 25, 2012.

  1. wow4201

    Patron

    1.4 Lag - Reasons/Analysis/Discussion

    Alright I run a 500 slot server Uberminecraft.com.
    In 1.3.x the server held 500 players with a constant 20 tps.
    (I reprogrammed every single plugin the server uses to remove/optimize cpu intensive functions)

    In 1.4 initially it held about 100 players.
    *I just want to note 1.4 was absolutely broken with bad packet id client errors.
    After analyzing and optimizing everything I got it to around 230 players.

    So what is new in 1.4 that has absolutely destroyed performance.
    Looking at my nolagg report, My plugins are absolutely optimized (even better then 1.3)
    The chunk loading again I got it optimized better then (1.3)(Kept all chunks loaded in memory).

    So my plugins are performing better then 1.3 and chunk loading/writing is performing better then 1.3. According to the NoLagg analyzer 1.4 should be running faster and allow more players.
    I've used both spigot/own custom modified craftbukkit core/Vanilla in 1.4 they all perform poorly.

    So what has been added in 1.4 that could be causing this lag.
    This is something that should be discussed so efforts are put in the right direction.
    This lag is absolutely discouraging, I personally don't want to program a single new plugin or feature knowing it will make this lag problem worse.

    Everything I say below could be wrong, these are just changes I noticed while going through the code.

    Chunk Compression
    Every single chunk packet that is sent is compressed using deflation. Basically in simple terms every chunk packet sent to the client is "zipped" literally.
    Code (Text):
    I say this because included is java.util.zip.Deflater;
    This compression method is used in many different applications such as apache. Using compression like this reduces the bandwidth the server uses by 200%-%1000. However it increases the cpu usage by about 10x. The chunk compression on vanilla uses a separate thread off the main thread (I believe)
    Code (Text):
     static final ThreadLocal<Deflater> localDeflater = new ThreadLocal<Deflater>() {
            @Override
            protected Deflater initialValue() {
                // Don't use higher compression level, slows things down too much
                return new Deflater(0); <-----0 = No compression 1-9 (1 = Less compr/ 9 = More compr)
            }
    No Lagg plugin has the ability to compress the packets on multiple threads (Defaulted to 2).
    Conclusion: Although Chunk compression is CPU intensive, It has the ability to be disabled, Runs off the main thread in vanilla craftbukkit (I think), and can be ran on multiple threads using nolagg plugin.

    Extension: So what are the benefits of serving lower bandwidth to your players? Obviously they receive the chunks in game significantly faster. Perhaps it takes load off the server because each download thread has less time being used (But does it matter if its off the main thread).

    Compression Conclusion: After trying every single variation of Disabling/Multithreading chunk compression, I did not notice any truly significant (maybe 10% increase). However while testing last night I noticed putting compression to 9 it caused the the core that the main thread was running to hit 100%. So I feel like maybe this compression does interfere with the core the main thread is running on. While disabling the compression doesn't "Fix" the 1.4 lag, it does significantly reduce lag compared to running the compression on "9".


    Packet56MapChunkBulk vs Packet51MapChunk

    1.3.x - Using Packet51MapChunk in EntityPlayer.java
    Code (Text):
                if (!arraylist.isEmpty()) {
                    for (Object object : arraylist) {
                        this.netServerHandler.sendPacket(new Packet51MapChunk((Chunk) object, true, 0xffff));
                    }

    1.4 - Using Packet56MapChunkBulk in EntityPlayer.java

    Code (Text):
                if (!arraylist.isEmpty()) {
                    this.netServerHandler.sendPacket(new Packet56MapChunkBulk(arraylist));
     
    Alright so what is Packet56MapChunkBulk when did it arrive and why are we using it over Packet51MapChunk?

    Packet56MapChunkBulk was coded but not implemented in 1.3.x Craftbukkit code. So this was something that has been planned.

    Packet56MapChunkBulk: obviously is used to send more chunk data in a single packet, because the packet is extremely large they decided to compress it. So what does this mean? The Client player no longer notices chunks of blocks missing, or no longer waits as long to receive their full view distance. This makes the client player use more CPU to decompress the chunks. The server spends more resources to send a larger amount of chunk data (and compressing it).

    This idea is great, for servers who have very few players on their server. More chunk data arriving much faster to the client.

    This idea is terrible for players without low end machines (They receive a CPU hit due to the chunk decompressing). Also I feel it puts an extremely heavier load on the server sending these massive packets out.

    These are just the observations I've made, I know in theory Packet56MapChunkBulk and compression/decompression could probably be used without seeing a performance hit using great multi-threading (Maybe java just isn't good with threads like C++ would be). Perhaps threads are being left open.

    Alright so why not switch back to the old method of sending Packet51MapChunk instead of Packet56MapChunkBulk. I did just that, while it works, there was a bug where all signs in the game had no text written on them (Possibly an easy fix? If you know how to fix this please do so). I feel like finding a method of reverting back to Packet51MapChunk should be found, just to measure the difference it would make.


    Personally I feel like there is some serious performance issues within the 1.4 logic. I'm asking the community to come together and discover what has changed in the 1.4 core that could be crippling the performance.

    Things to discuss.

    -New functions that are in the craftbukkit server core that could be affecting performance.
    -New client logic that could be affecting server performance (Perhaps the client->server packets)
    (Perhaps the client is sending packets that have to be decompressed in the server)
    -Why Packet56MapChunkBulk/Compression may be affecting main thread performance.
    -If possible how to revert back to using Packet51MapChunk w/o compression(1.3.x).

    -Developing/Using tools or changes in the code to monitor what craftbukkit code is affecting the main thread or causing lag.

    I want to thank the spigot community and Md_5 specifically (Got me started in making performance adjustments to the craftbukkit core).

    I'm Also looking forward to the Multi-threaded chunk loading. Seems interesting, my initial feeling is that it will increase performance but on paper I don't know if it would be a whole lot especially with these 1.4 changes causing severe performance drops. With that said I think the code behind it and the idea is simply brilliant and long over due.

    So let us developers and server owners come together and find solutions to these changes that have been made. I feel like mojang is going one direction making their product extremely optimized for tiny servers but crippled for large 200+ player servers.

    Not analyzing or finding solutions to these problems is hurting us all. So we need to come together on this one.


    Also on a side note:
    I think the most important feature that Spigot absolutely understands but craftbukkit does not, Is the ability to have variables that can be changed in a .config file. Instead of having to make changes to the craftbukkit core and recompile.

    For instance the compression value is hard coded. However that value makes drastic changes and should be configurable without recompiling.
     
    #1 wow4201, Nov 25, 2012
    Last edited: Nov 25, 2012
    • Informative Informative x 2
    • Like Like x 1
  2. ammar2

    ammar2 Retired Developer
    Retired

    First off, CraftBukkit compresses chunks on the network write thread for the player, it used to be that it would all be compressed by a single thread before, leading to bottlenecking. At this point map chunk bulk is used for all chunks loading and map chunk is used for chunk updates and chunk unloading, the only way to use map chunk to send full worlds involves an incredibly hacky hack which could burst apart at any update, so reverting to map chunk is a no no. Moving on to your questions:


    No such packets exist


    It doesn't, to a noticeable amount.

    Map chunk packets have been compressed for almost forever, vanilla did it on the main, craftbukkit used to do it all on one thread and now does it on the network writer thread
     
    • Informative Informative x 1
  3. wow4201

    Patron

    Any input on the following?

    Would you agree 1.4 has made changes that are effecting the server performance?
     
  4. ammar2

    ammar2 Retired Developer
    Retired

    I haven't looked into it too much but 1.4 did make quite a lot of changes, I don't have time to profile both independently but when I do I'll let you know what change caused it
     
  5. Note the compression on writer thread was a 1.4.4 change by CB team.

    That change lowers lag on the main thread (As it simply moves where the compression is done) but can cause some noticeable client side lag issues on teleport/login as chunks are sent.

    but once the chunks are sent all is well.
     
  6. md_5

    Administrator Developer

    Moved this to Discussion since developer ramblings isn't the place for this, and I permission fail let you post it.
     
  7. This is some good reading. So this is the cause of all the TPS drops i have been reading and i myself have been having on my server?

    What worries me is whats 1.5 going to bring when the new minecraft API comes out..... Oh the chaos!!
     
  8. I thought I was the only one with 1.4.x lag.
    Glad to hear it isn't just me!
     
    • Agree Agree x 1
  9. lol
     
  10. Usually the best approach to reduce lag: reduce view-distance in your config!
     
  11. This can reduce some lag, but he's talking about the lag that has been occurring since the switch from 1.3 to 1.4.
     
  12. My view distance is already at 5 :/