Mystery of the Lagging Server

Discussion in 'Performance Tweaking' started by jflory7, Oct 5, 2013.

  1. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Hello Spigot forums,


    Thanks for taking the time to read this post; it's going to be a lengthy one.


    Before I get into the issue itself, I am going to provide some background information so as to help diagnose the problem I am having.



    Background


    I am a server owner of a semi-vanilla Spigot server, and our server runs on minimal plugins and we do not use any heavy or memory-intensive commands (to my knowledge); however, we do NOT have a world limit, and we have players who travel out very far distances (ranging from thousands to hundred of thousands).


    Additionally, my server CURRENTLY has, on average, about 15-30 people online at one time. I do expect this number to go up with time, as before we had so many technical difficulties, we had around 70 players on at our peak.


    Additionally, here is the full list of plugins that my server has, with links to the respective BukkitDev pages. Also, I have taken the time to make sure every plugin is running the latest STABLE version available.
    Hardware

    I am currently an Intel Xeon E3-1230 V2 server from Limestone Networks, and our specific server is located in Texas. The stats on this server are as follows:
    • Core: Intel Xeon E3-1230 V2
    • RAM: 32GB DDR3 ECC
      • This is not to say the server is using all of the RAM -- scroll down to Diagnostics
    • Hard Drive: 750GB SATA
    • Uplink Port Speed: 100mbps
    • Bandwidth: 5TB
    • Operating System: CentOS 6.4 (64-bit)
    My actual server is managed on this Limestone Network dedicated server using the Multicraftcontrol panel. Additionally, I am using the paid version.



    The Problem

    The problem I have been having on my server is countless and several players logging on and complaining about significant amounts of lag.


    The thing that really bothers me about this is that the people complaining about lag have capable computers and usually exceptional Internet speeds.


    The lag varies from being so terrible that they complain about not being able to hardly move, to times where there are just severe lag spikes. It doesn't make sense to me as to why this is happening, especially for the fact that when I used to rent a server from Minecraft hosting company EnviousHost, the people who are complaining about lag now did not have issues before the switch. This is what really confuses me.


    Additionally, the same players say that it seems to be on and off. There are brief times when they can play without lag, but more often than not, the lag renders playing on my server impossible.


    Extra Note: I've had numerous players say that the lag is only occurring in the overworld and NOT in the Nether or the End. I assume this is highly relevant, but I am not sure what to make of it.



    Diagnostics

    I have tried countless different things to try to optimize the server for better performance.
    Firstly, as mentioned earlier, I have made sure all my plugins are up-to-date as possible, and at that, they are all using the most recent STABLE versions of the plugins. I am not using any developmental builds for any plugins.


    Secondly, I had some people perform ping tests and traceroutes to the server, and for most of these players, they are receiving pings of less than 70s and their traceroutes have apparently been reliable. You can see the ping test and traceroute of one of the players of my server by clicking the respective link.


    Thirdly, after recommendations of the #spigot IRC people, I had the problem players download MRT, the tool for combining ping tests and traceroutes together to test for packet loss. I had two players run this tool, and you can see their results here and here.


    Finally, I had originally had my server set to use 24GB of memory on Multicraft, but after the recommendation of the #spigot IRC people about garbage collection, I lowered it to 8GB; however, this is having little improvement, if at all, for the people who are complaining about lag.



    Timings


    I took a timing of my server last night, and here are the results:
    Server IP


    Finally, if you would like to log on for yourself to see if you experience lag on the server, the IP for the server is play.crystalcraftmc.com. I'm not trying to advertise, but I figure it might be helpful for some of you if you want to log on and see if you are having issues.


    What confuses me is that someone from the #spigot IRC channel logged on from New Zealand and he said he experience 0% lag, which really threw me a curveball… anyways, hopefully that may help you diagnose the issue if you want to log on and see.



    Conclusion


    I hope I have included all of the relevant information to help diagnose whatever issues it may be that my server is having. None of this making too much sense to me as to why the server is causing lag for our players… hopefully, this can help yield some results.


    If you need to know ANY more details, please just ask! Anything that may help is worth a shot.
     
    #1 jflory7, Oct 5, 2013
    Last edited: Oct 6, 2013
    • Like Like x 1
  2. This is a mystery my friend, are all your plugins up to date? And are you sure it's not a network issue?
    Players on my previously server were complaining about lag when I was on 100mbpsbandwidth, i upgraded to 1Gbps uplink and all lag was gone. However can we get your spigot.yml file?

    EDIT: I joined your server to take a look and I experienced a very big FPS drop (from 700fps to 53fps) The world is loading very slow aswell. Not sure what can be causing this but i'm sure it's not client side.
     
  3. joehot200

    Supporter

    Install a plugin to test ping and see the ping of the players. It may or may not be an issue.

    For developers:
    If there is no plugins out here that do this, just cast the player to CraftPlayer, then use the getHandle() method, and then use .ping; to get the players ping (returns an integer).
    int ping = ((CraftPlayer)theplayer).getHandle().ping;
    theplayer.sendMessage(ChatColor.YELLOW + "Ping: " + ping);
     
  4. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    TheMcGuider Certainly. Here it is as follows:
    Code (Text):
    config-version: 3
    settings:
      prevent-proxies: false
      bungeecord-addresses:
      - 127.0.0.1
      bungeecord: true
      timeout-time: 60
      restart-on-crash: true
      restart-script: ./start.sh
      log-filters:
      - ^(.*)(/login)(.*)$
      netty-threads: 3
    commands:
      spam-exclusions:
      - /skill
      log: true
      tab-complete: true
    listeners:
    - port: default
      host: default
      netty: true
    messages:
      whitelist: You are not whitelisted on this server!
      unknown-command: Unknown command. Type "/help" for help.
      server-full: The server is full!
      outdated-client: Outdated client!
      outdated-server: Outdated server!
    world-settings:
      default:
        verbose: true
        anti-xray:
          enabled: true
          engine-mode: 2
          blocks:
          - 1
          - 14
          - 15
          - 16
          - 21
          - 29
          - 33
          - 48
          - 49
          - 54
          - 55
          - 56
          - 73
          - 74
          - 129
          - 130
          - 171
        mob-spawn-range: 4
        growth:
          cactus-modifier: 100
          cane-modifier: 100
          melon-modifier: 100
          mushroom-modifier: 100
          pumpkin-modifier: 100
          sapling-modifier: 100
          wheat-modifier: 100
        entity-activation-range:
          animals: 32
          monsters: 32
          misc: 16
        entity-tracking-range:
          players: 48
          animals: 48
          monsters: 48
          misc: 32
          other: 64
        ticks-per:
          hopper-transfer: 8
          hopper-check: 8
        random-light-updates: false
        save-structure-info: true
        view-distance: 10
        chunks-per-tick: 650
        merge-radius:
          exp: 3.0
          item: 2.5
        item-despawn-rate: 6000
     
  5. Everything is fine inside of the spigot.yml (just a tip, disable "tab-complete").
    Well I'm afraid that your overworld is corrupted. Are you able to create backup of your current world and make a new one to test if there is still a massive fps drop?
     
    • Agree Agree x 1
  6. Recolance

    Supporter

    This is a very organized forum. You are awesome.
     
    • Agree Agree x 4
    • Friendly Friendly x 1
  7. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    TheMcGuider: Oh, damn… what is it specifically that sets off the warning bells for you? Also, I am running a 1GB server on the same node using Spigot WITHOUT x-ray protection, and it runs beautifully.

    Recolance: I tried, hahah! I figured it wouldn't be benefitting anyone if I was like, "i hav lag spigot sux fix my sever plz".
     
    • Funny Funny x 1
  8. Perhaps it is lagging on server save-all? If so this is because it is saving all structures. They started saving all structure after 1.6.4. Try disabling sever structures in spigot.yml. This fixed lag for me.

    Code (Text):
    save-structure-info: false
    For warning: This is what will happen if you disable structures:
    Ender Eyes will not find Strongholds properly
    Witch huts will not spawn witches
    Nether Fortresses will not spawn Wither Skeletons (can be fixed by regenerating your nether in 1.7)
     

  9. This is exactly what I would recommend doing, this is a confirmed problem by md_5 and it can generate massive lag, assuming you save your world every few minutes. Luckily there is a easy fix, that only involves you changing the spigot.yml

    Change:
    Code (Text):
    save-structure-info: true
     
    to
     
    save-structure-info: false
    Hoped this helped, and best of luck for your server.
     
  10. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    eNkrypt Silverbrit
    Alright, I will go ahead and do this to try to optimize performance. Hopefully this will cut back on lag significantly. Fingers crossed! What will be problems be in 1.7 if I have this disabled?

    neilnet
    Thanks for the link; I'll definitely go check that out and follow its advice. Thanks for the heads-up!


    Additionally, I adjusted my x-ray protection from Engine Mode 2 to 1 and it made a significant difference. The only problem is that people can x-ray to bases now, but I suppose it's a consequence I'll have to live with.
     
  11. That's fine, I've hired so many secret moderators just going in vanish mode and catching x-rayers :) The manual way is the effective way.
     
  12. This will cause structures in 1.7 to no longer be structures.

    Many people are doing a map reset upon 1.7
     
    • Informative Informative x 1
    • Friendly Friendly x 1
  13. Puremin0rez

    Moderator

    Your thread is very informative and easy to read, but I found the title to be very annoying. I changed it to standard english characters - hope you don't mind :)
     
  14. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    The manual way is effective, only time-consuming. And finding reliable staff members seems so hard nowadays.


    And that's what I think I'm just going to have to do. I'll do a transfer of the worlds by using Multiverse to let players move things in-between worlds for a month or so.

    That's understandable -- I won't do that in the future. Sorry!
     
  15. Recolance

    Supporter

    I still can't get over the fact that someone actually put time into their forum like this. It's amazing, let me study up a bit maybe I can come back with an answer to fix your server's latency issues.
     
    • Friendly Friendly x 1
  16. Recolance

    Supporter

    jflory, I have come online to even attempt to experience the lag and I haven't found any latency issues at all. I'd say your server itself is running quite a good performance! Blocks break and drop instantly, no glitchy running or walking, worldguard responds instantly. I wish I knew what to tell you.
     
  17. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    I made some changes yesterday, namely being switching to Engine Mode 1 for the x-ray protection and taking eNkrypt and Silverbrit's advice and turning off save-structure-info to false. Hopefully this will be the end of my problems! I will have to test continually and make sure that the lag seems to be resolved. (Oh, I also bumped the server's RAM to 12GB, but I'm not sure if that played much of a role here)
     
  18. Recolance

    Supporter

    Personally, I just shut xray protection off. It makes players not as interested in being on your server and causes latency.
     
  19. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Yes, but x-ray protection is something that's really needed in our case… x-raying is one of the worst problems on my server, and having a deterrent makes it just the little much easier for us to manage.