Solved [1.15.2] Server crashed during the night with no players connected.

Discussion in 'Spigot Help' started by Lud00, Feb 22, 2020.

  1. Hello Everyone,

    I got 2 server crash recently but I cannot figure out what is the issue.

    The first crash was 16 Feb @ 16:13:41 with 6 players online
    The second crash was this morning 22 Feb @ 06:15:38 with no players online

    I been thinking of too much entity in the spawn area but it's not the case.
    No red stone there too.

    The server is running on the last Spigot 1.15.2
    This server is 2 Years and follow the Spigot Updates since 1.13.

    I have the following plugins installed:
    Advanced Armor Stands 1.15.1
    bPermissions 2.12.3
    Coordinates HUD
    Core Protect 2.18.2
    Dead Souls
    Dynmap 3.0-beta-10
    EasyBackup 3.3.2
    Entity Tracker Fixer
    One PLayer Sleeping Is Enough
    Real LightWeight TPRequeses 0.2
    Ultimate Homes 1.3

    The server have 4 vCPU on my own dedicated host with 12 GB of RAM
    Its an Ubuntu server with Nginx and MySQL Installed (For Core CorProtect and Dynmap)

    All my configuration is documented on my blog:
    https://blog.stefaniak.fr/index.php...un-serveur-minecraft-spigot-sur-ubuntu-debian

    During the crash there is all cores at 100% (You will tell me its normal when you have a crash)

    Thanks for your help
     

    Attached Files:

  2. Does someone have debug skills here?

    Thanks
     
  3. Seems like EntityTrackerFixer is doing anything but "fixing" your server, it just appears to be crashing it.
    For like 2 hours all I see is:
    Code (Text):
    [06:21:01] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5850ms or 117 ticks behind
    [06:21:13] [Server thread/INFO]: [EntityTrackerFixer] Untracked 2 entities in BroCorp
    [06:21:13] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [06:21:13] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [06:21:36] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp
    [06:21:36] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [06:21:36] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [06:21:36] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5582ms or 111 ticks behind
    basically.... "fix, fix, fix, lag, lag, lag"
    Every time it runs your server is falling 5 seconds behind

    a little later on (just before the crash)
    Code (Text):
    [07:36:11] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [07:36:11] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [07:36:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 257109ms or 5142 ticks behind
    [07:39:50] [Server thread/INFO]: [EntityTrackerFixer] Untracked 1 entities in BroCorp
    [07:39:50] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [07:39:50] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [07:41:42] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 301525ms or 6030 ticks behind
    [07:43:49] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp
    [07:43:49] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [07:43:49] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [07:48:24] [Server thread/INFO]: [EntityTrackerFixer] Untracked 1 entities in BroCorp
    [07:48:24] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_nether
    [07:48:24] [Server thread/INFO]: [EntityTrackerFixer] Untracked 0 entities in BroCorp_the_end
    [07:48:24] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 371583ms or 7431 ticks behind
    Your server is 371 seconds behind.... THREE HUNDRED AND SEVENTY ONE SECONDS BEHIND... YIKES!
    Thats 6 minutes behind.... wtf is this plugin doin?!?!

    You might want to report this to the dev of that plugin, and in the meantime, remove that plugin, clearly it is doing nothing positive for your server.
     
  4. Hello Shane,

    Thank you for your time.

    I'm not sure the issue is with EntityTrackerFixer.

    We can see the issue triggered before @ 6:15

    Code (Text):
    [06:15:38] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5272ms or 105 ticks behind
    [06:16:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5011ms or 100 ticks behind
    [06:17:17] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5213ms or 104 ticks behind
    [06:17:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5142ms or 102 ticks behind
    Entity Tracker triggered because of the CPU @ 100%

    If the server is bellow 19.5 tps it will start that pluging to stop all mobs traking:
    https://www.spigotmc.org/resources/entitytrackerfixer-fix-1-14-4-1-15-entitytick-lag.70902/

    No players was online at that time.
    The last player disconnected @ 02:06
    Code (Text):
    [02:06:30] [Server thread/INFO]: Player_3 left the game
    I also checked that there is no farms at the spawn area.

    Do you know if there a way to that what caused that issue? Debug logs may be.

    No crash today. But I cannot predict when it could occure.
    [​IMG]
     
  5. looking at your original post again, and the log, it almost seems like maybe DynMap was overloading the server. I noticed it started up the rendering of the map, then the server kept falling behind, getting worse and worse each time.

    ex: I snipped out the entity tracker stuff
    Code (Text):

    [02:06:30] [Server thread/INFO]: Player_3 lost connection: Disconnected
    [02:06:30] [Server thread/INFO]: [dynmap] Full/radius render pause set to false
    [02:06:30] [Server thread/INFO]: [dynmap] Resume full/radius renders - below player limit
    [02:06:30] [Server thread/INFO]: Player_3 left the game
    [06:15:38] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5272ms or 105 ticks behind
    [06:16:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5011ms or 100 ticks behind
    [06:17:17] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5213ms or 104 ticks behind
    [06:17:57] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5142ms or 102 ticks behind
    [06:18:34] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5921ms or 118 ticks behind
    [06:19:11] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5349ms or 106 ticks behind
    EDIT:
    oh shit I read it again, it appears that the overload starts 4 hours after the dynmap thing.
    Honestly I have no clue.
     
  6. Hello Shane,

    Dynmap fullrender get paused when there is one player online. This is to prevent CPU usage when players are online.
    Normal dynmap render is still active until 5 players are online.

    Code (Text):
    [02:06:30] [Server thread/INFO]: [dynmap] Full/radius render pause set to false
    [02:06:30] [Server thread/INFO]: [dynmap] Resume full/radius renders - below player limit
    This only mean that fullrender can continue because no players are online.
    In my case the map was already full rendered so it didn't render anything.

    parallelrendercnt is also disabled so it only use one core.

    I enabled the spigot debug logging today. I'll wait for the next crash.

    Thank you very much for having a look at it anyway.
     
  7. Would you mind doing a timings report, have a few players online while doing it. That can make sure the full amount of entities & chunks loaded on the server.
     
  8. Sure:
    https://timings.spigotmc.org/?url=ayiyezikul

    Thanks
     
  9. Isn't your answer right there...
    There's a mob being ticked. You should remove it. You should also lower the values of entity-activation-range in spigot.yml (Figured it out from Activated Entities)
    upload_2020-2-24_21-55-26.png
     
  10. Thank you for the advices.

    The players are using Mobs farm. It usually not cause issue. (Farms are not in spawn area)

    I'll check the values when nobody is online to see if there is still mobs since it crash when it was empty.

    Thank you
     
  11. Hello,

    Just closing this since it didn't re occur.

    No change made.