Spigot 1.14.2 freezes on shutdown, just before saving players/world

Discussion in 'Spigot Help' started by Z0eff, Jun 23, 2019.

  1. Hello,

    For some reason my spigot servers sometimes do not stop properly, causing nightly restarts to hang. This doesn't happen on a consistent basis making diagnosis a bit tricky. When it does happen I end up waking up in the morning and having to kill and restart the screen that the server is running in.

    Here's the log file of this happening: https://pastebin.com/T0Wj4sgp

    Here's a log file of the same server shutting down properly: https://pastebin.com/4eBmqAQZ
    The only difference being is that it reports players and worlds being saved.

    I use the following bash script to restart a spigot server inside a screen after a plugin sends a stop command to the spigot console at a specific time of day. Could this be causing this issue?
    Code (Text):
    #!/bin/bash

    while true; do
      java -Xmx4g -Xms4g -jar spigot-1.14.2.jar
      echo "\0033[30;106m#### Server will restart in 5 seconds, press ctrl+c to stop ####\0033[0m"
      sleep 5
    done
    Does anybody know why this is happening?
     
    #1 Z0eff, Jun 23, 2019
    Last edited: Jun 23, 2019
  2. This has also been happening to me. Recently I have just had it run 'kick @a Server restarting' before doing the stop command and it has been shutting down correctly. This is only after a few tests, but previously it was failing about 3/4 of the time.
     
  3. CoreProtect locks up when shutting down the server. They claim it's a spigot issue though, but I've only had this issue with their plugin installed.

    EDIT: Someone in their support says they fixed this issue buy kicking all players off the server before the shutdown starts.
     
  4. Will try that, hope it works :)
     
  5. Having this exact same issue. Thought it was worldedit because it's the last plugin to shutdown before players start to save. Asked them about it, they said it's a bug with spigot probably. I'm going to start trying to kick all players before shutdowns and resets, but I'm worried that someone will connect the seconds in-between it kicking everyone and everything shutting down and finally getting to player saving.
     
    #5 MURD3RUS, Jun 29, 2019
    Last edited: Jun 29, 2019
  6. Also, I don't use CoreProtect.
     
  7. I'm using CMI to kick all players, with a 1 second delay before sending the stop command. Since I changed it to this a week back it's worked out but like you I also fear a player reconnecting before the server is actually shutting down completely.

    So With this not being a CoreProtect issue, could it potentially be another plugin that does this? Or is it definitely spigot/minecraft itself?
     
    • Informative Informative x 1
  8. As a 1.14.x takes so much longer to load up, I am wondering if this is a vanilla based problem?
    It seems to make sense that the server will take longer to process the shutdown, and maybe the process's are flawed?
     
  9. With 1.14 not being that stable yet, the occasional ticking world crash combined with this issue means that it won't restart after a crash. :(
    (Using 1.14.3 now)
     
  10. Typically on shutdown when a plugin is hung up, its typically the plugin that is listed last at fault, but only if its consistent every shutdown. Years ago I really dug deep into this issue and found plugins really grab for memory on shutdown to save their data, lack of free memory means it locks up.

    Another thing, if your home hosting on windows, nightly restarts help clear ram a ton. Learned that the hard way. Just make sure your server shuts down before the restart or is integrated as a service through a wrapper.
     
    • Informative Informative x 1
  11. What's a good way of checking the RAM usage in such a short time period as it shuts down?
     
    • Like Like x 1
  12. Been years since I've done it, but there is a way to get a heap dump on server stop, I think it's like a script that records once the stop command is fired
    I'm not sure how to do that anymore exactly, maybe others know how.
     
    • Informative Informative x 1
  13. Plugins such as HeapDump and spark can do RAM dumps, you'd just need to figure out a way to have it run the command midway through shutdown (plugin runs command onDisable?) or just type it yourself before the plugin itself shuts down.
     
    • Like Like x 1
    • Informative Informative x 1
  14. I'm having the exact same issue in 1.14.4.
    I guess none of you found any solution for this, and just went for the workaround?

    The issue seems totally random, from what I have seen so far. Now and then it succeeds, but it fails almost every time.

    Logs from a failed restart vs. a successful restart, is missing the following lines:

    Code (Text):
    [Server thread/INFO]: <PLAYERS> left the game
    [Server thread/INFO]: <PLAYERS> lost connection: Server Restarting. Check Back In A Few Seconds.
    [Server thread/INFO]: Saving players
    [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM1): All chunks are saved
    [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM-1): All chunks are saved
    [Server thread/INFO]: ThreadedAnvilChunkStorage (World): All chunks are saved
    [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM1): All chunks are saved
    [Server thread/INFO]: Saving chunks for level 'World_the_end'/minecraft:the_end
    [Server thread/INFO]: ThreadedAnvilChunkStorage (DIM-1): All chunks are saved
    [Server thread/INFO]: Saving chunks for level 'World_nether'/minecraft:the_nether
    [Server thread/INFO]: ThreadedAnvilChunkStorage (World): All chunks are saved
    [Server thread/INFO]: Saving chunks for level 'World'/minecraft:overworld
    [Server thread/INFO]: Saving worlds