POC Reduce Minigame bootstrap time

Discussion in 'Spigot Plugin Development' started by twanneke8, Jun 4, 2017.

  1. Hi everyone

    It is not uncommon for minigame server instances to have a very short lifetime (5 minutes). With that in might one of the biggest performance costs is the bootstrapping of the server.

    Do you guys think it is possible to CRIU a loaded instance of your server (a plugin can send a save signal as soon as it's loaded, and stop the server) and use it as a template? Or could we experience issues with the server clock?
  2. Inkzzz

    Resource Staff

    Have you taken a look into google cloud servers?
  3. electronicboy

    IRC Staff

    Well, unless you took steps to mitigate it, you'd trip the watchdog unless you disabled/remove it.

    it just seems like a hacky workaround that would likely cause some stupid issues elsewhere, if your servers are taking too long to start up, you should probably look into solving that instead of working around it with hacky measures.
  4. Don't boot on demand have a threshold of like 3 server instances that are already spun up and then spin more up to refill the threshold.
  5. electronicboy

    IRC Staff

    I was going to include this, but somehow it didn't make its way in there... *gets distracted and comes back in 20 minutes*
    if you wanna spin instances up and down a lot, you should be prepared to have a few instances laying around so that they can be used fast when needed, this; combined with keeping your instances in a condition so that they can boot up in a reasonable speed, is pretty much essential beyond having dozens of instances laying around, or hacky systems that will cause chaos when they break
  6. World loading is very bandwidth intensive which writes of any cloud hosting, anyway not sure how this is relevant :)

    What is this watchdog? I assume it's some kind of tick counter?
    Yeah you can provision around this though it will increase the economic tax even further (boot time is like the most cpu intensive moment of the server).

    Lol we're build on top of kubernetes (docker), so no hacky systems here. Anyway, it's not so much about the provisioning problem as it is about the CPU tax of booting servers while you shouldn't have to.
  7. electronicboy

    IRC Staff

    watchdog runs on a separate thread, which basically every second checks if the current time has passed x amount of time since the server last ticked.
    Doesn't docker let you limit resources so you can reduce the strain on the system?

    Also as a side note, cloud services really don't work for this, generally you spend much more on cloud services than on a decent priced dedicated server, unless you're throwing in a crapton of money on a dedi for no real reason beyond lack of market research and going with a "top tier company". MC really doesn't afford for scalability in the way that web services do.
  8. Docker (and kubernetes) do allow resource limits, but the bootstrapping still has to be done, whether it's bursted in a couple of seconds or done over 20 seconds the net amount of CPU required stays the same. So setting this to a safe amount will only allow you to better predict performance, but it won't save costs.

    Anyway the thing I'd like to discuss is this watchdog thread, what happens when a time leap has been made, does the server terminate?
  9. electronicboy

    IRC Staff

    Yes, once you pass the timeframe the watchdog will trip and stop the server. paper has a flag to disable it for development purposes, however it's kinda one of those things you really don't wanna disable on production servers as it means to give a chance for your server to actually shutdown (safely, albeit you probably don't care too much about this one) and pass players off properly as needed as opposed to just timing them out.

    as I've said, if servers taking too long to start is your concern, you should have instances sitting around instead of attempting to employ hacky measures such as this that will drastically impede the stability of services and likely cause issues.
    sane companies, such as hypixel, just ensure that they have instances sitting around ready.
    • Agree Agree x 1
  10. Sounds about right, thanks for considering this (we're working with idle instances right now btw) :)