Deploying Minecraft Servers using Docker and Git

Discussion in 'Systems Administration' started by dragonlord, Nov 30, 2015.

  1. Hi Everyone!

    First off, let me introduce myself. I am dragonlord, the system administrator for a server called Adobocraft. I'm here to share some of the good stuff we've been doing to run and manage our servers, using Docker and Git.

    What is Docker?

    Docker is the hottest technology to come out since Virtual Machines. Docker allows applications and its dependencies to be packaged and deployed regardless of the server environment. The main difference is that you don't have the overhead of a Virtual Machine. This allows applications to run at near-native performance.

    Docker is a Linux-only technology, open source, and is constantly being improved. The client can be used on Linux, OS X, and Windows. The server component is currently Linux only, and Microsoft has been hard at work bringing the technology to Windows Servers.

    Want more details? Check out their page here: https://www.docker.com/what-docker

    Why use Docker for running Minecraft Servers?

    Let's enumerate the steps on how to setup a Minecraft server on Linux. I'll use Spigot in this case:
    1. Install the Linux distribution of your choice
    2. Install Java. You will need the JDK for compiling Spigot.
    3. Create separate user to run Minecraft (because running a Minecraft Server as root is a BIG NO).
    4. Install screen/tmux to allow scripting of console commands
    5. Download Spigot Build Tools to a temporary directory
    6. Run Spigot Build Tools to start building the server
    7. Have coffee and go to the gym
    8. Copy the compiled jars to the Minecraft user
    9. Run Spigot server to test if it's running
    10. Create shell script to manage the Spigot server by wrapping the server instance in screen or tmux
    11. For distros that still use SysV, create another shell script to control the execution and shutdown of the Spigot server. For distros that use Systemd, create a Unit File for the shell script, then register it with the systemd daemon.
    12. Restart your server and see if the Spigot server automatically starts.
    WOW! THAT'S A LOT OF WORK FOR ONE SERVER!

    Try rolling out 3 server instances, and suddenly you have this complicated mess of shell scripts, configs, port conflicts. Try adding Bungee to the mix, and it becomes a crazy mess!

    There is always a better way.

    Let's take that same example above, and deploy a server using Docker:
    1. Install the Linux distribution of your choice
    2. Create a group with the name docker, and add your user to the group
    3. Install Docker and enable at startup. It is absolutely simple to install.
    4. Deploy a Spigot server with the following command:
    Code (Text):
    docker run \
        --name spigot-1 \
        -p 0.0.0.0:25565:25565 \
        -d \
        -it \
        -e DEFAULT_OP=dinnerbone \
        -e MINECRAFT_EULA=true \
        --restart always \
        dlord/spigot
    THAT'S IT! No additional management scripts. No need to worry about adding users. No need to worry where the files are stored. Spigot gets compiled and deployed automatically. It just works!

    Need another Spigot instance on the same server, running on port 25566? No problem! You only need to changed the port mapping from the outside world to the container like so:

    Code (Text):
    docker run \
        --name spigot-2 \
        -p 0.0.0.0:25566:25565 \
        -d \
        -it \
        -e DEFAULT_OP=dinnerbone \
        -e MINECRAFT_EULA=true \
        --restart always \
        dlord/spigot
    No need to change the server.properties for the port mapping!

    And this is just the beginning!

    Spigot Docker Image used in Adobocraft

    Running a Dockerized Spigot server, while it all sounds great, has its own set of problems:
    1. Spigot does not have a clear separation between plugin configuration and data - Most of the time, the data files are living inside the same folder, together with the config files. This breaks one of the rules of Docker: you package your application and configuration together, and the data is stored outside the application. Of course, this can be easily solved by configuring the plugin to use an external DB. However, not all plugins support that.
    2. Sending console commands for automation purposes - Most server admins would probably use screen or tmux to allow sending console commands via scripts. While you can attach to a Docker container's STDIN to send commands, this does not mean you can send commands to it in an automated fashion. To allow sending console commands via scripts, you will need to wrap the docker attach session inside screen or tmux. Very annoying if you want to manage multiple Spigot instances.
    3. Performing a safe shutdown - Safe shutdown can be achieved by sending the stop command via the console or chat. If you want to do this in an automated manner, you will need to do something like #2, which is not ideal if you're handling multiple Spigot instances.
    To address these 3 main problems, I developed a Spigot Docker image. This is publicly available for download here. We have been using this to rollout configuration changes, execute scheduled commands without installing tmux, screen, or Spigot plugins, and cleanly shutdown our server instances during server updates.

    Configuration Management using Docker and Git

    Most Minecraft server admins are used to adding plugins and changing server configurations the old-fashioned way: either they manually upload via FTP, or they remotely access the server to edit the files. Not only is this inefficient, it is very error prone. I'm pretty sure a lot of you have tons of *.bak files to ensure you can rollback to the last known version. And even with that, there's no guarantee that you can rollback your server completely.

    Our configuration management is done via Git, a popular source control developed by the father of the Linux kernel, Linus Torvalds. We check in all our plugins and configuration files into a local Git repository. Once somebody pushes their changes to our central Git repo, all of the artifacts are packaged into a Docker image. This image can then be deployed as our test server. Once we're happy with the changes, we simply tear down the production docker container, and then recreate it using the updated image.

    This setup has many benefits:
    1. Traceability and Accountability - We can easily track who changed what, and who screwed up.
    2. Development Environments - All personnel who has access to our Git repo can run our entire infrastructure in their local machines, as if they are running the whole thing in production. Since every piece of our infrastructure is running as a Docker container, our setup is portable across machines.
    3. Fast deployments - We are able to deploy to production in under 30 seconds. Users only know we're restarting. What they don't know is we're replacing the entire server with a new one, every single time. :D
    4. Creating server instances with the same configuration - We have a test and a production environment. If we wanted to provision a new environment with the same settings, we can just spin up another instance using the same Docker image we used on test or production.
    5. Ability to rollback to previous versions - sometimes we would encounter an issue on the production server that we were not able to catch on our test environment. We can easily rollback to any version we wish, all under 30 seconds.
    This setup has saved our asses ALOT OF TIMES NOW!

    Data Management

    As mentioned previously, the main problem with running a Spigot server is that data and configuration are intertwined. Until such time that Spigot and/or plugin authors store their data files on a location outside the plugins folder, managing data can be such a hassle.

    We've experimented with different ways to make this manageable with our setup, and we settled with the following:
    1. Data Volume Containers is our best friend - We do not store our data directly on the host system. Instead, we make use of Docker's Data Volumes. This allows us to deploy our whole infrastructure on a different server, without worrying about missing folders and other similar problems.
    2. World data is stored on a separate folder from Spigot's main folder - We were able to achieve this thanks to Spigot's ability to specify a different world directory. This is a feature baked into my Spigot Docker image, which makes it easier for us to backup and restore the world data.
    3. Configured every plugin we use to store data in a DB like MySQL - not all plugins offer this option. Some only use sqlite, which is practically a flat file. Those that can store in an external DB are immediately configured to do so. We then provision a database server instance for each environment using Docker.
    4. For plugins that store data as flat files, exclude the data files from the Git repo - This is a tedious process. We test each plugin we add to determine where they store their data files. Once we've determined them, we exclude them from the Git repo, so that they are not overwritten during deployment.
    Things could be better. For now, this setup works well.

    Conclusion

    Running a Dockerized Minecraft Server may seem like a lot of hard work, especially if you're used to using things like McMyAdmin. Docker works well with automation tools like Jenkins and Ansible. It has allowed us to monitor and control all Spigot server logs via systemd. We can even create complex scripts to execute console commands automatically, without the hassle of managing tmux or screen sessions.

    Overall, it has been a pleasant experience for all of us. We continue to refine how we do things at Adobocraft, and we'll be sharing more as we go along.
     
    • Winner x 6
    • Useful x 6
    • Like x 3
    • Agree x 1
    • Informative x 1
  2. This is literally one of the best things I've ever seen in the Spigot community!! I will try this out on my own server. Thanks for the helpfull information about using docker for minecraft!
     
    • Agree Agree x 1
  3. If you have any questions on how to run this kind of setup, feel free to ask. :D
     
  4. This looks simply amazing, but I have some questions since I don't use Git at all: how do you manage the config files and the rest? And when you reboot your server, dockerized minecraft servers run automatically?
    Simply, if I want to make a config change to my hub image and apply to all my hub, how can I do it?
     
  5. Let me try to answer each question:

    1. Git - it's a popular tool used by developers to track changes to their source code. We use that as part of our automation tools. I have one policy for all Adobocraft Admins: if it's not in source control, it does not exist. :)

    I highly recommend reading more about source control management. We basically treat config files like any other source code.

    2. Docker and server reboots - Docker itself runs as a daemon on the server, which we configure to run on startup. Docker containers can be setup to autorestart if a container dies. This not only keeps containers alive despite of a failure, it will autostart containers on server startup.

    3. Applying config changes - our Git repository for the server configs contains multiple Dockerfiles, one for each server. We use Docker Compose to package the configs as Docker Images, and are deployed automatically.

    If you want this kind of setup, you will need to play around with Git, Docker, and Docker Compose. These 3 are the key to automation heaven.

    We use a lot of tools in Adobocraft. All of them are Dockerized. :D
     
    • Winner Winner x 1
    • Useful Useful x 1
  6. I'm interested in learning about how to gracefully stop the Minecraft process when the docker container is stopped. Let's say I run it via PID 1, how would it work?

    Love to learn more from you. Thanks!
     
  7. This user hasn't logged in for 26 weeks and this thread is almost a year old...
     
  8. I don't know what you mean but:

    [​IMG]
    This is how a Docker Container work.... If the Docker Container stops or restarts the whole Minecraft Server restarts with it.
     
  9. I haven't been around because work got in the way of Minecraft. :(

    I'm finally out of hell.

    My spigot docker image has undergone a lot of improvements, and it can perform a safe shutdown (provided you set it up correctly). You can find out more here: https://hub.docker.com/r/dlord/spigot/ (this also contains the link to the github repo).

    Sorry for resurrecting an old thread.
     
  10. I know this is a old thread, but I am certain this is still one of the best ways to manage a minecraft server :)

    However could you maybe provide some examples of the docker files and docker-compose files for deployment of Configs and plugins?

    I would love to try out a fully automated process with docker Configs, but need some clue on how to get started.

    Thanks!
     
  11. You should mark this as a resource.
     
  12. dragonlord was last seen:
    1y 41w ago
     
    • Funny Funny x 1
  13. This was promising but it appears dragonlord has gone, as have his servers.
     
  14. If someone needs something more simple at least at the moment of the post creation:
    https://gitlab.com/ldath-core/docker/spigot
    I do not have bigger plans than just what is needed to migrate my private spigot server to the new place where I'm using docker engine to manage services I have on it.
    Maybe there will be in the end examples how to use it with ansible and/or Kubernetes. I normally have very limited time and just trying to have staff as simple as possible.