Minecraft security | Part 4; Reducing risks

Discussion in 'Systems Administration' started by MrDienns, May 11, 2020.

  1. Note: This series isn't finished yet, and I may refactor pages severely the upcoming days.

    A comprehensive series of posts explaining why Minecraft servers get hacked, and what you can do to prevent it. This post will speak of several things which are almost always the case in those classic YouTube videos where servers get hacked. I will explain in detail why this happens, how this happens, and what you can do to prevent it. This is a post designed towards everyone; from (public) plugin developers, to system administrators, to the professionals who have full custom networks.

    In cyber security, you shouldn't asked yourself if you get hacked, but when.

    These posts consists of the following parts;

    I am writing this because I (too often) see servers get hacked. Now, this on it's own, while problematic, isn't necessarily alarming. What is alarming to me is how, and why they get hacked. For that reason I'm writing a multi part security blog (or whatever you want to call it) raising awareness to pretty much all. Note that this guide isn't specific to Minecraft; this is software development and system administration in general.

    Reducing risks
    In the last part we've talked about identifying risks. In this part, we'll talk about reducing risks to a lower impact. Throughout this part I will post several example risks, and example implementations of reducing risks. What's interesting in this part is that there sometimes multiple routes to reducing a security risk. Some may be more beneficial to reducing the risk, but will pose a negative impact elsewhere in the organisation.

    This part, as the title suggests, will focus mostly on reducing risks rather than preventing them. Reason for this is the difference in complexity, and skill necessity. Lots of risks can be reduced by system administrators for example, rather than requiring a developer to update, fix or change a piece of software. Most people will either have knowledge themselves or have a system administrator. Fewer people will have developers for their project. For that reason, risks talked about in this part will mostly resolve around system admininstration rather than development.

    One has to understand why we reduce security risks first. In the previous part, I tried to steer you towards identifying a risks. These risks have both an impact and a likelihood. We don't want to have security risks lay around for someone to exploit. If the issue is severe enough, it could literally end your project. We don't want this to happen, so we should look at the previously identified risks and try to reduce as much of them to a lower risk.

    Below are numerous examples of common security risks which can be tackled through system administration. With every example, I will describe a situation. The situation poses numerous security risks which may pop up. While reading, try to think of all of the security risks you could come up with while reading the situation. Part of reducing & preventing security risks is being aware of what risks exist, and pop up.

    However, since IT is so insanely diverse, it's impossible to know all of the risks. Instead, try to understand, try to get a specific mindset. Place yourself in the opposite shoes of those that you're currently standing in. You're no longer the system administrator or the server owner; you're the attacker now. Be as abusive as you can be. The more security risks, the more oppertunities to you. Think about how you can break things. Rather than thinking "Cool, it works!", think "Okay, it works, cool, but what now? Is this how it's suppose to work, or can I stricten it? How could I abuse this?"

    Database access & usage
    Any reasonably sized project will use a database of some sort. Smaller projects, or those that are just getting into things, will often start with flatfile (like YAML files). However, as you grow, you start to understand the need for a centralised database. This example scenario resolves around a project that's trying to grow. They're moving their data from flatfile to a MySQL database. Since they're new to system administration, they barely managed to setup their MySQL database on a separate machine and they managed to connect Minecraft to it. With the growing community, they also decided to get a forums (or its own machine), which also uses MySQL. They decide to use the same database server.

    To summarize it: Small, inexperienced server is growing and needs to start using a database. They also add a forums. Minecraft, the database and the forums all have their own machine. They use the MySQL server both in Minecraft and the forums.

    As mentioned above, think of the risks that can occur here. I'll post a spoiler with the ones that I can come up with.
    • They could be using the root MySQL user;
    • MySQL users don't have host restrictions;
    • MySQL itself could be ran as root;
    • They could be using the same database within MySQL for both the forums and the Minecraft data;
    • The database is publicly accessible and not restricted by a firewall (prone to denial of services);
    • Software that's connected to it may have SQL injection vulnerabilities;
    • They're not taking any backups of their MySQL data;
    There's likely more risks, but these are some severe ones I could come up with. Now, considering you're reading part 4 of this guide, you should have read the previous 3 parts. Part 2 explained the exploit chain (or lateral movement in official terms). With that in mind, look at the above (or your own found issues) and think about how lateral movement could be used. I would consider all of the above mentioned risks to be quite severe, and can post a very serious impact when abused. I will mention all of them below, and what the options are of reducing (or when possible, preventing) these risks.

    They could be using the root MySQL user
    What I mean with this is, when connecting to the MySQL database, they may use the root user. MySQL, just like Unix, has a root user. This user, as you might guess, is a super-administrator. It can do everything, and has non restrictions what so ever. It's the most powerful user there is, just like in Unix. If any unauthorised person gets access to this user, it's game over. Data could be deleted, altered, exported, or created. Altering or creating data may result in further lateral movement. Think about what can be used as lateral movement here.

    For example. As we mentioned, we have both Minecraft and our forums connected to the same database here. It's not uncommon to use MySQL for a permissions & group plugin. Once an attacker has access to the root user of the database (or any priviledged user in this case), he could then modify the data thats stored in by your permissions plugin. He could make himself admin through your Minecraft permissions plugin, or potentially even administrator on your forums, if it's using the same database. He could then access your forums or Minecraft server, and cause further damage with his newly aquired admin rank.

    Solution: Don't use root. Never, ever. Root is there for initially setting up. The first thing you do when you see a root user is disable it. This does not apply to just MySQL, but also things like SSH. Now, don't delete the user entirely (if that's even possible), just make it inaccessible or significantly harder. For instance, in MySQL, you can set host restrictions on users. You could make it so that only localhost can log into the root user. You'd thus have to SSH into the machine in order to be able to use root in MySQL.

    Not using root in your database connections should reduce the risk of escalated priviledge abuse once your database user gets compromised. Don't hold back with creating new users, and be strict on what they need. Have a lucky crates plugin which has 2 MySQL tables? Sure, give this user access to just those two tables; nothing more, nothing less. Have a permissions plugin? Sure, just make sure to use a dedicated user for that too, and make sure he can only access what he's suppose to access.

    MySQL users don't have host restrictions
    This is an additional elaboration on the earlier described risk, but it comes down to the same principles. Configuring host restrictions on your MySQL users will put restrictions on which hosts are allowed to login to the specified user. If, for whatever reason, your credentials get leaked, then an attacker wouldn't simply be able to login to the database with those credentials if you have specifically whitelisted which host is allowed to login to the user. For example, you could configure that only your Minecraft machine is allowed to login to the users that you have for Minecraft.

    Configuring host restrictions on database users prevents the usage of database users that had their credentials compromised (unless they have also compromised the whitelisted host).

    MySQL itself could be ran as root
    Once again we run into root. Remember what we said earlier about root? Never use it. The Unix root user is no different. Perhaps to your surprise; MySQL can do a lot. Some functions raise a few eyebrows with me personally, but they probably have their use cases. For example, did you know that MySQL has a LOAD_FILE(...) function? Not sure why it exists, but it exists. For example;

    Code (MySQL):
    SELECT LOAD_FILE('/data/test.txt') AS Result;
    Could result in;
    Code (Text):
    | Result                                   |
    | This text is all that the file contains! |
    Now imagine that we have our database user compromised, or we have some SQL injection (because we should assume we get hacked). Now, let's also assume that the user that got compromised (or the user thats been used in an SQL injection) has access to these file queries (hint: which the MySQL root user has!). These queries could be abused to then read or write to any file in the system. And yes, I mean any. Why? Because MySQL is running as root, and root is unstoppable and can do everything. It could read and write to extremely crucial and confidential machine files. Think of modifying BashRC files to inject malware code (like a reverse shell), or gaining access to private keys stored on the server.

    Not using the Unix root user prevents extreme priviledge escalation and prevents the MySQL service from having unnecessary access to other parts of our Unix machine.

    They could be using the same database within MySQL for both the forums and the Minecraft data
    MySQL is a database. What may sound confusing to people that are new to MySQL is that you can create multiple databases within MySQL, or almost any other database for that manner. For MySQL, a database is something that containsa set of tables. Tables then contain rows and columns and you probably already know. You can assign different users to different databases. If a connection, say from a Minecraft plugin, is made to database A, then it can't simply access database B unless you've explicitly given permission for that (which is not what you want, unless you absolutely know what you're doing).

    The problem with using the same database for numerous different sets of unrelated data, again comes down to giving your users (your connections) access to data that they don't need to have access to. Let's take our scenario example; let's assume our little project here has one database for both our forums and our Minecraft data. If they've done that, it's safe to assume they didn't setup any table restrictions either. This means that if you have some kind of Minecraft plugin installed which is vulnerable to SQL injections (which, I'm sorry, way too many are!) then they could simply read the other tables of the database, which would now include our forums data. Usernames, IP addresses, passwords, email addresses, you name it.

    Not using the same database, and making different databases for different software components (including different Minecraft plugins!), in combination with setting up proper database access per user, can reduce the damage in case one of these connections or users get compromised as they wouldn't be able to access any more data than they originally were suppose to.

    The database is publicly accessible and not restricted by a firewall (prone to denial of services)
    This is one that people often underestimate, even with enough knowledge on the topic. Denial of services are everywhere. Especially in Minecraft, it's not uncommon to be attacked. However, these denial of service (like DDOS attacks) are often targetted towards the Bungeecord or Spigot servers. Little do some people realise that other crucial parts of your infrastructure can be attacked, causing collateral damage on the rest of the system.

    As we explained in previous parts; databases are crucial. They store information which is vital for the application to function correctly. If a database becomes inaccessible, then in almost every case, any service that uses that database will now be rendered useless. It can't access the data, so it breaks. In lots of cases, this isn't handled very nicely either. Plugins throw massive unexpected errors, websites fail to render at all, etc.

    Having your database port, or even the entire machine, accessible to the open internet opens up a risk for them to be attacked. Leaving open the port would allow the oppertunity for layer 7 (application layer) DDOS attacks, whereas not having a general firewall in place to restrict access could leave you more vulnerable to layer 3 (network layer) and layer 4 (transport layer) DDOS attacks. While this is less likely than having your user access points attacked, it's not one to over look. It's a vital component in your infrastructure, so it should be treated as such. Restrict who communicate with the machine and ports.

    Using a firewall, like IPtables or host specific firewalls, can reduce the chances of denial of service attacks taking down large chunks of your project.

    Software that's connected to it may have SQL injection vulnerabilities
    This one is rather problematic. It's a bit problematic because as a system administrator, there's not that much you can do about this. For those unaware (and you really should be aware); an SQL injection is a hacking technique where some form of direct or indirect user input is not safely pushed into a database through a query. I won't go into detail on how to prevent them or how they work, just be aware of the fact that way too many software components, especially Minecraft plugins on this site, are vulnerable to these vulnerabilities.

    SQL injections are just awful. They've existed for way too long, and in 2020, it's rather ridiculous that they still pop up. For IT standards, it's an ancient trick. It really shouldn't work anymore, but it still does. People somehow still miss them. It becomes worse because lots of people that post plugins here on this website are made by new, fairly inexperienced developers. It's great to see people getting into development, but never assume that just because a plugin has lots of downloads and good ratings, it's actually secure. The majority of the people, often including the developers, are just not aware of these issues and thus nobody bothers with them; until somebody finds out and starts to abuse it.

    Take a look at this example of a popular plugin where @clx_ found out that it was injectable (through sending a special character). @MiniDigger and me then tried to exploit this, and sure we could. We could post something in the chat that would freeze, and thus kill the entire server. We could've abused this over and over again. It's effectively a denial of service attack. I can assure you that in this case, any system administrator would have severe difficulties with figuring out what the hell would be going on if someone would abuse that. We reported the issue and it was fixed shortly afterwards, but this should hopefully prove 2 unfortunate points; point 1 being that too few people, even in 2020, know about SQL injections. And point 2 being that they really are a severe security risk once detected and abused. From altering data, deleting data, reading data that you shouldn't have access to, to killing the server altering files on the machine, all are possible with SQL injections.

    Knowing how SQL injections work, even as system administrator, can help you test plugins before you put them in production. Testing plugins for vulnerabilities before using them prevents these vulnerabilities from ending up in your production environments, and should prevent people from abusing them. Now for non developers, this isn't easy. But SQL injections are ancient, anyone can learn about it. And please, absolutely do learn about them.

    They're not taking any backups of their MySQL data
    This one should be obvious but it's worth mentioning. We've covered all of the above example risks, and by now you should have a good realisation of how many issues can pop up with just using a database. So many issues, so many routes to people screwing with your data. It should then not come to a surprise that I'm now going to mention the importance of backups. MySQL can easily be backed up as it has a simple export functionality which writes all of the tables and its data to a .sql file.

    However, making merely a backup isn't enough. Backups should be pushed to different environments, so that even if the entire machine gets compromised, you would still have your backups safely in another place. They're called offsite backups. This isn't specific to MySQL though, this goes with everything. Backup your data, backup all of your stuff, and save it somewhere offsite. Just think about how you'll save it offsite; you don't want a compromised machine to get access to your offsite backups as that kind of defeats the entire point. Download the backups, push them to some cloud storage (write only and no overriding of old ones); there's numerous ways to do so. And remember; a backup is useless if you can't or don't know how to restore it! Make backups, and practice disaster recovery. Make sure that your backups work.

    Having good, offsite backups will reduce the damage in case something does go wrong; and we should assume that this happens. If something gets compromised and your data gets tinkered with, then having good, frequent & functional backups will put you back into production in no time. The more frequently you take backups, the less data lose during the use of a backup. So, more frequent backups is better.

    Continue reading below...
    #1 MrDienns, May 11, 2020
    Last edited: May 13, 2020
    • Like Like x 2
    • Winner Winner x 2
    • Informative Informative x 1
  2. Using Bungeecord or similar proxies
    This example revolves around Bungeecord or similar proxies like Waterfall or Velocity. For this example, let's assume we have a project that's running 10 Spigot servers behind one Bungeecord. Just as in the previous example, think about what could be wrong here. Think about what you can abuse, how you can abuse it, and what kind of impact this would have. Below a few risks I can think of;
    • They didn't firewall their machines properly, so Bungeecord can be bypassed and abused;
    • They only have a single Bungeecord;
    • There's no live or rolling updates
    Now let's again go over all of the risks I could think of for now. Note that one could argue that some of the risks I've thought of above are more operations risks, but considering an outside security factor could initiate, influence or enlarge the problems, I want to at least mention them.

    They didn't firewall their machines properly
    This one is honestly just stupid. There are still people that only read half of the Bungeecord installation guide. Remember what I said in the previous parts? People tend to follow installation guides right up until the point where it starts to functionally work. They stop reading when Bungeecord is up and running and they can join stuff. They'll stop reading, and they especially won't read the maximum font size, bold, red, underscored text which says "Security warning!". Nah, that's not interesting to them, sadly.

    I probably don't have to explain this one. Bungeecord (and forks) has the issue that people can connect to your Spigot servers directly, because they are required to be in offline mode. Although I haven't used it yet myself (I plan to), the Velocity aims to improve the situation by adding a form of authentication. It works by configuring a password / key on the proxy and paper server. If the proxy forwarding the player connection doesn't provide the right password, then Paper will reject the connection. Alternatively, when using native Bungeecord (or forks), this plugin called BungeeGuard can be used. Unfortunately after all this time, Bungeecord doesn't support this natively. You'd think that with all of the people facing these issues (even if its their fault), you'd force them to take a few additional minutes to copy paste an auto generated key / password, but I guess not. Missed oppertunity towards supporting the community is what I'd call it. I'll leave my rant there.

    Firewalling your server, or using community forks, plugins or completely new proxies, can prevent people from bypassing your proxy and spoofing your Spigot servers into allowing hackers to join with administrative accounts.

    They only have a single Bungeecord
    Some may start thinking that this is overkill. Sure, to plenty it will be, but if you're starting to become a fairly established server (like 200+ players), then I'd argue you should already be looking into having multiple Bungeecords. Not because you can't handle it on a single one, but because having some load balancing is never a bad idea. A single point of entry is also a single point of failure. If you have one giant access point for all players, then it doesn't take much to make an entire network unavailable; just take down that single access point.

    Having multiple access points (proxies) can reduce the impact in case of a crash or denial of service. More about this in the next point.

    There's no live or rolling updates
    Like I mentioned earlier, this is arguably an operational risk, not a security risk. However, I do believe that operations of security go hand in hand when it comes to serving your platform. After all, availability is one of the elements of the CIA triad when categorising security risks. So in that sense, why wouldn't this be a security risk? If you're unable to roll out, live update, upscale or downscale your servers based on demand (or, to handle some guy doing funny stuff), then I would definitely consider that to be a security risk.

    Your server is all about availability. The problem that we face in Minecraft with proxies is that they are, well, proxies. It's a continuous connection going through a single gateway. If that one gateway dies, all players connected to it will disconnect. Naturally, in order to make your game available to as much players as possible, you want to ensure that your proxies don't die or need to restart unnecessarily. Or when they do need to restart, you do it in a controlled manner so you don't have to kick hundreds of players. This is where live patching or rolling updates come in.

    Live patching is really just a fancy word I use here for reloading your Bungee. We all know it's heavily discouraged in Spigot, but in Bungeecord it's far less of an impact because people on average have far less Bungeecord plugins. You can program your plugins to handle reloads without an issue. Being able to reload a plugin to address an issue so you don't have to kick potentially hundreds of players, would definitely be a good improvement.

    Furthermore, for enterprise servers, it may be a good idea to think about an up and downscaling strategy. Hypixel for instance has quite a setup when it comes down to their proxies and how people join through them. They have an incredible amount of Bungeecord (or similar) proxies which they rotate every x minutes. If I recall correctly, they try to have between 500 and 750 players on one proxy. Surely they could have more on a single proxy if they wanted to, but I doubt they want to. The more proxies, the more access points, the less impact when one dies (or gets killed). More about this in a potential bonus part of this guide regarding DDOS attacks.

    Having flexible access proxies, like Bungeecord, can reduce impact on availability, either caused by internal operations or outside influences like an attack. Having proxies that can be updated or changed without a restart will benefit availability. Having rolling updates, or a good cloud environment which can upscale when necessary, can also benefit availability. Remember; the more players on one proxy, the more people that get kicked when the proxy goes away for whatever reason. More proxies results in more load balancing, which results in fewer impact if one dies.

    Managing a website
    Almost all of us have a website. How we run our sites is all vastly different, but most of us will have one thing in common. A lot of us use Cloudflare. What's nice about Cloudflare is that they are a massive reverse proxy. They'll hide the IP address of the target system if you're connecting through them. Unless you're an enterprise customer, that'll only work on HTTP protocol (so, websites). The hiding of your origin server IP nice, but there's a catch.

    Hiding your IP address behind Cloudflare, without strictly enforcing this, is quite pointless. You may have heard of or seen the phrase "security through obscurity", and this is a prime example of that. Basically the phrase means securing something by hiding it. It's like securing your precious personal belonging in a closet, without actually putting a lock on it. Your security measure is then hoping nobody finds it, rather than ensuring nobody actually accesses it. Cloudflare usage is often the same. People tend to put things behind Cloudflare, but leave open the ability of connecting directly to the IP address of the server. Quite literally pointless as the server will still accept non-Cloudflare traffic. The hope is that people don't find the origin IP. That's what we mean with security through obscurity, and you should never rely on it.

    Not relying on security through obscurity, and actually enforcing the practice (such as rejecting traffic) will reduce numerous risks or impacts on your side, such as DDOS attacks, depending on where and how it's applied. Effectively, you're enforcing traffic go (in case of Cloudflare) through a much larger proxy, which has more power to deal with such attacks.

    Passwords and 2FA
    I shouldn't have to say this, just like a few other entries on this list. Passwords are still everywhere, and somehow, plenty of people still don't realise how important, yet vulnerable they are. Ensure that all of your passwords are rock solid. Use random generated ones where possible, and whereever possible, use 2FA. Even if your passwords get hacked, which can be completely outside of your control, you can still have 2FA has a 2nd layer of defense. Passwords may be leaked through other sites that weren't secure. You shouldn't assume your password never gets leaked. Always rely on another set of authentication, preferably hardware identification such as Google Authenticator.

    This was a really long thread, but hopefully it should give you an idea about what kind of issues you could face (which you may even have right now) and how you can reduce the risks through simple system administration. After reading this part, you may have learned something new about some of the risks you could face, and how you can tackle them right now or in the future. With these examples, I also hope to raise some awareness and give you a certain thought process. Having the mindset of a hacker will greatly help you secure your own networks. If you know your systems inside out, and you can't think of a single way of hacking it anymore, you've done a good job, but don't be confident of that too soon.

    As always, feedback is greatly appreciated.

    Next up, preventing security risks through software and infrastructure architecture.
    #2 MrDienns, May 12, 2020
    Last edited: May 13, 2020
    • Like Like x 2
    • Informative Informative x 1
  3. Nice series, spotted a typo:
    • Like Like x 1
  4. Andre_601


    (At the beginning, after the introduction)
    • Like Like x 1
  5. Thanks @MrDienns! Found a typo on the header:
    should be
    • Like Like x 1
  6. Bro you are a legend.
    Just like that you are a legend.
    I'm not owning a server but help people that have servers. You made 4 parts until now lest go for the 5th part now
    • Friendly Friendly x 1
  7. Thank you very much for your insights into IT security. ♡♡♡
    • Like Like x 1
  8. After I write part 5, I'm thinking of adding another series (or parts) to this called "Little exploit, big dreams" where I show the reality of a lot of the exploits or risks mentioned throughout part 3 and 4, and show the reality behind them and what damage can be done, and how (though not like a tutorial). Like, most people probably know SQL Injections can hack your database, but it would be interesting to see how you can get full root SSH access on the server like that (which, yes, you can). Would people be interested in this? Keep in mind they won't be tutorials, more posts to show the very harsh reality behind all of these things.
    • Like Like x 3
  9. Andre_601


    One thing I would always like to see is a post explaining how all those supposed "hacked server" videos are most likely fake.
    Quite a few plugins (most prominently LuckPerms) receive negative feedback because people believe it has some sort of backdoor or exploits which in most cases is just either horrible configuration or intentional/faked hacking.
  10. The problem with that is that it's easier to prove something being possible than it not being possible. Like I explained in part 2, lateral movement can often occur, at which point it may be a complete mystery how a certain hack occured. LuckPerms itself is likely not directly exploitable, but perhaps the exploit of another plugin made it possible. We don't have any insight into these server setups so it's near impossible to disprove it.
  11. Thank you for putting all this to paper. It's a valued resource to link to server owners who ask 'any tips?'
    • Like Like x 2