anti-DDOS/Load-Balancing of a Network

Discussion in 'BungeeCord Discussion' started by mkcoldwolf, Feb 11, 2020.

  1. Hello there, im here to discussion with you guys about the DDOS protection of a big Network, i have been searched thousand of threads on the internet but isn't really found a thread that talk about how to protect DDOS for a big network such as hypixel and mineplex, i dont know whether they don't need it or those owners dont even care about the availability of their server, also i hope this thread can be read by more people and help them protect their network. here i will share what i can think of and i know that with my tiny knowledge, there must be something more can be added to perfect the protection :L, i will ask some of the question i dont understand too

    1. At the beginning, i have an HA structure design pop out in my brain by using multiple BungeeCord setup at multiple VPS around the world, and then use round robin DNS to load balance for each bungeecord, by this, we can also increase the ping response from bungee to player (not the spigot response time), and this is which im currently using
    upload_2020-2-12_6-40-14.png
    while this design have the problem below and im going to deprecate it
    - Attackers can first lookup the DNS, pick an BungeeCord and focus his attack on that, which ignore the DNS load balancing and kill the BungeeCord
    - Onces if one of those 3 bungeecord is down or getting maintained, DNS is not able to check the health BungeeCord, and it still redirect player to a bungeecord which doens't work, unless we manually remove the BungeeCord from DNS or using an api, but DNS have cache so......


    2. after that, i want to know about wat actually is the HAProxy, BungeeCord seems to be support it officially with enabling "proxy_protocol" in the config, is it really just an software which redirect those traffic from player to BungeeCord? with this, is this also mean that i can hide my BungeeCord's ip behide? Can HAProxy check the health of BungeeCord b4 sending player to it?
    if all above is true or possible, i can then think of this design
    upload_2020-2-12_6-53-36.png
    This design can even protect the ip of BungeeCord and i can easily fork HAProxy for health check things maybe? there is no chche and the OVH vps can afford a much large traffic in same time, i can even add more HAProxy if i want, also, sinces OVH has a higher bandwidth, meaning that i can defend a larger DDOS, but here comes the problem.
    Do i really need more than one OVH behide the DNS? because as same as the 1st design, attackers can still focus his power to take down one OVH, but only half of players will be affacted
    And if the OVH VPS is powerfull enough, is that mean i dont need HAProxy on OVH anymore and i can just setup the BungeeCord on OVH vps?

    Finally, if i has anything wrong, plz correct me, and you can send ur design which can improve my answer too!!! its important for us to learn
     
  2. I have been searching for a similar setup to such as this. My current setup for my networking, which I will refer to as the "traditional setup," is as such:
    [​IMG]
    It is not too important to understand the setup within the socket bridge and spigot servers, but it's basically just a socket server that links all the spigot servers, which have socket clients running on them, to the database and discord bots running. I set up my own packet protocol and everything for this, it works nicely, and allows me to immediately update all servers when database changes are made. Anyway, that's beside the point.

    Currently I am paying for two droplets on DigitalOcean, both hosted in the NYC1 region, but we will refer to them as NYC1 and NYC2 to indicate they are separate machines. I do host my minecraft-related websites on the NYC1 server, however I have another server (NYC2) which hosts my other personal website. I have a local machine, which runs all the spigot servers, the database and the socket server (as described above). And yes, this does run at my house, so you could imagine me wanting to hide this IP address for a plethora of reasons (also yes, before you say it, I know this is bad practice and should be hosted remotely. Turns out, it's cheaper and worth the long term investment in my own machine. It also works really well for what it needs to do). This setup is not ideal for a couple of reasons:
    1. The BungeeCord and Websites have (at least) a 30ms latency on its calls to the spigot servers and database.
    2. I have to pay for two remote servers, which totals to about $10/mo (which isn't much, but still an amount that could hypothetically be unnecessary).
    I recently came up with a hypothetical solution in my head very similar to mkcoldwolf's question regarding DDoS protection and HAProxy. I saw them post a question on a discord server regarding this, and it intrigued me, and asked them to ping me in any followup they get. They tagged me in this, so here I am :). My hypothetical solution was that if I were to host all the websites (doing something like this), as well as the BungeeCord on my local machine, completely eliminating the delay between database calls and interactions within the server. However, without some sort of mask or proxy, this still allows people to have access to my personal IP address (and that is no bueno times 10, due to things like geolocation and DDoS attacks :eek:). So my idea, was to set up a proxy of sorts that would mask the IP for both the websites and the BungeeCord. This is the desired network setup:
    [​IMG]
    Nothing much different from the traditional setup, but it is much more desirable internally. I would also likely want to move the proxy to some region closer to my local server, as it would decrease the latency for people connecting through the proxy. This setup is more ideal for a few reasons:
    1. Only need to pay for one remote machine instead of two (which cuts the expense from $10/mo to $5/mo).
    2. The BungeeCord and Website(s) can access the database with no latency compared to the ~30ms from the traditional setup.
    3. I can close the database access to localhost only (more security! yay!).
    Given what mkcoldwolf has said on the topic of HAProxy, it looks like it would work perfectly for what I am aiming for. It would not only hide the IP of my server machine, but also establish a two-way connection. I do not need it entirely for the same reason as mkcoldwolf, but for a very similar reason nonetheless! I will continue to keep looking into this and see if I can find anything.

    Thanks for coming to my Ted Talk. :D:coffee:
     
    • Like Like x 1
  3. electronicboy

    IRC Staff

    The trick is to often use services more designed for pulling stuff in/out, or having a few dozen IPs, then you just perform health checking and pull the IPs out of the DNS zone if they're down
     
    • Agree Agree x 1
  4. Yes, the lower one seems to be the better choice if the NYC and afford large traffic, and i also think that multiple BungeeCord is needed, even u do hide ur IP of BungeeCord, HAProxy will still transfer the flood packet to the BungeeCord with may make BungeeCord down :L, unless ur NYC vps has a DDOS protection or something.

    I dont understand :thinking: what is point of having dozen IPs in DNS? attacker can just randomly pick one and focus his attack on that, it would still affact the player using that IP to disconnect, also pulling DNS is slow and not a good choice because it have cache, after i updated the DNS, player may still not getting it
     
    #4 mkcoldwolf, Feb 12, 2020
    Last edited: Feb 12, 2020
  5. electronicboy

    IRC Staff

    MC is too stateful for transparent transfers anyways outside of level 3 transfers (So, things like HAProxy is 100% irrelevant here, just, haproxy is less stateful so easier to scale up than bungee, but, bungee is going to be your weakest link in the terms of all of that setup bar the servers themselves), so no matter what you're going to drop people if a node is attacked to the point that it's killed, thats where OVHs ddos protection comes into play or services like TCPShield, basically; caching is 100% dependent on the network, roundrobin DNS has been used for this kinda thing for years, pulling records if they're under attack is just a reliability thing to try to prevent other players connecting too it, caching is an inconvenience here, but somewhat of a non-issue, just use a sane TTL
     
  6. If throughput and minimal player loss is your focus, and you want to make your own "solution", I would recommend accumulating a good number of OVH vps dotted throughout the world (e.g. 1 or 2 in low traffic regions, 3 or 4 in higher traffic regions such as NA). With this global ovh presence of 20 or so OVH vps, you could use Route 53's geo ip routing (for routing your players to the lowest latency PoP for them, all on one subdomain), along with utilizing their health checks for removing downed proxies, and readding when they're accessible again.
    With this setup, you'd have nearly the lowest latency available for your players, and you'd lose 5-10% of players in a worst case scenario DDOS. (it's been years since the last time DDOS was an issue for me when using OVH)
    Or, if you'd like, you could use TCPShield - it's free.
     
  7. So which means there is no ways to always keep every bungeecord to be alive, but player can rejoin for another bungeecord, then i would ask for it, HAProxy can check the health of an bungeecord before sending player and doesn't have DNS cache unlike Round Robin DNS, then why dont we use the second design which add an layer of HAProxys? is it because that it is actually no use and if attackers target the random HAProxy, it stlll make a point failure and it result same as the design 1 that some player will disconnect?


    Yea, thats a point for settingup Proxy/Bungeecord to decrease the ping
     
  8. For sure. That is definitely the route to go when running a much larger server like Hypixel as previously stated. However I'm not looking for insane scaling at the moment, I'm mainly just going for security. :)
     
  9. electronicboy

    IRC Staff

    your only as strong as your weakest link, it's easier to spread out actual bungee instances than it is to manage a layer of haproxy instances and a layer of bungee instances, also means that you have less stuff in the stack which are viable to knock outta the air
     
  10. or, instead of using haproxy, use route 53's health checks, as i mentioned earlier.
     
  11. Oh i see, so to be concluded, for smaller network, and the reason of less vps needed, we better use the design 1 even there is cache for RRDNS, or if the network getting bigger, we can try adding layer of Haproxy or route 53's health check to update bungeecord maintain/down state instantly (which maybe is not really a problem), but anyways or how ever i design my network, without DDOS washing like TCPShield or something, its not possible to keep everythings up, attackers can target one of the bungeecord to kick some of the players right? even hypixel and Minplex those network also have this problem?
     
  12. larger networks will typically have many endpoints - this is why i suggested 20+ ovh vps, depending on your desired level of redundancy. even in the event of a ddos going through OVH's protection (which is already rare in my experience), only a small percentage of players would be disconnected, and they'd be able to connect right back.
     
  13. If anyone feels like trying something new, I'd be interested to see how Linode holds up. They have load balancers at $10/m that can proxy traffic to your instances based on different algorithms, and have just revamped their DDoS mitigation, iirc it has at least 1.6tbps of capacity? Many locations, and no need for DNS based solutions.
     
  14. Others have shared good points already. To summarise things;

    Your access points are your weakest links (Bungee in this case, most likely). More access points means more points to attack. If they're all protected (like at OVH), this means its way harder to take down mulitple access points. More access points means more split connections. More split connections means reduced damage in case something does fall. I believe Hypixel has over 200 bungees.

    Round robin DNS can work, but be aware that DNS does not validate that the service you're connecting to (the given IP) is actually alive. This means you have to have something which synchronises health checks with your DNS. Load balancing IPs are nice to have in order to load balance your bungees. OVH sells those, but also providers like Cloudflare offer them to some extent. That makes things even less likely to go down.

    So, TL;DR;

    Client > DNS > LB IP cluster > Bungee cluster > Spigot

    Good luck taking that down.

    Do note, this is extreme. You have to see for yourself whether or not the massive infrastructure is worth the investment compared to the potential damage.
     
  15. One thing that has kind of confused me with multiple bungees is how does that work when getting the list of online players? I would imagine the list would not be consistent among all the different servers, as they are different connection instances. Unless it works opposite of how I'm thinking and the spigot servers tell the bungee server(s) what players are online...
     
    • Friendly Friendly x 1
  16. electronicboy

    IRC Staff

    General gist is that most servers large enough don't care about the player list, they just omit it in favor of not having to deal with maintaining a list of players somewhere, some even take advantage of that list to throw in other info; plugins like redisbungee will maintain the count for you, however
     
  17. The attacker can kick the players from each proxy server in turn. For me the most important attack "Layer 7 OSI model" is now.
     
  18. ok this is mostly the best summarise for now, ill use this design for my owner network

    now here i have one more question, how do i optimize the route of player's connection? like i have one bungee on japan, one on hong kong, i want the players at hong kong can use the hongkong bungeecord and japan players use it own, how would i do this? does DNS load balancing already did that?
     
  19. and Yes, Redis becomes a very important software, its a ram storage software faster than MySQL, server have multi bungeecord mostly develop their own plugin to transfer data between redis and bungeecords, such as Partys, Cross Bungee Message etc., with also the built-in push/sub feature by redis
     
    • Like Like x 1
  20. CloudFlare's round robin solution doesn't. You'll specifically need a Geo DNS solution, such as AWS Route 53