How to get rid of Bot Playerdata?

Discussion in 'Server & Community Management' started by Bloxxinator, Apr 23, 2017.

  1. Hello fellow Server Owners,

    I run my own Spigot 1.11.2 Server (Offline Mode) and have the following Problem:
    A few Weeks ago, there was a Time when my Server repeatedly got massively attacked by Join Bots / Fake Players (e.g. from mcspam.com). In the meantime I've installed an effective AntiBot Plugin which now protects my Server from such Attacks, but the Repercussions from that Time are still noticeable.

    There are now more than 13000 Playerdata Files (named by their respective UUID) in both the "playerdata" Folder of my Minecraft World (Directory: "/Minecraft_Spigot/world/playerdata") and the "userdata" Folder of the Essentials Plugin (Directory: "/Minecraft_Spigot/plugins/Essentials/userdata").
    I would estimate that around 60% of all those Files are Bots and not real human Players, and they are occupying a damn lot of Memory.

    So my Question is: How can I get rid of those Bot Files without deleting the Data of my actual Players?

    As for the "userdata" Folder of Essentials, I already thought about the following:
    These are ".yml" Files, so I could open them with Notepad++ and simply erase all Files with the Line money: '1000.0' in them, since the starting Balance each new Player on my Server gets is 1000, and Bots usually don't engage in the Virtual Economy System of the Server (because they immediately leave again after they've joined). The Problem here is though that it would take me Ages to check all of the over 13000 Files, delete the ones with Standard Balances, and then compare their UUID Names to the Files of the other Directory with the "playerdata" Folder, to delete the Counterparts there too.

    The Files of the "playerdata" Folder are written in a ".dat" Format, so I unfortunately cannot extract any valuable Information from them.

    My second Idea would be that I only delete all the Data of Players which have been online on my Server for less than three Minutes, because Bots only stay for a very short Time until they disconnect. But I don't know how to get the Playtime Information from the individual Files, and again, even if I managed to achieve that, it would still take me Ages to check and delete all the affected Files manually by Hand. I would need some special Software, Program or Plugin to execute the Task.

    Do you have any Suggestions what I could do? Any Help is highly appreciated!

    Thank you for reading :)
    - Bloxxinator
     
    • Informative Informative x 1
  2. the files in player data are uuid named iirc so you could simply gather the uuid's of your current players, copy them to safe place then delete everything else and move the real players back, should wipe everything the bots did
     
  3. I have much Players, so the Risk is that I forget some and all their Account-related Progress will be gone. Same Issue with old Players who don't play anymore, but maybe some Day like to come back. I don't want them to lose all their Data.
     
  4. maybe you could try deleting the playerdata based off of the time stamps on the files, if the attacks happened in a specific time and then stopped you could perhaps remove the files created/last modified during that period and just keep them in a zip file or something as a backup incase you accidentally remove a few you shouldnt have

    honestly not sure what the best way to do this would be, probably have to be manually done unless someones made a tool for this sorta thing
     
  5. There exist plugins that remove 'old player data', .dat and Essentials and probably other, but they don't distinguish by what the players may have been doing on your server. You'll likely need something specialized, unless it's ok for you to just remove all old data.

    I would automatized this for sure - question is what level of safety you need...
    • Collect UUIDS from .dat and plugin data (plugin can do)
    • Default money. (a plugin can do that)
    • OfflinePlayer: Played only a short time, played/left before you added the anti bot plugin, played during a known attack phase? This can be done by a plugin.
    • OfflinePlayer: Check location vs. world spawn(s) and perhaps public warps, if they were that sophisticated bots. (Needs an API for offline players)
    • Check other plugin data for reference - e.g. cross check last joined and such, if e.g. Essentials or another plugin keeps such data too, but more important: Check things like mcMMO levels for more safety.
    • Naturally more data can be checked (permissions groups, ender chest, chests, regions).
    • (Action needs to be distributed over time to not lag the server out.)
    Edit: Isn't the player.dat simply an nbt file? In that case it should be possible to scan them asynchronously and make some basic judgement, which uuids might be bots, then continue to check plugin / eco data and/or vice versa. An up to date NBT library and a little reference what to parse could help here...
     
    #5 asofold, Apr 24, 2017
    Last edited: Apr 24, 2017
  6. Thank you for your very comprehensive Answer! (What an Honour to talk to the Developer of NCP ...) :D

    I would prefer the "Having played less than three Minutes" Method, since it is probably the most accurate and harmless Way to do this. I've installed the AntiBot Plugin on February 28th, so this automatic Process should only affect Files which have last been changed before March 1st.

    I'm not far as competent as you are on IT Issues, my Knowledge only goes so deep that I can properly manage a Minecraft Server, but not develope my own Plugins/Mechanisms. Can you help me with that? And if so, would you do it for free or for charge? My Budget is not very huge though ...

    In any Case, thank you for replying.
    - Bloxxinator
     
  7. Can't promise much to follow up with quickly - might try to mock up something with existing plugin parts.

    The fastest thing would be to scan the economy files first and then delete all related files if appropriate, without hooking into Essentials/Vault. For higher reliability it probably would be better to cross check some things, e.g. position, and to use the removal function that Essentials should provide - perhaps in the second round... if the first attempt isn't as effective as hoped for.

    So the first go would probably be a plugin:
    • Have a configuration what to scan for and possibly what to exclude. Probably have named models to reference via command.
    • Have a command + permission to run the thing.
      • Read file names from essentials or player.dat.
      • Only process so and so many players per server tick, maximum of 2 milliseconds or so runtime per server tick.
      • Check essentials player data files for pre-selection (file system, parsing or dumb search).
      • Get the OfflinePlayer and check when they've played last and how long in total. (possibly first seen -> last seen too, for an option).
      • Remove files right away. -> suggestion is to move them to a folder within the plugin folder of this plugin, mimicking the other folder structure as if it was the server root. Could skip this at your own risk :).
      • Perhaps log either to an extra file or to server log, which players are removed. Server log should be ok, if it's tagged so it's easily extractable/searchable just in case.

    Apart from the direction:
    • Do you have Vault installed (Essentials Eco?)?
    • Are player.data and essentials player data the only concern? No other (mcmmo/...) ?
     
    • Useful Useful x 1
  8. Yes and yes.

    If you commit to make this and put it on SpigotMC.org as a (Premium or not - just not too expensive) Ressource I can download, I would be very thankful to you. And take your Time, it's also okay for me if you do it in a few Weeks, I don't need it immediately.
     
  9. Oh okay - i'll make some preliminary testing towards the weekend. Be ready to restore your entire server after running the pre-alpha, once it's up...
     
    • Friendly Friendly x 1
  10. Cool. My Hosting Company regularily makes Backups of everything, so that Part won't be hard.
     
  11. I'm not intending to destroy it :) - unfortunately i got lost closing 80 or so tickets - so i didn't make it this weekend. Unless you find someone to do it, i'll likely manage until next weekend. The simple heuristic with "played max. 3 minutes and joined last time before so and so + has the 1000$?-start money on essentials" for moving 2 files to a backup folder really isn't that difficult - of course it's more than nothing, because one doens't want to crash the server "skimming" through 13000 files :). (Bookmarked, so i don't lose track for 24 hours again.)
     
  12. No, I haven't found someone else to do it, I'm still counting on you. :D Thank you very much for your Effort, looking forward to Completition!
     
  13. Hey asofold, may I ask are you still working on this? I really need some Solution for my Problem in the near Future.

    Sincerely,
    Bloxxinator
     
  14. Sorry, i was too close to a cold and didn't do much of anything else either - i'll make a prototype for wednesday (latest)
     
  15. I wrote a shell script that checks the banned user data (with uuid and without) against the data we wish to prune.
    I can see if I can get permission to share it with you (no promises)
     
    • Like Like x 1
    • Agree Agree x 1
  16. I got a no on sharing the script, from 2 others involved with the servers ,sorry.
     
  17. Thank you, I'm already exited to run it :D

    No Problem, thanks for the Offer though.
     
  18. Players playtime is stored in their stats, so you can check every stat file and find their playtime, to generate a list of uuids that are no longer in use. I'd happily make this application if you want, just let me know. (It won't be a plugin, but a Java application that you run in the same folder as your Spigot while the server is offline.)