Server World Size

Discussion in 'Server & Community Management' started by IronAnvil, Apr 4, 2017.

  1. I have been searching for quite some time for an answer to my question. I have found several forums where my question was asked but never answered in a way I wanted to use.

    I run a spigot server with a world size larger then 50 GB. I know a large portion of the chunks and possibly several entire region files are chunks that are unchanged. People travel long distances on the server and simply walk through chunks without changing anything. I want these chunks gone. They can regenerate if someone visits them again.

    I am looking for a tool to find and delete these unchanged chunks and entire region files if nothing is changed in them. I have heard the worldborder answer. I don't want a border and I don't want to blindly delete everything outside of it. I have also found tools that search chunks for a list of block types and delete any other chunk. I don't like this answer either because of the potential to delete builds or old ruins. I have seen entire builds made of dirt. Also these tools have not been updated in a long time.

    Another problem I run into with most tools is the world size itself. Programs like NBTExplorer and MCEdit try to load the entire world and crash. I could load region files one at a time but there are tens of thousands of them. Compared to other servers(example 2b2t) my world file has to be small. I assume a tool like this would be useful to lots of server owners.

    Does a tool to perform this already exist? I realize it could take a long time to process a world of this size. I have no problems with that.

    Second request. NBTExplorer code is open source. Can someone add to it a function that is similar to Notepad++ Find in Files? I want to search my entire world but it would have to open region files one at a time and scan them and then output a list of all located instances. I have attempted to write it myself but I am not to savvy with that code.

    Any suggestions or input would be gladly appreciated. If there is an easy answer or something easy to find that I missed, feel free to point out my stupidity. I will appreciate that as well.
     
  2. Gaxan

    Supporter

    Mcedit will do it if you increase the ram in the graphics and options. You are going to be in for a long process.
    I used mcedit to generate a 40GB map to reduce server resource use. Admittedly it did take 12 days and 30GB of RAM to do it.
     
  3. Can MCEdit be used to scan worlds for unchanged chunks?
     
  4. If you use React by @cyberpwn, it has an option to unload the chunks (manually, idk about automatically) to clear up space and ram. Try it.
     
    • Like Like x 1
  5. When you say it has an option to unload chunks, does this just unload them from memory? Most of the chunks I am talking about never load into RAM. They were walked through once to get to a set of coordinates, not changed at all, and will probably never be loaded again. So I just want to delete them completely.
     
  6. Well they will be unloaded until they are loaded or walked into again, thus freeing up CPU, ram whatever.
     
  7. I am looking to reduce actual stored world size. I don't have a problem with RAM usage. I would imagine any server with a world border of 30mil and lots of players or a long uptime has the same problem. I would guess that most of my region files are unmodified chunks. If you walk from spawn to 1,000,000 you just saved 62,500 chunks to file. I have players beyond 10mil. I don't want to limit their journey, I just want to delete the files they will never walk through again.
     
  8. Each chunk has an "inhabited time" value for how many ticks players have been in it (multiplied for additional players), making a scanner to look for this and delete chunks with under, say, 200 ticks (10 seconds), would be fine.

    Incidentally, i am experimenting with compressing region files as part of a launcher i am working on for spigot. I have no idea how well it will work yet, i am still testing for unexpected side-effects, and it would definitely break stuff like mcedit and corrupt chunk fixers.
     
  9. I was looking into the nbt data for the InhabitedTime. That is actually a brilliant idea. I loaded three different random region files. 2 files were completely filled with chunks that had inhabited times of 0, meaning entire region file could be deleted. From my example files, 3 chunks out of a few hundred would be spared from being deleted. Though that is too small of a sample, the world size would be decreased an impressive amount. I have read that after deleting the chunks, the map would require a defragmentation to actually decrease the file sizes.

    In my example of a walk to 1,000,000 being 62,500 chunks, I forgot to think about the radius chunks loaded by that same player. Resulting in the inhabited time of 0. In the case of my server that radius is 6 or a path width of 11 if the journey were a straight line. If my evaluation of that is correct then it is actually 687,500 chunks that were saved to file. 625,000 of those chunks were never even inhabited. All so the player can build a base and exist in a few hundred chunks.

    If anyone is good enough with code to pull this off, I imagine I am not the only one who would appreciate it. I will attempt it myself.

    Thank you FlyingLlama
     
  10. Highly unlikely that any defrag would be needed on a modern file system, but it cant hurt.

    I can definitely make a plugin to do this, with a little time, this will be my task for today.
     
  11. Thank you. In plugin form would it just always be scanning or scan when a command is run? I was thinking about a desktop tool when I originally posted but a plugin would make it so I don't have to download, process, and upload.

    One thing. I have been playing with an offline world created for this purpose. I was trying to figure out how to determine unchanged chunks. You answered my quest with InhabitedTime, but offline it does not appear to update that value. I created a new world. Placed a command block at 8,8 so I knew I was in chunk 0,0. I verified the command block is in the file but the InhabitedTime is 0 even though I was in that chunk for over a minute. I don't want the tool for an offline world anyway. It was just something I noticed.
     
  12. I am starting work on a desktop gui + commandline tool right now. Making it a plugin would just invite all sorts of errors and issues.
     
  13. Apparently every jnbt-based library is broken/outdated and cant read mca files, i will need to figure out an alternative.
     
  14. GaIaxy

    Supporter

    One of my worlds is 30GB for some stupid reason. I've been transferring 13K files for the past two hours because of this.
     
  15. A desktop application would be my preference as well. Modifying world files in a loaded world sounds like a bad plan.

    NBTExplorer is open source and can open mca file but if I remember correctly it is a .NET C application. Google brought me to this page on region formatting, but I haven't implemented it in anything yet. I love code but I am still at the tip of the iceberg when it comes to java, I'm afraid.

    My guess, that stupid reason is the same thing I described above. I once had a player get onto the nether roof and use autorun and autoeat with a hack client. He did this all night long. My nether is 18GB, not just because of him.
     
  16. The spigot implementation seems to rely on spigot classes and about 5 nested input streams (OOP insanity). I am going to try to either use a modified and stripped-down implementation based on the vanilla server, or make my own. Needless to say this wont be done today.
     
  17. No problems here. I am thankful you are working on it at all. I have coded in a few languages, I understand it doesn't magically write itself.

    I found one other forum where someone asks the same question but he was talking about offline files and dealing in MB. He was good with the answer of MCEdit. I am kind of surprised I couldn't find more then that on the subject when it came to the larger server files.

    As you implied, at least in concept, it is a simple solution.
     
  18. Idea... But there are limitations to this.

    There isnt really a good way to tell what chunks have been modified or not without doing something when the chunk was generated in the first place. Also, the chunks are... technically modified (water loves to flow and update) most of the time.

    Region files are not chunks, they are 32x32 sections of chunks. So that may not be the best way of doing it.

    As for a plugin... i could make something for it, but there is one major problem right off the bat. I cannot know if a chunk has been modified or not unless i "track" all chunks that no player or player induced change has modified. So it would not fix your current world, but it would fix all future unused generated chunks. Maybe even put a file cap limit per world? So it will only start to delete the outmost ungenerated chunks once the file size of the world is met? I can make something like this for you, but just know it wont help your current situation (you need the plugin when you create the world and keep it on there.

    Just know that it wont improve performance, or anything. It may actually hurt performance a bit (since deleting chunks makes minecraft have to generate them again when a player is nearby) All it will do is reduce world sizes.

    Message me if you are interested.

    [​IMG]
     
    • Creative Creative x 1
  19. InhabitedTime.

    The only difficult part is loading a MCE file, which seems to be some sort of zip-like indexed series of NBT files, but the actual format is undocumented, as far as i can tell.
     
  20. FlyingLlama, did you see the link I posted about region file format? You would have to reinvent the wheel if you use it. It is what I am attempting to do as well. http://minecraft.gamepedia.com/Region_file_format It is a complicated file to read, but it is all there.