Solved Config size affect server performance?

Discussion in 'Spigot Plugin Development' started by enesmelda, Mar 24, 2020.

  1. Just a quick question I"ve been wondering about.
    Context:I am saving and loading a hashmap upon reload/restart,using the config.With that,the config gets a tiny bit bigger,because a new UUID and int gets added.
    My Question:Does a bigger config affect the server performance?Having multiple hashMaps and more people,would make the config bigger.Does it matter?Is there a limit(what would it be)?Does the server performance get affected?Or does it have nothing to do with server performance?
     
  2. The server performance would be affected because presumably, this entire hashmap is being stored in memory. I would recommend just loading the player's UUID when they join the server (if this is about players) and adding it to the hashmap, then removing it when they log off the server.
     
    • Like Like x 1
  3. What exactly do you mean with loading the players UUID or removing it?
     
  4. So when the player joins with PlayerJoinEvent, you add their data to the hashmap from the yaml config. When they leave with PlayerQuitEvent, you remove their data from the hashmap. This minimizes the amount of data that is in memory at any one time.
     
    • Like Like x 1
  5. wouldnt this be more memory? because onJoin and onLeave event would get triggered more than onEnable and onDisable?
     
  6. drives_a_ford

    Moderator

    If you've got a lot of data, you'd be better of storing it in a database rather than a flat file format like YAML.
    So it depends on what you're storing exactly.

    It's the IO (saving to disk and reading from disk) operations that can take a long time if you've got a large YAML file.

    I wouldn't really be too concerned with just the UUID instances. All a UUID stores is 2 long values. So each instance takes 24 bytes of memory (8 byes for either long and 8 bytes for the header). So at, say 10 000 UUIDs, you're using up around 240 kB of memory. The issue comes with other what other data is stored in connection to the UUIDs.
     
    • Winner Winner x 1
  7. Would it be huge if to the UUID a single int gets stored?(with the method I am using,saving and loading the maps on onenable and ondisable?)
     
  8. drives_a_ford

    Moderator

    If all you do is load in onEnable and save in onDisable and all you store is a map from a UUID to an integer, then that would probably work.

    However, if the value can change during operation, you'll need to save it while the server is running. Because servers do crash every now and then for various reasons. And if you've got a large file you're saving to often, that can create performance issues.
     
    • Winner Winner x 1
  9. I see,one last question.Let"s say something like 10 hashmaps get stored(UUID,int).And the Server reloads 3times a day.That would create quite some size.However,there is an option to delete the config every now and then,not affecting the plugin,but more likely resetting stuff.
    What would you suggest for the max kb/mb size of a config file to be,to prevent server issues.

    Sry for asking so many question..
     
  10. Assuming that the 10 hashmaps all contain the same players, not really that much. It's just a uuid with 10 ints. If the server restarts 3 times day it's enough if you just load it all in onEnable and save it onDisable. Don't know about a "max file size" but I believe you can save quite a lot of information before you have to worry.
     
    • Winner Winner x 1
  11. drives_a_ford

    Moderator

    Something like that would probably work for quite well for quite a large amount of UUIDs stored. I can't tell you the exact number, though. It depends on a lot of factors (i.e whether you're on a HDD or SSD).

    But what I can tell you is that if at some point you need more information saved per UUID (and you insisted on using a flat file approach, which I wouldn't recommend), you can easily just create a separate file for each UUID and load it as the players join. Such individual files would almost certainly not be large enough to cause problems.

    With that said, if the reason you want to keep all of the information is because you want to know the ranks of all players, then you'll need a slightly different approach.
     
    • Winner Winner x 1
  12. It is reasonably well known that Memory performance is many times faster than accessing disk storage performance.

    The process of saving or retrieving data from a disk stored file has an overhead for every access.
    With this in mind, smaller files can take less time to be processed due to the amount of data and clusters required. (How small, or how large I don't really know for any specific OS and disk drive).

    It is up to you to organise files into relevent data clusters.

    There is no point in saving a large file (of irrelevant, or unchanged data) for only one element change, where that element could be seperated to a medium or smaller category (file).

    I am not at all sure about the details of this part, but I understand that the smallest allocatable storage space (at this time) is in 1kb increments for Windows. (Any, non-occupied allocation within that cluster is ignored)
     
    #12 Goldentoenail, Mar 25, 2020
    Last edited: Mar 25, 2020
    • Winner Winner x 1