Store and use data from plugin memory or database/flatfile

Discussion in 'Spigot Plugin Development' started by Shin1gamiX, Jul 13, 2019 at 8:26 PM.

  1. I am quite curious for this one.

    I currently have everything loaded into plugin memory (basically a map with a (player) uuid as a key and a specific object (encapsulation) as a value). The value basically contains pretty much everything that would be needed for the player.

    In order to make this restart persistent, since map contents are removed (not removed but you get the point) when the plugin disables, I load contents to file and load them back on enable. When saving data, every content of the file is deleted so as not to cause any duplicate issues in case some data was removed from the map and not from file.

    Would it be less efficient (performance wise) to store data on file and load them back from it when I need to use it, everyitme?

    One benefit of this would be that in case of a crash, data would already be saved and I would no longer require to have a repeating task to save data every X seconds. The task basically acted as a fail safe for if a crash occured and the on disable method wouldn't execute. Data would already be saved so no issue there. Though, performance is being hurt with this task, a bit.

    Another benefit would be that I wouldn't load everything into memory and would only get something when I need it (though this may happen quite often).

    I am unsure whether I have a wrong perspective about something, so I'd appreciate any feedback and advice on this.

    Thank you in advance.
     
  2. Loading when needed would be more resource efficient, as long as you remember to decache the data from memory when it's no longer needed.
     
  3. The issue is, it is needed 80% of the time. Sometimes, some specific data may be required, other times the whole database may be needed, thus why I am concerned about performance.
     
  4. You should read it on load, you can use a repeating task to save it or everytime a change is made, write it, maybe async to not make a big charge
     
  5. YamlConfiguration works async an it has its own api, so you should give it a try
     
  6. If you're talking about your VoidStorage plugin, the issue here is that you keep all the data concerning all the players in one huge file. Since that file is so big, saving or parsing it takes a long time. Especially if a server gets older and there's thousands of players who've once joined the server and have a bunch of data yet will never do so again.

    Now, if you were to create a separate file for each UUID, you'd have a lot of smaller files. Loading or saving these individually would not take nearly as long. So whenever a player joins, you can load their data from their personal file. And whenever they leave, you can save said file. The files corresponding to players that only ever visited once, never even get touched.
     
    • Agree Agree x 1
  7. Changed are made nearly 80% of the time though. Having it done async wouldn't be much of a difference as far as I know since the file is only saved once after everything has already been set to file. This is what I do at least, sync.

    I am already using YamlConfiguration, however, that doesn't answer my questions.

    I am talking about it. I have thought of doing that, however, saving data when they leave would certainly be a good idea but what would happen in case of a crash? All data would be lost? I'd then need to have a repeating task save stats every X seconds which would still hurt performance since this time, all the files would need to be saved instead of just one.
     
  8. Try to use SQLite instead of files, that should be faster
     
  9. Would that be supported by all users? Meaning, would they be required to do something in order to use SQLite? Since they don't need to for yaml files.
     
  10. No need, you can set it up all for them, not even data/user/pass info if you want
     
  11. Is SQLite an online database or local?
     
  12. Local like yaml, json, or file systems
     
  13. Sure, you could still do something like that. But you'd still be saving a very small fraction of the amount of data.
    Let's say you've got 2 000 people who've ever joined the server. You've got 20 people online at the current time. In your method, you're saving all the data of all the 2 000 people every X ticks. If you only saved the data of the 20 online people, you'd only save 1% of the total data. Sure, there might be some overhead in saving multiple smaller files, but it won't be a hundred fold.
    Furthermore, you could easily lessen the strain by spreading out the data saving between different ticks (i.e even one config per tick, if you like). And that's simply not something you can do with a massive file containing all users' information.
     
  14. That seams reasonably better. Though, saving ~2000 people's data would be needed since some statistics would change even if they are not online at any point. (For as long as their stuff are loaded, and I am referring to chunks).