Database help

Discussion in 'Spigot Plugin Development' started by Hex_27, Jun 4, 2017.

  1. Let's say I have a database, and I load all the instances from the database to memory when the server starts. All these instances are stored in a hashmap (String, Object). How do I remove a specific set of the instances and erase it from database?

    Right now, I have a synchronised method that does it (loops through the hashmap, and checks if the object should be removed). But now, when the server has a lot more players, and everybody is constantly causing the hashmap to be used, the synchronised method can take forever to complete. Is there a better way for this?
     
  2. I am pretty sure the whole point of data in databases is that you don't load the data but access it using the database.

    you can DELETE FROM table WHERE some_condition or something like that
     
  3. But the data is constantly accessed, even at the point of every tick. Isn't it better to keep it in memory?
     
  4. Well databases are created for handling big amounts of traffic so unless the database is running on another machine (aka slow because its over the internet) I'd think the database will handle it better than you do in 99% of the time. How do they do it? No idea, never created a database.
     
  5. Database location varies in this case, it may be on another machine
     
  6. Mas

    Mas

    You really shouldn't be loading everything in the database into memory. Load data related to a player into memory when they join and then save it as they leave.
     
  7. It isn't player-related, it's chunk related, and when certain actions are done (i.e. breaking, placing, entering, leaving), the data is needed
     
  8. Just do all the stuff with your hashmap and save to the database every ~5 minutes, and on onDisable of course.
     
  9. That's not the problem, the issue is that it's already in the hashmap, but there'd be a concurrent modification error if I try to access it at the same time. To solve that, I used synchronised on certain methods, but that made the method take a long time to execute, presumably to queue up to get the instances from the hashmap
     
  10. Is it an option to use a ConcurrentHashMap? Sorry if it sounds stupid, this isn't really my cup of tea.
     
  11. What exactly does a concurrenthashmap do? I've tried looking for what it does before, but the explanations I found were overly complex. Any general explanation?
     
  12. electronicboy

    IRC Staff

    use a cache that loads data upon the chunk being loaded, and remove it from memory on unload. Guava has a pretty awesome cache system for handling this, however; it doesn't perfectly suit your needs as it's an evacutating cache and loves to try to remove stuff from memory itself, which, has advantages and disadvantages...

    a concurrent hashmap basically moves the highly complicated parts of synchronization to the inside of the implementation of a hashmap. it's basically designed in a way that will not throw a CME, at the cost of a small amount of overhead over a standard HashMap in order to try to garauntee that when an object is added to the map, it will actually be added to the map, or removed. (Thread safety is a bitch, and is generally a cost between complexity or performance)

    You should also be weary with synchronised methods, 99% of the time, people synchronise a full method when they only need to care about a small part of the method. "doing it properly" means that you can potentially reduce the amount of time spent locking, however; exactly what your method is doing can affect how ideal of a solution this is and if it's possible.
     
    • Like Like x 1
  13. What's the difference between caching and putting it in a hashmap?

    So I should remove synchronise from the method, and try using concurrent hashmap?

    EDIT: And the overhead refers to the initialisation, or putting a key and value into the hashmap?
     
  14. electronicboy

    IRC Staff

    caching is a concept or storing data that you want fast access to in memory. The idea is that you don't care about 99% of that data 99% of the time, you only care about data which is loaded in the chunk, thus; storing 100% of the data in memory is a complete and total waste. Guavas cache basically has a fair chunk of the logic you care about already implemented in a way which is guaranteed to be performant.

    any operation on a concurrent hashmap that has the potential to modify data stored in the maps indexes, e.g. remove, put... will have a potential additional overhead, they've limited the chance of locking to only cases where they actually need to, meaning that it is generally a safe option for most cases so long as you're not going to be performing large amounts of modifications to the map. (the use case of MC is generally not all too much of a concern on the performance of this, and will likely be *much* more performant over your implementation of synchronised methods). But, this does not guarantee any form of safety, e.g. you've removed the chance of it throwing an exception when you twiddle with it from two threads, however; you have no form of safety beyond knowing that 2 put operations will actually work. it is not a perfect solution and has many implications, such as the potential for race conditions. e.g. you run a put on another thread, can this thread actually see that that value has been put in the map yet, or is it still not in there?
     
  15. Well.. there is an issue here. We do care about a lot of the data at a lot of times. Like perhaps 70%-80%. There are a lot of repeated checks occurring in the plugin for all kinds of things
     
  16. electronicboy

    IRC Staff

    in those cases, the cache would keep those loaded in memory. if your database covers chunks that are loaded 100% of the time (e.g. you need to access a majority of the data in the database a majority of the time, and not just chunks that are loaded or the occasional lookup), using a database really isn't the ideal solution for your data storage, and if you really do require all that data to be loaded, you should probably look away from using a database, and consider looking into storing the actual data on blocks/chunks themselves
     
  17. How would that work?
     
  18. electronicboy

    IRC Staff

    You would basically need to look into the joys of reflection in order to store data using NBT, which isn't ideal.
    the other is to accept the fact that what you're doing is not suited for a database and look into using flat file storage. tools such as gson are nice for serializing lots of data to the disk, performance of that during saving is questionable and has its own downfalls, however there are ways of mitigating against that, e.g. each chunk has its own flat file, stored in a directory tree that attempts to reduce the number of single files in a directory.

    databases have downfalls and are "somewhat" redundant if you want all the information inside the database loaded at once in memory in the application connected to the database. I highly doubt that the data that you have in memory is being accessed to the degree you're stating that it's accessed, and if it actually is in such a way that using a proper cache based system won't work, you're pretty much-doing something wrong somewhere.