Custom Data Storage Idea

Discussion in 'Programming' started by Swedz, Apr 20, 2017.

  1. I've used a couple different database storage ways, and I've found them to be somewhat slow. So I've decided to try something new for my server. This system is going to be specifically coded to work with my network's setup, and not really designed to work in most instances.

    I had an idea, what if; you made a "database" using .properties (or .yml) files? I know, it sounds stupid and inefficient, but think about it. Files, directly in the root directory of your servers. You would of course, have different files for different things, to keep it organized and easy to gather information you want. I feel this could have quite a bit of potential since you would just have to cache the Properties variable on server boot--and update it whenever you need to. This way you could do something like:
    Code (Text):
    //getUUID() returns a Properties variable
    And that would return "Swedz", which would be updated whenever a player joins, keeping the cached Properties file, and the newly found data without having to re-read the information from the file, but also update the file itself. This way you could potentially have an async task that would automatically update the variable every 5 minutes or something like that, so it would always have the same data across all servers.

    I've asked for opinions from others but haven't gotten a response (other than from one who was very sarcastic and said it was a "Great idea, you should try using .txt files too!" :p Gave me a chuckle, but I decided to go ahead and ask the Spigot community as well, since many here have very good points/opinions and knowledge with Java in general.

    However, one fear I have of this is that the file could become so large it would be impossible to use. However I did a small amount of research and I saw the limit was Integer.MAX_INT, which is just over 2 billion, which is around 20 times the amount of accounts that have joined Hypixel, which I would like to believe is the world record at the moment, so I don't have much to fear. :eek: (Actually, after re-reading the post saying it, it looks like they were talking about the length of a single line)

    I'm not looking for help with making it or anything code wise--just wondering your thoughts on the idea. This could very well be a very bad idea and cause a lot of stress on the server, however I don't think it would if performed correctly (async tasks auto-updating and such).
    #1 Swedz, Apr 20, 2017
    Last edited: Apr 20, 2017
  2. 30-40k playerfiles can cause significant performance hits if you're trying to pull info out of all of them at once. Generally for small amounts of data flatfiles are completely fine however databases far excel at handling massive amounts of data.

    Sent from my iPhone using Tapatalk
  3. I'm not making multiple files for each player. I'm making about 5 different files, one for storing what uuids direct to what name, another for punishments, another for ranks, another for ips, another for stats.
  4. Again, small amounts of data is fine, but when those files get 30-40 thousand players in them you'll start seeing things slow down especially on something like stats that might get read/wrote fairly often. Any time you read/write to a file it takes much more time than it does to get / set that in ram. So you'll need to be careful about how you implement this or any storage method. Something that has to parse 30k player files to see whose got the highest stat in a category is going to be slower done via flat file than done via a database. Any way you implement it.

    And if any player can use a command to look up top stats your server will really feel it if they spam it.

    The performance hits will come when you're trying to access a lot of data from multiple rows of your file.

    If you need to find out what johnnysomeone's stat is in iron tools and johnnysomeone joined your sever last, and you have 20k users in your file. To find out that information the server has to read all 20k lines of that file every single time it needs to update or get johnnysomeones information.

    This is where a relational database excels. It's capable of making very complex queries over many tables to find the data quickly and efficiently.

    Most of the databases have been around quite a long time so unless you're working with very simplistic data or will never have a large amount of users / rows to deal with then it is unlikely you'll outperform any free database solution on the internet.

    Sent from my iPhone using Tapatalk
    • Agree Agree x 2
  5. You could load the files into RAM yourself and use more efficient methods of keeping the data, like using Sets instead of Lists for example, but at that point you're basically reinventing the wheel.

    In my opinion, you shouldn't be trying to make new ways to store data. What we need is an abstraction library that gives you a set of methods you can use to get and set data that work for multiple types of storage (SQL, Mongo, Redis, Flatfiles, etc etc). I tried making something like that a while ago and ran into complications and gave up. I may try again. Would be really useful for making plugins, especially if you are releasing them and want to let users use whatever database type they prefer.
  6. I'm not making a new Properties variable every time I want to get data, I'm using a variable that is only created on server start and potentially auto-updates asynchronously. It wouldn't be doing anything to get values other than Properties#getProperty(), which I don't think would take much server power, would it? Since it's basically a HashMap or something of that sorts (in the sense that it stores keys and values). Nonetheless, doesn't MySQL and things like that just store everything in some sort of file? All things are stored in files one way or another.

    And yes, I'm only really working with very simple data, such as punishments, ranks, ips and uuids. Not really anything other than that. Not any stats (at least not currently planned to) or anything more advanced or needed to be frequently updated other than those stated in the original post.

    Not making this public in any way shape or form, it's going to be private for my server only. I understand it would be more "reliable" in a sense to use a SQL or Mongo database, but it would in all honesty, be easier this way.

    I understand your point that using an official storage program would be much "better" for compatibility, but I'm not really worried about compatibility when I'm making something that I know will only ever be used on my server.
  7. md_5

    Administrator Developer

    Congratulations, you just invented a crappy database!
    If you want local data storage CraftBukkit bundles SQLite, but realistically a full database of some sort isn't a bad idea if you plan on scaling.
    • Funny Funny x 2
  8. I understand it's not really good for more advanced data, but would this work out for simple data? (which is only what I want)
  9. electronicboy

    IRC Staff

    Property files are slow as all heck, and having large amounts of data sitting in memory is just asking for issues, especially with data that is likely mostly just going to build up and sit there.
    a proper DB server will store the file on the disk, where you likely have *much* more free space than RAM, as well as stores information in a much more efficient manner than a flatfile

    Database servers also exist for a reason, Threading is hard, and that is in a single application where everything can talk without issues, handling that between two completely different processes is just asking for issues with sync, especially if you ever want more than a single server to actually write to these files.
  10. Oh I see, that would cause a lot of issues cross-server. Since one server would save the data-and then the other may not be aware of that new data-and override it.

    I guess I may just have to stick with MySQL or possibly look into trying SQLite.
    I'll probably look into SQLite.
  11. Would you not be safer using MySQL, MongoDB and possibly some Redis? In general, proper databases...? You could store the UUID as a Primary Key and then store various information about them there, etc.
  12. If you are editing/creating/deleting a lot of files at once, the OS will run out of file handles and your application will probably crash. Also, it would be fckn slow :)
  13. If you could read my last message on the post you could realize I decided off of it. Thanks for your pointless input though, since this is just a repeat of all of the information given before hand.
  14. Lol.
    see an interesting idea - share my thoughts - get insulted by the op
  15. I don't care what you have to say if you're going to be vulgar about it.

    Requesting lock.