Using MongoDB

Discussion in 'Wiki Discussion' started by jflory7, May 27, 2015.

  1. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Using MongoDB

    Using MongoDB

    How to use MongoDB in your plugins



    Note: Before editing this article, please ask in the discussion section for a second opinion – personal opinions and views should be kept out of wiki articles. Thank you!


    NoSQL Databases(top)


    NoSQL Databases don't follow a table-like structure of the SQL databases. Instead many use either a Document style format (see MongoDB, etc.) or a key->value structure (see Aerospike, Redis, etc)

    This article especially targets MongoDB, the database used by many Minecraft networks.

    The concept of MongoDB(top)


    Like in SQL, you have databases in MongoDB. In these databases there are things like tables, called collections. But these tables have no defined columns. Every collection is simply a list of little JSON objects. The JSON objects can contain whatever fields they want, they don't have to have one field in common. For faster searching you can define a field that can be found in every JSON object, for example the player UUID. This is called indexing. This way, the DB only keeps these values (the UUID) in memory and searches only them before loading the rest of the objects. You can still search for other values in the JSON object, but the search will not be as fast as a search for an index.

    Morphia(top)

    Besides just what is covered in this tutorial, there is another way to save and access data from MongoDB, and that is with an API called Morphia. It is completely optional, but it makes the process of creating new collections from Java Objects a very streamline process. The tutorial on Spigot can be found here: https://www.spigotmc.org/wiki/mongodb-with-morphia/

    Setup the database(top)


    When you are on Debian Linux you can simply run:
    Code (Java):
    apt-get install mongodb
    This will automatically install and setup your database.
    When not on Linux, you can get MongoDB from their homepage: Downloads

    Securing your database(top)


    By default, the database will bind on 127.0.0.1, which means that it can only be accessed from the root itself.
    If you want to change the bind ip to 0.0.0.0 (can be accessed from everywhere) and create a user and password for your db's, please follow these instructions: Enable Authentication
    0.0.0.0 is not "accessible" from anywhere. it's simply a binding to every local machine ip

    Begin using MongoDB(top)


    On your root simply enter "mongo" in console and you will enter the so called mongo shell. There you can simply dispatch commands. Our aim is to dispatch these commands with Java, so we continue downloading the Java Driver of MongoDB. Import the .jar as library to your project. Alternatively, more experienced developers can import it via. its maven artifact.

    I suggest using the old 2.x driver as it will be used here. If you want to try out the new one, feel free to do so. I didn't get the new driver to work because of missing references in it :/

    Connect to MongoDB using Java(top)


    Code (Java):
        private DBCollection players;
        private DB mcserverdb;
        private MongoClient client;
        public boolean connect(String ip, int port){
            //Connect to the specified ip and port
            //Default is localhost, 27017
            try {
                client = new MongoClient(ip, port);
            } catch (UnknownHostException e) {
                //When you end up here, the server the db is running on could not be found!
                System.out.println("Could not connect to database!");
                e.printStackTrace();
                return false;
            }
            //Get the database called "mcserver"
            //If it does not exist it will be created automatically
            //once you save something in it
            mcserverdb = client.getDB("mcserver");
            //Get the collection called "players" in the database "mcserver"
            //Equivalent to the table in MySQL, you can store objects in here
            players = mcserverdb.getCollection("players");
            return true;
        }

    Store player information(top)


    Code (Java):
        public void storePlayer(UUID uuid, String name, long tokens, String rank){
            //Lets store our first player!
            //This player has never played before and we just want to create a object for him
            DBObject obj = new BasicDBObject("uuid", uuid);
            obj.put("name", name);
            obj.put("tokens", tokens);
            obj.put("rank", rank);
            //Lets insert it in our collection:
            players.insert(obj);
        }

    Read player information(top)


    There are two ways here. You can either use a Cursor to loop through all objects, or use findOne() to only find one on the server side. For searching only one object, as we do now, findOne() is faster. For other operations, where you search multiple objects, Cursor is very useful.

    Option 1: Using findOne()
    Code (Java):
        public void readPlayer(UUID uuid){
            //Lets build a minimal object to get all objects in the
            //collection "players" containing the field
            //"uuid" with the value uuid (what we are
            //searching for)
            DBObject r = new BasicDBObject("uuid", uuid);
            //Use findOne to only get one object!
            DBObject found = players.findOne(r);
            if(found==null){
                //User not saved yet. Add him in the DB!
                return;
            }
            //The user was found! Lets get our values!
            //You can cast the objects to String/Long etc.
            //As they are being delivered as binary objects
            //not as String like in MySQL
            String name = (String) found.get("name");
            long tokens = (long) found.get("tokens");
            String rank = (String) found.get("rank");
        }
    Option 2: Using DBCursor
    Code (Java):
        public void readPlayer(UUID uuid){
            //Lets build a minimal object to get all objects in the
            //collection "players" containing the field
            //"uuid" with the value uuid (what we are
            //searching for)
            DBObject r = new BasicDBObject("uuid", uuid);
            //Create a cursor object to loop through all the results
            //Lets get all the objects and get the one object that we are searching for
            DBObject found = null;
            // try-with-resources will handle the closing of the resources
            try (DBCursor cursor = players.find(r)){
                   while(cursor.hasNext()) {
                       found = cursor.next();
                   }
            }

            if (found == null) {
                //User not saved yet. Add him in the DB!
                return;
            }
            //The user was found! Lets get our values!
            //You can cast the objects to String/Long etc.
            //As they are being delivered as binary objects
            //not as String like in MySQL
            String name = (String) found.get("name");
            long tokens = (long) found.get("tokens");
            String rank = (String) found.get("rank");
        }

    Updating player information in the DB(top)


    Updating works like reading. You search the object, modify it and send it back to the server.

    Option 1: Updating all the Objects.
    Code (Java):
        public void updatePlayer(UUID uuid, String name, long tokens, String rank) {
            //Lets build a minimal object to get all objects in the
            //collection "players" containing the field
            //"uuid" with the value uuid (what we are
            //searching for)
            DBObject r = new BasicDBObject("uuid", uuid);
            //Use findOne to only get one object!
            DBObject found = players.findOne(r);
            if (found == null){
                //User not saved yet. Add him in the DB!
                return;
            }
            //The user was found! Lets create a new replacement object!
            DBObject obj = new BasicDBObject("uuid", uuid);
            obj.put("name", name);
            obj.put("tokens", tokens);
            obj.put("rank", rank);
            //And update it! This simply replace our found object!
            players.update(found, obj);
        }
    Option 2: Updating only 1 Object.
    Code (Java):
        public void updatePlayer(UUID uuid, String rank) {
            //Lets build a minimal object to get all objects in the
            //collection "players" containing the field
            //"uuid" with the value uuid (what we are
            //searching for)
            DBObject r = new BasicDBObject("uuid", uuid);
            //Use findOne to only get one object!
            DBObject found = players.findOne(r);
            if (found == null){
                //User not saved yet. Add him in the DB!
                return;
            }
            //The user was found! Lets create a new replacement object!
            BasicDBObject set = new BasicDBObject("$set", r);
            set.append("$set", new BasicDBObject("rank", rank));
            //And update it! This simply replace our found object!
            players.update(found, set);
        }

    Using the MongoDB driver in your server(top)


    So, I'll assume that after reading this topic, you've made a plugin that successfully uses MongoDB as it's database system, yay!

    And now you put the plugin in the server, boot it up aaaaaaand... wait... what? an error?
    NoClassDefFound? NoClassFoundException: com.mongodb.MongoClient.?
    How is this possible?
    Everything should be working fi-
    Oooooh I forgot to put the driver into the plugins folder
    ~Places mongo-java-driver-{Version here}.jar into plugins folder~
    Now it should wor- Wait, it still doesn't works?
    ~Moves mongo-java-driver-{Version here}.jar to {Server folder}/lib~
    What is happening???

    Now, as you should know, the MongoDB Java Driver isn't loaded as a bukkit plugin, because it isn't one. So the question floating in the air is:
    Option 1: The --classpath flag

    There is this cool command line option that allow us to solve this, which is -classpath, what it does is that it tells the JRE:
    The usage for the argument is:
    Code (Text):
    java –classpath ${CLASSPATH} {MAIN CLASS}
    So we should use something among the lines of:
    Code (Text):
    java -classpath "spigot.jar:lib/*" org.bukkit.craftbukkit.Main
    This will load the jar files in {spigot.jar folder}/lib as dependencies! Now we can use MongoDB and be eternally happy with our lives!

    Note: If using Windows as your OS, you will need to use a semi-colon rather than a colon in the above command.

    Please note that when using the classpath option, you don't use -jar spigot.jar, the classpath will execute the server without the need for the -jar argument

    Option 2: Shading the java driver into the plugin.

    Shading has it's upsides and it's downsides. The downside is that the file size of the final compiled JAR will include the size of the MongoDB Java Driver. The upside is that there's no risk of version mismatching (As opposed to option 1).

    If you're using Maven, open the pom.xml file and add the Maven shade plugin to the plugins section.

    Code (XML):
    <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
        </configuration>
        <executions>
            <execution>
                <phase>deploy</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
    Set the MongoDB Driver dependency scope to "compile"

    Code (XML):
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongodb-driver-sync</artifactId>
        <version>4.0.1</version>
        <scope>compile</scope>
    </dependency>

    The End(top)


    Everything above should be enough to use MongoDB in your plugins. If you need further information, you can ask in this Wiki Page's discussion thread or go to the official MongoDB Wiki.
     
    #1 jflory7, May 27, 2015
    Last edited by a moderator: Jun 11, 2015
    • Like Like x 5
    • Informative Informative x 2
    • Optimistic Optimistic x 1
  2. People that need tutorials like these most likely won't understand the concept of concurrency. Shouldn't you be teaching them how to execute their operations in a way that won't block the main thread?
     
    • Agree Agree x 3
  3. Like @Komp pointed out, a properly configured SQL database in an easily accessible location is not resource intensive. I've seen thousands of queries/sec on a table with 1 billion rows take only ~15% CPU (please note the size of the table...).

    That said, if you do run into resource problems using SQL you are likely executing queries in places you probably haven't thought out. You should collect information and cache it and only store it when necessary, like when closing/terminating a process (or in minecrafts case, when a player logs out).

    Concurrency is a way of running SQL queries (even assuming you half ass the entire system) so that no matter what you do, there's no performance problem the end user can notice. At the same time, for queries that normally return the same information you should cache the result and give it a TTL (Time To Live) so you can update it after a specific time frame.

    This article should be updated to reflect the inaccuracies of resource hogging SQL performance to better explain to newer developers looking for the best database model to use.
     
    • Agree Agree x 2
  4. Yea, this article is clearly biased and not properly researched. There are reasons to use mongodb in some cases but the listed ones aren't them. This article sounds more like it was written by someone who didn't learn to use SQL databases properly and prefers mongodb due to its easy setup.
     
  5. Requesting this wiki page is removed until it's reviewed and rewritten in an unbiased way.
     
    • Agree Agree x 3
    • Optimistic Optimistic x 1
  6. Uh, last time I checked, regular Oracle MySQL (note: not mariadb) scaled better than MongoDB under highly intensive loads (running with mongo's lazy cache disabled, why does that even exist?).
     
    • Agree Agree x 1
  7. Tux

    Tux

    Excuse me? There's plenty of high-volume MySQL setups out there, and MongoDB is actually one of the poorest choices for a NoSQL setup, although certainly the most popular.

    For now, I will remove the part advocating for MongoDB and make it just a quick MongoDB overview, which I think is fair.
     
    • Agree Agree x 3
    • Like Like x 1
  8. Although I prefer mysql, great documentation for NoSQL, good jobs :)
     
  9. Yes, but why are you asking this on a wiki discussion article?
     
    • Agree Agree x 1
  10. Hey, i followed the tutorial but i got this error:
    Code (Text):
    [19:45:58] [Server thread/ERROR]: Could not load 'plugins/MongoTester.jar' in folder 'plugins'
    org.bukkit.plugin.InvalidPluginException: java.lang.NoClassDefFoundError: com/mongodb/DBObject
        at org.bukkit.plugin.java.JavaPluginLoader.loadPlugin(JavaPluginLoader.java:135) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.plugin.SimplePluginManager.loadPlugin(SimplePluginManager.java:329) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.plugin.SimplePluginManager.loadPlugins(SimplePluginManager.java:251) [spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.craftbukkit.v1_8_R3.CraftServer.loadPlugins(CraftServer.java:291) [spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at net.minecraft.server.v1_8_R3.DedicatedServer.init(DedicatedServer.java:198) [spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at net.minecraft.server.v1_8_R3.MinecraftServer.run(MinecraftServer.java:524) [spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
    Caused by: java.lang.NoClassDefFoundError: com/mongodb/DBObject
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_60]
        at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_60]
        at org.bukkit.plugin.java.PluginClassLoader.<init>(PluginClassLoader.java:64) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.plugin.java.JavaPluginLoader.loadPlugin(JavaPluginLoader.java:131) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        ... 6 more
    Caused by: java.lang.ClassNotFoundException: com.mongodb.DBObject
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_60]
        at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:101) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:86) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_60]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_60]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_60]
        at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_60]
        at org.bukkit.plugin.java.PluginClassLoader.<init>(PluginClassLoader.java:64) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        at org.bukkit.plugin.java.JavaPluginLoader.loadPlugin(JavaPluginLoader.java:131) ~[spigot.jar:git-Spigot-b2c2c63-a3cb1bc]
        ... 6 more
     
     
  11. I searched everywhere and i found it Thank you!
     
  12. I know what this error say, but i dont know how to fix it, i already added the java driver to my project
     
  13. But you need to include the mongo-java-driver in your plugin, too.
    Otherwise it won't be available at runtime.

    If you use Maven, take a look at the shading plugin.
     
  14. i don't use Maven ... any other ways?
     
  15. Would it be an idea to include the 2nd argument of find/findOne which allows you to set what fields you wish to retreive from the database? I have often seen this feature overlooked by multiple networks and for very often occuring queries it can improve overall performance quite a bit.
     
  16. you should use maven, other way is configure you build path and import the jar with de jdbc driver for mongodb... but don't go for this way, use maven!
     
  17. can you give me a step by step tutorial for including it with maven? i installed maven but i dont know how to use it in my eclipse-project...
     
  18. JamesJ

    Supporter

    http://lmgtfy.com/?=maven+eclipse+setup
     
    • Like Like x 1
  19. I searched everywhere and i found it Thanks :D