Chunkli | PreRelease Test | Insert 1 million blocks/second

Discussion in 'Spigot Plugin Development' started by Kademlia, Jun 9, 2015.

  1. Hello, let's blow your mind :D
    I am fairly new to this community but not to Minecraft. I´ve been developing plugins for several years now. This thread is about a plugin I have been working on for 1+ year now and plan to extend further.

    Whats Chunkli?
    If you ever worked with (Async)WorldEdit this speed-demo should be somewhat impressive.

    Additionally here is a trailer I made some time ago
    • Save your buildings online at runtime
      This includes Chests, Items on the Ground, Animals, Monsters, Ticking redstone etc.
    • Copy your buildings to another server at runtime
    • Move huge areas/buildings to another place without lag
    • Create pictures of your saved data and show your friends
    • Provide other players with your buildings with a simple command. No more file-downloads.
    • Copy Items, Animals, Monsters and even WorldGuard-Regions to another server/world!
    • Backup your Buildings, even if your server crashes you can undo those changes.
    • Compatible with Bukkit/Spigot 1.7_R4 and 1.8_R1, 1.8_R2, 1.8_R3

    Basic information:
    • Chunkli manipulates more than 1.000.000 Blocks per Second while the world is still ticking
    • Chunkli heavily utilizes Multithreading. Most of the work is done outside of the Minecraft-Process
    • Chunkli has a very small memory footprint, even if you insert millions of blocks
    • Chunkli is able to save your Stuff and restore even if your server crashed
    • Chunkli allows you to copy anything, anywhere at any time - even to another server at runtime!
    Online Storage!
    Chunkli manages your saved data 'in the cloud' - accessible from every server no matter where you saved. You can even command the service to render images of your saves. As an example these are my current saves: http://a7.chunk.li/area/Kademlia/named/html

    And a single saved areas perspectives:
    [​IMG] [​IMG] [​IMG] [​IMG]

    Share your best Buildings with the World!
    You can save your building and allow others to insert them on their server. This allows you to share your accomplishments with everyone within two seconds. As an example you could insert some 'ducks' on your server by using "/chunk paste kademlia:duck" (you access my save-file i made public)



    What is this topic?
    Essentially it's about feedback. Before actually releasing the plugin I would like to test it some more with dedicated users and programmers. If you have feature-requests or questions/ideas I might be able to address them directly. My goal for this plugin is to be the best one around with a huge margin.

    Programmers?
    The plugin is designed to work for you. You should be able to use chunkli as a 'backend solution' to do the heavy lifting (map manipulation) and not worry about lags from manipulating. This is the part I need the most feedback - if possible.
    The following code inserts an area at the location of the player if it's exactly 1 Chunk big.
    Code (Text):
      private ActionFuture<?> load2(final Player player){
         CommanderInfo info = SessionFacade.getCommanderInfo(player);
         IWorldChunkArea area = AreaFacade.getPlayerChunk(player);
         PasteAreaActionConfig config = ConfigFacade.getPasteAreaActionConfig(info, info.getAccount(), saveName, area);
         ActionFuture<?> future = ActionFacade.pasteChunkArea(config);
         return future;
       }
    This code is used to allow players (not admins!) to move there plots from A to B. In more detail:

    • A Callable is provided. The Callable gets called at the end of the action and provides feedback to the caller (The message: "unable to insert Plots that overwrite Chest at the edges of a chunk" if the action failed).
    • The Permission is changed to "land.load". This overwrites the default Chunkli Permission and allows players to use Chunkli without needing any chunkli permissions.
    • setDisallowedBorderBlocks are set. If any of these Blocks are found at the border the action will be cancelled.
    • SetAreaPluginDataConfig is provided. This allows to set 3rdparty Data. For example your own plugindata could be added. In this case WorldGuard regions are removed/inserted.
    • loadAreaAction.get(25000, TimeUnit.MILLISECONDS); waits for the action to finish
      Only use .get()
      from an asyncronous thread. DO NOT use .get within the Minecraft-Thread. If you do the Server will wait for Chunkli and Chunkli waits for Minecraft to advance to the next tick. You will get warnings in the console if you do tho.
    Code (Text):

       private boolean load(final Player player){
         ActionResult<Boolean> c = new ActionResult<Boolean>() {
           @Override
           public Boolean call() {
             if(!this.isSucsessful()){
               player.sendMessage(ChatColor.RED + "Plots with Chests/Pistons at their border cannot be overwritten. Remove those blocks and try again.");
             }
             return true; // The callable worked
           }
         };
         CommanderInfo info = SessionFacade.getCommanderInfo(player);
         IWorldChunkArea area = AreaFacade.getSingleWorldChunkArea(loc);
         PasteAreaActionConfig config = ConfigFacade.getPasteAreaActionConfig(info, info.getAccount(), saveName, area);
         config.setAllowExternalAreas(true);
         config.setCallable(c);
         config.setNeededPermission("land.load"); // overwrite the default permission to enable users ONLY via this command to use that chunkli action
         Map<Integer, Integer> disallowedBorderBlocks = new HashMap<>();
         disallowedBorderBlocks.put(54, 0); //chest
         disallowedBorderBlocks.put(146, 0); //lockedChest
         disallowedBorderBlocks.put(29, 0); //piston
         disallowedBorderBlocks.put(33, 0); //piston
         disallowedBorderBlocks.put(34, 0); //piston
         disallowedBorderBlocks.put(36, 0); //piston
         config.setDisallowedBorderBlocks(disallowedBorderBlocks);


         SetAreaPluginDataConfig setConfig = new SetAreaPluginDataConfig(info, area, area, null); // WorldEdit Regions! \o/
         setConfig.setPasteRegions(true);
         setConfig.setRenameExisting(true); // rename if we encounter a region with the same region name
    //     setConfig.setDeleteExisting(true); // dont delete, might be a existing region on another server owned by someone else!
         setConfig.setDeleteRegionsInTargetArea(true); // delete regions in the target-area. This should always be the region posessed by the current player
         config.setSetPluginDataConfig(setConfig);
         ActionFuture<AbstractWorldAction<?>> loadAreaAction = ActionFacade.pasteChunkArea(config);
         try {
           AbstractAction<?> abstractAction = loadAreaAction.get(25000, TimeUnit.MILLISECONDS);
           if(!abstractAction.wasSuccessfullyExecuted()){
             Bukkit.getServer().getLogger().info("ERROR Landli, Chunkli: Somehow the saveChunkSelection has failed for player:" + player);
           }
           return abstractAction.wasSuccessfullyExecuted();
         } catch (Exception e) {
           e.printStackTrace();
         }
         return false;
       }
     


    I need help setting up! I need help programming thing X with Chunkli!
    I will happily help you. I need to know all the problems new people have with the software in order to make things easier in the future. We can even meet in Teamspeak and I´ll help you to setup chunkli on your server. Unfortunately for you as a tester, there is no "wiki" as the software is absolutely new. But the plugin itself should help you quite a lot with it's syntax and examples.

    "/chunk" shows some basic information.

    More importantly its "/chunk <TAB>". Making a space and a TAB behind the /chunk command will show you all the available commands the plugin offers.

    If you want to know more about a specific command write a "?" at the end and execute the command.
    You will get the syntax, permission and examples for every command.
    "/chunk select?"
    "/chunk paste?"
    "/chunk stack?"
    "/chunk raise?"
    "/chunk biome?"
    "/chunk preview?"
    "/chunk save?"
    "/chunk relight?"
    "/chunk list?"
    "/chunk undo?"

    1. Download the latest Version here: http://chunk.li/versions/ (sort by date)
    2. Put it into your plugin-folder and start the server
    3. connect to "a7master.chunk.li" to get a name+secret

    [​IMG]

    As we are in alpha the answer should be "no" in any case. Apart from that:
    • The software has run on a Multi-Server/Bungee System for about 6 Month now (200-400 concurrent users).
    • I wrote two Plugins that use Chunkli as a Backend.
      • The "Furniture" Plugin uses the possibility to provide 3rdparty data to copy/save furniture if there is any on a specific chunk
      • The "Plots" Plugin hugely uses the capabilities of Chunkli. For example it allows the players to save their houses - switch the server - and insert heir house again at a different location and they can export their stuff to their own server (ores are removed as xray prevention).
    • Chunkli is used to "regenerate" map-areas that are not owned by players (plots). This process replaces all non-owned chunks with chunks from a backup world. Magnitude: Replacement of ~250.000 Chunks over night is no problem.
    Feel free to contact me regarding this topic.

    No

    The software is in alpha status and currently no modifications are allowed. This may change in future version but in the current state it is not allowed to modify the software.
    See http://chunk.li/versions/license.txt

    Conclusion
    Please keep in mind this plugin is in alpha development. Expect Bugs :oops:. Everything is subject to change even the API-Methods are far from finished and may change in their specific implementation on updates.
    If you are interested post here or write me a PM.

    This wall of text should be enough for now.:eek:
    I´ll provide more information upon request.

    Thank you and I hope some people would like to try this plugin early,
    Kademlia
     
    #1 Kademlia, Jun 9, 2015
    Last edited: Jun 9, 2015
    • Like Like x 12
    • Winner Winner x 4
    • Useful Useful x 1
  2. Hm not bad :O what specs does your test server run on?
     
  3. Looks fantastic, shame WorldEdit can't be used with it effectively like AsyncWorldEdit does :/
     
  4. Its a good machine but not the latest generation. Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz

    Uhm. You can use both if that is what you mean.
    Edit: Oh i think i know now. Yes direct block-manipulation is not yet implemented an will not be in the near future. But if it fits a chunk you can stack/copy/raise etc.
     
  5. I'd love to see this, and I think it would be a real turning point for the future of this project. Do you have an ETA? :)
     
  6. You should try using it. The reason why direct block manipulation is at low priority is because i rarely need it.
    Most things in smaller sizes can be done by WorldEdit without lag. If you start manipulation large areas it would be best to stick to the Chunk-Grid anyway and thus use chunkli.

    I've now used the Chunkli&WorldEdit combination in production several month and rarely needed those features in Chunkli.
     
  7. I'd want to select an area, move it, copy it, paste it, rotate etc... I'd need a wand for a precise selection
     
  8. I would love this for building(has some advantages I can see over async)...will try later and say how it works out.
     
  9. Hehe yes i guess you´ll have to see for yourself if the plugin in its current state can be any help for you.
     
  10. Sadly it doesn't which is why I asked for an ETA :/
     
  11. I´d argue you still have to at least try it to get to that conclusion. The explanation before is all the ETA i can provide atm
     
  12. Impressive.
     
  13. samczsun

    Supporter

    @Kademlia

    Let me guess. Direct NMS modification + packet sending?
     
    • Like Like x 1
  14. @samczsun
    Yes parts of the software work that way. But keep in mind everytime you manipulate something there are actual changes to the world and everything that gets changed is backed up before alteration. Even if your server would crash after that alteration /chunk undo would still work upon the next server boot :)
     
  15. samczsun

    Supporter

    @Kademlia

    In that case, it's not that difficult at all. Save a snapshot of every chunk you're modifying (just a byte array) and search up by timestamp.

    Also, your opsec could use a bit of work. I can see easy ways of bypassing your pay-per-use system you have implemented
     
  16. 1 million blocks a second? If I go totally asshole towards other plugins, I can replace about 187449230 blocks in 50ms (1 tick). Yes people, that's 187 million blocks. Note that there's reduction for preparing tile entities and all that, but I'm sure I can provide 18mil with ease.

    (Note: this was done from a 3-4 year old gaming pc. I am a total hardware noob, so if you need specs, give me directions and I'll get them for you)
     
    • Informative Informative x 1
  17. @samczsun
    I'll invite you to try ;)
    opsec: Thats not the goal and the system is generally free but has options for heavy users. As I am hugely invested in this (fulltime job) i need to pay rent in some way at the end of the day.

    @DarkSeraphim
    I would love to see that and bet against it. You did read the stuff about not lagging even on crowded servers right?
     
    • Agree Agree x 1
  18. samczsun

    Supporter

    @Kademlia

    Sure. I have chunkli-a7.150.jar. Is that the version you want me to crack, or is there a more updated version that I should use
     
    • Funny Funny x 1
  19. @samczsun
    You do realize you wrote two different things and i ansered them in the same order?
    Obviously the first part regards your sentence about the implementation. As the second part highlights the partial currently not important obfuscation. Why would you assume that if im already telling you its not the challenge?
     
    • Copy arrays backing the data
    • Start a thread / async task
    • Do whatever you want with the bytes. In the async task you can even splice the arrays to update only arbitrary parts of the chunk - you can apply math / basic programming to modify the indices.
    • Prepare the arrays for an arraycopy call (simple POJO should do fine to store invocation data)
    • use Scheduler#runTask to hop back on the main thread
    • Copy the arrays back using arraycopy.
    You can even slow it down and/or split it up for some fancier merging strategies (like a pessimistic merge, to preserve after-copy init changes). All your actual hard data crunching is done on the other thread.

    And all that for a couple of MBs (about 1MB per million blocks per byte of storage) of temp RAM storage.

    Any further questions?