1. Guest, as per the stickied thread, this forum has not been in use since 2014. All bugs and feature requests should be posted to JIRA.

Rejected LongHashSet is consuming far too much memory

Discussion in 'Bugs & Feature Requests' started by bergerkiller, Apr 10, 2013.

  1. Made an account just for this...

    Anyway, in TrainCarts I make heavy use of LongHashSets to store chunk coordinate information of trains/minecarts. A few days ago, dockter messaged me about a memory issue. A stunning 3 gb was 'consumed' by the offline train storage system. I then spent a few days debugging this issue.

    Now, after profiling the memory usage per code block with Dockter's help, we have identified the issue, and that is the LongHashSet. A single LongHashSet instance, with an initial size of about 30 elements, consumed 4 mb of RAM memory, persistently! On CraftBukkit the ENTIRE storage consumes less than 1 mb. How can a single element (out of 400 of them) consume well over that amount, 8 mb?! (I use two Long Hashsets)

    Now, I looked at Spigots changelog, and well...here we go.

    I have no idea why a single LongHashSet instance uses a 2-dimensional Object[1024][1024] array to check whether elements are contained. md_5 will have to explain that himself I suppose.

    And yes, a HashSet<Long> consumed far less memory than this implementation.
     
    #1 bergerkiller, Apr 10, 2013
    Last edited: Apr 10, 2013
  2. md_5

    Administrator Developer

    1 element uses no ram over the base.
    The base is 4mb.
    There are 2 instances per world.
    So on a 16 world server, thats 32 instances == 128mb ram.

    Please explain how that is far too much.
     
    • Like Like x 1
  3. md_5
    I already knew by IRC that you were going to expect LongHashSet/Maps to be used as long-term storage only. For that matter, I have 'bypassed' the LongHashSet code of Spigot in BKCommonLib so it uses a HashSet instead. If it is every worth it, I'll just copy-paste the original code in there. Or, easier, write a Dummy FlatMap that doesn't do anything.