Solved Simulating chat tabs, how to solve users ending up with GBs of logs?

Discussion in 'Spigot Plugin Development' started by Bjorno43, Feb 8, 2020.

  1. General Info
    Hey everyone. I'm working on a plugin. Part of it changes the chat system to allow multiple chat channels which are divided into (clicktable) tabs. New channels can be created / removed on the fly and it also creates tabs for DMs with other users.

    Proof of concept
    [​IMG]

    The way it works

    Because this is done purely server side, there are obviously a lot of limitations on what you can actually do with the chatsystem. It's not possible to create fixed elements within the chat. So to solve this problem, it basically copies the visible lines; removes the top message from the copy and adds the new message at the bottom of it on every message event. Then it just pastes the whole copy to the chat, making it look like a working chat with tabs.

    The problem with this approach
    Minecraft clients automatically log everything that happens within the chat. It seems there's no way to turn this off and there's no option to have it automatically remove logs at a certain age either. Due to the way this works, such logs will obviously start stacking up fast! I imagine this can potentially result in gigabytes of logs in a matter of days on busy networks.

    Question
    I'm looking for ideas on how to solve that issue. Or at the very least minify the problem as much as possible.
     
    • Creative Creative x 1
  2. 1Byte is one letter
    For each letter that the log saves it would add 1Byte of size
    1Gigabyte = 10^9 bytes. You would need 10^9 letters to get to a gigabyte and I think that when minicraft is closed the log is compressed so I think you don't need to worry

    I have just created a 25972Byte log (25972 letters) and then I closed minecraft and the log didn't compressed, I oppend minecraft again and it compressed the log to create a new latest.log and the log now has a size of 4000Byte (aprox)
    Minecraft reduced its size to a 15%(aprox). If a player somehow manages to get a log with the size of a gigabyte, as soon as he starts minecraft again the log would be compressed and it would just be 1'5e8Bytes (0,1Gigabyte aprox)
    So yeah, no need to worry

    Also, I like your plugin idea
     
    #2 Nemo_64, Feb 8, 2020
    Last edited: Feb 8, 2020
    • Informative Informative x 3
  3. Assuming around 50 characters per line and (currently) 20 lines per message, that's 1KB per message. That's going to increase because it limits chat history too much for users. Say I want users to be able to scroll back up to 50 lines, you're looking at 2.5KB per message. On busy networks you easily reach 1000+ messages per session, but for simplicity, let's keep it at 1000. So that's 25MB of log data per Minecraft session. So in that case it would take little over a month to reach 1GB of logs, assuming only a single session per day.

    Those are just low assumptions. In reality, every single message will be that large. So not just user messages, but also plugin messages, server messages, anything that appears in chat! Add all other console logs to that, including any OpenGL errors etc. It can easily increase to 100MB per session!

    Also, I'm happy you like it, but I'm not gonna take credit for something that isn't my original idea. The plugin was partially developed some years ago and left abandoned on Github. Luckily it was MIT licensed, allowing me to pick it up and finish the project / get it working in today's versions :)
     
  4. Did you take into account the compression in the month calculation? If it's compressed with gzip it will drastically reduce the size of the logs because the demonstrated functionality uses a lot of duplicate messages (which are mostly compressed away)
     
    • Like Like x 1
    • Agree Agree x 1
  5. No I didn't take any compression into the calculation yet. Seems you guys are right though. It's not that much of an issue. It'll be fine to continue developing this ^^ Thanks a lot guys!
     
    • Agree Agree x 1