A concerning stacktrace about too many open files?

Discussion in 'Spigot Help' started by jflory7, May 28, 2015.

  1. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Hello Spigot forums,

    A day or two ago, I was watching my server from console and I noticed that a user logged in for the first time. However, after joining, several stacktraces started firing off immediately after he joined, and stopped as soon as he disconnected. It seemed clear to me that this user was trying to do something malicious or something he shouldn't have been (he also didn't respond to any warm welcome messages).

    This stacktrace makes little sense to me. Here is the Pastebin link for it, but just for SEO purposes, I also have the stacktrace included below.

    Code (Text):
    [06:45:07 INFO]: [R] DJ2790: Welcome!
    [06:45:46 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:45:50 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:45:51 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:45:52 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:45:53 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:45:54 WARN]: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.io.IOException: Error during accept(...): Too many open files
            at io.netty.channel.epoll.Native.accept(Native Method) ~[paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollServerSocketChannel$EpollServerSocketUnsafe.epollInReady(EpollServerSocketChannel.java:102) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [paperspigot.jar:git-PaperSpigot-ebb90ce-529e4cd]
            at java.lang.Thread.run(Thread.java:745) [?:?]
    [06:46:37 INFO]: FERNAND1811 lost connection: TextComponent{text='Disconnected', siblings=[], style=Style{hasParent=false, color=null, bold=null, italic=null, underlined=null, obfuscated=null, clickEvent=null, h
    overEvent=null, insertion=null}}
    Any idea what could have happened? Thanks in advance!

    Cheers,
    - jflory7​
     
  2. JamesJ

    Supporter

    Edit: never mind

    Why are you running PaperSpigot? :eek:
     
  3. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    It has features and changes that work better for my survival server and it allows me to alter some of the behaviors of vanilla Minecraft that Spigot doesn't change (and rightfully shouldn't).
     
  4. JamesJ

    Supporter

    Hm.
    Very strange error. Has it happened with any other players?
     
  5. Looks like you got a bot user.
     
  6. I'm going to go out on a limb here and say the system your survival server is on probably doesn't have enough file descriptors for the user running the server to access.

    Check the limit with ulimit -n while on the user that runs the server, and increase it as necessary in system configuration files.

    As for the user at hand, maybe he was trying to open a ton of connections through a script.
     
  7. I once asked a player to try bypassing my CAPTCHA plugin with his bot and he accidentally used a different bot function on his bot and I got the exact exceptions.
     
  8. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Not as far as I know – it was only with this user that I've seen this happen.

    What kind of actions do you think they were trying to perform based on these stacktraces?

    You're referring to something more in the operating system? What exactly do you mean by file descriptors for the user running the server? I do run my server as an underprivileged user (intentionally), so it wouldn't be a major surprise if there were something that the user did not have privileges or access to.
     
  9. JamesJ

    Supporter

    This is just a random guess now, but is system (your dedicated machine, if you have one?) running out of ram? Not the actual server, but the machine. Or have you ran out of disk space.
    They're the only explanations that I could think of, however I think that if it was to do with disk space there you would be more errors in console.
     
  10. jflory7

    jflory7 Retired Moderator
    Retired Benefactor

    Hmm, I would be surprised if it was running out of memory. It's a 16GB machine, but the server is only allotted 6GB. And it's a 2TB hard drive, which I haven't even begun to fill.

    This is really weird. :(
     
  11. JamesJ

    Supporter

    Very strange :3

    Edit:
    After doing some googling, multiple results have mentioned it being a program fault, however none of them were directly related to PaperSpigot, I would speak to whoever maintains PaperSpigot, @jflory7
     
  12. File descriptors are basically pointers to files currently being accessed. Each process has a set amount they are allowed to open in total. This value can be raised or even set to unlimited.

    Because of how minecraft handles many files at once, possibly dozens or hundreds depending on the system, it's recommended you set an unlimited value.

    This will end up using more RAM (because open file pointers use RAM obviously) but because you may be running multiple servers, possibly a panel, maybe screen for console, output logs, etc. it's recommended you raise the ulimit (file descriptor) cap for the system and user.

    Because this seemed to happen with only one user, it's either a complete fluke or was indeed a bot attempt. In either case the limit should be raised and the developers should be notified because it's probably s bug in their software.
     
    #12 CraftedFury, May 29, 2015
    Last edited: May 30, 2015
    • Useful Useful x 1
  13. Up

    I have the same problems with spigot 1.8.8 :/