1.8.8 Sending and receiving packets for newer versions at byte level

Discussion in 'Spigot Plugin Development' started by 3ricL, May 4, 2021.

  1. For a client we are currently tasked with making the recipe book function for 1.12+ players on a 1.8.8 server with ViaVersion. By default it will not be usable for them, mainly as the server never communicates about the recipes so it will be empty (the book didn't exist until 1.12). I have already figured out most of the packet protocol, by experimenting with packets on my 1.16 test server.

    My main problem is that these packets obviously don't exist on 1.8.8. This means that the only possible way (I know of) to do this is by (pre)converting the packets to bytes and sending these 'raw' instead.

    First issue: sending these packets: -- SEE BELOW
    If you have an NMS packet instance (in 1.16 for example), I found out you can do:
    Code (Java):
    ByteBuf buf = Unpooled.buffer(1000);
    PacketDataSerializer serializer = new PacketDataSerializer(buf);
    try {
       myPacket.b(serializer); //where myPacket is the packet I want to convert to its bytes
    } catch (IOException ex){
        ex.printStackTrace();
    }
    getLogger().info("buffer: " + Arrays.toString(Arrays.copyOf(buf.array(), buf.readableBytes())));
    And it prints out the exact bytes of the packet. As I always need to send the same exact packet, my plan was to send these same bytes on 1.8.8. Now to send this I tried to send raw bytes using ViaVersions API:
    Code (Java):
    byte[] array = new byte[]{....bytes of a packet as printed above };
    ByteBuf buf = Unpooled.copiedBuffer(array);
    Player player = ...;
    Via.getAPI().sendRawPacket(player.getUniqueId(), buf);
    But this makes the client error out, failing to decompile the packet (both on a 1.16 server as a 1.8.8 server with viaversion and 1.16 client).
    My best guess is that the packets are missing the "packet identifier", so the client will incorrectly take the first byte and use that as the incorrect identifier, resulting in this breakage. As when I log the bytes of a packet and compare it with https://wiki.vg/Protocol the first couple of bytes are already boolean values and the client will say it received an invalid packet with an id of 0 or 1.
    Now my question is: how do I properly decompose a packet so that I can send it using ViaVersions API?

    Second issue: receiving these packets: -- SOLVED
    If I get past the first issue, the client will start responding with packets that also don't exist on 1.8.8 and that I need to read. Now how can I 'listen' for these packets? I understand that they cannot be converted to their true server representation (and that I need to write their decompiler from bytes myself), but how can I detect them coming in at all (and get their bytes)? Does ViaVersion help me here? I assume that it has to do something with these packets, to prevent the server from receiving a packet that it cannot understand.

    PS: Stuff like Protocollib is not an option as it requires the packet classes to exist on the server. Switching server versions is also not possible.

    As this thread is highly specific and about ViaVersion (somewhat), I could really use your expert knowledge on this @FormallyMyles :)


    EDIT: Read last post for current issue.
     
    #1 3ricL, May 4, 2021
    Last edited: May 6, 2021 at 10:11 PM
    • Useful Useful x 1
  2. First issue: from my understanding of the source, sendRawPacket sends the given bytes sans any processing, which means you will need to implement the encoding steps in the protocol yourself... More specifically, you will need to add the header in addition to potentially also compression and encryption, though I’m not for certain on the latter two. That being said, Via’s source is pretty straightforward so it shouldn’t be too much of a hassle to look into it further.

    Second issue: no idea of Via has any new API since I’ve last looked at it. The way I’ve been doing it is here: https://github.com/AgentTroll/viapr...gmail/woodyc40/viaprotohack/ViaProtoHack.java. Should emphasize hack, this can very easily break since it relies upon proxying Via classes and tricking it into thinking they are its own, but it does the trick for a closed ecosystem. You can probably get around this by replacing a channel initializer somewhere and prepending your own channel handler before Via’s magic code. That would probably be the approach I would take if you’re looking for the “right” way of doing things.
     
    • Informative Informative x 1
    • Useful Useful x 1
  3. Thank you so much for your elaborate answer!
    For the first issue, do you happen to know what exactly I should put in front of my byte array? I know it should contain the packet id (which should be 0x35 in hexadecimal given it is the unlock recipes packet) but unsure about the formatting.

    Wow thank you for that second class, that is just what I am looking for indeed! No problem it is hacky, I am literally sending packets as raw bytes so long past clean code:) and its custom made for client anyway, so if it changes will just update it.
     
    • Useful Useful x 1
  4. https://wiki.vg/Protocol#Packet_format

    If that still doesn’t work, I would disable packet compression and turn off online mode in the server.properties file for the purpose of eliminating compression and encryption from the equation and then factor that in once you get everything working. Wireshark is also very helpful for this.
     
  5. Okay I tried following this but really can't get it to work. This is what I tried:
    Code (Java):
    public void sendRawPacket(Player player, int id, byte[] data){
        byte[] idEncoded = writeVarInt(id);
        byte[] lengthEncoded = writeVarInt(data.length + idEncoded.length);
        byte[] prefix = ArrayUtils.addAll(lengthEncoded, idEncoded);
        ByteBuf buf = Unpooled.copiedBuffer(ArrayUtils.addAll(prefix, data));

        getLogger().info("buffer: " + Arrays.toString(Arrays.copyOf(buf.array(), buf.readableBytes())));
        Via.getAPI().sendRawPacket(player.getUniqueId(), buf);
    }

    public byte[] writeVarInt(int value) {
        byte[] result = {};
        do {
            byte temp = (byte)(value & 0b01111111);
            // Note: >>> means that the sign bit is shifted with the rest of the number rather than being left alone
            value >>>= 7;
            if (value != 0) {
                temp |= 0b10000000;
            }
            result = ArrayUtils.add(result, temp);
        } while (value != 0);
        return result;
    }
    And then sending the packet with id 0x35 (so that becomes 53 as integer) and the exact same data as printed when deserialising the 1.16 packet. This code prints the following:
    Code (Java):
    buffer: [34, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 21, 109, 105, 110, 101, 99, 114, 97, 102, 116, 58, 97, 99, 97, 99, 105, 97, 95, 100, 111, 111, 114, 0]
    But crashes my client, giving the error message: "Internal Exception: DecoderException: IndexOutOfBoundsException: readIndex(34) + length(4) exceeds writerIndex(35): PooledUnsafeDirectByteBuf(ridx:34, widx: 35, cap: 35)". What am I doing wrong here?

    I have my server set in offline mode and packet compression threshold is 256 so this packet shouldn't be compressed. I also tried omitting the length part and then it doesn't crash, but the packet also doesn't seem to have any effect.


    EDIT: for reading the packets adapting your second solution totally works! So just the first issue with sending these packets that is left now.
     
    #5 3ricL, May 6, 2021 at 9:39 AM
    Last edited: May 6, 2021 at 11:20 AM
  6. This message generally means the packet is too short.

    That makes a difference because the packet format with compression isn’t the same as the packet format without compression. If you disabled compression entirely, what you are doing right now would work. Otherwise, check the “with compression format” on the wiki.vg page.
     
  7. I disabled compression entirely, I set network-compression-threshold=-1 and online-mode=false in my server.properties.

    I find it rather strange that it is indeed too short, as my length of 34 does match up with what the amount of bytes but the length(4) it mentions makes little sense to me.

    Okay I just was looking through ViaVersions code and found out I can create version agnostic packets with it (which would be way better than hardcoding bytes). I did the following (to attempt to send an "Unlock Recipes" packet):
    Code (Java):
    PacketWrapper wrap = new PacketWrapper(0x35, null, Via.getManager().getConnection(player.getUniqueId()));
    wrap.write(Type.VAR_INT, 0);
    for (int i = 0; i<8; i++){
        wrap.write(Type.BOOLEAN, false);
    }
    wrap.write(Type.VAR_INT, 1);
    wrap.write(Type.STRING, "minecraft:acacia_door");
    wrap.write(Type.VAR_INT, 0);

    ByteBuf buf = Unpooled.buffer();

    try {
        wrap.writeToBuffer(buf);
        getLogger().info("buffer: " + Arrays.toString(Arrays.copyOf(buf.array(), buf.readableBytes())));
        wrap.send(Protocol1_16_4To1_16_3.class); // also tried the deprecated wrap.send(), same result
    } catch (Exception e) {
        e.printStackTrace();
    }
    The interesting thing is that my bytes are printed out as:
    Code (Java):
    buffer: [53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 21, 109, 105, 110, 101, 99, 114, 97, 102, 116, 58, 97, 99, 97, 99, 105, 97, 95, 100, 111, 111, 114, 0]
    . This is exactly the same as what I tried to achieve manually earlier, except for the length part which is just gone. All bytes but the first one are the same.

    Unfortunately though, it seems to do nothing... The client doesn't complain at all, but also doesn't show the recipe when I open my crafting bench. Sending a regular old PacketPlayOutRecipe in my 1.16.4 testing server to a 1.16.4 client does work, so I know this packet is the correct one. And the data (everything after the first byte) exactly matches up with the deserialised bytes from the first code snippet on this thread, so those are correct I think.
    Do you (or others ;)) have any clue what I am still doing wrong? I feel like I am so close but can't figure out what is missing still.

    The packet does get send btw, because when I write a random string to it in an incorrect place it does make the client error out.
     
  8. I am doubtful that the packet encoding is the issue then. The standard Minecraft client is fairly fickle when it comes to protocol adherence, so if it accepts the packet, chances are that its correct. I’m guessing the problem with the client not producing the correct response has something to do with something else such as not receiving some other packet like recipe declaration or something along those lines.
     
  9. Okay, thank you. I do know that that packet on its own is enough to trigger a recipe appearing (by logging all recipe related packets on 1.16 test server). However the normal 1.16 server also sends this packet during the log-in phase I believe (and then I send it 30 ticks later with other payload). Can I use ViaVersion to send the packet earlier too? Because it mentions that during log in it might not have wrapped the player yet (but it must have right, how else will it translate the login packets etc)?

    Btw what is the recipe declaration packet for? I thought that was just for custom recipes so not relevant, or is it?
     
  10. I’m not completely sure. It’s very likely that the API doesn’t provide any connection to use until the player completes the login process even if everything needed to adapt the channel has been completed internally.

    I’m not totally sure myself, it’s just a guess. I’m supposing that it isn’t what is being sent and more that it is sent that makes a difference; but again, take my word with a grain of salt. I haven’t actually played this game in over a year. It could be something entirely different and I’m just leading you down the wrong path lol.
     
    • Friendly Friendly x 1
  11. Bump! See the post below for a breakdown of the current issue, which is sending the "Unlock Recipe" packet to 1.16 players from a 1.8 server (using ViaVersion's API):
     
  12. Curiosity got the better of me and I decided to check the client source. Looks like my hunch was correct, it’s the declare recipes packet. If you send that ahead of time, it will work.
     
    • Winner Winner x 1
    • Useful Useful x 1
  13. You absolute legend!:) Thank you so much, I will definitely implement that today and I will be sure to credit you in the final plugin!

    EDIT: Do you happen to know how to create this packet using ViaVersion's API? Of course I could send the raw bytes but that will be quite ugly (there are 850+ recipes, so its an enormous packet when I log it). Looking at wiki.vg the packet contains more complex parts to encode the recipe. I feel like ViaVersion must have these wrapped somewhere, but can't figure out how to do it elegantly.

    Btw you were indeed totally right, for the meantime I have used the raw packet version and that makes it function on a 1.8.8 server and 1.16 client.
     
    #13 3ricL, May 10, 2021 at 8:41 AM
    Last edited: May 10, 2021 at 12:23 PM
  14. If I’m going to be totally honest I wouldn’t bother constructing the packet. When I tested it I just dumped the hexstream to a file and loaded the bytes; the packet is 87k or thereabouts bytes long so it isn’t worth doing it in code with a wrapper in my opinion. Otherwise, you’re on your own man.
     
  15. Haha that is the exact thing I did indeed (after realizing how complex this packet is), I just write the bytes to a file on one MC version and read that in again on 1.8. Especially given that the packet I need to send is the same each time anyway it really isn't worth the trouble. Thank you again for all your help with this!:)