Standalone Generic SQL UUID updater 1.2

Migrate UUIDs in any SQL database.

  1. CyberTiger submitted a new resource:

    Generic SQL UUID updater - Migrate UUIDs in any SQL database.

    Read more about this resource...
     
  2. LihP

    Supporter

    Hey, thanks for sharing this resource, its really helpful.

    I have run into a little problem that I hope you might be able to help with.
    Yesterday I had success running the converter on a old backup database, 57000 player records where updated without using any rate settings. Today I am trying to run the conversion on the current database with 69000 records, but with and without rate settings I run into the following error which I assume means I have reach the rate limit.
    Code (Text):
    Selecting all player names from the database with: select player_name from tbl_stats
    Exception in thread "main" java.io.IOException: Server returned HTTP response code: 429 for URL: https://api.mojang.com/profiles/minecraft
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
            at org.cyberiantiger.minecraft.sqlconverter.Main.getOnlineUUIDs(Main.java:417)
            at org.cyberiantiger.minecraft.sqlconverter.Main.main(Main.java:323)
    I first tried no rate limit as I did with the test data yesterday, then I tired -R 6000 and finally -R 550. I waited an hour or two between each run.
    Any suggestions? I feel like the rate limits made no difference, can you talk me through how the converter limits the requests? It seems like it took the same amount of time (20-30 seconds) to give the error with an without a rate limit setting. I didn't see any pause in the requests, do you know how many records are requested per second and what will be displayed while the requests are paused for 10 mins?

    Thanks.
     
    #2 LihP, Feb 16, 2015
    Last edited: Feb 16, 2015
  3. 429 html response code means Too Many Requests
     
  4. It defaults to 600 requests / 10 minutes, and it'll do 100 uuids lookups per query (which used to work fine).

    Either Mojang has changed the rate limit or you've other processes on the host requesting a lot of UUIDs, which are causing you to hit the rate limit.

    Using anything over -R 600 will cause you to hit the rate limit. I suggest lowering it massively and see if it still errors, -R 300 *should* be ok, the worst case is Mojang have changed the rate limiting to account for the number of queries per a request (I really hope they haven't), in which case you'll need to use -R 6. Good luck, let me know if you figure it out, I'll change the default if I need to.
     
  5. LihP

    Supporter

    Thanks for the info, I just tried 300 but got the same result, I might try 6 in an hour or so but does that mean I will be requesting 6 UUID's every 10 minutes? If I do test to see if that works is it possible to cancel the process?

    So far I have been running the requests on the server so I assume any limits mojang has place on me are attached to the IP? If I use a remote location is there anything I should consider that might cause a problem?
     
  6. LihP

    Supporter

    Can you tell me a little about the rate option please, if I set it low like 6 should it pause almost immediately and display a message before running again in 10 minutes? Cos I am finding it takes about 30 seconds and then gives me the too many requests error.
    I had such good test runs a few days ago, I managed to convert 50000 uuids in a old test database because I used it without a rate limit the first time I used it. I was shocked how quick it got that many, I think it took less than a few minutes. In fact I managed to get them all for a second time because I thought the format setting "mojang" was what I wanted but it turned out I needed "hexadecimal" so I ran the conversation again using the -F option. I guess I should have used a less populated backup to test on as I cannot get the converter to run on any rate setting now and I desperately want to convert the real database. I also tried using remote locations so I would have a different IP but they all got the 429 error with a 500 rate limit. I am confused how mojang associated the IPs, unless it's the order they are requested in.

    Is it possible to add an option to grab as many as it can then populate the database and wait 10 minutes before trying again, or randomize the order they are requested in? This way I would be making progress slowly.
    Thanks again.
     
    #6 LihP, Feb 18, 2015
    Last edited: Feb 18, 2015
  7. Run the conversion on a machine which isn't running minecraft servers or anything else likely to be abusing Mojang's API at the same time? Mojang's limit is machine wide, if it's erroring even at -R 6, I'm out of sensible suggestions. Assuming you're using mysql you can port forward the database port over ssh. Any errors will likely also be affecting anything else on the machine doing Mojang API queries.
     
  8. cowgod

    Supporter

    @CyberTiger I tried using this on a mysql database with null uuid's and it didn't work. For every username it just outputted "Skipping row with name = (username was here) uuid = ; Invalid uuid" I don't know why it's checking for a UUID if they are all null to begin with? Am I using it incorrectly?
     
  9. The UUID was not null, it was an empty string I guess, try again with -F ? (make a backup first if you didn't already).

    I'll look at updating the code to consider empty strings the same as nulls.
     
  10. cowgod

    Supporter

    Ah, yes they were empty strings. And I didn't try -F but I will give that a shot. Thanks for the quick reply!
     
  11. If you're still stuck, I uploaded a new version with a new flag to force updating of rows with invalid UUIDs.
     
    • Useful Useful x 1
  12. cowgod

    Supporter

    Great, thanks!
     
  13. Get the updated version and use the 'binary' type.
     
    • Like Like x 1
  14. Hello, i really really needs this, but the download link in the resouce page is broken :c
     
  15. Hello, I got this error. Does anyone know how to solve that? Thanks!
    Selecting all player names from the database with: select realname from playertestdata
    Exception in thread "main" java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
    at org.cyberiantiger.minecraft.sqlconverter.Main.getOnlineUUIDs(Main.java:473)
    at org.cyberiantiger.minecraft.sqlconverter.Main.main(Main.java:375)
     
  16. I have solved that, still thanks though.
     
  17. Hi, I got this error when converting online UUID to Offline UUID

    com.mysql.jdbc.NotUpdatable: Result Set not updatable (referenced table has no primary keys).This result set must come from a statement that was created with a result set type of ResultSet.CONCUR_UPDATABLE, the query must select only one table, can not use functions and must select all primary keys from that table. See the JDBC 2.1 API Specification, section 5.6 for more details.
    at com.mysql.jdbc.UpdatableResultSet.generateStatements(UpdatableResultSet.java:590)
    at com.mysql.jdbc.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1463)
    at com.mysql.jdbc.UpdatableResultSet.updateString(UpdatableResultSet.java:2408)
    at org.cyberiantiger.minecraft.sqlconverter.Main$UUIDFormat$1.update(Main.java:55)
    at org.cyberiantiger.minecraft.sqlconverter.Main.main(Main.java:417)