JavaFx Webview Cookies

Discussion in 'Programming' started by Stef, Apr 14, 2017.

  1. For the last couple of weeks I've been experimenting with JavaFX and as a fun project made a web browser with it, with the Webview component. It's working decent, most important stuff (for me at least) is working and the Webview does a great job at loading web pages (not all pages work, but that's fine). One thing that's really bothering me however, is cookies. The webview component uses the CookieHandler class and stores cookies directly in there. As long as I'm running the application cookies work fine even across multiple tabs. But when I exit the program they are obviously gone. So what I do is save these cookies to a file and then when the program starts it loads them back into the cookiemanager. But for some reason, no webpage seems to be 'accepting' my cookies except for one page which retrieves the username (that's how I know they are actually loaded). However that page normally also saves the password so you can easily log in, but it doesn't do that. I'm completely stuck at this point. I debugged my program multiple times, checked the cookie manager, file formats etc. but I haven't made any progress. Here is the code that handles the cookies:
    Code (Text):
    CookieManager manager = new CookieManager(new CustomCookieStore(), CookiePolicy.ACCEPT_ALL);

    try {
        for (String line : Files.readAllLines(Values.COOKIESFILE.toPath())) {
            String[] values = line.split("~");

            for (String header : values) {
                HttpCookie cookie = CookieUtil.fromString(header);

                if (cookie == null || cookie.hasExpired())
                    continue;

                manager.getCookieStore().add(new URI(header.split("|")[0]), cookie);
            }
        }
    } catch (IOException | URISyntaxException e) {
        e.printStackTrace();
    }

    CookieHandler.setDefault(manager);
     
    Code (Text):

    CookieManager man = (CookieManager) CookieHandler.getDefault();
    CookieStore store = man.getCookieStore();
    try {
        Path cookieFilePath = Values.COOKIESFILE.toPath();

        Files.write(cookieFilePath, ("").getBytes(), StandardOpenOption.TRUNCATE_EXISTING);

        for (URI uri : store.getURIs()) {
            for (HttpCookie cookie : store.get(uri)) {
                if (cookie.hasExpired())
                    continue;

                Files.write(cookieFilePath, (uri.toString() + "|" + CookieUtil.toString(cookie) + "~").getBytes(), StandardOpenOption.APPEND);
            }
        }

        Files.write(cookieFilePath, "\n".getBytes(), StandardOpenOption.APPEND);
    } catch (IOException e) {
        e.printStackTrace();
    }
     
    Code (Text):
    public static String toString(HttpCookie cookie) {
        try {
            Class<? extends HttpCookie> clazz = cookie.getClass();
            Field whenCreated = clazz.getDeclaredField("whenCreated");
            whenCreated.setAccessible(true);
            Field header = clazz.getDeclaredField("header");
            header.setAccessible(true);

            return cookie.getName() + "|" + cookie.getValue() +
                    ";Comment=" + cookie.getComment() +
                    ";CommentURL=" + cookie.getCommentURL() +
                    ";Discard=" + cookie.getDiscard() +
                    ";Domain=" + cookie.getDomain() +
                    ";Header=" + header.get(cookie) +
                    ";Max-Age=" + cookie.getMaxAge() +
                    ";Path=" + cookie.getPath() +
                    ";Port=" + cookie.getPortlist() +
                    ";Secure=" + cookie.getSecure() +
                    ";HttpOnly=" + cookie.isHttpOnly() +
                    ";Version=" + cookie.getVersion() +
                    ";WhenCreated=" + whenCreated.get(cookie);
        } catch (IllegalAccessException | NoSuchFieldException e) {
            e.printStackTrace();
            return null;
        }
    }

    public static HttpCookie fromString(String string) {
        String[] split = string.split(";");
        String[] nameValue = split[0].split("|");

        if (nameValue.length < 2)
            return null;

        HttpCookie cookie = new HttpCookie(nameValue[1], nameValue[2]);

        for (int i = 1; i < split.length; i++) {
            String[] value = split[i].split("=");

            if (value.length < 2)
                continue;

            if (value[1].equals("null"))
                value[1] = null;

            try {
                switch (value[0]) {
                    case "Comment":
                        cookie.setComment(value[1]);
                        break;
                    case "CommentURL":
                        cookie.setCommentURL(value[1]);
                        break;
                    case "Discard":
                        cookie.setDiscard(Boolean.parseBoolean(value[1]));
                        break;
                    case "Domain":
                        cookie.setDomain(value[1]);
                        break;
                    case "Header":
                        Field header = cookie.getClass().getDeclaredField("header");
                        header.setAccessible(true);
                        header.set(cookie, value[1]);
                        break;
                    case "Max-Age":
                        cookie.setMaxAge(Long.parseLong(value[1] == null ? "0" : value[1]));
                        break;
                    case "Path":
                        cookie.setPath(value[1]);
                        break;
                    case "Port":
                        cookie.setPortlist(value[1]);
                        break;
                    case "Secure":
                        cookie.setSecure(Boolean.parseBoolean(value[1]));
                        break;
                    case "HttpOnly":
                        cookie.setHttpOnly(Boolean.parseBoolean(value[1]));
                        break;
                    case "Version":
                        cookie.setVersion(Integer.parseInt(value[1] == null ? "0" : value[1]));
                        break;
                    case "WhenCreated":
                        Field whenCreated = cookie.getClass().getDeclaredField("whenCreated");
                        whenCreated.setAccessible(true);
                        whenCreated.set(cookie, Long.parseLong(value[1]));
                        break;
                }
            } catch (IllegalAccessException | NoSuchFieldException e) {
                e.printStackTrace();
            }
        }
        return cookie;
    }
     
    Code (Text):
    public class CustomCookieStore implements CookieStore {

        // the in-memory representation of cookies
        private List<HttpCookie> cookieJar = null;

        // the cookies are indexed by its domain and associated uri (if present)
        // CAUTION: when a cookie removed from main data structure (i.e. cookieJar),
        //          it won't be cleared in domainIndex & uriIndex. Double-check the
        //          presence of cookie when retrieve one form index store.
        private Map<String, List<HttpCookie>> domainIndex = null;
        private Map<URI, List<HttpCookie>> uriIndex = null;

        // use ReentrantLock instead of syncronized for scalability
        private ReentrantLock lock = null;


        /**
         * The default ctor
         */
        public CustomCookieStore() {
            cookieJar = new ArrayList<>();
            domainIndex = new HashMap<>();
            uriIndex = new HashMap<>();

            lock = new ReentrantLock(false);
        }

        /**
         * Add one cookie into cookie store.
         */
        public void add(URI uri, HttpCookie cookie) {
            // pre-condition : argument can't be null
            if (cookie == null) {
                throw new NullPointerException("cookie is null");
            }


            lock.lock();
            try {
                // remove the ole cookie if there has had one
                cookieJar.remove(cookie);

                // add new cookie if it has a non-zero max-age
                if (cookie.getMaxAge() != 0) {
                    cookieJar.add(cookie);
                    // and add it to domain index
                    if (cookie.getDomain() != null) {
                        addIndex(domainIndex, cookie.getDomain(), cookie);
                    }
                    if (uri != null) {
                        // add it to uri index, too
                        addIndex(uriIndex, getEffectiveURI(uri), cookie);
                    }
                }
            } finally {
                lock.unlock();
            }
        }


        /**
         * Get all cookies, which:
         *  1) given uri domain-matches with, or, associated with
         *     given uri when added to the cookie store.
         *  3) not expired.
         * See RFC 2965 sec. 3.3.4 for more detail.
         */
        public List<HttpCookie> get(URI uri) {
            // argument can't be null
            if (uri == null) {
                throw new NullPointerException("uri is null");
            }

            List<HttpCookie> cookies = new ArrayList<>();
            boolean secureLink = "https".equalsIgnoreCase(uri.getScheme());
            lock.lock();
            try {
                // check domainIndex first
                getInternal1(cookies, domainIndex, uri.getHost(), secureLink);
                // check uriIndex then
                getInternal2(cookies, uriIndex, getEffectiveURI(uri), secureLink);
            } finally {
                lock.unlock();
            }

            return cookies;
        }

        /**
         * Get all cookies in cookie store, except those have expired
         */
        public List<HttpCookie> getCookies() {
            List<HttpCookie> rt;

            lock.lock();
            try {
                Iterator<HttpCookie> it = cookieJar.iterator();
            } finally {
                rt = Collections.unmodifiableList(cookieJar);
                lock.unlock();
            }

            return rt;
        }

        /**
         * Get all URIs, which are associated with at least one cookie
         * of this cookie store.
         */
        public List<URI> getURIs() {
            List<URI> uris = new ArrayList<>();

            lock.lock();
            try {
                Iterator<URI> it = uriIndex.keySet().iterator();
                while (it.hasNext()) {
                    URI uri = it.next();
                    List<HttpCookie> cookies = uriIndex.get(uri);
                    if (cookies == null || cookies.size() == 0) {
                        // no cookies list or an empty list associated with
                        // this uri entry, delete it
                        it.remove();
                    }
                }
            } finally {
                uris.addAll(uriIndex.keySet());
                lock.unlock();
            }

            return uris;
        }


        /**
         * Remove a cookie from store
         */
        public boolean remove(URI uri, HttpCookie ck) {
            // argument can't be null
            if (ck == null) {
                throw new NullPointerException("cookie is null");
            }

            boolean modified;
            lock.lock();
            try {
                modified = cookieJar.remove(ck);
            } finally {
                lock.unlock();
            }

            return modified;
        }


        /**
         * Remove all cookies in this cookie store.
         */
        public boolean removeAll() {
            lock.lock();
            try {
                if (cookieJar.isEmpty()) {
                    return false;
                }
                cookieJar.clear();
                domainIndex.clear();
                uriIndex.clear();
            } finally {
                lock.unlock();
            }

            return true;
        }


        /* ---------------- Private operations -------------- */


        /*
         * This is almost the same as HttpCookie.domainMatches except for
         * one difference: It won't reject cookies when the 'H' part of the
         * domain contains a dot ('.').
         * I.E.: RFC 2965 section 3.3.2 says that if host is x.y.domain.com
         * and the cookie domain is .domain.com, then it should be rejected.
         * However that's not how the real world works. Browsers don't reject and
         * some sites, like yahoo.com do actually expect these cookies to be
         * passed along.
         * And should be used for 'old' style cookies (aka Netscape type of cookies)
         */
        private boolean netscapeDomainMatches(String domain, String host)
        {
            if (domain == null || host == null) {
                return false;
            }

            // if there's no embedded dot in domain and domain is not .local
            boolean isLocalDomain = ".local".equalsIgnoreCase(domain);
            int embeddedDotInDomain = domain.indexOf('.');
            if (embeddedDotInDomain == 0) {
                embeddedDotInDomain = domain.indexOf('.', 1);
            }
            if (!isLocalDomain && (embeddedDotInDomain == -1 || embeddedDotInDomain == domain.length() - 1)) {
                return false;
            }

            // if the host name contains no dot and the domain name is .local
            int firstDotInHost = host.indexOf('.');
            if (firstDotInHost == -1 && isLocalDomain) {
                return true;
            }

            int domainLength = domain.length();
            int lengthDiff = host.length() - domainLength;
            if (lengthDiff == 0) {
                // if the host name and the domain name are just string-compare euqal
                return host.equalsIgnoreCase(domain);
            } else if (lengthDiff > 0) {
                // need to check H & D component
                String H = host.substring(0, lengthDiff);
                String D = host.substring(lengthDiff);

                return (D.equalsIgnoreCase(domain));
            } else if (lengthDiff == -1) {
                // if domain is actually .host
                return (domain.charAt(0) == '.' &&
                        host.equalsIgnoreCase(domain.substring(1)));
            }

            return false;
        }

        private void getInternal1(List<HttpCookie> cookies, Map<String, List<HttpCookie>> cookieIndex,
                                  String host, boolean secureLink) {
            // Use a separate list to handle cookies that need to be removed so
            // that there is no conflict with iterators.
            ArrayList<HttpCookie> toRemove = new ArrayList<>();
            for (Map.Entry<String, List<HttpCookie>> entry : cookieIndex.entrySet()) {
                String domain = entry.getKey();
                List<HttpCookie> lst = entry.getValue();
                for (HttpCookie c : lst) {
                    if ((c.getVersion() == 0 && netscapeDomainMatches(domain, host)) ||
                            (c.getVersion() == 1 && netscapeDomainMatches(domain, host))) {
                        if ((cookieJar.indexOf(c) != -1)) {
                            // don't add twice and make sure it's the proper
                            // security level
                            if (!cookies.contains(c)) {
                                cookies.add(c);
                            }
                        } else {
                            // the cookie has beed removed from main store,
                            // so also remove it from domain indexed store
                            toRemove.add(c);
                        }
                    }
                }
                // Clear up the cookies that need to be removed
                for (HttpCookie c : toRemove) {
                    lst.remove(c);
                    cookieJar.remove(c);

                }
                toRemove.clear();
            }
        }

        // @param cookies           [OUT] contains the found cookies
        // @param cookieIndex       the index
        // @param comparator        the prediction to decide whether or not
        //                          a cookie in index should be returned
        private <T> void getInternal2(List<HttpCookie> cookies,
                                      Map<T, List<HttpCookie>> cookieIndex,
                                      Comparable<T> comparator, boolean secureLink)
        {
            for (T index : cookieIndex.keySet()) {
                if (comparator.compareTo(index) == 0) {
                    List<HttpCookie> indexedCookies = cookieIndex.get(index);
                    // check the list of cookies associated with this domain
                    if (indexedCookies != null) {
                        Iterator<HttpCookie> it = indexedCookies.iterator();
                        while (it.hasNext()) {
                            HttpCookie ck = it.next();
                            if (cookieJar.indexOf(ck) != -1) {
                                // don't add twice
                                if (!cookies.contains(ck))
                                    cookies.add(ck);
                            } else {
                                // the cookie has beed removed from main store,
                                // so also remove it from domain indexed store
                                it.remove();
                            }
                        }
                    } // end of indexedCookies != null
                } // end of comparator.compareTo(index) == 0
            } // end of cookieIndex iteration
        }

        // add 'cookie' indexed by 'index' into 'indexStore'
        private <T> void addIndex(Map<T, List<HttpCookie>> indexStore,
                                  T index,
                                  HttpCookie cookie)
        {
            if (index != null) {
                List<HttpCookie> cookies = indexStore.get(index);
                if (cookies != null) {
                    // there may already have the same cookie, so remove it first
                    cookies.remove(cookie);

                    cookies.add(cookie);
                } else {
                    cookies = new ArrayList<>();
                    cookies.add(cookie);
                    indexStore.put(index, cookies);
                }
            }
        }


        //
        // for cookie purpose, the effective uri should only be http://host
        // the path will be taken into account when path-match algorithm applied
        //
        private URI getEffectiveURI(URI uri) {
            URI effectiveURI;
            try {
                effectiveURI = new URI("http",
                        uri.getHost(),
                        null,  // path component
                        null,  // query component
                        null   // fragment component
                );
            } catch (URISyntaxException ignored) {
                effectiveURI = uri;
            }

            return effectiveURI;
        }
    }
    Any help would be appreciated.
     
    #1 Stef, Apr 14, 2017
    Last edited: Apr 18, 2017
  2. Have you debugged the String after loading? (Like lines, values, actual values, header, etc)

    And have you matched your received/sent cookies with a normal request? (check a browser's dev console for incoming Cookie headers and outgoing Set-Cookie headers)
     
  3. I did, but I'll try it again. I definitely made some changes to the system last time I checked.

    EDIT: I did some more research and apparently the cookiestore rejects cookies with an url which starts with a '.', because "RFC 2965 section 3.3.2 says that if host is x.y.domain.com and the cookie domain is .domain.com, then it should be rejected.", however it also states that it won't reject it, because "some sites, like yahoo.com do actually expect these cookies to be passed along.". However the code only checks this old method when the version number is zero while many websites use version number one while still using the old technique.
    So I decided to create my own cookiestore, exactly the same as the InMemoryCookieStore, but it does accept these values. Still no luck. I also saw that the character I used as a seperator was used in some strings from the cookie and the parsing was therefore incorrect, but when changed the cookies still wouldn't work correctly.
    I updated the code in my OP to reflect the changes I made. If anyone has an idea why this is happening, please tell me.
     
    #3 Stef, Apr 18, 2017
    Last edited: Apr 18, 2017
  4. Bump, I still need help.