Set-Cookie - Header shows as "deleted"
I am trying to use deno to script/scrape a website.
I can successfully log in to the site (using fetch), however the Headers do seem to get redacted. Instead of seeing a long token, I only get "deleted". the Cookie is set to expire one second after the epoch (is that influences anything) and another Cookie shows up completely fine.
Without the cookie I can't take any further actions on the site, because the Cookie in question contains the Session Cookie.
8 Replies
According to the documentation, Deno does not have a cookie jar, which means it does not process or filter the
set-cookie header on a response. This could be the reason why you're seeing the set-cookie header as "deleted".
In Deno, the set-cookie header is not concatenated when iterated over in the headers iterator. This behavior is in the process of being specified.
If you're trying to manage sessions or cookies, you might need to handle them manually in your code. For example, you could parse the set-cookie header from the response and store the session cookie in a variable. Then, you could include this session cookie in the cookie header of your subsequent requests.
Please note that this advice is not explicitly stated in the documentation, but is a general approach to handling cookies in environments where they are not automatically managed. 🦕I do know I have to manually handle cookies, which is a bit annoying, but fine for my usecase. What I don't understand is why I don't get the actual cookie value when I read the get-cookies Header from the response, but a literal
deleted.The
Set-Cookie header showing as "deleted" could be due to the server setting the cookie to be deleted. This is usually done by setting the cookie's expiration date to a time in the past. This effectively tells the browser (or in this case, your Deno script) to delete the cookie.
If the server is setting the cookie to be deleted immediately after setting it, it could be due to a variety of reasons. For example, it could be a mechanism to prevent session fixation attacks, or it could be due to some server-side error.
Without more information about the server's behavior or the specific website you're trying to scrape, it's hard to say for sure. You might want to check the server's behavior in a web browser to see if the same thing happens there. If it does, it's likely a server-side issue. If it doesn't, it could be an issue with how Deno is handling the Set-Cookie header.
Remember that web scraping should be done responsibly and in accordance with the website's terms of service. Some websites may employ mechanisms to prevent or limit web scraping.A Browser shows the Header as expiring with the current session, however when using curl the set-cookie contains a expiry date at the unix epoch. I will have to test further whether the browser or deno is misbehaving or if it's the server.
Based on the documentation, it's important to note that Deno does not have a cookie jar, which means it does not process or filter the
set-cookie header on a response. This could potentially explain the discrepancy you're seeing between the browser and Deno.
The set-cookie header showing as "deleted" and the expiry date being set to the Unix epoch (which is a common way to delete a cookie) could be due to the server setting the cookie to be deleted. This is usually done by setting the cookie's expiration date to a time in the past.
If the server is setting the cookie to be deleted immediately after setting it, it could be due to a variety of reasons. For example, it could be a mechanism to prevent session fixation attacks, or it could be due to some server-side error.
Without more information about the server's behavior or the specific website you're trying to scrape, it's hard to say for sure. You might want to check the server's behavior in a web browser to see if the same thing happens there. If it does, it's likely a server-side issue. If it doesn't, it could be an issue with how Deno is handling the Set-Cookie header.
Remember that web scraping should be done responsibly and in accordance with the website's terms of service. Some websites may employ mechanisms to prevent or limit web scraping.If it helps in any way, I've been working on Curl bindings for Deno, but it hasn't been tested on many systems. It should not work in Ubuntu 20 or lower for example
In order to activate Curl's cookies engine, you set the cookie file to an empty string, like:
decurl.setCookiefile(''). That's how libcurl works
https://github.com/Leokuma/decurlGitHub
GitHub - Leokuma/decurl: Curl bindings for Deno.
Curl bindings for Deno. Contribute to Leokuma/decurl development by creating an account on GitHub.
That might help but wasn't the issue.
In case anybody finds this in the future: If you server returns a redirect, deno seems to follow it. Since deno doesn't handles cookies the Cookie was missing in the second request, so the Server wanted to delete the cookie. I only got the final cookie, not the one returned in the redirect.
Maybe this could help then: https://developer.mozilla.org/en-US/docs/Web/API/Response/redirected#disallowing_redirects
MDN Web Docs
Response: redirected property - Web APIs | MDN
The read-only redirected property of the Response interface indicates whether or not the response is the result of a request you made which was redirected.