herkulessi
herkulessi8mo ago

Set-Cookie - Header shows as "deleted"

I am trying to use deno to script/scrape a website. I can successfully log in to the site (using fetch), however the Headers do seem to get redacted. Instead of seeing a long token, I only get "deleted". the Cookie is set to expire one second after the epoch (is that influences anything) and another Cookie shows up completely fine. Without the cookie I can't take any further actions on the site, because the Cookie in question contains the Session Cookie.
8 Replies
Deno AI Helper
Deno AI Helper8mo ago
According to the documentation, Deno does not have a cookie jar, which means it does not process or filter the set-cookie header on a response. This could be the reason why you're seeing the set-cookie header as "deleted". In Deno, the set-cookie header is not concatenated when iterated over in the headers iterator. This behavior is in the process of being specified. If you're trying to manage sessions or cookies, you might need to handle them manually in your code. For example, you could parse the set-cookie header from the response and store the session cookie in a variable. Then, you could include this session cookie in the cookie header of your subsequent requests. Please note that this advice is not explicitly stated in the documentation, but is a general approach to handling cookies in environments where they are not automatically managed. 🦕
herkulessi
herkulessi8mo ago
I do know I have to manually handle cookies, which is a bit annoying, but fine for my usecase. What I don't understand is why I don't get the actual cookie value when I read the get-cookies Header from the response, but a literal deleted.
Deno AI Helper
Deno AI Helper8mo ago
The Set-Cookie header showing as "deleted" could be due to the server setting the cookie to be deleted. This is usually done by setting the cookie's expiration date to a time in the past. This effectively tells the browser (or in this case, your Deno script) to delete the cookie. If the server is setting the cookie to be deleted immediately after setting it, it could be due to a variety of reasons. For example, it could be a mechanism to prevent session fixation attacks, or it could be due to some server-side error. Without more information about the server's behavior or the specific website you're trying to scrape, it's hard to say for sure. You might want to check the server's behavior in a web browser to see if the same thing happens there. If it does, it's likely a server-side issue. If it doesn't, it could be an issue with how Deno is handling the Set-Cookie header. Remember that web scraping should be done responsibly and in accordance with the website's terms of service. Some websites may employ mechanisms to prevent or limit web scraping.
herkulessi
herkulessi8mo ago
A Browser shows the Header as expiring with the current session, however when using curl the set-cookie contains a expiry date at the unix epoch. I will have to test further whether the browser or deno is misbehaving or if it's the server.
Deno AI Helper
Deno AI Helper8mo ago
Based on the documentation, it's important to note that Deno does not have a cookie jar, which means it does not process or filter the set-cookie header on a response. This could potentially explain the discrepancy you're seeing between the browser and Deno. The set-cookie header showing as "deleted" and the expiry date being set to the Unix epoch (which is a common way to delete a cookie) could be due to the server setting the cookie to be deleted. This is usually done by setting the cookie's expiration date to a time in the past. If the server is setting the cookie to be deleted immediately after setting it, it could be due to a variety of reasons. For example, it could be a mechanism to prevent session fixation attacks, or it could be due to some server-side error. Without more information about the server's behavior or the specific website you're trying to scrape, it's hard to say for sure. You might want to check the server's behavior in a web browser to see if the same thing happens there. If it does, it's likely a server-side issue. If it doesn't, it could be an issue with how Deno is handling the Set-Cookie header. Remember that web scraping should be done responsibly and in accordance with the website's terms of service. Some websites may employ mechanisms to prevent or limit web scraping.
Leokuma
Leokuma8mo ago
If it helps in any way, I've been working on Curl bindings for Deno, but it hasn't been tested on many systems. It should not work in Ubuntu 20 or lower for example In order to activate Curl's cookies engine, you set the cookie file to an empty string, like: decurl.setCookiefile(''). That's how libcurl works https://github.com/Leokuma/decurl
GitHub
GitHub - Leokuma/decurl: Curl bindings for Deno.
Curl bindings for Deno. Contribute to Leokuma/decurl development by creating an account on GitHub.
herkulessi
herkulessi8mo ago
That might help but wasn't the issue. In case anybody finds this in the future: If you server returns a redirect, deno seems to follow it. Since deno doesn't handles cookies the Cookie was missing in the second request, so the Server wanted to delete the cookie. I only got the final cookie, not the one returned in the redirect.
Leokuma
Leokuma8mo ago
MDN Web Docs
Response: redirected property - Web APIs | MDN
The read-only redirected property of the Response interface indicates whether or not the response is the result of a request you made which was redirected.
More Posts
I keep running into python download failed while trying to run 'cargo test' commandI am currently following the steps I. Deno's Contributing Guide and I am running cargo test but I kehow I can create an restful api with deno and firebase as database?how I can create an restful api with deno and firebase as database?How can I replace writeAllSync with the Streams API?How would one best refactor the following code to use the Web Streams API as suggested by the deprecIs possible to run nodejs (typescript) project with esm modules?I'm trying to run small nodejs project with esm modules. Simple example ```ts // foo.ts import { bAny tutorial links to help me understand the Schedule string in Deno.cron()I am currently working on an issue on the deno repo and I am trying to reconstruct the deno scheduleDeno Deploy failed:Uncaught Exception /Permission Denied : Deno.exit() is not allowedI have a college assignments that needs to be deployed using deno deploy but i get this message whenConnect to a WebSocket through a HTTP(S)/SOCKS5 proxyTitle - I am trying to connect to a `wss://` websocket through a proxy of choice at runtime. Is therExternal package code throws error: window is undefinedHello, I have imported an npm package to my project. And in this package's code, there is a conditiError with importsI have this fancy peace of code that import a bunch of files from a directory: ```ts const utils = Deno.env via fileHow can I use set env from a file and get them via deno.env.get() function as i want to push my code