Putting a `Cookie` in a `CookieJar`
A Requests Session
will receive and send cookies.
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'
(The code above is stolen from Session Objects.)
If you want cookies to persist on disk between runs of your code, you can directly use a CookieJar
and save/load them:
from http.cookiejar import LWPCookieJar
import requests
cookie_file = '/tmp/cookies'
jar = LWPCookieJar(cookie_file)
# Load existing cookies (file might not yet exist)
try:
jar.load()
except:
pass
s = requests.Session()
s.cookies = jar
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")
# Save cookies to disk, even session cookies
jar.save(ignore_discard=True)
Then look in the file /tmp/cookies:
#LWP-Cookies-2.0
Set-Cookie3: sessioncookie=123456789; path="/"; domain="httpbin.org"; path_spec; discard; version=0
How to add a cookie to the cookiejar in python requests library
I found out a way to do it by importing CookieJar, Cookie, and cookies. With help from @Lukasa, he showed me a better way. However, with his way I was not able to specify the "port_specified", "domain_specified", "domain_initial_dot" or "path_specified" attributes. The "set" method does it automatically with default values. I'm trying to scrape a website and their cookie has different values in those attributes. As I am new to all of this I'm not sure if that really matters yet.
my_cookie = {
"version":0,
"name":'COOKIE_NAME',
"value":'true',
"port":None,
# "port_specified":False,
"domain":'www.mydomain.com',
# "domain_specified":False,
# "domain_initial_dot":False,
"path":'/',
# "path_specified":True,
"secure":False,
"expires":None,
"discard":True,
"comment":None,
"comment_url":None,
"rest":{},
"rfc2109":False
}
s = requests.Session()
s.cookies.set(**my_cookie)
Adding cookies to a cookiejar
The session.cookies
object gives you a mapping interface; to add cookies, just set a value for the cookie name:
session.cookies['cookie_name'] = 'cookie_value'
and leave it to requests
.
In the other direction, using session.cookies['cookie_name']
gives you the just the value for that cookie, without a need for you to learn about how the Python CookieJar
object works and tracks all the other per-cookie metadata (such as host names and expiration information).
TypeError for cookielib CookieJar cookie in requests Session
You don't want to set the value of a single cookie in cookies
to a CookieJar
: it already is a CookieJar
:
>>> s = requests.Session()
>>> type(s.cookies)
<class 'requests.cookies.RequestsCookieJar'>
You'll probably have a better time by simply setting s.cookies
to your cookiejar:
def new_cookie(self):
br = mechanize.Browser()
cookie_jar = cookielib.CookieJar()
br.set_cookiejar(cookie_jar)
# Acquire cookies by logging in with mechanize browser
self.s.cookies = cookie_jar
Setting up cookies for Guzzle CookieJar
The source code provided the answer I needed.
The CookieJar class provides a method for building cookies from an associative array. Example:
$domain = 'example.org';
$values = ['users_token' => '2c26b46b68ffc68ff99b453c1d30113413422d706483bfa0f98a5e886266e7ae'];
$cookieJar = \GuzzleHttp\Cookie\CookieJar::fromArray($values, $domain);
$client = new \GuzzleHttp\Client([
'base_uri' => 'https://example.org',
'cookies' => $cookieJar
]);
Requests setting cookies in cookieJar including subdomains
I had the same problem recently. The solution is to do this:
cookies.push(new ToughCookie.Cookie({
domain: 'facebook.com'
key: cookie.name,
value: cookie.value,
secure: cookie.secure || false,
path: cookie.path,
httpOnly: cookie.httpOnly || false,
extensions: extensions
}));
The leading .
in .facebook.com
isn't required. This was discussed on the tough-cookie github issue too.
Save cookies between two curl requests
Use the --cookie-jar
or --dump-header
parameter to save received cookies to a file. The --cookie
parameter can read back the cookies from that file later.
-b, --cookie <name=data>
(HTTP) Pass the data to the HTTP server as a cookie. It is supposedly the data previously received from the server in a "Set-Cookie:" line. The data should be in the format "NAME1=VALUE1; NAME2=VALUE2".
If no '=' symbol is used in the line, it is treated as a filename to use to read previously stored cookie lines from, which should be used in this session if they match. Using this method also activates the cookie engine which will make curl record incoming cookies too, which may be handy if you're using this in combination with the -L, --location option. The file format of the file to read cookies from should be plain HTTP headers (Set-Cookie style) or the Netscape/Mozilla cookie file format.
The file specified with -b, --cookie is only used as input. No cookies will be written to the file. To store cookies, use the -c, --cookie-jar option.
Exercise caution if you are using this option and multiple transfers may occur. If you use the NAME1=VALUE1; format, or in a file use the Set-Cookie format and don't specify a domain, then the cookie is sent for any domain (even after redirects are followed) and cannot be modified by a server-set cookie. If the cookie engine is enabled and a server sets a cookie of the same name then both will be sent on a future transfer to that server, likely not what you intended. To address these issues set a domain in Set-Cookie (doing that will include sub-domains) or use the Netscape format.
If this option is used several times, the last one will be used.
-c, --cookie-jar <file name>
(HTTP) Specify to which file you want curl to write all cookies after a completed operation. Curl writes all cookies previously read from a specified file as well as all cookies received from remote server(s). If no cookies are known, no data will be written. The file will be written using the Netscape cookie file format. If you set the file name to a single dash, "-", the cookies will be written to stdout.
This command line option will activate the cookie engine that makes curl record and use cookies. Another way to activate it is to use the -b, --cookie option.
If the cookie jar can't be created or written to, the whole curl operation won't fail or even report an error clearly. Using -v will get a warning displayed, but that is the only visible feedback you get about this possibly lethal situation.
Since 7.43.0 cookies that were imported in the Set-Cookie format without a domain name are not exported by this option.
If this option is used several times, the last specified file name will be used.
-D, --dump-header <file>
Write the protocol headers to the specified file.
This option is handy to use when you want to store the headers that an HTTP site sends to you. Cookies from the headers could then be read in a second curl invocation by using the -b, --cookie option! The -c, --cookie-jar option is a better way to store cookies.
When used in FTP, the FTP server response lines are considered being "headers" and thus are saved there.
If this option is used several times, the last one will be used
Alternatively, instead of using the command-line cURL app, write some code that uses the libCurl library. That will give you more direct control over cookie handling. libCurl has several features related to HTTP cookies:
Options for curl_easy_getinfo()
:
- CURLINFO_COOKIELIST - get all known cookies
Options for curl_easy_setopt()
:
CURLOPT_COOKIE - set contents of HTTP Cookie header
CURLOPT_COOKIEFILE - file name to read cookies from
CURLOPT_COOKIEJAR - file name to store cookies to
CURLOPT_COOKIESESSION - start a new cookie session
CURLOPT_COOKIELIST - add to or manipulate cookies held in memory
Then you can store the cookies however you want, and assign them as needed to later HTTP sessions.
Related Topics
Child Processes Created with Python Multiprocessing Module Won't Print
Lambda Function Don't Closure the Parameter in Python
When Should Iteritems() Be Used Instead of Items()
How to Write Strategy Pattern in Python Differently Than Example in Wikipedia
Python Command Line Input in a Process
Nltk Named Entity Recognition to a Python List
Parse Key Value Pairs in a Text File
Import Module Works in Terminal But Not in Idle
Web Scraping Dynamic Content with Python
How to Do Row-To-Column Transposition of Data in CSV Table
Replace Invalid Values with None in Pandas Dataframe
Pyplot Move Alternative Y Axis to Background
Why Do I Need to Deploy a "Default" App Before I Can Deploy Multiple Services in Gae
How to Enumerate a Range of Numbers Starting at 1
Why Is Pip Installing an Old Version of My Package
Python - Using the Multiply Operator to Create Copies of Objects in Lists