Using MultipartPostHandler to POST form-data with Python
It seems that the easiest and most compatible way to get around this problem is to use the 'poster' module.
# test_client.py
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
import urllib2
# Register the streaming http handlers with urllib2
register_openers()
# Start the multipart/form-data encoding of the file "DSC0001.jpg"
# "image1" is the name of the parameter, which is normally set
# via the "name" parameter of the HTML <input> tag.
# headers contains the necessary Content-Type and Content-Length
# datagen is a generator object that yields the encoded parameters
datagen, headers = multipart_encode({"image1": open("DSC0001.jpg")})
# Create the Request object
request = urllib2.Request("http://localhost:5000/upload_image", datagen, headers)
# Actually do the request, and get the response
print urllib2.urlopen(request).read()
This worked perfect and I didn't have to muck with httplib. The module is available here:
http://atlee.ca/software/poster/index.html
Trying to post multipart form data in python, won't post
import urllib, urllib2
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
def queXF():
register_openers()
url = "http://lilix2/trunk/admin/new.php"
values = {'form':open('test.pdf'),
'bandingxml':open('banding.xml'),
'desc':'description'}
data, headers = multipart_encode(values)
headers['User-Agent'] = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
request = urllib2.Request(url, data, headers)
request.unverifiable = True
response = urllib2.urlopen(request)
the_page = response.read()
Adding headers['User-Agent']
and request.unverifiable = True
seems to have fixed it.
Do I need to use a with statement when sending multipart form data in requests?
tl;dr: Your intuition is right.
Without a with
statement (or an explicit payload.close()
call), you're not closing the file.
Of course it's always possible that some function you call, passing the file as an argument, might close
that file. But in general, they won't; and, in particular, requests.post
doesn't. (Since that's pretty rare and unusual behavior, it would probably be documented if a function did that. But if you need to be absolutely sure, you can always test it—call the function and then check f.closed
—or read the source.)
So, what happens if you don't close it? It just sits there open until the garbage collector deletes the file object. When does that happen? Well, you have a global variable named files
that holds a dict that holds the file object. So the file object is accessible all the way until you quit the program. It never becomes garbage, so it never gets deleted, so the file never gets closed, until you exit
.
Since you aren't writing anything to the file, this doesn't result in anything horrible like unflushed data sitting around in a buffer when you expected it to be on disk.
But it does mean you're wasting resources. The OS kernel has to keep some kind of structure for every open file, and the the process needs some other thing to reference that kernel structure. If you open thousands of files without closing them, eventually, one of those tables fills up, and the next time you try to open a file, you get an error about having too many files open.
And the kinds of programs that do a lot of work with requests
tend to be exactly the kinds of programs that need to work with thousands of files, so this can be a real problem.
So, do you need a with
statement here? Maybe not. But you definitely want one.
Python Scrapy override content type to be multipart/form-data on post Request
Just use scrapy.http.FormRequest instead of scrapy.Request, passing the parameters in the formdata argument.
Sample code:
import scrapy
from scrapy.http import FormRequest
class MySpider(scrapy.Spider):
# ...
def start_requests(self):
yield FormRequest(some_post_url,
formdata=dict(param1='value1', param2='value2'))
Read more:
- Request usage examples
- FormRequest objects
Multipart form encoding and posting with urllib3
If you wanted to use urllib3 for this, it would look something like this:
import urllib3
http = urllib3.PoolManager()
headers = urllib3.make_headers(user_agent='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6')
url = "http://www.ipm.ucdavis.edu/WEATHER/textupload.cgi"
csv_data = open("weather_scrape.csv").read()
params = {
"FILENAME": csv_data,
'CGIREF': '/calludt.cgi/DDFILE1',
'USE': 'MODEL',
'MODEL': 'CM',
'CROP': 'APPLES',
'METHOD': 'SS',
'UNITS' : 'E',
'LOWTHRESHOLD': '50',
'UPTHRESHOLD': '88',
'CUTOFF': 'H',
'COUNTY': 'AL',
'ACTIVE': 'Y',
'FROMMONTH': '3',
'FROMDAY': '15',
'FROMYEAR': '2013',
'THRUMONTH': '5',
'THRUDAY': '13',
'THRUYEAR': '2013',
'DATASOURCE' : 'FILE',
}
response = http.request('POST', url, params, headers)
I couldn't test this with your target url and csv data set, so it may have some small bugs in it. But that's the general idea.
Make an http POST request to upload a file using Python urllib/urllib2
After some digging around, it seems this post solved my problem. It turns out I need to have the multipart encoder setup properly.
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
import urllib2
register_openers()
with open("style.css", 'r') as f:
datagen, headers = multipart_encode({"file": f})
request = urllib2.Request("http://jigsaw.w3.org/css-validator/validator", \
datagen, headers)
response = urllib2.urlopen(request)
Related Topics
Difference Between Returns and Printing in Python
Python Method for Reading Keypress
Retrieving a Foreign Key Value with Django-Rest-Framework Serializers
Pandas: Cast Column to String Does Not Work
How to Run Pygame or Pyglet in a Browser
Is There a Description of How _Cmp_ Works for Dict Objects in Python 2
How to Reverse Lists in Python, Getting "Nonetype" as List
Python String.Strip Stripping Too Many Characters
Standard Way to Embed Version into Python Package
Update Row Values Where Certain Condition Is Met in Pandas
Scrape Website with Dynamic Mouseover Event
What Is the Correct Way to Set Python's Locale on Windows
Can Existing Virtualenv Be Upgraded Gracefully
How to Use Virtualenv with Python
Verifying Compatibility in Compiling Extension Types, and Using Them with Cdef