Using Multipartposthandler to Post Form-Data with Python

Using MultipartPostHandler to POST form-data with Python

It seems that the easiest and most compatible way to get around this problem is to use the 'poster' module.

# test_client.py
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
import urllib2

# Register the streaming http handlers with urllib2
register_openers()

# Start the multipart/form-data encoding of the file "DSC0001.jpg"
# "image1" is the name of the parameter, which is normally set
# via the "name" parameter of the HTML <input> tag.

# headers contains the necessary Content-Type and Content-Length
# datagen is a generator object that yields the encoded parameters
datagen, headers = multipart_encode({"image1": open("DSC0001.jpg")})

# Create the Request object
request = urllib2.Request("http://localhost:5000/upload_image", datagen, headers)
# Actually do the request, and get the response
print urllib2.urlopen(request).read()

This worked perfect and I didn't have to muck with httplib. The module is available here:
http://atlee.ca/software/poster/index.html

Trying to post multipart form data in python, won't post

import urllib, urllib2
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers

def queXF():
register_openers()
url = "http://lilix2/trunk/admin/new.php"
values = {'form':open('test.pdf'),
'bandingxml':open('banding.xml'),
'desc':'description'}
data, headers = multipart_encode(values)
headers['User-Agent'] = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
request = urllib2.Request(url, data, headers)
request.unverifiable = True
response = urllib2.urlopen(request)
the_page = response.read()

Adding headers['User-Agent'] and request.unverifiable = True seems to have fixed it.

Do I need to use a with statement when sending multipart form data in requests?

tl;dr: Your intuition is right.


Without a with statement (or an explicit payload.close() call), you're not closing the file.

Of course it's always possible that some function you call, passing the file as an argument, might close that file. But in general, they won't; and, in particular, requests.post doesn't. (Since that's pretty rare and unusual behavior, it would probably be documented if a function did that. But if you need to be absolutely sure, you can always test it—call the function and then check f.closed—or read the source.)

So, what happens if you don't close it? It just sits there open until the garbage collector deletes the file object. When does that happen? Well, you have a global variable named files that holds a dict that holds the file object. So the file object is accessible all the way until you quit the program. It never becomes garbage, so it never gets deleted, so the file never gets closed, until you exit.

Since you aren't writing anything to the file, this doesn't result in anything horrible like unflushed data sitting around in a buffer when you expected it to be on disk.

But it does mean you're wasting resources. The OS kernel has to keep some kind of structure for every open file, and the the process needs some other thing to reference that kernel structure. If you open thousands of files without closing them, eventually, one of those tables fills up, and the next time you try to open a file, you get an error about having too many files open.

And the kinds of programs that do a lot of work with requests tend to be exactly the kinds of programs that need to work with thousands of files, so this can be a real problem.

So, do you need a with statement here? Maybe not. But you definitely want one.

Python Scrapy override content type to be multipart/form-data on post Request

Just use scrapy.http.FormRequest instead of scrapy.Request, passing the parameters in the formdata argument.

Sample code:

import scrapy
from scrapy.http import FormRequest

class MySpider(scrapy.Spider):
# ...
def start_requests(self):
yield FormRequest(some_post_url,
formdata=dict(param1='value1', param2='value2'))

Read more:

  • Request usage examples
  • FormRequest objects

Multipart form encoding and posting with urllib3

If you wanted to use urllib3 for this, it would look something like this:

import urllib3

http = urllib3.PoolManager()

headers = urllib3.make_headers(user_agent='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6')
url = "http://www.ipm.ucdavis.edu/WEATHER/textupload.cgi"
csv_data = open("weather_scrape.csv").read()

params = {
"FILENAME": csv_data,
'CGIREF': '/calludt.cgi/DDFILE1',
'USE': 'MODEL',
'MODEL': 'CM',
'CROP': 'APPLES',
'METHOD': 'SS',
'UNITS' : 'E',
'LOWTHRESHOLD': '50',
'UPTHRESHOLD': '88',
'CUTOFF': 'H',
'COUNTY': 'AL',
'ACTIVE': 'Y',
'FROMMONTH': '3',
'FROMDAY': '15',
'FROMYEAR': '2013',
'THRUMONTH': '5',
'THRUDAY': '13',
'THRUYEAR': '2013',
'DATASOURCE' : 'FILE',
}

response = http.request('POST', url, params, headers)

I couldn't test this with your target url and csv data set, so it may have some small bugs in it. But that's the general idea.

Make an http POST request to upload a file using Python urllib/urllib2

After some digging around, it seems this post solved my problem. It turns out I need to have the multipart encoder setup properly.

from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
import urllib2

register_openers()

with open("style.css", 'r') as f:
datagen, headers = multipart_encode({"file": f})
request = urllib2.Request("http://jigsaw.w3.org/css-validator/validator", \
datagen, headers)
response = urllib2.urlopen(request)


Related Topics



Leave a reply



Submit