_Csv.Error: Field Larger Than Field Limit (131072)

_csv.Error: field larger than field limit (131072)

The csv file might contain very huge fields, therefore increase the field_size_limit:

import sys
import csv

csv.field_size_limit(sys.maxsize)

sys.maxsize works for Python 2.x and 3.x. sys.maxint would only work with Python 2.x (SO: what-is-sys-maxint-in-python-3)

Update

As Geoff pointed out, the code above might result in the following error: OverflowError: Python int too large to convert to C long.
To circumvent this, you could use the following quick and dirty code (which should work on every system with Python 2 and Python 3):

import sys
import csv
maxInt = sys.maxsize

while True:
    # decrease the maxInt value by factor 10 
    # as long as the OverflowError occurs.

    try:
        csv.field_size_limit(maxInt)
        break
    except OverflowError:
        maxInt = int(maxInt/10)

Python 3.7 64 bit on Windows 7 (64 bit): CSV - field larger than field limit (131072)

You could replace sys.maxsize by the c integer max value, which is 2147483647.

I know the sys.maxsize should take care of it, but I think using a value inferior to that roof, like 1.000.000 should resolve your issue.

A nicer way to do it could be min(sys.maxsize, 2147483646)

The _csv library is a compiled extension, then it uses the c variables.

Difficult CSV import

One Side Note:

In Python, regardless of where the " are, having them within a string is bothersome.

When you set

x = "He said "hi there!" to me"

This will return an error, because it parses the string "He said " and then errors out with hi there!. This is an issue you may run into depending on how you parse your csv file.

This would return a different error than you are receiving, so it might not pose an immediate problem, but if the row you uploaded in your question actually has "Hello, Im text. "Thats a silly way to do it" is what the guy said." in it you might run into issues down the road.

Back to the problem at hand:

To solve the error you are receiving, you can try running:

import sys
import csv

csv.field_size_limit(sys.maxsize)

This should increase the size of the field read_csv() accepts.

Hope it helps!

Source

Cassandra CQLSH TEXT field limit on COPY FROM CSV (field larger than field limit (131072))

Take a look at this answer:

_csv.Error: field larger than field limit (131072)

You will need to add this solution to the top of the cqlsh file. So after:

import csv
import getpass

csv.field_size_limit(sys.maxsize)

csv.field_size_limit not worked

You will need to call field_size_limit() with the new limit.

csv.field_size_limit(10000000)  # 10 megabytes, give or take

---

Related Topics