Remove Bom () from Imported .CSV File

Remove BOM () from imported .csv file

Try this:

function removeBomUtf8($s){
  if(substr($s,0,3)==chr(hexdec('EF')).chr(hexdec('BB')).chr(hexdec('BF'))){
       return substr($s,3);
   }else{
       return $s;
   }
}

How to remove BOM from UTF-8 CSV file imported into SQLITE

I figured it out myself.

I applied the encoding to the Open() statement.

class csvrd(object):
    # To be combined into connect()

    def csvFile(self):

        self.readFile('Acronyms.csv')

    def readFile(self, filename):
        conn = sqlite3.connect("database.db")
        cur = conn.cursor()
        cur.execute("CREATE TABLE IF NOT EXISTS Actions (Acronym TEXT, Detail TEXT, Tuple INTEGER PRIMARY KEY)")
        print(filename)
        cur.execute("DELETE FROM Actions")
        with open(filename, encoding = 'utf-8-sig') as f:
            reader = csv.reader(f)
            for field in reader:
                cur.execute("INSERT INTO Actions VALUES (?,?,NULL);", field)
        cur.execute(("Update Actions SET Tuple = Tuple - 1 WHERE Tuple > 0 "))

        conn.commit()
        conn.close()

how remove the BOM() characters from a UTF 8 encoded csv?

Here is a function that does this:

    public static void SaveAsUTF8WithoutByteOrderMark(string fileName)
    {
        SaveAsUTF8WithoutByteOrderMark(fileName, null);
    }

    public static void SaveAsUTF8WithoutByteOrderMark(string fileName, Encoding encoding)
    {
        if (fileName == null)
            throw new ArgumentNullException("fileName");

        if (encoding == null)
        {
            encoding = Encoding.Default;
        }

        File.WriteAllText(fileName, File.ReadAllText(fileName, encoding), new UTF8Encoding(false));
    }

Weird characters added to first column name after reading a toad-exported csv file

Try this:

d <- read.csv("test_file.csv", fileEncoding="UTF-8-BOM")

This works in R 3.0.0+ and removes the BOM if present in the file (common for files generated from Microsoft applications: Excel, SQL server)

Importing txt file into R Studio includes unwanted BOM characters 

What I found was that by saving the file as a ANSI encoded txt file this cleared the issue up.

  items         transactionID
1 {test1,test2} 1            
2 {test1,test3} 2  

You can use the following r studio code to convert your file to ANSI format:

writeLines(iconv(readLines("Old File Name"), from = "UTF8", to = "ANSI_X3.4-1986"), 
           file("New File Name", encoding="ANSI_X3.4-1986"))

Hope this helps someone else if they have the same issue.

---

Related Topics