How to Remove  from the Beginning of a File

How do I remove  from the beginning of a file?

Three words for you:

Byte Order Mark (BOM)

That's the representation for the UTF-8 BOM in ISO-8859-1. You have to tell your editor to not use BOMs or use a different editor to strip them out.

To automatize the BOM's removal you can use awk as shown in this question.

As another answer says, the best would be for PHP to actually interpret the BOM correctly, for that you can use mb_internal_encoding(), like this:

 <?php
   //Storing the previous encoding in case you have some other piece 
   //of code sensitive to encoding and counting on the default value.      
   $previous_encoding = mb_internal_encoding();

   //Set the encoding to UTF-8, so when reading files it ignores the BOM       
   mb_internal_encoding('UTF-8');

   //Process the CSS files...

   //Finally, return to the previous encoding
   mb_internal_encoding($previous_encoding);

   //Rest of the code...
  ?>

How do I remove the character  from the beginning of a text file in C++?

That's UTF-8's BOM

You need to read the file as UTF-8. If you don't need Unicode and just use the first 127 ASCII code points then save the file as ASCII or UTF-8 without BOM

Remove BOM () from imported .csv file

Try this:

function removeBomUtf8($s){
  if(substr($s,0,3)==chr(hexdec('EF')).chr(hexdec('BB')).chr(hexdec('BF'))){
       return substr($s,3);
   }else{
       return $s;
   }
}

How to remove  from a PHP page

Your header file contains a Byte Order Mark header, not clearly visible be text editors which understand them and do not print them.

Compilers/parser, on the other hand, expect full ASCII, and don't understand BOM marks.

Solution: Remove the 3 first bytes of your header file with an hex editor or a text editor set to ASCII or UTF8 without BOM encoding and it will work.

below screenshot of the file encoded in UTF8 with BOM with 3 first chars highlighted (using frHed)

If I remove the 3 first bytes it works.
OR: open with Notepad++ and change encoding to ASCII, save, voilà.

Batch script remove BOM () from file

This is because the type command will preserve the UTF-8 BOM, so when you combine multiple files which have the BOM, the final file will contain multiple BOMs in various places in middle of the file.

If you are certain that all the SQL files that you want to combine, start with the BOM, then you can use the following script to remove the BOM from each of them before actually combining them.

This is done by piping the output of type. The other side of pipe will consume the first 3 bytes (The BOM) with the help of 3 pause commands. each pause will consume one byte. The rest of stream will be send to the findstr command to append it to final script.

Since the SQL files are encoded UTF-8 and they may contain any characters in the Unicode range, certain code pages will interfere with the operation and may cause the final SQL script to be corrupted.

So this has been taken into account and the batch file will be restarted with code page 437 which is safe for accessing any binary sequence.

@echo off
setlocal DisableDelayedExpansion


setlocal EnableDelayedExpansion
for /F "tokens=*" %%a in ('chcp') do for %%b in (%%a) do set "CP=%%~nb"
if  !CP! NEQ 437 if !CP! NEQ 65001 chcp 437 >nul && (

    REM for file operations, the script must restatred in a new instance.
    "%COMSPEC%" /c "%~f0"

    REM Restoring previous code page
    chcp !CP! >nul
    exit /b
)
endlocal


set "RemoveUTF8BOM=(pause & pause & pause)>nul"
set "echoNL=echo("
set "FinalScript=C:\FinalScript\AllScripts.sql"

:: If you want the final script to start with UTF-8 BOM (This is optional)
:: Create an empty file in NotePad and save it as UTF8-BOM.txt with UTF-8 encoding.
:: Or Create a file in your HexEditor with this byte sequence: EF BB BF
:: and save it as UTF8-BOM.txt
:: The file must be exactly 3 bytes with the above sequence.
(
    type "UTF8-BOM.txt" 2>nul

    REM This assumes that all sql files start with UTF-8 BOM
    REM If not, then they will loose their first 3 otherwise legitimate characters.
    REM Resulting in a final corrupted script.
    for %%A in (*.sql) do (type "%%~A" & %echoNL%)|(%RemoveUTF8BOM% & findstr "^")

)>"%FinalScript%"

Python: pandas dataframe: Remove  BOM character

Given your comment, I suppose that you ended up having two BOMs.

Let's look at a small example.
I'm using built-in open instead of pd.read_csv/pd.to_csv, but the meaning of the encoding parameter is the same.

Let's create a file saved as UTF-8 with a BOM:

>>> text = 'foo'
>>> with open('/tmp/foo', 'w', encoding='utf-8-sig') as f:
...     f.write(text)

Now let's read it back in.
But we use a different encoding: "utf-8" instead of "utf-8-sig".
In your case, you didn't specify the encoding parameter at all, but the default value is most probably "utf-8" or "cp-1252", which both keep the BOM.
So the following is more or less equivalent to your code snippet:

>>> with open('/tmp/foo', 'r', encoding='utf8') as f:
...     text = f.read()
... 
>>> text
'\ufefffoo'
>>> with open('/tmp/foo_converted', 'w', encoding='utf-8-sig') as f:
...     f.write(text)

The BOM is read as part of the the text; it's the first character (here represented as "\ufeff").

Let's see what's actually in the files, using a suitable command-line tool:

$ hexdump -C /tmp/foo
00000000  ef bb bf 66 6f 6f                                 |...foo|
00000006
$ hexdump -C /tmp/foo_converted 
00000000  ef bb bf ef bb bf 66 6f  6f                       |......foo|
00000009

In UTF-8, the BOM is encoded as the three bytes EF BB BF.
Clearly, the second file has two of them.
So even a BOM-aware program will find some non-sense character in the beginning of foo_converted, as the BOM is only stripped once.

 characters appended to the beginning of each file

OK, I've debugged your code.

BOM marks appear in the source stream when the files are being read from the disk:

byte[] bytes = File.ReadAllBytes(physicalPath);
// TODO: Convert unicode files to specified encoding. For now, assuming
// files are either ASCII or UTF8

If you read the files properly, you can get rid of the marks.

What's  sign at the beginning of my source file?

You propably save the files as UTF-8 with BOM. You should save them as UTF-8 without BOM.

Why does  appear in my data?

It looks like you're opening a file with a UTF-8 encoded Byte Order Mark using the ISO-8859-1 encoding (presumably because this is the default encoding on your OS).

If you open it as bytes and read the first line, you should see something like this:

>>> next(open('pi_million_digits.txt', 'rb'))
b'\xef\xbb\xbf3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679\n'

… where \xef\xbb\xbf is the UTF-8 encoding of the BOM. Opened as ISO-8859-1, it looks like what you're getting:

>>> next(open('pi_million_digits.txt', encoding='iso-8859-1'))
'3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679\n'

… and opening it as UTF-8 shows the actual BOM character U+FEFF:

>>> next(open('pi_million_digits.txt', encoding='utf-8'))
'\ufeff3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679\n'

To strip the mark out, use the special encoding utf-8-sig:

>>> next(open('pi_million_digits.txt', encoding='utf-8-sig'))
'3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679\n'

The use of next() in the examples above is just for demonstration purposes. In your code, you just need to add the encoding argument to your open() line, e.g.

with open(filename, encoding='utf-8-sig') as file_object:
    # ... etc.

---

Related Topics