Securely Erasing Password in Memory (Python)

Python doesn't have that low of a level of control over memory. Accept it, and move on. The best you can do is to del password after calling mail.login so that no references to the password string object remain. Any solution that purports to be able to do more than that is only giving you a false sense of security.

Python string objects are immutable; there's no direct way to change the contents of a string after it is created. Even if you were able to somehow overwrite the contents of the string referred to by password (which is technically possible with stupid ctypes tricks), there would still be other copies of the password that have been created in various string operations:

by the getpass module when it strips the trailing newline off of the inputted password
by the imaplib module when it quotes the password and then creates the complete IMAP command before passing it off to the socket

You would somehow have to get references to all of those strings and overwrite their memory as well.

Prevent a plain text password in RAM (Python)

If you have an OS with process memory protection (all modern OSes have this) then any code that is running in the same process will have access to the password. Other processes will not have access to data unless you grant specific access to a page of memory in an OS specific manner: this is one method RPG is done. The kernel has access to your memory and thus to the password, but if an attack vector comes through that path you have some serious problems.

If you have an OS with virtual memory then the page that contains the password may be written to a file that the root user has access. So a processes that is running in root could read the password from there. But if you have a rogue process running as root you have more serious problems to worry about.

Private members in objects is a language level protection that is only enforced at compilation or interpretation of the code. It has no effect on the RAM access of the data.

So in summary the password is secure while it is in the running process. You only need to be concerned if the password were saved to a file under your control or written to a stream in some manner under your control.

Completely erase object from memory in python

Ok, apparently, there's no real way to do this in Python, since string objects are immutable. References to these strings exist all over the place. Best one can do, is del and gc.collect(). As specified in this link, "accept it, or move on. Anything else will give you a false sense of security"

I need to securely store a username and password in Python, what are my options?

I recommend a strategy similar to ssh-agent. If you can't use ssh-agent directly you could implement something like it, so that your password is only kept in RAM. The cron job could have configured credentials to get the actual password from the agent each time it runs, use it once, and de-reference it immediately using the del statement.

The administrator still has to enter the password to start ssh-agent, at boot-time or whatever, but this is a reasonable compromise that avoids having a plain-text password stored anywhere on disk.

Mark data as sensitive in python

Edit

I have made a solution that uses ctypes (which in turn uses C) to zero memory.

import sys
import ctypes

def zerome(string):
    location = id(string) + 20
    size     = sys.getsizeof(string) - 20

    memset =  ctypes.cdll.msvcrt.memset
    # For Linux, use the following. Change the 6 to whatever it is on your computer.
    # memset =  ctypes.CDLL("libc.so.6").memset

    print "Clearing 0x%08x size %i bytes" % (location, size)

    memset(location, 0, size)

I make no guarantees of the safety of this code. It is tested to work on x86 and CPython 2.6.2. A longer writeup is here.

Decrypting and encrypting in Python will not work. Strings and Integers are interned and persistent, which means you are leaving a mess of password information all over the place.

Hashing is the standard answer, though of course the plaintext eventually needs to be processed somewhere.

The correct solution is to do the sensitive processes as a C module.

But if your memory is constantly being compromised, I would rethink your security setup.

How can I retrieve a password such that it can securely be deleted from memory later?

What do you mean by "hexadecimals"? Do you want to convert pass to a string containing #pass*2 hexadecimal characters? Then you want this:

function toHex(s)
    return (string.gsub(s, ".", function (c)
        return string.format("%02X", string.byte(c))
      end))
end
print(toHex('password')) --> 70617373776F7264

Or do you want a table of numbers, where each number is one character code (byte)? Then you want this:

function toBytes(s)
    return {string.byte(s, 1, #s)}
end
print(table.concat(toBytes('password'), ',')) --> 112,97,115,115,119,111,114,100

How do you securely move sensitive data from memory to a local disk?

There are several ways to accomplish this and I will be providing several examples that either use cleartext data or encrypted data and encrypted compressed files with password protection.

pyzipper

Here is one method to accomplish this use case using pyzipper

import pyzipper

# generate data
secret_data = b'xyz' * 10

# password for the zip file
secret_password = b'super secret password'

# Create encrypted password protected ZIP file on disk
with pyzipper.AESZipFile('password_protected.zip',
                         'w',
                         compression=pyzipper.ZIP_LZMA,
                         encryption=pyzipper.WZ_AES) as zf:
    zf.setpassword(secret_password)
    zf.writestr('secret_data.txt', secret_data)

# Read encrypted password protected ZIP file from disk
with pyzipper.AESZipFile('password_protected.zip', 'r') as zf:
    zf.setpassword(secret_password)
    my_secrets = zf.read('secret_data.txt')
    print(my_secrets)
    # output
    xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz

When I was reviewing the code for pyzipper it wasn't clear if the secret data was being protected during the read and write processes. For extra security I decided to encrypt the secret data using the cryptography package. This package was first released in 2014 and has been continually maintained by hundreds of contributors.

import os
import base64
import pyzipper
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC

# password for the cryptographic process
password = b"password"

# random salt for the cryptographic process
salt = os.urandom(16)

# key derivation function (KDF) for the cryptographic process
kdf = PBKDF2HMAC(
    algorithm=hashes.SHA256(),
    length=32,
    salt=salt,
    iterations=390000,
)

# create the encryption key for the cryptographic process
encryption_key = base64.urlsafe_b64encode(kdf.derive(password))

# https://cryptography.io/en/latest/fernet/
f = Fernet(encryption_key)

# generate data
secret_data = b'xyz' * 10

encrypted_data = f.encrypt(secret_data)

# password for the zip file
secret_password = b'super secret password'

# Create encrypted password protected ZIP file on disk
with pyzipper.AESZipFile('password_protected.zip',
                         'w',
                         compression=pyzipper.ZIP_LZMA,
                         encryption=pyzipper.WZ_AES) as zf:
    zf.setpassword(secret_password)
    zf.writestr('encryption_key.txt', encryption_key)
    zf.writestr('encrypted.txt', encrypted_data)

# Read encrypted password protected ZIP file from disk
# and decrypt data
with pyzipper.PyZipFile('password_protected.zip', 'r') as zf:
    zf.setpassword(secret_password)
    my_secrets = zf.read('encrypted.txt')
    my_key = zf.read('encryption_key.txt')
    f2 = Fernet(my_key)
    decrypt_message = f2.decrypt(my_secrets)
    print(decrypt_message)
    # output
    b'xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz'

For an added layer of security an in-memory file system can be used. In this scenario the secret data is encrypted and then written to an in-memory file, which is then written to an encrypted password protected ZIP file.

import fs
import os
import base64
import pyzipper
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC

# create in-memory file system
mem_fs = fs.open_fs('mem://')
mem_fs.makedir('hidden_dir')

# password for the cryptographic process
password = b"password"

# random salt for the cryptographic process
salt = os.urandom(16)

# key derivation function (KDF) for the cryptographic process
kdf = PBKDF2HMAC(
    algorithm=hashes.SHA256(),
    length=32,
    salt=salt,
    iterations=390000,
)

# create the encryption key for the cryptographic process
encryption_key = base64.urlsafe_b64encode(kdf.derive(password))

# https://cryptography.io/en/latest/fernet/
f = Fernet(encryption_key)

# generate data
secret_data = b'xyz' * 10

encrypted_data = f.encrypt(secret_data)

# password for the zip file
secret_password = b'super secret password'

# write encryption key to an in-memory file
with mem_fs.open('hidden_dir/encryption_key.txt', 'wb') as in_file_in_memory:
    in_file_in_memory.write(encryption_key)
    in_file_in_memory.close()

# write encrypted data to an in-memory file
with mem_fs.open('hidden_dir/encrypted.txt', 'wb') as in_file_in_memory:
    in_file_in_memory.write(encrypted_data)
    in_file_in_memory.close()

# Create encrypted password protected ZIP file on disk
with pyzipper.AESZipFile('password_protected.zip',
                         'w',
                         compression=pyzipper.ZIP_LZMA,
                         encryption=pyzipper.WZ_AES) as zf:
    zf.setpassword(secret_password)
    zf.writestr('encryption_key.txt', mem_fs.readbytes('hidden_dir/encryption_key.txt'))
    zf.writestr('encrypted.txt', mem_fs.readbytes('hidden_dir/encrypted.txt'))

# Read encrypted password protected ZIP file from disk
# and decrypt data
with pyzipper.PyZipFile('password_protected.zip', 'r') as zf:
    zf.setpassword(secret_password)
    my_secrets = zf.read('encrypted.txt')
    my_key = zf.read('encryption_key.txt')
    f2 = Fernet(my_key)
    decrypt_message = f2.decrypt(my_secrets)
    print(decrypt_message)
    b'xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz'

7zip

Another way to accomplish the use case is by using 7zip and the Python subprocess module. For this example I show how to create the in-memory file system, encrypt the data, write the data and the encryption key to a directory in memory, write these files to an encrypted password protected 7zip file on the disk. I also showed how to extract the files from this 7zip archive and decrypt the secret data.

import fs
import os
import base64
from subprocess import Popen, PIPE
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC

# create in-memory file system
mem_fs = fs.open_fs('mem://')
mem_fs.makedir('hidden_dir')

# password for the cryptographic process
password = b"password"

# random salt for the cryptographic process
salt = os.urandom(16)

# key derivation function (KDF) for the cryptographic process
kdf = PBKDF2HMAC(
    algorithm=hashes.SHA256(),
    length=32,
    salt=salt,
    iterations=390000,
)

# create the encryption key for the cryptographic process
encryption_key = base64.urlsafe_b64encode(kdf.derive(password))

# https://cryptography.io/en/latest/fernet/
f = Fernet(encryption_key)

# generate data
secret_data = b'xyz' * 10

encrypted_data = f.encrypt(secret_data)

# password for the zip file
secret_password = b'super secret password'

# write encryption key to an in-memory file
with mem_fs.open('hidden_dir/encryption_key.txt', 'wb') as in_file_in_memory:
    in_file_in_memory.write(encryption_key)
    in_file_in_memory.close()

# write encrypted data to an in-memory file
with mem_fs.open('hidden_dir/encrypted.txt', 'wb') as in_file_in_memory:
    in_file_in_memory.write(encrypted_data)
    in_file_in_memory.close()

# Create encrypted password protected 7ZIP file on disk
file_names = ['encryption_key.txt', 'encrypted.txt']
data_elements = [mem_fs.readbytes('hidden_dir/encryption_key.txt'), mem_fs.readbytes('hidden_dir/encrypted.txt')]
for file_name, data_element in zip(file_names, data_elements):
    args = [f'{full_path}/7zz', 'a', '-mem=AES256', '-y', f'-p{secret_password}', f'-si{file_name}', f'{zip_archive_name}']
    proc_zip = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
    proc_zip.stdin.write(data_element)
    proc_zip.communicate()

# Unzip encrypted password protected 7ZIP file from disk
outfile_directory_name = 'test_files'
args = [f'{full_path}/7zz', "x", f'-p{secret_password}', f"{zip_archive_name}", f'-o{outfile_directory_name}']
proc_unzip = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
proc_unzip.communicate()

# decrypt data from disk
with open(f'{full_path}/{outfile_directory_name}/encryption_key.txt', 'rb') as key_file:
    with open(f'{full_path}/{outfile_directory_name}/encrypted.txt', 'rb') as text_file:
        f = Fernet(key_file.read())
        print(f.decrypt(text_file.read()))
        # output
        b'xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz'

Securely Erasing Password in Memory (Python)