Encrypt Data Bag from Inside of Ruby Without Relying on Knife

Encrypt data bag from inside of ruby without relying on knife

I worked out how to encrypt inside of ruby, just use this code:

require 'chef/knife'
#require 'chef/encrypted_data_bag_item' #you need to do this in chef version 12, they've moved it out of knife and into it's own section
require 'json'

secret = Chef::EncryptedDataBagItem.load_secret Secret_Key_Path

to_encrypt = JSON.parse(json_to_encrypt)

encrypted_data = Chef::EncryptedDataBagItem.encrypt_data_bag_item to_encrypt, secret

Answer achieved with information from this answer, here is the code in question:

namespace 'databag' do
  desc 'Edit encrypted databag item.'
  task :edit, [:databag, :item, :secret_file] do |t, args|
    args.with_defaults :secret_file => "#{ENV['HOME']}/.chef/encrypted_data_bag_secret"
    secret = Chef::EncryptedDataBagItem.load_secret args.secret_file
    item_file = "data_bags/#{args.databag}/#{args.item}.json"
    tmp_item_file = "/tmp/#{args.databag}_#{args.item}.json"
    begin
      #decrypt data bag into tmp file
      raw_hash = Chef::JSONCompat.from_json IO.read item_file
      databag_item = Chef::EncryptedDataBagItem.new raw_hash, secret
      IO.write tmp_item_file, Chef::JSONCompat.to_json_pretty( databag_item.to_hash )
      #edit tmp file
      sh "#{ENV['EDITOR']} #{tmp_item_file}"
      #encrypt tmp file data bag into original file
      raw_hash = Chef::JSONCompat.from_json IO.read tmp_item_file
      databag_item = Chef::EncryptedDataBagItem.encrypt_data_bag_item raw_hash, secret
      IO.write item_file, Chef::JSONCompat.to_json_pretty( databag_item )
    ensure
      ::File.delete tmp_item_file #ensure tmp file deleted.
    end
  end
end

Data bag encryption encrypts on Chef server, but how to encrypt local copy?

I have a following bash (called encrypted-databag.sh) in my chef working directory:

#!/bin/bash -e

knife data bag $1 $2 $3 --secret-file ~/.chef/encrypted_data_bag_secret
if [ "$1" == "edit" ] ; then
    knife data bag show $2 $3 -Fj > "./data_bags/$2/$3.json"
fi

It saves me typing every time I knife to show me the encrypted data bag. And it automatically updates/saves it into repository, when I edit it.

Updated on 30.08.2013

The drawback of the script above is that you edit your data bag straight on chef-server. But there is a problem when you are still working on some cookbook and haven't uploaded it, but the data bag already there and is used by the older version of the cookbook. This way when chef-client is run on some node, it may lead to some errors.

So I was thinking about editing the encrypted data bag locally, without chef-server and then upload the new version of it together with new version of cookbook (after the tests have passed). So here is the rake task I use now to edit encrypted data bags.

namespace 'databag' do
  desc 'Edit encrypted databag item.'
  task :edit, [:databag, :item, :secret_file] do |t, args|
    args.with_defaults :secret_file => "#{ENV['HOME']}/.chef/encrypted_data_bag_secret"
    secret = Chef::EncryptedDataBagItem.load_secret args.secret_file
    item_file = "data_bags/#{args.databag}/#{args.item}.json"
    tmp_item_file = "/tmp/#{args.databag}_#{args.item}.json"
    begin
      #decrypt data bag into tmp file
      raw_hash = Chef::JSONCompat.from_json IO.read item_file
      databag_item = Chef::EncryptedDataBagItem.new raw_hash, secret
      IO.write tmp_item_file, Chef::JSONCompat.to_json_pretty( databag_item.to_hash )
      #edit tmp file
      sh "#{ENV['EDITOR']} #{tmp_item_file}"
      #encrypt tmp file data bag into original file
      raw_hash = Chef::JSONCompat.from_json IO.read tmp_item_file
      databag_item = Chef::EncryptedDataBagItem.encrypt_data_bag_item raw_hash, secret
      IO.write item_file, Chef::JSONCompat.to_json_pretty( databag_item )
    ensure
      ::File.delete tmp_item_file #ensure tmp file deleted.
    end
  end
end

Now to edit encrypted data bag I use:

rake databag:edit[my_databag,item_in_databag]

How do I hide my secret key for data bag encryption on a standalone node using Chef Client local mode

This isn't what encrypted data bags do. The purpose of that feature is to prevent disclosing the contents to the Chef Server. From the PoV of the client, it is in the clear because it has to have the decryption key. If you have only a single node, there isn't much value in the encryption for Chef. It might still be useful if you are storing that data in a git repo or similar, but in those cases you are probably better off with another solution. Check out https://coderanger.net/chef-secrets/ for a summary of the options.

Chef Configuration Constructs by Example

What is a cookbook ?

You have a lot of parts in a cookbook, corresponding to directories:

• Attributes : This directory will contain ruby files to define default attributes used in the recipes later.
• Recipes : Contain ruby files defining resources (directory, files, services) and the desired state for them.
• files : Here you'll store static files you wish to deploy on hosts (a login banner for example)
• resources and providers, library They are used to defined custom resources or helpers to use in recipes
• templates : Here you'll define files templates, they are a way to define files on host which are host dependent, you can give variables to the templates and use the node attributes inside (set the number of thread in mysql to be 3 times the number of cpu for exemple could be computed like node['cpu']['real']*3

At the root of the cookbook you'll find a mandatory file named metadata.rb, which define the cookbook name, its version and its dependencies.

What are dependencies ?

Let's take the database and mysql cookbooks.

In the mysql cookbook there's specific resources defined like mysql_user, mysql_database and so on. The database cookbook use this resources, it depends on the code located in the mysql cookbook.

That's why you'll find a line in its metadata.rb like this depends 'mysql', '>= 5.0.0'. This line tells chef to load the mtsql cookbook in version 5.0.0 or higher if available on the chef-server when the database cookbook is loaded.

What is a node ?

Short one: a computer on which chef-client is run.
Long one: A target system where chef-client is run to get the system in the desired state. The node is also the object in chef where we store the run_list and where is stored attributes about the node (ram,cpu, jdk version, mysql version, etc.)

The attributes on the nodes are made from different sources, automatic attributes gathered by ohai ( ram, number of cpu, disks, os type, etc) are merged with attributes from loaded cookbooks, attributes from roles and attributes from environment the node belongs to. See here for the details on this part;

What is the runlist ?

The runlist is an attribute of the node which list the recipes and roles we want to apply to this node.

What is a Role ?

A role is a kind of helper, it gives you a way to define attributes and a runlist to apply to one or many nodes. the consensus is to avoid setting attributes in roles as they're not version-controlled on the server side and are generally cross environment.

A easy to understand drawback is a password change on mysql when you have an already on line setup. You have 3 server in dev/QA and PROD. If you make a change on the role, it will be applied on all environments when you probably wanted to restrict to dev then QA and then PROD once the tests are OK.

The workaround is to use a cookbook to do the same, use depends in the metadata.rb and include_recipe in this wrapper cookbook to define the runlist, and use this cookbooks attributes files to set the common attributes as in a role.

What is an environment ?

An environment is a logical group for your nodes, you can follow the dev/QA/PROD as your working envs or you may go by kind of system (web-servers/db-servers) and eventually mix both( Dev_web-servers, Dev_db-server, and so on). A node can belong to only one environment.

An environment can host attributes too, usually a dns server, a smtp server specific to this environment, etc. Same warning as roles, they are not version controlled, but the scope is less wide here as it targets a logical group of node.

The main interest of environment is the cookbook version limitation. You can control which version of which cookbook is available on each environment. This come handy when you're working a new version of a cookbook but don't want it to be applied on all your servers. If you change some myslq parameter in a my-mysql cookbook attributes file, you'll wish to restrict when the changes are made on all environment, the limitation will help you there, have QA and PROD limiting your my-mysql cookbook in version A when your dev environment is allowed to use version B.

What is a DataBag ?

As it names imply, a databag is a store of data. It's a group of Json files, each being a DataBagItem and containing json converted to a mash to be used in recipes when loaded.

DataBags goal is to store read-only data, updating a databag from a recipe is dangerous, each item being saved as a whole, two nodes trying to write the same object at the same time will go into a race condition and one change will be lost.

The main purpose of databags is to store common objects (a list of admin users, etc) you don't wish to set in a cookbook/role.

So all glued together

I tried to give a simple view of this documentation on the chef-run here on this paragraph.

We have a node running chef-client, it will ask the chef-server (or read the command line) to know it's runlist. This runlist will then be expanded (search which recipes to load in the roles). In this early stage, ohai will be executed to gather the automatic attributes.

After that load all the cookbooks from the server (or disk in solo/local mode) taking care of limitations and dependencies and read all the attributes files.

Now compile the recipes, in this phase the recipe's ruby code is executed, and a collection of resources is built. At this time nothing has changed on the machine.

Once all recipes have been compiled, the resource collection is ready, go over it and try to get each resource on the desired state. I;e: convergence phase.

The directories will be created if they don't exist. Templates will be rendered and then compared to their target, if they don't match, the target will be replaced. Services are checked for their state, if the desired state is :start and the service is stopped, chef will try to start it.

Once the convergence is done, the node state will be saved on the chef-server.

How can I use a Chef resource within a library module? (Or should I...)?

Libraries are a way to abstract complex Ruby code away from a cookbook recipe.

To group resources (Chef DSL code) you should use either

• Definitions (simplest option) which you can use like a regular Chef resource in your recipes; or
• LWRPs which are more complex but support different actions (think service resources that you can :start, :stop, :restart, etc; or package resources that you can :install, :upgrade, :remove, etc).

---

Update

A definition that would solve your example problem:

# cookbooks/common/definitions/common_stop_services.rb
define :common_stop_services, :services => [] do
  params[:services].each do |svc|
    service svc do
      action :stop
    end
  end
end

then use it like this:

# my_cookbook/recipes/my_recipe.rb
common_stop_services "my_recipe_services" do
  services [ 'svc1', 'svc2' ]
end

Obs: It's probably worth it asking yourself if you really want to wrap multiple service stoppage in a common method. Usually a service action is notified by some other resource (classic example is a configuration file change notifying the reconfigured service to restart, but other patterns apply as well).

Obs2: CamelCase is only used for Classes and Modules in Ruby code. I recommend reading the style guide.

Obs3: Even if you were to implement that code in a library, you probably don't want to use a Class. You're never going to instantiate MyHelper, so what you want is a Module instead.

---

Related Topics