Use Yaml With Variables

Use placeholders in yaml

Context

  • YAML version 1.2
  • user wishes to
    • include variable placeholders in YAML
    • have placeholders replaced with computed values, upon yaml.load
    • be able to use placeholders for both YAML mapping keys and values

Problem

  • YAML does not natively support variable placeholders.
  • Anchors and Aliases almost provide the desired functionality, but these do not work as variable placeholders that can be inserted into arbitrary regions throughout the YAML text. They must be placed as separate YAML nodes.
  • There are some add-on libraries that support arbitrary variable placeholders, but they are not part of the native YAML specification.

Example

Consider the following example YAML. It is well-formed YAML syntax, however it uses (non-standard) curly-brace placeholders with embedded expressions.

The embedded expressions do not produce the desired result in YAML, because they are not part of the native YAML specification. Nevertheless, they are used in this example only to help illustrate what is available with standard YAML and what is not.

part01_customer_info:
cust_fname: "Homer"
cust_lname: "Himpson"
cust_motto: "I love donuts!"
cust_email: homer@himpson.org

part01_government_info:
govt_sales_taxrate: 1.15

part01_purchase_info:
prch_unit_label: "Bacon-Wrapped Fancy Glazed Donut"
prch_unit_price: 3.00
prch_unit_quant: 7
prch_product_cost: "{{prch_unit_price * prch_unit_quant}}"
prch_total_cost: "{{prch_product_cost * govt_sales_taxrate}}"

part02_shipping_info:
cust_fname: "{{cust_fname}}"
cust_lname: "{{cust_lname}}"
ship_city: Houston
ship_state: Hexas

part03_email_info:
cust_email: "{{cust_email}}"
mail_subject: Thanks for your DoughNutz order!
mail_notes: |
We want the mail_greeting to have all the expected values
with filled-in placeholders (and not curly-braces).
mail_greeting: |
Greetings {{cust_fname}} {{cust_lname}}!

We love your motto "{{cust_motto}}" and we agree with you!

Your total purchase price is {{prch_total_cost}}

Explanation

  • Below is an inline image that illustrates the example with colored regions in green, yellow and red.

  • The substitutions marked in GREEN are readily available in standard YAML, using anchors, aliases, and merge keys.

  • The substitutions marked in YELLOW are technically available in standard YAML, but not without a custom type declaration, or some other binding mechanism.

  • The substitutions marked in RED are not available in standard YAML. Yet there are workarounds and alternatives; such as through string formatting or string template engines (such as python's str.format).

Image explaining the different types of variable substitution in YAML

Details

A frequently-requested feature for YAML is the ability to insert arbitrary variable placeholders that support arbitrary cross-references and expressions that relate to the other content in the same (or transcluded) YAML file(s).

YAML supports anchors and aliases, but this feature does not support arbitrary placement of placeholders and expressions anywhere in the YAML text. They only work with YAML nodes.

YAML also supports custom type declarations, however these are less common, and there are security implications if you accept YAML content from potentially untrusted sources.

YAML addon libraries

There are YAML extension libraries, but these are not part of the native YAML spec.

  • Ansible
    • https://docs.ansible.com/ansible-container/container_yml/template.html
    • (supports many extensions to YAML, however it is an Orchestration tool, which is overkill if you just want YAML)
  • https://github.com/kblomqvist/yasha
  • https://bitbucket.org/djarvis/yamlp

Workarounds

  • Use YAML in conjunction with a template system, such as Jinja2 or Twig
  • Use a YAML extension library
  • Use sprintf or str.format style functionality from the hosting language

Alternatives

  • YTT YAML Templating essentially a fork of YAML with additional features that may be closer to the goal specified in the OP.
  • Jsonnet shares some similarity with YAML, but with additional features that may be closer to the goal specified in the OP.

See also

Here at SO

  • YAML variables in config files
  • Load YAML nested with Jinja2 in Python
  • String interpolation in YAML
  • how to reference a YAML "setting" from elsewhere in the same YAML file?
  • Use YAML with variables
  • How can I include a YAML file inside another?
  • Passing variables inside rails internationalization yml file
  • Can one YAML object refer to another?
  • is there a way to reference a constant in a yaml with rails?
  • YAML with nested Jinja
  • YAML merge keys
  • YAML merge keys

Outside SO

  • https://learnxinyminutes.com/docs/yaml/
  • https://github.com/dreftymac/awesome-yaml#variables
  • https://duckduckgo.com/?q=yaml+variables+in+config+file&t=h_&ia=web

Parse yaml list and save it to variables in python

I'm not quite certain what you want, but I think you're after something like this:

# test.yml
- app1:
name: example1
host: example1
tags:
- "user"
- "age"
- "gender"

- app2:
name: example2
host: example4
tags:
- "user"
- "age"
- "height"
import yaml
from pathlib import Path

with Path("test.yml").open() as f:
data = yaml.load(f)

out = []

for entry in data:
app, *_ = entry.keys()
tags = {k: k for k in entry[app]["tags"]}
out.append(tags)

print(out)

You could easily save these by the app name instead, by making out a dict.

Note that I don't know why you are after redundant key/value pairs, and if tags is actually supposed to contain data (unlike in your example) you will have to do something like:

tags = {k:v for k, v in entry[app]["tags"].items()}

Lastly you can of course dump back to yaml with yaml.dump(out).

** Unpacking

It occurs to me you might want to unpack this dict into a function. You can do so like this:

def my_fn(user=None, age=None, gender=None, height=None):
print("user", user, "age", age, "gender", gender, "height", height)

out = [{'user': 'user', 'age': 'age', 'gender': 'gender'},
{'user': 'user', 'age': 'age', 'height': 'height'}]

for row in out:
my_fn(**row)

Thus ** unpacks the dict into keyword arguments, which might be what you were looking for. (At any rate it makes more sense to me than generating variables with programmatic names). Note the app, *_ which unpacked entry.keys() so we could get the first item without having to cast it to an indexable type. dict.keys() isn't a generator, so you can't use next(dict.keys()). This particular unpacking is a recent (and very useful) addition to python.

EDIT: using these variables

yaml variable (e.g. host) will be populated with the values from Elasticsearch. If we define in tags "age", I'll search Elasticsearch for "age" and grab value from document, so it becomes "age": "20".

Right. So you want to see what entries are in tags, and then populate them with the right values. So you want to do something like this:

for entry in data:
tags = {}
app, *_ = entry.keys()
for tag in in entry[app]["tags"]:
val = get_tag_value_from_elasticsearch(tag)
tags[tag] = val

After that, this "age" tag will be called in sh script

So---still in the same loop for simplicity, but you can always append tags to a list and then iterate that list later:

    cmd = ["command"]
for param, val in tags.items():
cmd += [f"--{param}", str(val)]
res = subprocess.run(cmd, capture_output=True)
print(res.stdout)

Note a few things here: I've avoided the use of .get() to lookup a value in a dict when I can just use [] access; I've used subprocess.run instead of Popen (if you want to background you'll have to go back to Popen), and I've build the command up directly in a list, rather than building a string and splitting it.

At this point anything else you want to do with it is I think another question, so I'm going to leave this answer here.

How to use enviroment variables in Envoy

1 Option

You need to use envsubst, a tool that helps you put variables into envoy.yaml. You can do it this way

cat /tmpl/envoy.yaml.tmpl | envsubst \$ARG_1,\$ARG_2 > /etc/envoy.yaml

In the path tmpl/envoy.yaml.tmpl we save our temporary config where we prescribed where $ARG_1 and $ARG_2 will be used. Then we take these variables out of .env and rewrite $ARG_1 and $ARG_2 in the new config to their values. So our final config that we can run will be in /etc/envoy.yaml.

If you would like to learn more about envsubst, I recommend reading the following articles:

  • https://skofgar.ch/dev/2020/08/how-to-quickly-replace-environment-variables-in-a-file/
  • https://www.baeldung.com/linux/envsubst-command

2 Option

Also you can use jinja2 + python to render your template.j2 files to yaml. You can find more useful information in Google or read this article:
Generate yaml file with python and jinja2.

In Azure yaml, how to use local variables, conditional variables and template variables together?

How can I combine these 3?

In Pipeline YAML, variables can only be defined once at a stage or root level.

To meet your requirements, you need to define three type of variables in the same variables field.

Here is an example:

parameters:
- name: environment
displayName: Test
type: string
values:
- dev
- test
- prod

variables:
- name: buildPlatform
value: 'Any CPU'
- name: buildConfiguration
value: 'Release'
- template: variables.yml


- ${{ if eq(parameters.environment, 'dev') }}:
- name: environment
value: development
- ${{ if eq(parameters.environment, 'test') }}:
- name: environment
value: test
- ${{ if eq(parameters.environment, 'prod') }}:
- name: environment
value: prod

Passing bash script variables to .yml file for use as child and subdirectories

Does envsubst solve your problem?

For example, if I have a test-yaml.yml that contains $foo:

cat test-yaml.yml 

output:

general:
$foo: argument_from_bash_script
rawdatadir: '/some/data/directory/$foo'
input: '/my/input/directory/$foo/input'
output: '/my/output/directory/$foo/output'

You can replace $foo inside test-yaml.yml with shell variable $foo by envsubst:

export foo=123
envsubst < test-yaml.yml

output:

general:
123: argument_from_bash_script
rawdatadir: '/some/data/directory/123'
input: '/my/input/directory/123/input'
output: '/my/output/directory/123/output'


Related Topics



Leave a reply



Submit