Use placeholders in yaml
Context
- YAML version 1.2
- user wishes to
- include variable placeholders in YAML
- have placeholders replaced with computed values, upon
yaml.load
- be able to use placeholders for both YAML mapping keys and values
Problem
- YAML does not natively support variable placeholders.
- Anchors and Aliases almost provide the desired functionality, but these do not work as variable placeholders that can be inserted into arbitrary regions throughout the YAML text. They must be placed as separate YAML nodes.
- There are some add-on libraries that support arbitrary variable placeholders, but they are not part of the native YAML specification.
Example
Consider the following example YAML. It is well-formed YAML syntax, however it uses (non-standard) curly-brace placeholders with embedded expressions.
The embedded expressions do not produce the desired result in YAML, because they are not part of the native YAML specification. Nevertheless, they are used in this example only to help illustrate what is available with standard YAML and what is not.
part01_customer_info:
cust_fname: "Homer"
cust_lname: "Himpson"
cust_motto: "I love donuts!"
cust_email: homer@himpson.org
part01_government_info:
govt_sales_taxrate: 1.15
part01_purchase_info:
prch_unit_label: "Bacon-Wrapped Fancy Glazed Donut"
prch_unit_price: 3.00
prch_unit_quant: 7
prch_product_cost: "{{prch_unit_price * prch_unit_quant}}"
prch_total_cost: "{{prch_product_cost * govt_sales_taxrate}}"
part02_shipping_info:
cust_fname: "{{cust_fname}}"
cust_lname: "{{cust_lname}}"
ship_city: Houston
ship_state: Hexas
part03_email_info:
cust_email: "{{cust_email}}"
mail_subject: Thanks for your DoughNutz order!
mail_notes: |
We want the mail_greeting to have all the expected values
with filled-in placeholders (and not curly-braces).
mail_greeting: |
Greetings {{cust_fname}} {{cust_lname}}!
We love your motto "{{cust_motto}}" and we agree with you!
Your total purchase price is {{prch_total_cost}}
Explanation
Below is an inline image that illustrates the example with colored regions in green, yellow and red.
The substitutions marked in GREEN are readily available in standard YAML, using anchors, aliases, and merge keys.
The substitutions marked in YELLOW are technically available in standard YAML, but not without a custom type declaration, or some other binding mechanism.
The substitutions marked in RED are not available in standard YAML. Yet there are workarounds and alternatives; such as through string formatting or string template engines (such as python's
str.format
).
Details
A frequently-requested feature for YAML is the ability to insert arbitrary variable placeholders that support arbitrary cross-references and expressions that relate to the other content in the same (or transcluded) YAML file(s).
YAML supports anchors and aliases, but this feature does not support arbitrary placement of placeholders and expressions anywhere in the YAML text. They only work with YAML nodes.
YAML also supports custom type declarations, however these are less common, and there are security implications if you accept YAML content from potentially untrusted sources.
YAML addon libraries
There are YAML extension libraries, but these are not part of the native YAML spec.
- Ansible
- https://docs.ansible.com/ansible-container/container_yml/template.html
- (supports many extensions to YAML, however it is an Orchestration tool, which is overkill if you just want YAML)
- https://github.com/kblomqvist/yasha
- https://bitbucket.org/djarvis/yamlp
Workarounds
- Use YAML in conjunction with a template system, such as Jinja2 or Twig
- Use a YAML extension library
- Use
sprintf
orstr.format
style functionality from the hosting language
Alternatives
- YTT YAML Templating essentially a fork of YAML with additional features that may be closer to the goal specified in the OP.
- Jsonnet shares some similarity with YAML, but with additional features that may be closer to the goal specified in the OP.
See also
Here at SO
- YAML variables in config files
- Load YAML nested with Jinja2 in Python
- String interpolation in YAML
- how to reference a YAML "setting" from elsewhere in the same YAML file?
- Use YAML with variables
- How can I include a YAML file inside another?
- Passing variables inside rails internationalization yml file
- Can one YAML object refer to another?
- is there a way to reference a constant in a yaml with rails?
- YAML with nested Jinja
- YAML merge keys
- YAML merge keys
Outside SO
- https://learnxinyminutes.com/docs/yaml/
- https://github.com/dreftymac/awesome-yaml#variables
- https://duckduckgo.com/?q=yaml+variables+in+config+file&t=h_&ia=web
Parse yaml list and save it to variables in python
I'm not quite certain what you want, but I think you're after something like this:
# test.yml
- app1:
name: example1
host: example1
tags:
- "user"
- "age"
- "gender"
- app2:
name: example2
host: example4
tags:
- "user"
- "age"
- "height"
import yaml
from pathlib import Path
with Path("test.yml").open() as f:
data = yaml.load(f)
out = []
for entry in data:
app, *_ = entry.keys()
tags = {k: k for k in entry[app]["tags"]}
out.append(tags)
print(out)
You could easily save these by the app name instead, by making out a dict.
Note that I don't know why you are after redundant key/value pairs, and if tags
is actually supposed to contain data (unlike in your example) you will have to do something like:
tags = {k:v for k, v in entry[app]["tags"].items()}
Lastly you can of course dump back to yaml with yaml.dump(out)
.
** Unpacking
It occurs to me you might want to unpack this dict into a function. You can do so like this:
def my_fn(user=None, age=None, gender=None, height=None):
print("user", user, "age", age, "gender", gender, "height", height)
out = [{'user': 'user', 'age': 'age', 'gender': 'gender'},
{'user': 'user', 'age': 'age', 'height': 'height'}]
for row in out:
my_fn(**row)
Thus ** unpacks the dict into keyword arguments, which might be what you were looking for. (At any rate it makes more sense to me than generating variables with programmatic names). Note the app, *_
which unpacked entry.keys()
so we could get the first item without having to cast it to an indexable type. dict.keys()
isn't a generator, so you can't use next(dict.keys())
. This particular unpacking is a recent (and very useful) addition to python.
EDIT: using these variables
yaml variable (e.g. host) will be populated with the values from Elasticsearch. If we define in tags "age", I'll search Elasticsearch for "age" and grab value from document, so it becomes "age": "20".
Right. So you want to see what entries are in tags
, and then populate them with the right values. So you want to do something like this:
for entry in data:
tags = {}
app, *_ = entry.keys()
for tag in in entry[app]["tags"]:
val = get_tag_value_from_elasticsearch(tag)
tags[tag] = val
After that, this "age" tag will be called in sh script
So---still in the same loop for simplicity, but you can always append tags
to a list and then iterate that list later:
cmd = ["command"]
for param, val in tags.items():
cmd += [f"--{param}", str(val)]
res = subprocess.run(cmd, capture_output=True)
print(res.stdout)
Note a few things here: I've avoided the use of .get()
to lookup a value in a dict when I can just use []
access; I've used subprocess.run
instead of Popen (if you want to background you'll have to go back to Popen), and I've build the command up directly in a list, rather than building a string and splitting it.
At this point anything else you want to do with it is I think another question, so I'm going to leave this answer here.
How to use enviroment variables in Envoy
1 Option
You need to use envsubst, a tool that helps you put variables into envoy.yaml. You can do it this way
cat /tmpl/envoy.yaml.tmpl | envsubst \$ARG_1,\$ARG_2 > /etc/envoy.yaml
In the path tmpl/envoy.yaml.tmpl we save our temporary config where we prescribed where $ARG_1 and $ARG_2 will be used. Then we take these variables out of .env and rewrite $ARG_1 and $ARG_2 in the new config to their values. So our final config that we can run will be in /etc/envoy.yaml.
If you would like to learn more about envsubst, I recommend reading the following articles:
- https://skofgar.ch/dev/2020/08/how-to-quickly-replace-environment-variables-in-a-file/
- https://www.baeldung.com/linux/envsubst-command
2 Option
Also you can use jinja2 + python to render your template.j2 files to yaml. You can find more useful information in Google or read this article:
Generate yaml file with python and jinja2.
In Azure yaml, how to use local variables, conditional variables and template variables together?
How can I combine these 3?
In Pipeline YAML, variables can only be defined once at a stage or root level.
To meet your requirements, you need to define three type of variables in the same variables field.
Here is an example:
parameters:
- name: environment
displayName: Test
type: string
values:
- dev
- test
- prod
variables:
- name: buildPlatform
value: 'Any CPU'
- name: buildConfiguration
value: 'Release'
- template: variables.yml
- ${{ if eq(parameters.environment, 'dev') }}:
- name: environment
value: development
- ${{ if eq(parameters.environment, 'test') }}:
- name: environment
value: test
- ${{ if eq(parameters.environment, 'prod') }}:
- name: environment
value: prod
Passing bash script variables to .yml file for use as child and subdirectories
Does envsubst
solve your problem?
For example, if I have a test-yaml.yml
that contains $foo
:
cat test-yaml.yml
output:
general:
$foo: argument_from_bash_script
rawdatadir: '/some/data/directory/$foo'
input: '/my/input/directory/$foo/input'
output: '/my/output/directory/$foo/output'
You can replace $foo
inside test-yaml.yml
with shell variable $foo
by envsubst
:
export foo=123
envsubst < test-yaml.yml
output:
general:
123: argument_from_bash_script
rawdatadir: '/some/data/directory/123'
input: '/my/input/directory/123/input'
output: '/my/output/directory/123/output'
Related Topics
How to Get Argument Names Using Reflection
Using Sinatra For Larger Projects Via Multiple Files
Ruby Function to Remove All White Spaces
Save Image from Url by Paperclip
How to Split (Chunk) a Ruby Array into Parts of X Elements
How Does Instance_Eval Work and Why Does Dhh Hate It
Ruby: Kind_Of? Vs. Instance_Of? Vs. Is_A
Why Does Installing Nokogiri on MAC Os Fail With Libiconv Is Missing
Continuously Read from Stdout of External Process in Ruby
How to Create an Average from a Ruby Array
Rails: Access to Current_User from Within a Model in Ruby on Rails
Ruby: Inherit Code That Works With Class Variables
Avoiding Applescript Through Ruby: Rb-Appscript or Rubyosa