Can one YAML object refer to another?
Some yaml objects do refer to the others:
irb> require 'yaml'
#=> true
irb> str = "hello"
#=> "hello"
irb> hash = { :a => str, :b => str }
#=> {:a=>"hello", :b=>"hello"}
irb> puts YAML.dump(hash)
---
:a: hello
:b: hello
#=> nil
irb> puts YAML.dump([str,str])
---
- hello
- hello
#=> nil
irb> puts YAML.dump([hash,hash])
---
- &id001
:a: hello
:b: hello
- *id001
#=> nil
Note that it doesn't always reuse objects (the string is just repeated) but it does sometimes (the hash is defined once and reused by reference).
YAML doesn't support string interpolation - which is what you seem to be trying to do - but there's no reason you couldn't encode it a bit more verbosely:
intro: Hello, dear user
registration:
- "%s Thanks for registering!"
- intro
new_message:
- "%s You have a new message!"
- intro
Then you can interpolate it after you load the YAML:
strings = YAML::load(yaml_str)
interpolated = {}
strings.each do |key,val|
if val.kind_of? Array
fmt, *args = *val
val = fmt % args.map { |arg| strings[arg] }
end
interpolated[key] = val
end
And this will yield the following for interpolated:
{
"intro"=>"Hello, dear user",
"registration"=>"Hello, dear user Thanks for registering!",
"new_message"=>"Hello, dear user You have a new message!"
}
how to reference a YAML setting from elsewhere in the same YAML file?
I don't think it is possible. You can reuse "node" but not part of it.
bill-to: &id001
given : Chris
family : Dumars
ship-to: *id001
This is perfectly valid YAML and fields given and family are reused in ship-to block. You can reuse a scalar node the same way but there's no way you can change what's inside and add that last part of a path to it from inside YAML.
If repetition bother you that much I suggest to make your application aware of root property and add it to every path that looks relative not absolute.
How can I include a YAML file inside another?
No, standard YAML does not include any kind of "import" or "include" statement.
Include One Yaml File Inside Another
In YAML you cannot mix scalars, mapping keys and sequence elements. This is invalid YAML:
- abc
d: e
and so is this
some_file_name
a: b
and that you have that scalar quoted, and provide a tag does of course not change the fact
that it is invalid YAML.
As you can already found out, you can trick the loader into returning a dict instead of
the string (just like the parser already has built in constructors for non-primitive types like datetime.date).
That this:
!!python/object:foo.Bar
x: 1.0
y: 3.14
works is because the whole mapping is tagged, where you just tag a scalar value.
What also would be invalid syntax:
!include base.yaml
foo: 2
baz: baz
but you could do:
!include
filename: base.yaml
foo: 2
baz: baz
and process the 'filename' key in a special way, or make
the !include tag an empty key:
!include : base.yaml # : is a valid tag character, so you need the space
foo: 2
baz: baz
I would however look at using merge keys, as merging is essentially what
you are trying to do. The following YAML works:
import sys
import ruamel.yaml
from pathlib import Path
yaml_str = """
<<: {x: 42, y: 196, foo: 3}
foo: 2
baz: baz
"""
yaml = ruamel.yaml.YAML(typ='safe')
yaml.default_flow_style = False
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)
which gives:
baz: baz
foo: 2
x: 42
y: 196
So you should be able to do:
<<: !load base.yaml
foo: 2
baz: baz
and anyone with knowledge of merge keys would know what happens if base.yaml does include the key foo with value 3,
and would also understand:
<<: [!load base.yaml, !load config.yaml]
foo: 2
baz: baz
(As I tend to associate "including" with textual including as in the C preprocessor, I think `!load' might be a
more appropriate tag, but that is probably a matter of taste).
To get the merge keys to work, it is probably easiest to just sublass the Constructor, as merging is done before tag resolving:
import sys
import ruamel.yaml
from ruamel.yaml.nodes import MappingNode, SequenceNode, ScalarNode
from ruamel.yaml.constructor import ConstructorError
from ruamel.yaml.compat import _F
from pathlib import Path
class MyConstructor(ruamel.yaml.constructor.SafeConstructor):
def flatten_mapping(self, node):
# type: (Any) -> Any
"""
This implements the merge key feature http://yaml.org/type/merge.html
by inserting keys from the merge dict/list of dicts if not yet
available in this node
"""
merge = [] # type: List[Any]
index = 0
while index < len(node.value):
key_node, value_node = node.value[index]
if key_node.tag == 'tag:yaml.org,2002:merge':
if merge: # double << key
if self.allow_duplicate_keys:
del node.value[index]
index += 1
continue
args = [
'while constructing a mapping',
node.start_mark,
'found duplicate key "{}"'.format(key_node.value),
key_node.start_mark,
"""
To suppress this check see:
http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys
""",
"""\
Duplicate keys will become an error in future releases, and are errors
by default when using the new API.
""",
]
if self.allow_duplicate_keys is None:
warnings.warn(DuplicateKeyFutureWarning(*args))
else:
raise DuplicateKeyError(*args)
del node.value[index]
if isinstance(value_node, ScalarNode) and value_node.tag == '!load':
file_path = None
try:
if self.loader.reader.stream is not None:
file_path = Path(self.loader.reader.stream.name).parent / value_node.value
except AttributeError:
pass
if file_path is None:
file_path = Path(value_node.value)
# there is a bug in ruamel.yaml<=0.17.20 that prevents
# the use of a Path as argument to compose()
with file_path.open('rb') as fp:
merge.extend(ruamel.yaml.YAML().compose(fp).value)
elif isinstance(value_node, MappingNode):
self.flatten_mapping(value_node)
print('vn0', type(value_node.value), value_node.value)
merge.extend(value_node.value)
elif isinstance(value_node, SequenceNode):
submerge = []
for subnode in value_node.value:
if not isinstance(subnode, MappingNode):
raise ConstructorError(
'while constructing a mapping',
node.start_mark,
_F(
'expected a mapping for merging, but found {subnode_id!s}',
subnode_id=subnode.id,
),
subnode.start_mark,
)
self.flatten_mapping(subnode)
submerge.append(subnode.value)
submerge.reverse()
for value in submerge:
merge.extend(value)
else:
raise ConstructorError(
'while constructing a mapping',
node.start_mark,
_F(
'expected a mapping or list of mappings for merging, '
'but found {value_node_id!s}',
value_node_id=value_node.id,
),
value_node.start_mark,
)
elif key_node.tag == 'tag:yaml.org,2002:value':
key_node.tag = 'tag:yaml.org,2002:str'
index += 1
else:
index += 1
if bool(merge):
node.merge = merge # separate merge keys to be able to update without duplicate
node.value = merge + node.value
yaml = ruamel.yaml.YAML(typ='safe', pure=True)
yaml.default_flow_style = False
yaml.Constructor = MyConstructor
yaml_str = """\
<<: !load base.yaml
foo: 2
baz: baz
"""
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)
print('---')
file_name = Path('test.yaml')
file_name.write_text("""\
<<: !load base.yaml
bar: 2
baz: baz
""")
data = yaml.load(file_name)
yaml.dump(data, sys.stdout)
this prints:
bar:
- 2
- 3
baz: baz
foo: 2
---
bar: 2
baz: baz
foo: 1
Notes:
- don't open YAML files as text. They are written binary (UTF-8), and you should load them as such (
open(filename, 'rb')). - If you had included a full working program in your question (or at least included the text of
IncludeLoader, it
would have
been possible to provide a full working example with the merge keys (or find out for you that it
doesn't work for some reason) - as it is, it is unclear if your
yaml.load()is an instance method call (import ruamel.yaml; yaml = ruamel.yaml.YAML()) or calling a function (from ruamel import yaml). You should not use the latter as it is deprecated.
YAML / SnakeYAML: how refer to the same object multiple times
Using YAML you may represent objects of arbitrary graph-like structures. If you want to refer to the same object from different parts of a document, you need to use anchors and aliases
http://code.google.com/p/snakeyaml/wiki/Documentation#Aliases
Use placeholders in yaml
Context
- YAML version 1.2
- user wishes to
- include variable placeholders in YAML
- have placeholders replaced with computed values, upon
yaml.load - be able to use placeholders for both YAML mapping keys and values
Problem
- YAML does not natively support variable placeholders.
- Anchors and Aliases almost provide the desired functionality, but these do not work as variable placeholders that can be inserted into arbitrary regions throughout the YAML text. They must be placed as separate YAML nodes.
- There are some add-on libraries that support arbitrary variable placeholders, but they are not part of the native YAML specification.
Example
Consider the following example YAML. It is well-formed YAML syntax, however it uses (non-standard) curly-brace placeholders with embedded expressions.
The embedded expressions do not produce the desired result in YAML, because they are not part of the native YAML specification. Nevertheless, they are used in this example only to help illustrate what is available with standard YAML and what is not.
part01_customer_info:
cust_fname: "Homer"
cust_lname: "Himpson"
cust_motto: "I love donuts!"
cust_email: homer@himpson.org
part01_government_info:
govt_sales_taxrate: 1.15
part01_purchase_info:
prch_unit_label: "Bacon-Wrapped Fancy Glazed Donut"
prch_unit_price: 3.00
prch_unit_quant: 7
prch_product_cost: "{{prch_unit_price * prch_unit_quant}}"
prch_total_cost: "{{prch_product_cost * govt_sales_taxrate}}"
part02_shipping_info:
cust_fname: "{{cust_fname}}"
cust_lname: "{{cust_lname}}"
ship_city: Houston
ship_state: Hexas
part03_email_info:
cust_email: "{{cust_email}}"
mail_subject: Thanks for your DoughNutz order!
mail_notes: |
We want the mail_greeting to have all the expected values
with filled-in placeholders (and not curly-braces).
mail_greeting: |
Greetings {{cust_fname}} {{cust_lname}}!
We love your motto "{{cust_motto}}" and we agree with you!
Your total purchase price is {{prch_total_cost}}
Explanation
Below is an inline image that illustrates the example with colored regions in green, yellow and red.
The substitutions marked in GREEN are readily available in standard YAML, using anchors, aliases, and merge keys.
The substitutions marked in YELLOW are technically available in standard YAML, but not without a custom type declaration, or some other binding mechanism.
The substitutions marked in RED are not available in standard YAML. Yet there are workarounds and alternatives; such as through string formatting or string template engines (such as python's
str.format).
Details
Templates with variable placeholders is a frequently-requested YAML feature.
Routinely, developers want to cross-reference content in the same YAML file or transcluded YAML file(s).
YAML supports anchors and aliases, but this feature does not support arbitrary placement of placeholders and expressions anywhere in the YAML text. They only work with YAML nodes.
YAML also supports custom type declarations, however these are less common, and there are security implications if you accept YAML content from potentially untrusted sources.
YAML addon libraries
There are YAML extension libraries, but these are not part of the native YAML spec.
- Ansible
- https://docs.ansible.com/ansible-container/container_yml/template.html
- (supports many extensions to YAML, however it is an Orchestration tool, which is overkill if you just want YAML)
- https://github.com/kblomqvist/yasha
- https://bitbucket.org/djarvis/yamlp
Workarounds
- Use YAML in conjunction with a template system, such as Jinja2 or Twig
- Use a YAML extension library
- Use
sprintforstr.formatstyle functionality from the hosting language
Alternatives
- YTT YAML Templating essentially a fork of YAML with additional features that may be closer to the goal specified in the OP.
- Jsonnet shares some similarity with YAML, but with additional features that may be closer to the goal specified in the OP.
See also
Here at SO
- YAML variables in config files
- Load YAML nested with Jinja2 in Python
- String interpolation in YAML
- how to reference a YAML "setting" from elsewhere in the same YAML file?
- Use YAML with variables
- How can I include a YAML file inside another?
- Passing variables inside rails internationalization yml file
- Can one YAML object refer to another?
- is there a way to reference a constant in a yaml with rails?
- YAML with nested Jinja
- YAML merge keys
- YAML merge keys
Outside SO
- https://learnxinyminutes.com/docs/yaml/
- https://github.com/dreftymac/awesome-yaml#variables
- https://duckduckgo.com/?q=yaml+variables+in+config+file&t=h_&ia=web
How to reference an existing variable in YAML?
Here's a way of doing it the uses the di() function defined in the answer to another question. It takes the integer value returned from the built-in id() function and converts it to a string. The yaml.load() function will call a custom constructor which then does the reverse of that process to determine the object returned.
Caveat: This takes advantage of the fact that, with CPython at least, the id() function returns the address of the Python object in memory—so it may not work with other implementations of the interpreter.
import _ctypes
import yaml
def di(obj_id):
""" Reverse of id() function. """
return _ctypes.PyObj_FromPtr(obj_id)
def py_object_constructor(loader, node):
return di(int(node.value))
yaml.add_constructor(u'!py_object', py_object_constructor)
class Items(object): pass
def Ball(): return 42
ITEMS = Items()
setattr(Items, 'BALL', Ball()) # Set attribute to result of calling Ball().
yaml_text = "item1: !py_object " + str(id(ITEMS.BALL))
yaml_items = yaml.load(yaml_text)
print(yaml_items['item1']) # -> 42
If you're OK with using eval(), you could formalize this and make it easier to use by monkey-patching the yaml module's load() function to do some preprocessing of the yaml stream:
import _ctypes
import re
import yaml
#### Monkey-patch yaml module.
def _my_load(yaml_text, *args, **kwargs):
REGEX = r'@@(.+)@@'
match = re.search(REGEX, yaml_text)
if match:
obj = eval(match.group(1))
yaml_text = re.sub(REGEX, str(id(obj)), yaml_text)
return _yaml_load(yaml_text, *args, **kwargs)
_yaml_load = yaml.load # Save original function.
yaml.load = _my_load # Change it to custom version.
#### End monkey-patch yaml module.
def di(obj_id):
""" Reverse of id() function. """
return _ctypes.PyObj_FromPtr(obj_id)
def py_object_constructor(loader, node):
return di(int(node.value))
yaml.add_constructor(u'!py_object', py_object_constructor)
class Items(object): pass
def Ball(): return 42
ITEMS = Items()
setattr(Items, 'BALL', Ball()) # Set attribute to result of calling Ball().
yaml_text = "item1: !py_object @@ITEMS.BALL@@"
yaml_items = yaml.load(yaml_text)
print(yaml_items['item1']) # -> 42
Related Topics
Unpermitted Parameters for Dynamic Forms in Rails 4
Ruby: Could Not Find a Temporary Directory
Which Equality Test Does Ruby's Hash Use When Comparing Keys
Ruby on Rails - Doesn't Create Script/Server
How to Rescue from a Require "Gem_Name" When the Gem Is Not Installed
Eliminate Consecutive Duplicates of List Elements
Axlsx Merge Cells Inside a Style
Ruby Dot Parenthesis Call Syntax
How to Find If Range Is Contained in an Array of Ranges
How to Check Ruby Syntax Error in Ruby Code
How to Access the Current Node from a Library in a Chef Cookbook
How to Get the Current Test Filename from Rspec
Calendar Events Notification Not Received on Server
Rbenv Install Ruby Build Failed
How Does Activerecord Define Methods Compared to Attr_Accessor
How to Ensure Ruby Gems Are Installed in Right Place to Be Executed by Bundler