!import-all-parameterized tag

Constructor

ImportAllParameterizedConstructor dataclass

Custom PyYAML constructor for the !import-all-parameterized tag, which loads all files that match a glob pattern into the current document, including merging the named wildcards into each result.

As a Constructor, it can be called with a yaml.Loader and a yaml.Node to attempt to construct a given node tagged as !import-all-parameterized into a Python object. In any valid use of the tag, this node should always be a scalar string, and it should be in the form of a valid path pattern, optionally with named wildcards, e.g.:

my-data: !import-all-parameterized data/{file_name:*}.yml
more-data: !import-all-parameterized data/{sub_dirs:**}/info.yml

To standardize the parsing of the tag's argument, the Constructor uses an ImportAllParameterizedSpec dataclass to hold the path pattern object to be matched after it's been parsed from the string in the YAML document.

Methods:

Name	Description
__call__	Construct a node tagged as !import-all-parameterized into a Python object.
load	Using a specified loader type, load the contents of the files that match the pattern into a sequence of objects, including merging the named wildcards into each result.

Source code in yaml_extras/yaml_import.py

@dataclass
class ImportAllParameterizedConstructor:
    """Custom PyYAML constructor for the `!import-all-parameterized` tag, which loads all files that
    match a glob pattern into the current document, including merging the named wildcards into each
    result.

    As a Constructor, it can be called with a `yaml.Loader` and a `yaml.Node` to attempt to
    construct a given node tagged as `!import-all-parameterized` into a Python object. In any valid
    use of the tag, this node should always be a scalar string, and it should be in the form of a
    valid path pattern, optionally with named wildcards, e.g.:

    ```yaml
    my-data: !import-all-parameterized data/{file_name:*}.yml
    more-data: !import-all-parameterized data/{sub_dirs:**}/info.yml
    ```

    To standardize the parsing of the tag's argument, the Constructor uses an
    [`ImportAllParameterizedSpec`](./#yaml_extras.yaml_import.ImportAllParameterizedSpec) dataclass
    to hold the path pattern object to be matched after it's been parsed from the string in the YAML
    document.

    Methods:
        __call__: Construct a node tagged as `!import-all-parameterized` into a Python object.
        load: Using a specified loader type, load the contents of the files that match the pattern
            into a sequence of objects, including merging the named wildcards into each result.

    """

    def __call__(self, loader: yaml.Loader, node: yaml.Node) -> list[Any]:
        """Using the specified loader, attempt to construct a node tagged as
        `!import-all-parameterized` into a sequence of Python objects. For any valid use of the tag,
        the node should always be a scalar string, and it should be in the form of a valid path
        pattern, optionally with named wildcards.

        Args:
            loader (yaml.Loader): YAML loader
            node (yaml.Node): `!import-all-parameterized`-tagged node

        Returns:
            list[Any]: List of objects loaded from the files that match the pattern, including
                merging the named wildcards into each result.
        """
        import_spec: ImportAllParameterizedSpec
        if isinstance(node, yaml.ScalarNode):
            val = loader.construct_scalar(node)
            if isinstance(val, str):
                import_spec = ImportAllParameterizedSpec.from_str(val)
            else:
                raise TypeError(f"!import-all Expected a string, got {type(val)}")
        else:
            raise TypeError(f"!import-all Expected a string scalar, got {type(node)}")
        return self.load(type(loader), import_spec)

    def load(
        self, loader_type: Type[yaml.Loader], import_spec: ImportAllParameterizedSpec
    ) -> list[Any]:
        """Utility function which, using the specified loader type and the
        `ImportAllParameterizedSpec`, attempts to load the contents of the files that match the
        pattern into a sequence of objects, including merging the named wildcards into the results.

        Args:
            loader_type (Type[yaml.Loader]): YAML loader type
            import_spec (ImportAllParameterizedSpec): Dataclass containing the path pattern to be
                matched.

        Returns:
            list[Any]: List of objects loaded from the files that match the pattern, including
                merging the named wildcards into each result.
        """
        # Find and load all files that match the pattern into a sequence of objects, including
        # merging the named wildcards into the results.
        import_results: dict[PathWithMetadata, Any] = {
            path_w_metadata: yaml.load(path_w_metadata.path.open("r"), loader_type)
            for path_w_metadata in import_spec.path_pattern.results()
        }
        _to_object = lambda content: content if isinstance(content, dict) else {"content": content}
        return [
            _to_object(content) | (path_w_metadata.metadata or {})
            for path_w_metadata, content in import_results.items()
        ]

__call__(loader, node)

Using the specified loader, attempt to construct a node tagged as !import-all-parameterized into a sequence of Python objects. For any valid use of the tag, the node should always be a scalar string, and it should be in the form of a valid path pattern, optionally with named wildcards.

Parameters:

Name	Type	Description	Default
loader	Loader	YAML loader	required
node	Node	!import-all-parameterized-tagged node	required

Returns:

Type	Description
list[Any]	list[Any]: List of objects loaded from the files that match the pattern, including merging the named wildcards into each result.

Source code in yaml_extras/yaml_import.py

def __call__(self, loader: yaml.Loader, node: yaml.Node) -> list[Any]:
    """Using the specified loader, attempt to construct a node tagged as
    `!import-all-parameterized` into a sequence of Python objects. For any valid use of the tag,
    the node should always be a scalar string, and it should be in the form of a valid path
    pattern, optionally with named wildcards.

    Args:
        loader (yaml.Loader): YAML loader
        node (yaml.Node): `!import-all-parameterized`-tagged node

    Returns:
        list[Any]: List of objects loaded from the files that match the pattern, including
            merging the named wildcards into each result.
    """
    import_spec: ImportAllParameterizedSpec
    if isinstance(node, yaml.ScalarNode):
        val = loader.construct_scalar(node)
        if isinstance(val, str):
            import_spec = ImportAllParameterizedSpec.from_str(val)
        else:
            raise TypeError(f"!import-all Expected a string, got {type(val)}")
    else:
        raise TypeError(f"!import-all Expected a string scalar, got {type(node)}")
    return self.load(type(loader), import_spec)

load(loader_type, import_spec)

Utility function which, using the specified loader type and the ImportAllParameterizedSpec, attempts to load the contents of the files that match the pattern into a sequence of objects, including merging the named wildcards into the results.

Parameters:

Name	Type	Description	Default
loader_type	Type[Loader]	YAML loader type	required
import_spec	ImportAllParameterizedSpec	Dataclass containing the path pattern to be matched.	required

Returns:

Type	Description
list[Any]	list[Any]: List of objects loaded from the files that match the pattern, including merging the named wildcards into each result.

Source code in yaml_extras/yaml_import.py

def load(
    self, loader_type: Type[yaml.Loader], import_spec: ImportAllParameterizedSpec
) -> list[Any]:
    """Utility function which, using the specified loader type and the
    `ImportAllParameterizedSpec`, attempts to load the contents of the files that match the
    pattern into a sequence of objects, including merging the named wildcards into the results.

    Args:
        loader_type (Type[yaml.Loader]): YAML loader type
        import_spec (ImportAllParameterizedSpec): Dataclass containing the path pattern to be
            matched.

    Returns:
        list[Any]: List of objects loaded from the files that match the pattern, including
            merging the named wildcards into each result.
    """
    # Find and load all files that match the pattern into a sequence of objects, including
    # merging the named wildcards into the results.
    import_results: dict[PathWithMetadata, Any] = {
        path_w_metadata: yaml.load(path_w_metadata.path.open("r"), loader_type)
        for path_w_metadata in import_spec.path_pattern.results()
    }
    _to_object = lambda content: content if isinstance(content, dict) else {"content": content}
    return [
        _to_object(content) | (path_w_metadata.metadata or {})
        for path_w_metadata, content in import_results.items()
    ]

---

Utility dataclass

ImportAllParameterizedSpec dataclass

Small utility dataclass for typing the parsed argument to the !import-all-parameterized tag as a specialized PathPattern type. E.g.,

my-data: !import-all-parameterized data/{file_name:*}.yml

Shall be parsed as,

ImportAllParameterizedSpec(PathPattern("data/{file_name:*}.yml", ...))

Attributes:

Name	Type	Description
path_pattern	PathPattern	Pattern for matching files to be imported, optionally using named wildcards.

Methods:

Name	Description
from_str	Parse a string into an ImportAllParameterizedSpec dataclass.

Source code in yaml_extras/yaml_import.py

@dataclass
class ImportAllParameterizedSpec:
    """Small utility dataclass for typing the parsed argument to the `!import-all-parameterized` tag
    as a specialized `PathPattern` type. E.g.,

    ```yaml
    my-data: !import-all-parameterized data/{file_name:*}.yml
    ```

    Shall be parsed as,

    ```python
    ImportAllParameterizedSpec(PathPattern("data/{file_name:*}.yml", ...))
    ```

    Attributes:
        path_pattern (PathPattern): Pattern for matching files to be imported, optionally using
            named wildcards.

    Methods:
        from_str: Parse a string into an `ImportAllParameterizedSpec` dataclass.

    """

    path_pattern: PathPattern

    @classmethod
    def from_str(cls, path_pattern_str: str) -> "ImportAllParameterizedSpec":
        """Parse a string into an `ImportAllParameterizedSpec` dataclass. It is expected that the
        string contains a valid path pattern, optionally with named wildcards for extracting
        some metadata strings from the matched file paths.

        Args:
            path_pattern_str (str): String containing a path pattern to be parsed.

        Raises:
            ValueError: If the PathPattern fails to be parsed from the provided string.

        Returns:
            ImportAllParameterizedSpec: Dataclass containing the path pattern to be matched.
        """
        try:
            return cls(PathPattern(path_pattern_str, get_import_relative_dir()))
        except Exception as e:
            raise ValueError(f"Failed to form path pattern: {path_pattern_str}") from e

from_str(path_pattern_str) classmethod

Parse a string into an ImportAllParameterizedSpec dataclass. It is expected that the string contains a valid path pattern, optionally with named wildcards for extracting some metadata strings from the matched file paths.

Parameters:

Name	Type	Description	Default
path_pattern_str	str	String containing a path pattern to be parsed.	required

Raises:

Type	Description
ValueError	If the PathPattern fails to be parsed from the provided string.

Returns:

Name	Type	Description
ImportAllParameterizedSpec	ImportAllParameterizedSpec	Dataclass containing the path pattern to be matched.

Source code in yaml_extras/yaml_import.py

@classmethod
def from_str(cls, path_pattern_str: str) -> "ImportAllParameterizedSpec":
    """Parse a string into an `ImportAllParameterizedSpec` dataclass. It is expected that the
    string contains a valid path pattern, optionally with named wildcards for extracting
    some metadata strings from the matched file paths.

    Args:
        path_pattern_str (str): String containing a path pattern to be parsed.

    Raises:
        ValueError: If the PathPattern fails to be parsed from the provided string.

    Returns:
        ImportAllParameterizedSpec: Dataclass containing the path pattern to be matched.
    """
    try:
        return cls(PathPattern(path_pattern_str, get_import_relative_dir()))
    except Exception as e:
        raise ValueError(f"Failed to form path pattern: {path_pattern_str}") from e