!import-all tag

Constructor

ImportAllConstructor dataclass

Custom PyYAML constructor for the !import-all tag, which loads all files that match a glob pattern into the current document.

As a Constructor, it can be called with a yaml.Loader and a yaml.Node to attempt to construct a given node tagged as !import-all into a Python object. In any valid use of the tag, this node should always be a scalar string, and it should be in the form of a path glob with no named wildcards, e.g.:

my-data: !import-all data/*.yml

To standardize the parsing of the tag's argument, the Constructor uses an ImportAllSpec dataclass to hold the path pattern object to be matched after it's been parsed from the string in the YAML document.

Source code in yaml_extras/yaml_import.py

@dataclass
class ImportAllConstructor:
    """Custom PyYAML constructor for the `!import-all` tag, which loads all files that match a
    glob pattern into the current document.

    As a Constructor, it can be called with a `yaml.Loader` and a `yaml.Node` to attempt to
    construct a given node tagged as `!import-all` into a Python object. In any valid use of the
    tag, this node should always be a scalar string, and it should be in the form of a path glob
    _with no named wildcards_, e.g.:

    ```yaml
    my-data: !import-all data/*.yml
    ```

    To standardize the parsing of the tag's argument, the Constructor uses an
    [`ImportAllSpec`](./#yaml_extras.yaml_import.ImportAllSpec) dataclass to hold the path pattern
    object to be matched after it's been parsed from the string in the YAML document.
    """

    def __call__(self, loader: yaml.Loader, node: yaml.Node) -> list[Any]:
        """Using the specified loader, attempt to construct a node tagged as `!import-all` into a
        sequence of Python objects.

        For any valid use of the tag, the node should always be a scalar string, and it should be in
        the form of a path glob _with no named wildcards_.

        Args:
            loader (yaml.Loader): YAML loader
            node (yaml.Node): `!import-all`-tagged node

        Returns:
            list[Any]: List of objects loaded from the files that match the pattern.
        """
        import_spec: ImportAllSpec
        if isinstance(node, yaml.ScalarNode):
            val = loader.construct_scalar(node)
            if isinstance(val, str):
                import_spec = ImportAllSpec.from_str(val)
            else:
                raise TypeError(f"!import-all Expected a string, got {type(val)}")
        else:
            raise TypeError(f"!import-all Expected a string scalar, got {type(node)}")
        return self.load(type(loader), import_spec)

    def load(self, loader_type: Type[yaml.Loader], import_spec: ImportAllSpec) -> list[Any]:
        """Utility function which, using the specified loader type and the `ImportAllSpec`, attempts
        to load all files that match the pattern into a list of objects (one per each file matched).

        Args:
            loader_type (Type[yaml.Loader]): YAML loader type
            import_spec (ImportAllSpec): Dataclass containing the path pattern to be matched.

        Returns:
            list[Any]: List of objects loaded from the files that match the pattern.
        """
        # Find and load all files that match the pattern into a sequence of objects
        return [
            yaml.load(path_w_metadata.path.open("r"), loader_type)
            for path_w_metadata in import_spec.path_pattern.results()
        ]

__call__(loader, node)

Using the specified loader, attempt to construct a node tagged as !import-all into a sequence of Python objects.

For any valid use of the tag, the node should always be a scalar string, and it should be in the form of a path glob with no named wildcards.

Parameters:

Name	Type	Description	Default
loader	Loader	YAML loader	required
node	Node	!import-all-tagged node	required

Returns:

Type	Description
list[Any]	list[Any]: List of objects loaded from the files that match the pattern.

Source code in yaml_extras/yaml_import.py

def __call__(self, loader: yaml.Loader, node: yaml.Node) -> list[Any]:
    """Using the specified loader, attempt to construct a node tagged as `!import-all` into a
    sequence of Python objects.

    For any valid use of the tag, the node should always be a scalar string, and it should be in
    the form of a path glob _with no named wildcards_.

    Args:
        loader (yaml.Loader): YAML loader
        node (yaml.Node): `!import-all`-tagged node

    Returns:
        list[Any]: List of objects loaded from the files that match the pattern.
    """
    import_spec: ImportAllSpec
    if isinstance(node, yaml.ScalarNode):
        val = loader.construct_scalar(node)
        if isinstance(val, str):
            import_spec = ImportAllSpec.from_str(val)
        else:
            raise TypeError(f"!import-all Expected a string, got {type(val)}")
    else:
        raise TypeError(f"!import-all Expected a string scalar, got {type(node)}")
    return self.load(type(loader), import_spec)

load(loader_type, import_spec)

Utility function which, using the specified loader type and the ImportAllSpec, attempts to load all files that match the pattern into a list of objects (one per each file matched).

Parameters:

Name	Type	Description	Default
loader_type	Type[Loader]	YAML loader type	required
import_spec	ImportAllSpec	Dataclass containing the path pattern to be matched.	required

Returns:

Type	Description
list[Any]	list[Any]: List of objects loaded from the files that match the pattern.

Source code in yaml_extras/yaml_import.py

def load(self, loader_type: Type[yaml.Loader], import_spec: ImportAllSpec) -> list[Any]:
    """Utility function which, using the specified loader type and the `ImportAllSpec`, attempts
    to load all files that match the pattern into a list of objects (one per each file matched).

    Args:
        loader_type (Type[yaml.Loader]): YAML loader type
        import_spec (ImportAllSpec): Dataclass containing the path pattern to be matched.

    Returns:
        list[Any]: List of objects loaded from the files that match the pattern.
    """
    # Find and load all files that match the pattern into a sequence of objects
    return [
        yaml.load(path_w_metadata.path.open("r"), loader_type)
        for path_w_metadata in import_spec.path_pattern.results()
    ]

---

Utility dataclass

ImportAllSpec dataclass

Small utility dataclass for typing the parsed argument to the !import-all tag as a specialized PathPattern type. E.g.,

my-data: !import-all data/*.yml

Shall be parsed as,

ImportAllSpec(PathPattern("data/*.yml", ...))

Attributes:

Name	Type	Description
path_pattern	PathPattern	Pattern for matching files to be imported.

Methods:

Name	Description
from_str	Parse a string into an ImportAllSpec dataclass.

Source code in yaml_extras/yaml_import.py

@dataclass
class ImportAllSpec:
    """Small utility dataclass for typing the parsed argument to the `!import-all` tag as a
    specialized `PathPattern` type. E.g.,

    ```yaml
    my-data: !import-all data/*.yml
    ```

    Shall be parsed as,

    ```python
    ImportAllSpec(PathPattern("data/*.yml", ...))
    ```

    Attributes:
        path_pattern (PathPattern): Pattern for matching files to be imported.

    Methods:
        from_str: Parse a string into an `ImportAllSpec` dataclass.
    """

    path_pattern: PathPattern

    @classmethod
    def from_str(cls, path_pattern_str: str) -> "ImportAllSpec":
        """Parse a string into an `ImportAllSpec` dataclass. It is expected that the string is a
        valid path pattern with no named wildcards.

        Args:
            path_pattern_str (str): String containing a path pattern to be parsed.

        Raises:
            ValueError: If the user supplies named wildcards in the path pattern, which should only
                be used in `!import-all-parameterized`.

        Returns:
            ImportAllSpec: Dataclass containing the path pattern to be matched.
        """
        path_pattern = PathPattern(path_pattern_str, get_import_relative_dir())
        if path_pattern.names != []:
            raise ValueError(
                "Named wildcards are not supported in !import-all. Use !import-all-parameterized "
                "instead."
            )
        return cls(PathPattern(path_pattern_str, get_import_relative_dir()))

from_str(path_pattern_str) classmethod

Parse a string into an ImportAllSpec dataclass. It is expected that the string is a valid path pattern with no named wildcards.

Parameters:

Name	Type	Description	Default
path_pattern_str	str	String containing a path pattern to be parsed.	required

Raises:

Type	Description
ValueError	If the user supplies named wildcards in the path pattern, which should only be used in !import-all-parameterized.

Returns:

Name	Type	Description
ImportAllSpec	ImportAllSpec	Dataclass containing the path pattern to be matched.

Source code in yaml_extras/yaml_import.py

@classmethod
def from_str(cls, path_pattern_str: str) -> "ImportAllSpec":
    """Parse a string into an `ImportAllSpec` dataclass. It is expected that the string is a
    valid path pattern with no named wildcards.

    Args:
        path_pattern_str (str): String containing a path pattern to be parsed.

    Raises:
        ValueError: If the user supplies named wildcards in the path pattern, which should only
            be used in `!import-all-parameterized`.

    Returns:
        ImportAllSpec: Dataclass containing the path pattern to be matched.
    """
    path_pattern = PathPattern(path_pattern_str, get_import_relative_dir())
    if path_pattern.names != []:
        raise ValueError(
            "Named wildcards are not supported in !import-all. Use !import-all-parameterized "
            "instead."
        )
    return cls(PathPattern(path_pattern_str, get_import_relative_dir()))