Data Sync Model

Data models for data synchronization operations.

---

Data synchronization models.

Defines the request and response models for data sync operations between S3, EFS, and local file systems.

GetDataPathStatsRequest dataclass

GetDataPathStatsRequest(
    path=custom_field(mm_field=(DataPathField())),
)

Bases: WithDataPath

Request for getting statistics about a data path.

Attributes:

Name	Type	Description
path	DataPath	The data path to get statistics for.

GetDataPathStatsResponse dataclass

GetDataPathStatsResponse(
    path=custom_field(mm_field=(DataPathField())),
    path_stats=custom_field(
        mm_field=(PathStats.as_mm_field())
    ),
    children=custom_field(
        mm_field=(
            DictField(
                keys=(StringField()),
                values=(PathStats.as_mm_field()),
            )
        )
    ),
)

Bases: WithDataPath

Response containing data path statistics.

Attributes:

Name	Type	Description
path	DataPath	The data path.
path_stats	PathStats	Statistics for the path.
children	Dict[str, PathStats]	Statistics for child paths keyed by name.

ListDataPathsRequest dataclass

ListDataPathsRequest(
    path=custom_field(mm_field=(DataPathField())),
    include=custom_field(
        default=None,
        mm_field=(
            UnionField(
                [
                    (str, StringField()),
                    (list, ListField(StringField())),
                ]
            )
        ),
    ),
    exclude=custom_field(
        default=None,
        mm_field=(
            UnionField(
                [
                    (str, StringField()),
                    (list, ListField(StringField())),
                ]
            )
        ),
    ),
)

Bases: WithDataPath

Request for listing files under a data path.

Supports filtering with include/exclude regex patterns.

Attributes:

Name	Type	Description
path	DataPath	The data path under which to list files.
include	Optional[Union[str, List[str]]]	Optional regex pattern(s) for files to include. If multiple patterns, includes files matching any pattern.
exclude	Optional[Union[str, List[str]]]	Optional regex pattern(s) for files to exclude. Exclude patterns take precedence over include patterns.

ListDataPathsResponse dataclass

ListDataPathsResponse(
    paths=custom_field(
        default_factory=list,
        mm_field=(ListField(DataPathField())),
    )
)

Bases: SchemaModel

Response containing listed data paths.

Attributes:

Name	Type	Description
paths	List[DataPath]	List of data paths found.

OutdatedDataPathScannerRequest dataclass

OutdatedDataPathScannerRequest(
    path=custom_field(mm_field=(DataPathField())),
    days_since_last_accessed=custom_field(
        default=0, mm_field=(FloatField())
    ),
    max_depth=custom_field(
        default=None, mm_field=(IntegerField())
    ),
    min_depth=custom_field(
        default=None, mm_field=(IntegerField())
    ),
    min_size_bytes_allowed=custom_field(
        default=0, mm_field=(IntegerField())
    ),
    current_time=custom_field(
        default_factory=get_current_time,
        mm_field=(CustomAwareDateTime()),
    ),
)

Bases: WithDataPath

Request for scanning outdated data paths.

Scans for paths that haven't been accessed within a specified time.

Attributes:

Name	Type	Description
path	DataPath	The root path to scan.
days_since_last_accessed	float	Minimum days since last access to be outdated.
max_depth	Optional[int]	Maximum directory depth to scan.
min_depth	Optional[int]	Minimum directory depth to scan.
min_size_bytes_allowed	int	Minimum size threshold for paths to include.
current_time	datetime	Reference time for calculating age.

OutdatedDataPathScannerResponse dataclass

OutdatedDataPathScannerResponse(
    paths=custom_field(
        default_factory=list,
        mm_field=(ListField(DataPathField())),
    )
)

Bases: SchemaModel

Response containing outdated data paths.

Attributes:

Name	Type	Description
paths	List[DataPath]	List of paths identified as outdated.

RemoveDataPathsRequest dataclass

RemoveDataPathsRequest(
    paths=custom_field(
        default_factory=list,
        mm_field=(ListField(DataPathField())),
    )
)

Bases: SchemaModel

Request for removing data paths.

Attributes:

Name	Type	Description
paths	List[DataPath]	List of data paths to remove.

RemoveDataPathsResponse dataclass

RemoveDataPathsResponse(
    size_bytes_removed=custom_field(),
    paths_removed=custom_field(
        default_factory=list,
        mm_field=(ListField(DataPathField())),
    ),
)

Bases: SchemaModel

Response from removing data paths.

Attributes:

Name	Type	Description
size_bytes_removed	int	Total bytes removed.
paths_removed	List[DataPath]	List of paths that were removed.

WithDataPath dataclass

WithDataPath(path=custom_field(mm_field=(DataPathField())))

Bases: SchemaModel

Base class for models that contain a data path.

Provides convenience properties for accessing the path as different types.

Attributes:

Name	Type	Description
path	DataPath	The data path (S3, EFS, or local).

efs_path property

efs_path

Get the path as an EFS path if applicable.

Returns:

Type	Description
Optional[EFSPath]	The EFS path or None if not an EFS path.

local_path property

local_path

Get the path as a local path if applicable.

Returns:

Type	Description
Optional[Path]	The local path or None if not a local path.

s3_uri property

s3_uri

Get the path as an S3 URI if applicable.

Returns:

Type	Description
Optional[S3Path]	The S3 path or None if not an S3 path.