Data Sync Model¶
Data models for data synchronization operations.
Data synchronization models.
Defines the request and response models for data sync operations between S3, EFS, and local file systems.
GetDataPathStatsRequest ¶
Bases: WithDataPath
Request for getting statistics about a data path.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
DataPath
|
The data path to get statistics for. |
GetDataPathStatsResponse ¶
Bases: WithDataPath
Response containing data path statistics.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
DataPath
|
The data path. |
path_stats |
PathStats
|
Statistics for the path. |
children |
dict[str, PathStats]
|
Statistics for child paths keyed by name. |
ListDataPathsRequest ¶
Bases: WithDataPath
Request for listing files under a data path.
Supports filtering with include/exclude regex patterns.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
DataPath
|
The data path under which to list files. |
include |
str | list[str] | None
|
Optional regex pattern(s) for files to include. If multiple patterns, includes files matching any pattern. |
exclude |
str | list[str] | None
|
Optional regex pattern(s) for files to exclude. Exclude patterns take precedence over include patterns. |
ListDataPathsResponse ¶
Bases: PydanticBaseModel
Response containing listed data paths.
Attributes:
| Name | Type | Description |
|---|---|---|
paths |
list[DataPath]
|
List of data paths found. |
OutdatedDataPathScannerRequest ¶
Bases: WithDataPath
Request for scanning outdated data paths.
Scans for paths that haven't been accessed within a specified time.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
DataPath
|
The root path to scan. |
days_since_last_accessed |
float
|
Minimum days since last access to be outdated. |
max_depth |
int | None
|
Maximum directory depth to scan. |
min_depth |
int | None
|
Minimum directory depth to scan. |
min_size_bytes_allowed |
int
|
Minimum size threshold for paths to include. |
current_time |
datetime
|
Reference time for calculating age. |
OutdatedDataPathScannerResponse ¶
Bases: PydanticBaseModel
Response containing outdated data paths.
Attributes:
| Name | Type | Description |
|---|---|---|
paths |
list[DataPath]
|
List of paths identified as outdated. |
RemoveDataPathsRequest ¶
Bases: PydanticBaseModel
Request for removing data paths.
Attributes:
| Name | Type | Description |
|---|---|---|
paths |
list[DataPath]
|
List of data paths to remove. |
RemoveDataPathsResponse ¶
Bases: PydanticBaseModel
Response from removing data paths.
Attributes:
| Name | Type | Description |
|---|---|---|
size_bytes_removed |
int
|
Total bytes removed. |
paths_removed |
list[DataPath]
|
List of paths that were removed. |
WithDataPath ¶
Bases: PydanticBaseModel
Base class for models that contain a data path.
Provides convenience properties for accessing the path as different types.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
DataPath
|
The data path (S3, EFS, or local). |
efs_path
property
¶
efs_path
Get the path as an EFS path if applicable.
Returns:
| Type | Description |
|---|---|
EFSPath | None
|
The EFS path or None if not an EFS path. |
local_path
property
¶
local_path
Get the path as a local path if applicable.
Returns:
| Type | Description |
|---|---|
Path | None
|
The local path or None if not a local path. |
s3_path
property
¶
s3_path
Get the path as an S3 path if applicable.
Returns:
| Type | Description |
|---|---|
S3Path | None
|
The S3 path or None if not an S3 path. |