Data Sync

Data synchronization models and utilities.

---

BatchDataSyncRequest

Bases: PydanticBaseModel

Request for a batch of data sync operations.

BatchDataSyncResponse

Bases: PydanticBaseModel

Response from a batch data sync operation.

add_failed_request

add_failed_request(request)

Add a request to the list of failed requests.

Source code in src/aibs_informatics_core/models/data_sync.py

def add_failed_request(self, request: DataSyncRequest) -> None:
    """Add a request to the list of failed requests."""
    if self.failed_requests is None:
        self.failed_requests = []
    self.failed_requests.append(request)

BatchDataSyncResult

Bases: DataSyncResult

Aggregated result metrics for a batch data sync.

increment_failed_requests_count

increment_failed_requests_count(increment=1)

Increment the failed and total request counters.

Source code in src/aibs_informatics_core/models/data_sync.py

def increment_failed_requests_count(self, increment: int = 1) -> None:
    """Increment the failed and total request counters."""
    self.failed_requests_count += increment
    self.total_requests_count += increment

increment_successful_requests_count

increment_successful_requests_count(increment=1)

Increment the successful and total request counters.

Source code in src/aibs_informatics_core/models/data_sync.py

def increment_successful_requests_count(self, increment: int = 1) -> None:
    """Increment the successful and total request counters."""
    self.successful_requests_count += increment
    self.total_requests_count += increment

DataSyncConfig

Bases: PydanticBaseModel

Configuration options for data sync operations.

Attributes:

Name	Type	Description
max_concurrency	int	Maximum number of concurrent transfer operations.
retain_source_data	bool	Whether to keep the source data after syncing.
delete	bool	Whether the sync deletes destination paths that are not present in the (filtered) source -- i.e. whether the destination is made to mirror the source rather than merely receive from it. .. warning:: This interacts destructively with :class:DataSyncFilterConfig. Filters narrow what the sync considers to be "the source", so with delete=True any file already at the destination that the filters exclude is treated as unexpected and deleted. Syncing a filtered subset into a directory that holds an earlier unfiltered copy will therefore remove the non-matching files. This is deliberately *not* guarded by validation -- ``delete`` is the gate, and mirroring remains a legitimate use of a filtered sync. Callers that pass a ``filter_config`` are expected to pass ``delete=False`` unless they specifically want the destination mirrored to the filtered subset.
require_lock	bool	Whether to acquire a lock on the destination path.
force	bool	Whether to transfer regardless of existing destination content.
size_only	bool	Whether to compare only file sizes when deciding to transfer.
fail_if_missing	bool	Whether to raise if the source path does not exist.
include_detailed_response	bool	Whether to compute detailed transfer metrics.
remote_to_local_config	RemoteToLocalConfig	Options specific to remote-to-local syncs.

DataSyncFilterConfig

Bases: PydanticBaseModel

Include/exclude filters restricting which files a data sync moves.

Patterns follow the shared contract in :mod:aibs_informatics_core.utils.filters: they are regular expressions (not globs) matched with fullmatch against the path relative to the filter root, and exclude patterns take precedence over include patterns. An absent or empty include includes everything.

Attributes:

Name	Type	Description
include	str | list[str] | None	Optional regex pattern(s) for files to include. If multiple patterns, includes files matching any pattern.
exclude	str | list[str] | None	Optional regex pattern(s) for files to exclude. Exclude patterns take precedence over include patterns.

from_patterns classmethod

from_patterns(include=None, exclude=None)

Build a config from raw patterns, or None when nothing is filtered.

The single definition of "are these filters actually filtering anything". Empty counts as absent -- None, "" and [] all yield None rather than a config that matches everything.

That distinction is load-bearing for callers that branch on whether filters are present. DemandExecutionParameters.sanitize_serialized_params serializes a resolvable one way when it has filters and another way when it does not, and the result feeds an execution hash -- so an empty-but-present config must not read as "filtered", or it would change the hash while filtering nothing.

Callers holding raw include/exclude should route through here rather than writing the emptiness check themselves; it has already been written two different ways in this codebase.

Parameters:

Name	Type	Description	Default
include	str | list[str] | None	Optional regex pattern(s) for files to include.	None
exclude	str | list[str] | None	Optional regex pattern(s) for files to exclude.	None

Returns:

Type	Description
DataSyncFilterConfig | None	A config, or None if neither argument carries a pattern.

Source code in src/aibs_informatics_core/models/data_sync.py

@classmethod
def from_patterns(
    cls,
    include: str | list[str] | None = None,
    exclude: str | list[str] | None = None,
) -> "DataSyncFilterConfig | None":
    """Build a config from raw patterns, or ``None`` when nothing is filtered.

    The single definition of "are these filters actually filtering anything". Empty
    counts as absent -- ``None``, ``""`` and ``[]`` all yield ``None`` rather than a
    config that matches everything.

    That distinction is load-bearing for callers that branch on whether filters are
    present. ``DemandExecutionParameters.sanitize_serialized_params`` serializes a
    resolvable one way when it has filters and another way when it does not, and the
    result feeds an execution hash -- so an empty-but-present config must not read as
    "filtered", or it would change the hash while filtering nothing.

    Callers holding raw ``include``/``exclude`` should route through here rather than
    writing the emptiness check themselves; it has already been written two different
    ways in this codebase.

    Args:
        include: Optional regex pattern(s) for files to include.
        exclude: Optional regex pattern(s) for files to exclude.

    Returns:
        A config, or ``None`` if neither argument carries a pattern.
    """
    if not include and not exclude:
        return None
    return cls(include=include, exclude=exclude)

DataSyncRequest

Bases: DataSyncConfig, DataSyncTask

Combined request model for a single data sync operation.

config property

config

Extract the configuration portion of this request.

Note

Fields are enumerated by hand -- any field added to :class:DataSyncConfig must be added here too, or it will be silently dropped.

task property

task

Extract the task portion of this request.

Note

Fields are enumerated by hand -- any field added to :class:DataSyncTask must be added here too, or it will be silently dropped.

DataSyncResponse

Bases: PydanticBaseModel

Response from a single data sync operation.

DataSyncResult

Bases: PydanticBaseModel

Result metrics for a data sync operation.

add_bytes_transferred

add_bytes_transferred(bytes_transferred)

Increment the bytes transferred counter.

Source code in src/aibs_informatics_core/models/data_sync.py

def add_bytes_transferred(self, bytes_transferred: int) -> None:
    """Increment the bytes transferred counter."""
    self.bytes_transferred += bytes_transferred

add_files_transferred

add_files_transferred(files_transferred)

Increment the files transferred counter.

Source code in src/aibs_informatics_core/models/data_sync.py

def add_files_transferred(self, files_transferred: int) -> None:
    """Increment the files transferred counter."""
    self.files_transferred += files_transferred

DataSyncTask

Bases: PydanticBaseModel

Defines source and destination paths for a data sync operation.

Attributes:

Name	Type	Description
source_path	S3Path | EFSPath | Path	Path to sync data from.
destination_path	S3Path | EFSPath | Path	Path to sync data to.
source_path_prefix	S3KeyPrefix | None	Optional S3 key prefix scoping the source.
filter_config	DataSyncFilterConfig | None	Optional include/exclude filters describing what to move. Filters live on the task rather than the config because they change the set of data transferred, not how the transfer runs.
filter_root	str | None	Root that filter patterns are matched relative to. Internal plumbing -- set by the prepare handler, never by users. The distributed sync workflow splits a sync of s3://b/run1/ into sub-requests rooted at s3://b/run1/sampleA/. Each sub-request re-lists from its own root, so patterns written against run1/ would silently stop matching. Sub-requests therefore carry the original root here. None means "not set" rather than a resolved default: this model applies no fallback, and consumers are expected to treat None as "anchor to source_path". Resolving it here instead would materialize the source path into the serialized task, which to_dict() currently omits while the field is None.

GetJSONFromFileRequest

Bases: JSONReference

Request to read JSON content from a file.

GetJSONFromFileResponse

Bases: JSONContent

Response containing JSON content read from a file.

JSONContent

Bases: PydanticBaseModel

Model containing raw JSON content.

JSONReference

Bases: PydanticBaseModel

Model containing a reference to a JSON file.

PrepareBatchDataSyncRequest

Bases: DataSyncRequest

Request to prepare a batch of data sync operations from a single sync task.

PrepareBatchDataSyncResponse

Bases: PydanticBaseModel

Response containing prepared batch data sync requests.

PutJSONToFileRequest

Bases: JSONContent

Request to write JSON content to a file.

PutJSONToFileResponse

Bases: JSONReference

Response from writing JSON content to a file.

RemoteToLocalConfig

Bases: PydanticBaseModel

Configuration for syncing remote data to local filesystem.