Skip to content

cdm_data_loaders.utils.file_transfer.s3.transfer_config

Compute a safe boto3 TransferConfig for a given (or unknown) file size.

Kept free of any S3/HTTP I/O so the chunksize arithmetic can be unit tested directly against plain integers.

compute_multipart_chunksize 

compute_multipart_chunksize(
    file_size: int | None,
    requested_chunksize: int | None = None,
) -> int

Compute a multipart chunksize that will not exceed S3's 10,000-part limit.

If file_size would require more than S3_MAX_PARTS parts at requested_chunksize, the chunksize is scaled up just enough to bring the part count back within S3's limit; a warning is logged in that case, since it usually indicates an unusually large file. The result is always at least S3_MIN_PART_SIZE, since S3 rejects smaller multipart parts.

Parameters:

Name Type Description Default
file_size int | None

size of the file to be uploaded, in bytes, or None if unknown

 required
requested_chunksize int | None

chunksize to use as a starting point, in bytes, defaults to DEFAULT_MULTIPART_CHUNKSIZE if not given

 None

Returns:

Type Description
int

a chunksize, in bytes, safe to use as TransferConfig.multipart_chunksize for a file of file_size bytes
Source code in src/cdm_data_loaders/utils/file_transfer/s3/transfer_config.py 
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
def compute_multipart_chunksize(file_size: int | None, requested_chunksize: int | None = None) -> int:
    """Compute a multipart chunksize that will not exceed S3's 10,000-part limit.

    If `file_size` would require more than `S3_MAX_PARTS` parts at
    `requested_chunksize`, the chunksize is scaled up just enough to bring
    the part count back within S3's limit; a warning is logged in that case,
    since it usually indicates an unusually large file. The result is always
    at least `S3_MIN_PART_SIZE`, since S3 rejects smaller multipart parts.

    :param file_size: size of the file to be uploaded, in bytes, or None if unknown
    :type file_size: int | None
    :param requested_chunksize: chunksize to use as a starting point, in
        bytes, defaults to `DEFAULT_MULTIPART_CHUNKSIZE` if not given
    :type requested_chunksize: int | None, optional
    :return: a chunksize, in bytes, safe to use as `TransferConfig.multipart_chunksize`
        for a file of `file_size` bytes
    :rtype: int
    """
    chunksize = requested_chunksize or DEFAULT_MULTIPART_CHUNKSIZE

    if file_size is not None and file_size > 0:
        required_chunksize = math.ceil(file_size / S3_MAX_PARTS)
        if required_chunksize > chunksize:
            logger.warning(
                "File size (%d bytes) would require more than %d parts at chunksize=%d bytes; "
                "increasing multipart_chunksize to %d bytes to stay within S3's part limit",
                file_size,
                S3_MAX_PARTS,
                chunksize,
                required_chunksize,
            )
            chunksize = required_chunksize

    return max(chunksize, S3_MIN_PART_SIZE)

build_transfer_config 

build_transfer_config(
    file_size: int | None, **overrides: Any
) -> TransferConfig

Build a boto3.s3.transfer.TransferConfig sized appropriately for file_size.

Always computes a safe multipart_chunksize via compute_multipart_chunksize, using any explicitly requested multipart_chunksize in overrides as the starting point (and scaling it up if necessary), rather than silently ignoring the caller's preference.

Parameters:

Name Type Description Default
file_size int | None

size of the file to be uploaded, in bytes, or None if unknown ahead of time (e.g. no Content-Length header available)

 required
overrides Any

any other TransferConfig keyword arguments to pass through (e.g. multipart_threshold, max_concurrency); if multipart_chunksize is included, it is treated as a minimum rather than an absolute value

 {}

Returns:

Type Description
TransferConfig

a TransferConfig with a chunksize guaranteed not to exceed S3's 10,000-part limit for a file of file_size bytes
Source code in src/cdm_data_loaders/utils/file_transfer/s3/transfer_config.py 
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
def build_transfer_config(file_size: int | None, **overrides: Any) -> TransferConfig:
    """Build a `boto3.s3.transfer.TransferConfig` sized appropriately for `file_size`.

    Always computes a safe `multipart_chunksize` via `compute_multipart_chunksize`,
    using any explicitly requested `multipart_chunksize` in `overrides` as the
    starting point (and scaling it up if necessary), rather than silently
    ignoring the caller's preference.

    :param file_size: size of the file to be uploaded, in bytes, or None if
        unknown ahead of time (e.g. no Content-Length header available)
    :type file_size: int | None
    :param overrides: any other `TransferConfig` keyword arguments to pass
        through (e.g. `multipart_threshold`, `max_concurrency`); if
        `multipart_chunksize` is included, it is treated as a minimum rather
        than an absolute value
    :type overrides: Any
    :return: a TransferConfig with a chunksize guaranteed not to exceed
        S3's 10,000-part limit for a file of `file_size` bytes
    :rtype: TransferConfig
    """
    requested_chunksize = overrides.pop("multipart_chunksize", None)
    base_chunksize = requested_chunksize if file_size is not None else UNKNOWN_SIZE_CHUNKSIZE
    chunksize = compute_multipart_chunksize(file_size, base_chunksize)

    overrides.setdefault("multipart_threshold", DEFAULT_MULTIPART_THRESHOLD)

    return TransferConfig(multipart_chunksize=chunksize, **overrides)