OXIESEC PANEL
- Current Dir:
/
/
opt
/
gsutil
/
gslib
/
utils
/
__pycache__
Server IP: 2a02:4780:11:1594:0:ef5:22d7:a
Upload:
Create Dir:
Name
Size
Modified
Perms
📁
..
-
02/11/2025 08:19:48 AM
rwxr-xr-x
📄
__init__.cpython-39.pyc
126 bytes
02/11/2025 08:19:48 AM
rw-r--r--
📄
acl_helper.cpython-39.pyc
9.38 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
arg_helper.cpython-39.pyc
1.01 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
boto_util.cpython-39.pyc
16.46 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
cat_helper.cpython-39.pyc
4.21 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
cloud_api_helper.cpython-39.pyc
3.29 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
constants.cpython-39.pyc
2.88 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
copy_helper.cpython-39.pyc
101.64 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
encryption_helper.cpython-39.pyc
5.84 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
execution_util.cpython-39.pyc
1.06 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
hashing_helper.cpython-39.pyc
15.71 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
iam_helper.cpython-39.pyc
8.61 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
ls_helper.cpython-39.pyc
12.4 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
metadata_util.cpython-39.pyc
3.1 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
parallelism_framework_util.cpython-39.pyc
11.79 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
posix_util.cpython-39.pyc
12.46 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
retention_util.cpython-39.pyc
8.77 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
retry_util.cpython-39.pyc
2.5 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rsync_util.cpython-39.pyc
1.6 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
shim_util.cpython-39.pyc
19.8 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
signurl_helper.cpython-39.pyc
4.21 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
stet_util.cpython-39.pyc
3.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
system_util.cpython-39.pyc
7.9 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
temporary_file_util.cpython-39.pyc
915 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
text_util.cpython-39.pyc
12.62 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
translation_helper.cpython-39.pyc
25.67 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
unit_util.cpython-39.pyc
7.01 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
update_util.cpython-39.pyc
3.39 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
user_agent_helper.cpython-39.pyc
1.49 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
version_check.cpython-39.pyc
1.39 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
wrapped_credentials.cpython-39.pyc
5.81 KB
02/11/2025 08:19:49 AM
rw-r--r--
Editing: hashing_helper.cpython-39.pyc
Close
a +(WgHI � @ sv d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl m Z ddl Z ddlmZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ dZdZdZdZdZdZ dZ dZ dZ g d�Z dZ!dZ"dd� Z#d d � Z$d d � Z%d d!� Z&d:d"d#�Z'd$d%� Z(d&d'� Z)d(d)� Z*d*d+� Z+d,d-� Z,d.d/� Z-d0d1� Z.d;d3d4�Z/d<d6d7�Z0G d8d9� d9e1�Z2dS )=z+Helper functions for hashing functionality.� )�absolute_import)�print_function)�division)�unicode_literalsN)�config)�CommandException)�UsingCrcmodExtension)�DEFAULT_FILE_BUFFER_SIZE)�MIN_SIZE_COMPUTE_LOGGING)�TRANSFER_BUFFER_SIZE)�UTF8z� WARNING: You have requested checksumming but your crcmod installation isn't using the module's C extension, so checksumming will run very slowly. For help installing the extension, please see "gsutil help crcmod". a� WARNING: gsutil rsync uses hashes when modification time is not available at both the source and destination. Your crcmod installation isn't using the module's C extension, so checksumming will run very slowly. If this is your first rsync since updating gsutil, this rsync can take significantly longer than usual. For help installing the extension, please see "gsutil help crcmod". a| WARNING: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod". To disable slow integrity checking, see the "check_hashes" option in your boto config file. a: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod". To download regardless of crcmod performance or to skip slow integrity checks, see the "check_hashes" option in your boto config file. NOTE: It is strongly recommended that you not disable integrity checks. Doing so could allow data corruption to go undetected during uploading/downloading.aT WARNING: This download will not be validated since your crcmod installation doesn't use the module's C extension, so the hash computation would likely throttle download performance. For help in installing the extension, please see "gsutil help crcmod". To force integrity checking, see the "check_hashes" option in your boto config file. �if_fast_else_fail�if_fast_else_skip�always�never) � � � � i iAo� ivE�:i Wi�P�Yl T/U l ?O i��Ni.�%i'��hi��nTi��\Fib�&l �"�, iGWL;i�gl �b ! l <�c l �!� l �N,Q l �2�! l =X� i��D3iK�S}i�2�=l �m�y i�x�Bl Ao�= � c C s |s| S t | d| �|A S )a� Computes CRC32C for concat(A, B) given crc(A), crc(B) and len(B). An explanation of the algorithm can be found at crcutil.googlecode.com/files/crc-doc.1.0.pdf. Args: crc_a: A 32-bit integer representing crc(A) with least-significant coefficient first. crc_b: Same as crc_a. num_bytes_in_b: Length of B in bytes. Returns: CRC32C for concat(A, B) � )�_ExtendByZeros)Zcrc_aZcrc_bZnum_bytes_in_b� r �)/opt/gsutil/gslib/utils/hashing_helper.py� ConcatCrc32cj s r c C sN d}dt > }tt �D ]4}| d@ r(||N }|dK }||@ r@|tN }| dL } q|S )z�Multiplies two polynomials together modulo CASTAGNOLI_POLY. Args: p: The first polynomial. q: The second polynomial. Returns: Result of the multiplication. r � )�DEGREE�range�CASTAGNOLI_POLY)�p�q�resultZtop_bit�_r r r � _CrcMultiply s r# c C sX dd� }|| �} d}|dkrL|d@ r:t | t|tt� �} |d7 }|dL }q|| �} | S )z�Given crc representing polynomial P(x), compute P(x)*x^num_bits. Args: crc: crc respresenting polynomial P(x). num_bits: number of bits in crc. Returns: P(x)*x^num_bits c S s t dj| dd�d d d� d�S )Nz{0:032b}r )�width���r )�int�format)�crcr r r �_ReverseBits32� s z&_ExtendByZeros.<locals>._ReverseBits32r r )r# �X_POW_2K_TABLE�len)r( �num_bitsr) �ir r r r � s r c C s2 d|i}| � d� t| |� | � d� |d �� S )a& Calculates a base64 digest of the contents of a seekable stream. This function resets the file pointer to position 0. Args: fp: An already-open file object. hash_alg: Instance of hashing class initialized to start state. Returns: Hash of the stream in hex string format. � placeholderr )�seek�CalculateHashesFromContents� hexdigest)�fp�hash_alg� hash_dictr r r �_CalculateHashFromContents� s r5 c C s\ | � t�}|sqXtjr*t|t�r*|�t�}t�|�D ]}|� |� q4|r |� t |�� q dS )a} Calculates hashes of the contents of a file. Args: fp: An already-open file object (stream will be consumed). hash_dict: Dict of (string alg_name: initialized hashing class) Hashing class will be populated with digests upon return. callback_processor: Optional callback processing class that implements Progress(integer amount of bytes processed). N) �readr �six�PY3� isinstance�str�encoder � itervalues�updateZProgressr+ )r2 r4 Zcallback_processor�datar3 r r r r0 � s r0 c C s t | tj�d��S )a Calculates a base64 CRC32c checksum of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: CRC32c checksum of the file in base64 format. �crc-32c)�$_CalculateB64EncodedHashFromContents�crcmod� predefined�Crc�r2 r r r �%CalculateB64EncodedCrc32cFromContents� s �rE c C s t | t� �S )z�Calculates a base64 MD5 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: MD5 digest of the file in base64 format. )r@ �GetMd5rD r r r �"CalculateB64EncodedMd5FromContents� s rG c C s t | t� �S )z�Calculates a base64 MD5 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: MD5 digest of the file in hex format. )r5 rF rD r r r �CalculateMd5FromContents� s rH c C s t �t�| ��}|�d��t�S )zAReturns the base64-encoded version of the input hex digest value.� )�base64� b64encode�binascii� unhexlify�rstrip�decoder )Z digest_valueZ encoded_bytesr r r �Base64EncodeHash s rP c C s t �| �d��t��}t�|�S )z�Returns the hex digest value of the input base64-encoded hash. Args: base64_hash: Base64-encoded hash, which may contain newlines and single or double quotes. Returns: Hex digest of the input argument. z "')rJ � b64decode�stripr; r rL �hexlify)Z base64_hashZ decoded_bytesr r r �Base64ToHexHash s rT c C s t t| |��S )a4 Calculates a base64 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. hash_alg: Instance of hashing class initialized to start state. Returns: Hash of the stream in base64 format. )rP r5 )r2 r3 r r r r@ s r@ c C s" t �ddt�} | dkri S dtiS )a? Returns a dict of hash algorithms for validating an uploaded object. This is for use only with single object uploads, not compose operations such as those used by parallel composite uploads (though it can be used to validate the individual components). Returns: dict of (algorithm_name: hash_algorithm) �GSUtil� check_hashesr �md5)r �get� CHECK_HASH_IF_FAST_ELSE_FAILrF )�check_hashes_configr r r �GetUploadHashAlgs( s �r[ Fc C s� t �ddt�}|tkri S i }|r,t|d< nj|r�t� rDdd� |d< nR|s�|tkrZtt��n<|tkrn| � t � n(|t kr�| � t � dd� |d< ntd��|S ) a� Returns a dict of hash algorithms for validating an object. Args: logger: logging.Logger for outputting log messages. consider_md5: If True, consider using a md5 hash. consider_crc32c: If True, consider using a crc32c hash. Returns: Dict of (string, hash algorithm). Raises: CommandException if hash algorithms satisfying the boto config file cannot be returned. rU rV rW c S s t j�d�S �Nr? �rA rB rC r r r r �<lambda>U � z%GetDownloadHashAlgs.<locals>.<lambda>�crc32cc S s t j�d�S r\ r] r r r r r^ ] r_ z8Your boto config 'check_hashes' option is misconfigured.) r rX rY �CHECK_HASH_NEVERrF r r �_SLOW_CRC_EXCEPTION_TEXT� CHECK_HASH_IF_FAST_ELSE_SKIP�warn�_NO_HASH_CHECK_WARNING�CHECK_HASH_ALWAYS� _SLOW_CRCMOD_DOWNLOAD_WARNING)�loggerZ consider_md5Zconsider_crc32crZ � hash_algsr r r �GetDownloadHashAlgs9 s. � �rj r_ c C s2 z t �| �W S ty, t j| dd� Y S 0 dS )aN Returns md5 object, avoiding incorrect FIPS error on Red Hat systems. Examples: GetMd5(b'abc') GetMd5(bytes('abc', encoding='utf-8')) Args: byte_string (bytes): String in bytes form to hash. Don't include for empty hash object, since md5(b'').digest() == md5().digest(). Returns: md5 hash object. F)�usedforsecurityN)�hashlibrW � ValueError)� byte_stringr r r rF e s rF c @ sT e Zd ZdZdd� Zedd� �Zddd�Zd d � Zd d � Z e j fd d�Z dd� Z dS )�HashingFileUploadWrappera� Wraps an input stream in a hash digester and exposes a stream interface. This class provides integrity checking during file uploads via the following properties: Calls to read will appropriately update digesters with all bytes read. Calls to seek (assuming it is supported by the wrapped stream) using os.SEEK_SET will catch up / reset the digesters to the specified position. If seek is called with a different os.SEEK mode, the caller must return to the original position using os.SEEK_SET before further reads. Calls to seek are fast if the desired position is equal to the position at the beginning of the last read call (we only need to re-hash bytes from that point on). c C st |st d��n |st d��|| _|| _|| _|| _d| _i | _| jD ]}| j| �� | j|< qDd| _d| _ || _ dS )a� Initializes the wrapper. Args: stream: Input stream. digesters: dict of {string: hash digester} containing digesters, where string is the name of the hash algorithm. hash_algs: dict of {string: hash algorithm} for resetting and recalculating digesters. String is the name of the hash algorithm. src_url: Source FileUrl that is being copied. logger: For outputting log messages. z0HashingFileUploadWrapper used with no digesters.z0HashingFileUploadWrapper used with no hash_algs.Nr ) r �_orig_fp� _digesters�_src_url�_logger� _seek_away�_digesters_previous�copy�_digesters_previous_mark�_digesters_current_mark� _hash_algs)�self�stream� digestersri Zsrc_urlrh �algr r r �__init__� s z!HashingFileUploadWrapper.__init__c C s t | jdd�S )z<Returns the mode of the underlying file descriptor, or None.�modeN)�getattrrp �rz r r r r � s z HashingFileUploadWrapper.moder% c C s� | j durtd��| j�|�}t|tj�r4|�t�}| j | _ | j D ](}| j | � � | j |< | j | �|� qB| j t|�7 _ |S )ai "Reads from the wrapped file pointer and calculates hash digests. Args: size: The amount of bytes to read. If ommited or negative, the entire contents of the file will be read, hashed, and returned. Returns: Bytes from the wrapped stream. Raises: CommandException if the position of the wrapped stream is unknown. Nz\Read called on hashing file pointer in an unknown position; cannot correctly compute digest.)rt r rp r6 r9 r7 � text_typer; r rx rw rq rv ru r= r+ )rz �sizer> r} r r r r6 � s z HashingFileUploadWrapper.readc C s | j �� S )z$Returns the current stream position.)rp �tellr� r r r r� � s z HashingFileUploadWrapper.tellc C s | j �� S )z'Returns true if the stream is seekable.)rp �seekabler� r r r r� � s z!HashingFileUploadWrapper.seekablec C s |t jkr| j�� | _n�d| _|| jk rd| jD ]}| j| � | j|< q.d| _| j� d� | � |� n�|| jkr�| j| _| jD ]}| j | | j|< q|nl|| jk r�| j| _| jD ]}| j | | j|< q�| j� | j� | � || j � n | j� | j� | � || j � | j� ||�S )z�Seeks in the wrapped file pointer and catches up hash digests. Args: offset: The offset to seek to. whence: os.SEEK_CUR, or SEEK_END, SEEK_SET. Returns: Return value from the wrapped stream's seek call. Nr ) �os�SEEK_SETrp r� rt rw rq ry rx r/ �_CatchUpru )rz �offset�whencer} r r r r/ � s, z HashingFileUploadWrapper.seekc C s� | j �� | jkr(td| j �� | jf ��| jD ]4}|tkrN| j�d|| jj � | j| � � | j |< q.| j| _ |}t |t�}|r�| j �|�}t|tj�r�|�t�}||8 }| jD ]}| j| �|� q�t |t�}qz| j|7 _dS )ak Catches up hashes, but does not return data and uses little memory. Before calling this function, digesters_current_mark should be updated to the current location of the original stream and the self._digesters should be current to that point (but no further). Args: bytes_to_read: Number of bytes to catch up from the original stream. zJInvalid mark when catching up hashes. Stream position %s, hash position %szCatching up %s for %s...N)rp r� rx r rq r rs �debugrr � url_stringrv ru rw �minr r6 r9 r7 r� r; r r= )rz Z bytes_to_readr} Zbytes_remainingZbytes_this_roundr> r r r r� s0 �� � z!HashingFileUploadWrapper._CatchUpN)r% )�__name__� __module__� __qualname__�__doc__r~ �propertyr r6 r� r� r� r� r/ r� r r r r ro | s 0ro )N)FF)r_ )3r� � __future__r r r r rJ rL rl r� r7 �botor rA �gslib.exceptionr �gslib.utils.boto_utilr �gslib.utils.constantsr r r r ZSLOW_CRCMOD_WARNINGZSLOW_CRCMOD_RSYNC_WARNINGrg rb re rY rc rf ra r* r r r r# r r5 r0 rE rG rH rP rT r@ r[ rj rF �objectro r r r r �<module> sX ,