OXIESEC PANEL
- Current Dir:
/
/
opt
/
gsutil
/
gslib
/
utils
/
__pycache__
Server IP: 2a02:4780:11:1594:0:ef5:22d7:a
Upload:
Create Dir:
Name
Size
Modified
Perms
📁
..
-
02/11/2025 08:19:48 AM
rwxr-xr-x
📄
__init__.cpython-39.pyc
126 bytes
02/11/2025 08:19:48 AM
rw-r--r--
📄
acl_helper.cpython-39.pyc
9.38 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
arg_helper.cpython-39.pyc
1.01 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
boto_util.cpython-39.pyc
16.46 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
cat_helper.cpython-39.pyc
4.21 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
cloud_api_helper.cpython-39.pyc
3.29 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
constants.cpython-39.pyc
2.88 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
copy_helper.cpython-39.pyc
101.64 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
encryption_helper.cpython-39.pyc
5.84 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
execution_util.cpython-39.pyc
1.06 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
hashing_helper.cpython-39.pyc
15.71 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
iam_helper.cpython-39.pyc
8.61 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
ls_helper.cpython-39.pyc
12.4 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
metadata_util.cpython-39.pyc
3.1 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
parallelism_framework_util.cpython-39.pyc
11.79 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
posix_util.cpython-39.pyc
12.46 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
retention_util.cpython-39.pyc
8.77 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
retry_util.cpython-39.pyc
2.5 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rsync_util.cpython-39.pyc
1.6 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
shim_util.cpython-39.pyc
19.8 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
signurl_helper.cpython-39.pyc
4.21 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
stet_util.cpython-39.pyc
3.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
system_util.cpython-39.pyc
7.9 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
temporary_file_util.cpython-39.pyc
915 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
text_util.cpython-39.pyc
12.62 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
translation_helper.cpython-39.pyc
25.67 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
unit_util.cpython-39.pyc
7.01 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
update_util.cpython-39.pyc
3.39 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
user_agent_helper.cpython-39.pyc
1.49 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
version_check.cpython-39.pyc
1.39 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
wrapped_credentials.cpython-39.pyc
5.81 KB
02/11/2025 08:19:49 AM
rw-r--r--
Editing: boto_util.cpython-39.pyc
Close
a +(Wgh � @ sT d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddl Z ddlmZ dd lmZ dd lmZ dd lmZ ddlZdd lmZ dd lmZ ddlmZ ddl m Z ddl m Z ddl m Z ddl m Z ddl!m"Z" ddl!m#Z# ddl$Z$ddl%m&Z& e j'�r<e(Z)e �*ddd�Z+e �*ddd�Z,da-da.dd� Z/d d � Z0d d � Z1d d!� Z2d"d#� Z3d$d%� Z4d&d'� Z5d(d)� Z6d*d+� Z7d,d-� Z8d.d/� Z9d0d1� Z:d2d3� Z;d4d5� Z<d6d7� Z=e$j>fd8d9�Z?d:d;� Z@d<d=� ZAd>d?� ZBd@dA� ZCdBdC� ZDdDdE� ZEdFdG� ZFdHdI� ZGdJdK� ZHdLdM� ZIdUdOdP�ZJdQdR� ZKdSdT� ZLdS )Vz�Shared utility structures and methods that require importing boto. This module also imports httplib2 (as it is Boto's http transport and closely tied to some of Boto's core functionality) and oauth2client. � )�absolute_import)�print_function)�division)�unicode_literalsN)�config)�NoAuthHandlerFound)� GSConnection)�Provider)�BotoConfigLocations)�context_config)�CommandException)� system_util)� DEFAULT_GCS_JSON_API_VERSION)�DEFAULT_GSUTIL_STATE_DIR)�SSL_TIMEOUT_SEC)�UTF8)�HumanReadableToBytes)�ONE_MIB)� HAS_CRYPTO�Boto� is_secureT�https_validate_certificatesc C s� t j�ddd�} | dkr dS | s�ts�tj�tj�tj dd��atj� t�s�t � dd�}|sbt d ��t�|�}tjd d d �\}}t�|d �}|�|� |�� |atat} | S )z�Configures and returns the CA Certificates file. If one is already configured, use it. Otherwise, use the cert roots distributed with gsutil. Returns: string filename of the certs file to use. r Zca_certificates_fileN�system�dataz cacerts.txt�gslibzdata/cacerts.txtzACertificates file not found. Please reinstall gsutil from scratchz.txtzgsutil-cacerts)�suffix�prefix�w)�botor �get�configured_certs_file�os�path�abspath�joinr � GSLIB_DIR�exists�pkgutil�get_datar �six� ensure_str�tempfile�mkstemp�fdopen�write�close�temp_certs_file)Z certs_fileZ certs_data�fd�fname�f� r4 �$/opt/gsutil/gslib/utils/boto_util.py�ConfigureCertsFileB s* � r6 c C sZ t � sVt�dd�rJtsJt�� r4td�t� d����qVtd�t� d����n ddl m } dS ) zASets up no-op auth handler if no boto credentials are configured.� Credentials�gs_service_client_id� aI Your gsutil is configured with an OAuth2 service account, but you do not have PyOpenSSL or PyCrypto 2.6 or later installed. Service account authentication requires one of these libraries; please reactivate your service account via the gcloud auth command and ensure any gcloud packages necessary for service accounts are present.a( Your gsutil is configured with an OAuth2 service account, but you do not have PyOpenSSL or PyCrypto 2.6 or later installed. Service account authentication requires one of these libraries; please install either of them to proceed, or configure a different type of credentials with "gsutil config".r ��no_op_auth_pluginN) �HasConfiguredCredentialsr � has_optionr r �InvokedViaCloudSdkr r$ �textwrap�wrapr r; r: r4 r4 r5 �ConfigureNoOpAuthIfNeededl s ��� �� rA c C s t S �N)r r4 r4 r4 r5 � GetCertsFile� s rC c C s t r t gS g S )zHReturns a list of temp files to delete (if possible) when program exits.)r0 r4 r4 r4 r5 �GetCleanupFiles� s rD c C s� t } dtjv r | �tjd � tj�tj�d�dd�}tj�|�rL| �|� g }| D ]P}z8t|d�� |�|� W d � n1 s�0 Y W qT t y� Y qT0 qT|S )zFReturns a list of the path(s) to the boto config file(s) to be loaded.�AWS_CREDENTIAL_FILE�~z.aws� credentials�rN) r r! �environ�appendr" r$ � expanduser�isfile�open�IOError)Zpotential_config_pathsZ aws_cred_file�readable_config_pathsr" r4 r4 r5 �GetConfigFilePaths� s , rP c C s t � } t| �dkr | �d� | S )zGLike GetConfigFilePaths but returns a not-found message if paths empty.r zNo config found)rP �lenrJ )rO r4 r4 r5 �GetFriendlyConfigFilePaths� s rR c C s t �ddt�} t�| � | S )a� Returns the location of the directory for gsutil state files. Certain operations, such as cross-process credential sharing and resumable transfer tracking, need a known location for state files which are created by gsutil as-needed. This location should only be used for storing data that is required to be in a static location. Returns: Path to directory for gsutil static state files. �GSUtilZ state_dir)r r r r �CreateDirIfNeeded)Zconfig_file_dirr4 r4 r5 �GetGsutilStateDir� s rU c C s t j�t� d�S )NZ credstore2�r! r" r$ rU r4 r4 r4 r5 �GetCredentialStoreFilename� s rW c C s t j�t� d�S )NZ gcecredcacherV r4 r4 r4 r5 � GetGceCredentialCacheFilename� s rX c C s t �ddt�S )NrS �json_api_version)r r r r4 r4 r4 r5 �GetGcsJsonApiVersion� s rZ c C sP t �ddtd��} | dkr$td�} n(| td� dkrL| td�| td� 7 } | S )NrS �json_resumable_chunk_sizei @r i )r �getint�long)� chunk_sizer4 r4 r5 �GetJsonResumableChunkSize� s � r_ c C s t j�t� d�S )Nz.last_software_update_checkrV r4 r4 r4 r5 �*GetLastCheckedForGsutilUpdateTimestampFile� s r` c C sH t � } dt }| | d d|d d d }t� | }|dkrDd}|S )z�Gets the max concurrent transport compressed uploads allowed in parallel. Returns: The max number of concurrent transport compressed uploads allowed in parallel without exceeding the max_upload_compression_buffer_size. � � � � i�? r )r_ r �!GetMaxUploadCompressionBufferSize)Zupload_chunk_sizeZcompression_chunk_sizeZtotal_upload_sizeZmax_concurrent_uploadsr4 r4 r5 �!GetMaxConcurrentCompressedUploads� s ��rf c C s t �ddd�S )Nr �max_retry_delay� �r r\ r4 r4 r4 r5 �GetMaxRetryDelay� s rj c C s t t�ddd��S )zEGet the max amount of memory compressed transport uploads may buffer.rS Z"max_upload_compression_buffer_sizeZ2GiB)r r r r4 r4 r4 r5 re � s �re c K s� t �ddd�}|t �ddd�t �dd�t �ddd�t �ddd�t �dd |rNd nd�d �}t|�}t� |d < t|d < | f d|i|��}t �dd� |_t� � }|r�|j r�|j |j |j d|j d� |S )z�Creates and returns a new httplib2.Http instance. Args: http_class: Optional custom Http class to use. **kwargs: Arguments to pass to http_class constructor. Returns: An initialized httplib2.Http instance. r �proxyN� proxy_type�http� proxy_port� proxy_user� proxy_pass� proxy_rdnsT�� proxy_hostrl rn ro rp rq �ca_certs�timeout� proxy_infor � )�key�cert�domain�password)r r r\ � SetProxyInforC r �getbool�"disable_ssl_certificate_validationr �get_context_config�use_client_certificate�add_certificate�client_cert_path�client_cert_password)� http_class�kwargsrs �boto_proxy_configrv rm Zglobal_context_configr4 r4 r5 � GetNewHttp s. � � �r� c C s t �ddd�S )Nr � num_retries� ri r4 r4 r4 r5 � GetNumRetries8 s r� c C s t j�t� d�S )Nztab-completion-logsrV r4 r4 r4 r5 �GetTabCompletionLogFilename< s r� c C s, t j�t� d�} tj| dd� t j�| d�S )Nztab-completioni� )�mode�cache)r! r" r$ rU r rT )Ztab_completion_dirr4 r4 r5 � GetTabCompletionCacheFilename@ s r� c C s� t �dd�ot �dd�} t �dd�o.t �dd�}t �dd�}t �dd�}t �dd�}tont �dd �ont �dd �}| s�|s�|s�|s�|s�|r�d S d }z<tjjtjt td �dgd�}dt t |dd �dd �kr�d }W n t y� Y n0 |S )z1Determines if boto credential/config file exists.r7 �gs_access_key_id�gs_secret_access_key�aws_access_key_id�aws_secret_access_key�gs_oauth2_refresh_tokenZgs_external_account_fileZ(gs_external_account_authorized_user_filer8 �gs_service_key_fileTN�google�s3��requested_capabilityZNoOpAuth� __class__�__name__) r r= r r �authZget_auth_handlerr � DefaultHostr �getattrr )Zhas_goog_credsZhas_amzn_credsZhas_oauth_credsZhas_external_credsZ*has_external_account_authorized_user_credsZhas_service_account_credsZvalid_auth_handlerr4 r4 r5 r< G sX � ���� � � ��� � � r< c C s t �ddd �} | d uS )NrS r[ )r r )Zchunk_size_definedr4 r4 r5 � JsonResumableChunkSizeDefinedq s r� c sh ddl �tjj�d��fdd� } | tj_ddl�ddl�tjj� tjj�� ���fdd�}|tjj _ dS )a� Apply gsutil-specific patches to Boto. Here be dragons. Sorry. Note that this method should not be used as a replacement for contributing fixes to the upstream Boto library. However, the Boto library has historically not been consistent about release cadence, so upstream fixes may not be available immediately in a version which we can pin to. Also, some fixes may only be applicable to gsutil. In such cases, patches should be applied to the Boto library here (and removed if/when they are included in the upstream repository and included in an official new release that we pull in). This method should be invoked after all other Boto-related initialization has been completed. r Nc s\ �| |d�}�j j�j jf� t� fdd�|D �dd� d�t� fdd�|D �dd� d� }|S ) Nr� c s g | ]}|� vr|�qS r4 r4 ��.0rH �Zxml_oauth2_handlersr4 r5 � <listcomp>� � zDMonkeyPatchBoto.<locals>._PatchedGetPluginMethod.<locals>.<listcomp>c S s | j S rB �r� �Z handler_tr4 r4 r5 �<lambda>� r� zBMonkeyPatchBoto.<locals>._PatchedGetPluginMethod.<locals>.<lambda>)rx c s g | ]}|� v r|�qS r4 r4 r� r� r4 r5 r� � r� c S s | j S rB r� r� r4 r4 r5 r� � r� )Z oauth2_pluginZOAuth2ServiceAccountAuthZ OAuth2Auth�sorted)�clsr� Zhandler_subclassesZ new_result)�gcs_oauth2_boto_plugin�orig_get_plugin_methodr� r5 �_PatchedGetPluginMethod� s" ������ z0MonkeyPatchBoto.<locals>._PatchedGetPluginMethodc s` t | d�r"��| j| jf| j�}n��| j| jf�}d}| jrN|d| j 7 }n|d7 }tj�|� t �d��rt �dd��r�� �j �}�j |_ | jr�|�| j� | jr�|�| j| j� |j|| jd�| _| j| j_| j| j_|j | j_�j | j_| j| j_d | j_n �j|| j| j�j | jd �| _| j�� }| j�d d �d }�||��s\� ||d | ��d S ) Nru zwrapping ssl socket; zCA certificate file=%sz using system provided SSL certs� SSLContext�HAS_SNIF)�server_hostname)�keyfile�certfile� cert_reqsrt �:r z/remote hostname "%s" does not match certificate)�hasattr�create_connection�host�portru rt r �log�debugr� r� �PROTOCOL_SSLv23� CERT_REQUIRED� verify_mode�load_verify_locations� cert_file�load_cert_chain�key_file� wrap_socket�sockr� r� r� � ssl_version�ciphers� getpeercert�split)�selfr� �msg�contextry �hostname)�InvalidCertificateException�ValidateCertificateHostname�socket�sslr4 r5 �_PatchedConnectMethod� sH � ��z.MonkeyPatchBoto.<locals>._PatchedConnectMethod)N) r� r �plugin� get_pluginr� r� Zhttps_connectionr� r� Z CertValidatingHTTPSConnection�connect)r� r� r4 )r� r� r� r� r� r� r5 �MonkeyPatchBotov s ��-�r� c C sj t j�| �}|r | �� �d�s0t�tjjdd�S | �� � d�d }|�� �d�s\|d | }tj ||d�S )a Reads proxy info from the environment and converts to httplib2.ProxyInfo. Args: proxy_env_var: Environment variable string to read, such as http_proxy or https_proxy. Returns: httplib2.ProxyInfo constructed from the environment string. rm Nr �_z://)�method) r! rI r �lower� startswith�httplib2� ProxyInfo�socks�PROXY_TYPE_HTTPr� �proxy_info_from_url)� proxy_env_varZ proxy_urlZproxy_protocolr4 r4 r5 �ProxyInfoFromEnvironmentVar� s r� c C s t �dddt �S )NrS �resumable_threshold� )r r\ r r4 r4 r4 r5 �ResumableThreshold s r� c C s� ddddd�}|� | � d��� |d �}| � d�}| � d�}| � d �}| � d �}t| � d ��}tj||||||d �}|j|d ks�d |_|jr�|js�dD ]8} | t j v r�t j | r�t | �}| � d �dkr�d|_ q�q�|S )z�Sets proxy info from boto and environment and converts to httplib2.ProxyInfo. Args: dict: Values read from the .boto file Returns: httplib2.ProxyInfo constructed from boto or environment variable string. rd � � )Zsocks4Zsocks5rm �httpsrl rm rs rn ro rp rq rr F)� http_proxy� https_proxy� HTTPS_PROXYNT) r r� �boolr� r� rl rq rs rn r! rI r� ) r� Zproxy_type_specrl rs rn ro rp rq rv r� r4 r4 r5 r| s4 � � r| c C s> t j�ddd �} | d ur | S tdt� t� dgd�}t|dd�S )NrS Ztest_assume_fast_crcmodz crcmod.crcmodZ_usingExtensionr F)r r r � __import__�globals�localsr� )Zboto_optZ nested_crcmodr4 r4 r5 �UsingCrcmodExtension5 s �r� Fc C s | j jj}|s*|jdv r dS |�d�r*dS d|j kr@dkr�n nv|�d�| _| j}t|t�rj|� t �}|�dd�}|�d d�}|du r�|du r�| jd | kr�|� d | j| jf ��dS |jd kr�|� � }|� |j|j|�}|jd v r�tjjd|j |d��dS )zFReplaces boto.s3.key's should_retry() to handle KMS-encrypted objects.)i� i� T�location�� i+ �etagz/x-amz-server-side-encryption-customer-algorithmNz x-goog-encryption-kms-key-namez"%s"z2ETag from S3 did not match computed MD5. %s vs. %si� )ZRequestTimeoutzSaw %s, retrying)�responseF)�bucket� connection�provider�status� getheaderr� �md5� isinstance�bytes�decoder Zstorage_data_error�readZstorage_response_error�reason� error_coder � exception�PleaseRetryException) r� r� Zchunked_transferr� r� Z-amz_server_side_encryption_customer_algorithmZ goog_customer_managed_encryption�body�errr4 r4 r5 �_PatchedShouldRetryMethodJ sP ��� �� � �r c C s< t j�ddd �} t�t jjjj�}| d ur8|t�| �kS dS )Nr7 �gs_hostF) r r r r) r* �gsr� r r� )r Z default_hostr4 r4 r5 �HasUserSpecifiedGsHost� s r c C sR t j} | �dd�}| �dd�o(| �dd�}| �dd�o@| �dd�}| oP| oP|S )Nr7 r� r8 r� r� r� )r r r= )r Zhas_refresh_tokenZ has_service_account_credentialsZhas_hmac_credentialsr4 r4 r5 � UsingGsHmac� s � � �r )F)M�__doc__� __future__r r r r r! r' r+ r? r) r r Z boto.auth�boto.exceptionr �boto.gs.connectionr Z boto.providerr �boto.pyami.configr r r �gslib.exceptionr � gslib.utilsr �gslib.utils.constantsr r r r �gslib.utils.unit_utilr r r� �oauth2client.clientr �PY3�intr] r �BOTO_IS_SECURE� CERTIFICATE_VALIDATION_ENABLEDr r0 r6 rA rC rD rP rR rU rW rX rZ r_ r` rf rj re �Httpr� r� r� r� r< r� r� r� r� r| r� r r r r4 r4 r4 r5 �<module> s� �* 5*x/ ;