OXIESEC PANEL
- Current Dir:
/
/
lib
/
python3.9
/
site-packages
/
S3
/
__pycache__
Server IP: 2a02:4780:11:1594:0:ef5:22d7:a
Upload:
Create Dir:
Name
Size
Modified
Perms
📁
..
-
10/22/2025 07:58:12 AM
rwxr-xr-x
📄
ACL.cpython-39.opt-1.pyc
8.44 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
ACL.cpython-39.pyc
8.44 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
AccessLog.cpython-39.opt-1.pyc
3.26 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
AccessLog.cpython-39.pyc
3.26 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
BaseUtils.cpython-39.opt-1.pyc
8.33 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
BaseUtils.cpython-39.pyc
8.33 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
BidirMap.cpython-39.opt-1.pyc
1.82 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
BidirMap.cpython-39.pyc
1.82 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
CloudFront.cpython-39.opt-1.pyc
24.28 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
CloudFront.cpython-39.pyc
24.28 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Config.cpython-39.opt-1.pyc
16.61 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Config.cpython-39.pyc
16.61 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
ConnMan.cpython-39.opt-1.pyc
8.11 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
ConnMan.cpython-39.pyc
8.11 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Crypto.cpython-39.opt-1.pyc
9.2 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Crypto.cpython-39.pyc
9.2 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Custom_httplib27.cpython-39.opt-1.pyc
4.91 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Custom_httplib27.cpython-39.pyc
4.91 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Custom_httplib3x.cpython-39.opt-1.pyc
6.04 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Custom_httplib3x.cpython-39.pyc
6.04 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Exceptions.cpython-39.opt-1.pyc
4.65 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Exceptions.cpython-39.pyc
4.65 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
ExitCodes.cpython-39.opt-1.pyc
1.65 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
ExitCodes.cpython-39.pyc
1.65 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
FileDict.cpython-39.opt-1.pyc
2.1 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
FileDict.cpython-39.pyc
2.1 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
FileLists.cpython-39.opt-1.pyc
14.46 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
FileLists.cpython-39.pyc
14.5 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
HashCache.cpython-39.opt-1.pyc
2.33 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
HashCache.cpython-39.pyc
2.33 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
MultiPart.cpython-39.opt-1.pyc
7.93 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
MultiPart.cpython-39.pyc
7.93 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
PkgInfo.cpython-39.opt-1.pyc
593 bytes
12/15/2023 11:05:18 AM
rw-r--r--
📄
PkgInfo.cpython-39.pyc
593 bytes
12/15/2023 11:05:18 AM
rw-r--r--
📄
Progress.cpython-39.opt-1.pyc
5.67 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Progress.cpython-39.pyc
5.67 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
S3.cpython-39.opt-1.pyc
55.27 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
S3.cpython-39.pyc
55.29 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
S3Uri.cpython-39.opt-1.pyc
8.04 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
S3Uri.cpython-39.pyc
8.04 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
SortedDict.cpython-39.opt-1.pyc
3.08 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
SortedDict.cpython-39.pyc
3.08 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Utils.cpython-39.opt-1.pyc
9.02 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
Utils.cpython-39.pyc
9.02 KB
12/15/2023 11:05:18 AM
rw-r--r--
📄
__init__.cpython-39.opt-1.pyc
138 bytes
12/15/2023 11:05:18 AM
rw-r--r--
📄
__init__.cpython-39.pyc
138 bytes
12/15/2023 11:05:18 AM
rw-r--r--
Editing: S3.cpython-39.pyc
Close
a ��wel� � @ s d dl mZmZ d dlZd dlZd dlZd dlZd dlZd dlZd dl Z d dl m Z d dl m Z d dlmZmZmZmZ d dlmZmZmZmZ zd dlmZ W n ey� d dlmZ Y n0 d dl Z dd l m Z m Z m Z m!Z!m"Z"m#Z#m$Z$ dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ dd l,m,Z, dd l-m-Z- dd l.m.Z.m/Z/ ddl0m0Z0 ddl1m1Z1 ddl2T ddl3m4Z4 ddl5m5Z5 ddl6m6Z6 ddl7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=m>Z>m?Z? z�d dl@mAZA d dlBZBzeBjCdd�ZDdd� ZEW n� eF�y. z eB�GeBjH�ZDeD�I� dd� ZEW n* eJ�y( eBjCeBjHd�ZDd d� ZEY n0 Y n0 eJ�y\ eB�GeBjH�ZDeD�I� d d� ZEY n0 W n^ eeKf�y� ZL z@eMeL�ZNd eNv �r�d ZOnd eN ZOeOd!7 ZOd"aPd#d� ZEW Y dZL[Ln dZL[L0 0 d$d%� ZQd&ZRd'ZSg ZTG d(d)� d)eU�ZVG d*d+� d+eU�ZWeT�Xd+� d,d-� ZYdS ).� )�absolute_import�divisionN)�saxutils)�timeout)�debug�info�warning�error)�ST_SIZE�ST_MODE�S_ISDIR�S_ISREG)�urlparse� )�getListFromXml�getTextFromXml�getRootTagName�decode_from_s3� encode_to_s3�md5�s3_quote)� convertHeaderTupleListToDict� unicodise� deunicodise�check_bucket_name� check_bucket_name_dns_support�getHostnameFromBucket)� SortedDict)� AccessLog)�ACL�GranteeLogDelivery)�BidirMap)�Config)�*)�MultiPartUpload)�S3Uri)�ConnMan)�sign_request_v2�sign_request_v4�checksum_sha256_file�checksum_sha256_buffer�generate_content_md5� hash_file_md5�calculateChecksum�format_param_str)� ArgumentErrorT)Zmimec C s t �| �S �N)�magic_Z from_file��file� r4 �)/usr/lib/python3.9/site-packages/S3/S3.py�mime_magic_file< s r6 c C s8 z t �| �W S tttfy2 t �t| �� Y S 0 d S r0 )r1 r3 �UnicodeDecodeError�UnicodeEncodeErrorr/ r r2 r4 r4 r5 r6 I s )�flagsc C s t �| �S r0 )r1 Z id_filenamer2 r4 r4 r5 r6 S s c C s t �t| ��S r0 )r1 r3 r r2 r4 r4 r5 r6 \ s �magicz%Module python-magic is not available.z'Module python-magic can't be used (%s).z. Guessing MIME types based on file extensions.Fc C s t stt� da t�| �d S )NTr )� magic_warnedr � magic_message� mimetypes� guess_typer2 r4 r4 r5 r6 g s c C sj dd� }|| �}|d urZt |t�rZd|v rR|�d�\}}|td�d � }||f}n|d f}|d u rfd}|S )Nc S s t | �}|S r0 )r6 )r3 Z magictyper4 r4 r5 � _mime_magicp s z mime_magic.<locals>._mime_magic�;�charset)NN)� isinstance�str�split�len)r3 r? �resultZmimetyperA r4 r4 r5 � mime_magicn s rG � i c @ sF e Zd Zi Zi Zddd�Zdd� Zdd� Zdd � Zd d � Z d d � Z dS )� S3RequestNc C sn || _ t|pi dd�| _t| j jj�dkrF| j j�� | j jj| jd< || _|| _|pXi | _ || _ | � � d S )NT�Z ignore_caser zx-amz-security-token) �s3r �headersrE �configZ access_tokenZ role_refresh�resource� method_string�params�body�requester_pays)�selfrK rO rN rL rQ rP r4 r4 r5 �__init__� s zS3Request.__init__c C s" | j jjr | jdv r d| jd< d S )N)�GET�POST�PUT�HEADZ requesterzx-amz-request-payer)rK rM rR rO rL �rS r4 r4 r5 rR � s zS3Request.requester_paysc C s, d| j v r| j d= t�dt�� �| j d< d S )N�datez%a, %d %b %Y %H:%M:%S +0000z x-amz-date)rL �time�strftime�gmtimerY r4 r4 r5 �update_timestamp� s zS3Request.update_timestampc C s& | j jr dS | j jjs | j jr"dS dS )NFT)rK � endpoint_requires_signature_v4rM Z signature_v2�fallback_to_signature_v2rY r4 r4 r5 �use_signature_v2� s zS3Request.use_signature_v2c C s | j �d�}| �� rVtd� |r4d|| j d f }n | j d }t| j|| j| j�| _n�td� | j� | j d �}|r�|t j v r�t j �|d�� d| �r�|t j vr�t t� j|�s�d|| j d f }n | j d }t j�| j d t� j�}t| j||| j|| j| j�| _d S )N�bucketzUsing signature v2�/%s%s�urizUsing signature v4� �%s.)rN �getra r r' rO rP rL rK � get_hostnamerI � redir_map� startswithr r" � host_bucket� region_map�bucket_locationr( rQ )rS � bucket_name� resource_uri�hostnameZ bucket_regionr4 r4 r5 �sign� s0 ��� � �zS3Request.signc C sT | � � | �� t| j�}t|d ddd�|d< |d t| j�7 < | j|| jfS )Nrd FT�Zquote_backslashesZunicode_output) r^ rq �dictrN r r. rP rO rL )rS rN r4 r4 r5 � get_triplet� s zS3Request.get_triplet)N) �__name__� __module__� __qualname__rl ri rT rR r^ ra rq rt r4 r4 r4 r5 rI � s !rI c @ s� e Zd Zeddddddd�Zedd d d d d �Zeded ed B ed ed B ed ed B ed ed B ed ed B ed ed B ed ed B ed ed B ed ed B ed ed B d� Zdddd �Zd d � Zd d � Z d!d"� Z d#d$� Z d�d&d'�Z d(d)� Z d�d+d,�Zd�d-d.�Zd�d/d0�Zd�d1d2�Zd3d4� Zd�d6d7�Zd8d9� Zd:d;� Zd<d=� Zd>d?� Zd@dA� ZdBdC� Zd�dDdE�Zd�dFdG�Zd�dHdI�Z d�dJdK�Z d�dLdM�Z dNdO� Z dPdQ� Z dRdS� Z!d�dTdU�Z"dVdW� Z#d�dYdZ�Z$d�d[d\�Z%d]d^� Z&d_d`� Z'dadb� Z(dcdd� Z)dedf� Z*d�dgdh�Z+d�didj�Z,d�dkdl�Z-dmdn� Z.dodp� Z/dqdr� Z0dsdt� Z1dudv� Z2dwdx� Z3dydz� Z4d{d|� Z5d}d~� Z6dd�� Z7d�d�� Z8d�d�� Z9d�d�� Z:d�d�� Z;d�d�� Z<d�d�� Z=d�d�� Z>d�d�� Z?d�d�� Z@d�d�� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�d��ZEd�d�d��ZFd�d�d��ZGd�d�d��ZHd�d�d��ZId�d�� ZJd�d�� ZKd�d�� ZLd�d�d��ZMd�d�d��ZNd�d�� ZOd�d�� ZPd�d�� ZQd�d�� ZRd�d�� ZSd�d�d��ZTd�d�d��ZUd�d�d��ZVd�d�d��ZWd�d�dZXd�d�dĄZYd%S )��S3r rH � � � � )rU rW rX �DELETErV �MASK� i i i i )�SERVICE�BUCKET�OBJECT�BATCHr~ r r� rU r� rW r} r� rX rV r� ) Z UNDEFINED�LIST_ALL_BUCKETS� BUCKET_CREATE� BUCKET_LIST� BUCKET_DELETE� OBJECT_PUT� OBJECT_GET� OBJECT_HEAD� OBJECT_DELETE� OBJECT_POST� BATCH_DELETEzBucket '%s' does not existz Access to bucket '%s' was deniedzBucket '%s' already exists)Z NoSuchBucket� AccessDeniedZBucketAlreadyExistsc C s || _ d| _d| _d| _d S )NF)rM r` r_ � expect_continue_not_supported)rS rM r4 r4 r5 rT � s z S3.__init__c C s( d}| j jdkr| j jS | j jr$d}|S )NZSTANDARDre ZREDUCED_REDUNDANCY)rM � storage_classZreduced_redundancy)rS �clsr4 r4 r5 r� s zS3.storage_classc C s� |r|t jv rt j| }n(|r6t| jj|�r6t|�}n | jj�� }| jjrb|� d�rx|d d� }n|� d�rx|d d� }t d||f � |S )Nz:443���z:80���zget_hostname(%s): %s) rI ri r rM rk r � host_base�lower� use_https�endswithr )rS rb �hostr4 r4 r5 rh s zS3.get_hostnamec C s |� � tj|< d S r0 )r� rI ri )rS rb �redir_hostnamer4 r4 r5 � set_hostname" s zS3.set_hostnameNc C s� |� d�}|rd|tjv r0tj� |d��d| �rH|tjvrdt| jj|�sddt|ddd�|d f }n|d }|r|d ||f }| jjdkr�| jj s�d | � |�|f }t d | � |S ) Nrb re rf rc FTrr rd z%s%sz http://%s%szformat_uri(): ) rg rI ri rj r rM rk r Z proxy_hostr� rh r )rS rN � base_pathrn rd r4 r4 r5 � format_uri% s, ��� ��� z S3.format_uric C s* | � d�}| �|�}t|d d�|d< |S )Nr� �dataZBucket�list)�create_request� send_requestr )rS �request�responser4 r4 r5 �list_all_buckets: s zS3.list_all_buckets���c C sZ g }g }| � |||||�D ] \}} } |�| � |�| � qi } || d< || d< || d< | S )Nr� Zcommon_prefixes� truncated)�bucket_list_streaming�extend) rS rb �prefix� recursive� uri_params�limitZ item_list�prefixesr� �dirsZobjectsr� r4 r4 r5 � bucket_list@ s zS3.bucket_listc # sJ dd� }dd� }dd� }� fdd�} |r0|� � p2i }d } g } d } d } |}| �rF| �|||||�� |� d �}|� d �}| t|�7 } | t|�7 } || | kr�|| | }|� d �} | �r8|d ks�| | |k �r(|r�| � d |d �|d< n,|�r| � d |d�|d< nd||fV �qFtd|d � n| ||fV �qF| ||fV qHdS )ze Generator that produces <dir_list>, <object_list> pairs of groups of content of a specified bucket. c S s t | d�p d}|�� dkS )N�.//IsTruncated�false)r r� )r� Z is_truncatedr4 r4 r5 �_list_truncatedO s z1S3.bucket_list_streaming.<locals>._list_truncatedc S s t | d�S )NZContents�r �r� r4 r4 r5 � _get_contentsT s z/S3.bucket_list_streaming.<locals>._get_contentsc S s t | d�S )NZCommonPrefixesr� r� r4 r4 r5 �_get_common_prefixesW s z6S3.bucket_list_streaming.<locals>._get_common_prefixesc s t � d d�p|d | S )Nr� Z NextMarkerr� )r )r� Z current_elts�key�r� r4 r5 �_get_next_markerZ s z2S3.bucket_list_streaming.<locals>._get_next_markerTr r� r� �KeyZmarkerZPrefixF� Listing continues after '%s'N)�copy�bucket_list_noparserE r )rS rb r� r� r� r� r� r� r� r� r� r� � num_objectsZ num_prefixes�max_keysZ current_listZcurrent_prefixesr4 r� r5 r� M sF � �� zS3.bucket_list_streamingc C sn |d u r i }|r||d< | j js,|s,d|d< |dkr@t|�|d< | j jrPd|d< | jd||d �}| �|�}|S ) Nr� �/Z delimiterr� zmax-keys�truezallow-unorderedr� �rb r� )rM r� rC Zlist_allow_unorderedr� r� )rS rb r� r� r� r� r� r� r4 r4 r5 r� � s zS3.bucket_list_noparsec C s� t dd�}|r|�|� d}|r�|�� �� dkr�|�� �� dkr�|�� }|�� dkr\|�� }d}||7 }|d7 }td | � t|dd � n t|d d � | jjr�d |d < d|d< | j d|||d�}| � |�}|S )NTrJ re �US� us-east-1�EUz/<CreateBucketConfiguration><LocationConstraint>z1</LocationConstraint></CreateBucketConfiguration>zbucket_location: )Z dns_strictF� public-read� x-amz-aclZ ObjectWriterzx-amz-object-ownershipr� )rb rL rQ ) r �update�strip�upperr� r r rM � acl_publicr� r� )rS rb rm � extra_headersrL rQ r� r� r4 r4 r5 � bucket_create� s( $ zS3.bucket_createc C s | j d|d�}| �|�}|S )Nr� )rb �r� r� )rS rb r� r� r4 r4 r5 � bucket_delete� s zS3.bucket_deleteFc C sB |� � }| jd|� � dd id�}tj�|d�}tj�|d�}z~|rb|rJ|sb| jjtj|< dtj|< | �|�}W ||v r�|tj|< n|tjv r�tj|= ||v r�|tj|< n|tjv r�tj|= nP||v r�|tj|< n|tjv r�tj|= ||v r�|tj|< n|tjv �r tj|= 0 t |d d�}|�r*|dv �r0d}n|d k�r>d }|S ) Nr� �locationr� re r� r� ZLocationConstraint)re r� r� z eu-west-1) rb r� rI ri rg rl rM r� r� r ) rS rd �force_us_defaultrb r� Zsaved_redir_mapZsaved_region_mapr� r� r4 r4 r5 �get_bucket_location� sB � � zS3.get_bucket_locationc C sF | j d|�� dd id�}| �|�}|�dd�}|r>t|d�}nd }|S )Nr� �requestPaymentr� r� re ZPayer�r� rb r� rg r )rS rd r� r� � resp_dataZpayerr4 r4 r5 �get_bucket_requester_pays� s � z S3.get_bucket_requester_paysc C sT t dd�}d}|| }td| � t|�|d< | jd|||dd id�}| �|�}|S ) NTrJ z�<OwnershipControls xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ObjectOwnership>%s</ObjectOwnership></Rule></OwnershipControls>zset_bucket_ownership(%s)� content-md5r� �ownershipControls�rd rL rQ r� �r r r+ r� r� )rS rd � ownershiprL rQ r� r� r4 r4 r5 �set_bucket_ownership� s � zS3.set_bucket_ownershipc C sF | j d|�� dd id�}| �|�}|�dd�}|r>t|d�}nd }|S )Nr� r� r� r� re z.//Rule//ObjectOwnershipr� )rS rd r� r� r� r� r4 r4 r5 �get_bucket_ownership� s � zS3.get_bucket_ownershipc C s� t dd�}d}dD ]*}|�|d�r&dp(d}|d|||f 7 }q|d 7 }td | � t|�|d < | jd |||d d id�}| �|�}|S )NTrJ zP<PublicAccessBlockConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">��BlockPublicAcls�IgnorePublicAcls�BlockPublicPolicy�RestrictPublicBucketsFr� r� z <%s>%s</%s>z!</PublicAccessBlockConfiguration>z"set_bucket_public_access_block(%s)r� r� �publicAccessBlockr� )r rg r r+ r� r� ) rS rd r9 rL rQ �tag�valr� r� r4 r4 r5 � set_bucket_public_access_block� s � z!S3.set_bucket_public_access_blockc C sr | j d|�� dd id�}| �|�}|�dd�}|rjt|d�dkt|d�dkt|d �dkt|d �dkd �}ni }|S ) Nr� r� r� r� re r� r� r� r� r� r� r� )rS rd r� r� r� r9 r4 r4 r5 � get_bucket_public_access_block s � �z!S3.get_bucket_public_access_blockc C s� i }| � |�|d< d| jfd| jfd| jffD ]F\}}z||�||< W q. tyr } zd ||< W Y d }~q.d }~0 0 q.z| �|�|d< W n, ty� } zi |d< W Y d }~n d }~0 0 |S )Nzbucket-locationzrequester-pays� versioningr� zpublic-access-block)r� r� �get_versioningr� �S3Errorr� )rS rd r� r� �func�er4 r4 r5 � bucket_info s � zS3.bucket_infoc C s� |� � }| jd|dd id�}zT| �|�}t|d d�|d< t|d d�|d< | jj|� � | �|�d � |d < |W S ty� } z.|jd kr�t d � W Y d }~d S � W Y d }~n d }~0 0 d S ) Nr� �websiter� r� z.//IndexDocument//SuffixZindex_documentz.//ErrorDocument//KeyZerror_document)rb r� �website_endpoint� zICould not get /?website - website probably not configured for this bucket) rb r� r� r rM r� r� r� �statusr )rS rd rm rb r� r� r� r4 r4 r5 � website_info/ s$ � � zS3.website_infoc C s� |� � }d}|d7 }|d| jj 7 }|d7 }| jjrT|d7 }|d| jj 7 }|d7 }|d7 }| jd ||d d id �}| �|�}td | � |S ) NzF<WebsiteConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">z <IndexDocument>z <Suffix>%s</Suffix>z </IndexDocument>z <ErrorDocument>� <Key>%s</Key>z </ErrorDocument>z</WebsiteConfiguration>r� r� �rb rQ r� �Received response '%s')rb rM Z website_indexZ website_errorr� r� r )rS rd rm rb rQ r� r� r4 r4 r5 �website_createB s � zS3.website_createc C sN |� � }| jd|dd id�}| �|�}td| � |d dkrJtd| ��|S )Nr� r� r� r� r� �� zExpected status 204: %s)rb r� r� r �S3ResponseError�rS rd rm rb r� r� r4 r4 r5 �website_deleteU s � zS3.website_deletec C s� |� � }| jd|dd id�}z| �|�}W nf ty� } zN|jdkrZtd� W Y d }~d S |jdkrztd� W Y d }~d S � W Y d }~n d }~0 0 t|d �}|d kr�td |� d S t|d d �|d < t|d d �|d< t|d d�|d< |S )Nr� � lifecycler� r� zMCould not get /?lifecycle - lifecycle probably not configured for this bucket� zKCould not get /?lifecycle - lifecycle support not implemented by the serverr� ZLifecycleConfigurationz7Could not get /?lifecycle - unexpected xml response: %sz.//Rule//Prefixr� z.//Rule//Expiration//DaterZ z.//Rule//Expiration//DaysZdays)rb r� r� r� r� r r r )rS rd rm rb r� r� r� Z root_tag_namer4 r4 r5 �expiration_infob s< � ����zS3.expiration_infoc C s� | j jr| j jrtd��| j js^| j js^| j jr8td��td� |�� }| jd|dd id�}n | �|�}| � |�}td| � |S )Nz+Expect either --expiry-day or --expiry-datezdel bucket lifecycler� r� r� r� ) rM � expiry_date� expiry_days�ParameterError� expiry_prefixr rb r� �_expiration_setr� r� r4 r4 r5 �expiration_set� s � zS3.expiration_setc C s� t d� d}|d7 }|d7 }|d| jj 7 }|d7 }|d7 }|d7 }| jjr^|d | jj 7 }n| jjrv|d | jj 7 }|d 7 }|d 7 }|d 7 }tdd�}t|�|d< |�� }| jd|||dd id�}|S )Nzput bucket lifecyclez<LifecycleConfiguration>z <Rule>z <Filter>z <Prefix>%s</Prefix>z </Filter>z <Status>Enabled</Status>z <Expiration>z <Date>%s</Date>z <Days>%s</Days>z </Expiration>z </Rule>z</LifecycleConfiguration>TrJ r� r� r� �rb rL rQ r� ) r rM r r� r� r r+ rb r� )rS rd rQ rL rb r� r4 r4 r5 r � s. �zS3._expiration_setc C s� | j j}d }|dkr$| j js$td��| j jrl| j jrHttj�t |���}| j j r^t |�\}}nt � |�\}}|sx| j j}||fS )N�-zRYou must specify --mime-type or --default-mime-type for files uploaded from stdin.)rM �default_mime_typer� Zguess_mime_type�follow_symlinksr �os�path�realpathr Zuse_mime_magicrG r= r> )rS �filename� content_type�content_charsetr4 r4 r5 �_guess_content_type� s zS3._guess_content_typec C s, | j j}|s| j j}|d| j j�� 7 }|S )N� ; charset=)rM � mime_typer �encodingr� )rS r r4 r4 r5 �stdin_content_type� s zS3.stdin_content_typec C sl | j j}d }|dkr | �� S |r&d}n|s8| �|�\}}|sH| j j�� }| �||�rh|d urh|d | }|S )Nr zapplication/x-directoryr )rM r r r r r� � add_encoding)rS r �is_dirr r r4 r4 r5 r � s zS3.content_typec C sb d|v r dS | j j�d�}|d dkr*dS |�dd�}t|�dk rFdS |d }||v rZd S dS d S ) Nzcharset=F�,r re �.rH r T)rM Zadd_encoding_extsrD �rsplitrE )rS r r Zexts�parts�extr4 r4 r5 r � s zS3.add_encodingre c C s� |j dkrtd|j ��z�d}d}|dkrLd}tjtj�� ddd�}d |_ndd}t|�} t � | �} | t } t | �r�d}t� d �}n(t| �s�td ��ntj| dd �}| t }||_W n6 ttfy� } ztd | j ��W Y d } ~ n d } ~ 0 0 tdd�} |�r| �|� | jj�rd| d< | jj�r4d| d< | jj| d< | j||d�| d< | jj�rXd| d< | �� | d< d}| jj �s�|�r�t d��| jj �r�|| jj t k�s�|�r�d}|| jj | jj t k�r�t d| jj | jj f ��|�r�| �!|| |||�S | jj"�r�z| �#|�}W n t$�y d }Y n0 |d u�r�t%|d d �}|d d �&d �}||k�r�t'd |d|| jj(�}||k�r�t)d | � d S t)d |||f � nt)d!|||f � t*|�| d< | j+d"|| d#�}||�,� |d$�}| �-|||�}|S )%NrK � Expected URI type 's3', got '%s'Fr r T�rb)�mode�closefdz<stdin>� zNot a regular file)r �%srJ �AES256� x-amz-server-side-encryption�aws:kms�+x-amz-server-side-encryption-aws-kms-key-id)r r � content-typer� r� �x-amz-storage-classz2Multi-part upload is required to upload from stdinzZChunk size %d MB results in more than %d chunks. Please increase --multipart-chunk-size-mbrL �content-length�etag�"'re z,Put: size and md5sum match for %s, skipping.zBMultiPart: checksum (%s vs %s) does not match for %s, reuploading.z>MultiPart: size (%d vs %d) does not match for %s, reuploading.r� �rd rL ��sourceZ destinationZextra).�type� ValueError�io�open�sys�stdin�fileno� stream_namer r �statr r �BytesIOr �InvalidFileErrorr �IOError�OSError�strerrorr r� rM �server_side_encryption�kms_keyr r� r� �enable_multipartr� Zmultipart_chunk_size_mb�SIZE_1MBZmultipart_max_chunks�send_file_multipartZ put_continue� object_info� Exception�intr� r- � send_chunkr rC r� rd � send_file)rS r rd r� � extra_labelr �sizeZ is_streamZ src_streamZfilename_bytesr3 r r� rL � multipartr Z remote_sizeZremote_checksumZchecksumr� �labelsr� r4 r4 r5 � object_put� s� $ � �� z S3.object_putc C sJ |j dkrtd|j ��| jd|d�}|�� ||d�}| �||||�}|S )NrK r r� �rd r) )r+ r, r� rd � recv_file) rS rd �streamZ dest_name�start_positionrC r� rF r� r4 r4 r5 � object_getO s z S3.object_getc s � fdd�� D �}| � |�S )z" Batch delete given a remote_list c s g | ]}� | d �qS )Zobject_uri_strr4 )�.0�item�� remote_listr4 r5 � <listcomp>Y r z*S3.object_batch_delete.<locals>.<listcomp>)� object_batch_delete_uri_strs)rS rP �urisr4 rO r5 �object_batch_deleteW s zS3.object_batch_deletec C sv dd� }|}t |�dkr td��t|d ��� }|||�}tt|�dd�dd�}| jd |||d d id �}| �|�}|S ) z* Batch delete given a list of object uris c S s� d}|D ]f}t |�}|jdkr,td|j ��|�� s@td| ��|�� | krTtd��t�|�� �}|d| 7 }q|d7 }t|�}|S )Nz.<?xml version="1.0" encoding="UTF-8"?><Delete>rK r zURI '%s' has no objectz2The batch should contain keys from the same bucketz <Object><Key>%s</Key></Object>z </Delete>) r% r+ r, � has_objectrb r �escape�objectr )rb Zkey_listrQ r� rd rW r4 r4 r5 �compose_batch_del_xml^ s z>S3.object_batch_delete_uri_strs.<locals>.compose_batch_del_xmlr zKey list is empty�application/xml)r� r# TrJ r� �deleteNr )rE r, r% rb r r+ r� r� ) rS rS rX Zbatchrb Z request_bodyrL r� r� r4 r4 r5 rR \ s" ��� z S3.object_batch_delete_uri_strsc C s4 |j dkrtd|j ��| jd|d�}| �|�}|S )NrK r r� rH )r+ r, r� r� �rS rd r� r� r4 r4 r5 � object_delete{ s zS3.object_deletec C s� |j dkrtd|j ��| jjdk r,td��| jjdvr@td��d}|d| jj 7 }|d 7 }|d | jj 7 }|d 7 }|d 7 }| jd ||dd id�}| �|�}td| � |S )NrK r r z*You must restore a file for 1 or more days)ZStandardZ ExpeditedZBulkz7Valid restoration priorities: bulk, standard, expeditedz@<RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/">z <Days>%s</Days>z <GlacierJobParameters>z <Tier>%s</Tier>z </GlacierJobParameters>z</RestoreRequest>r� Zrestore�rd rQ r� r� ) r+ r, rM Z restore_daysr� Zrestore_priorityr� r� r )rS rd rQ r� r� r4 r4 r5 �object_restore� s$ � zS3.object_restorec C s4 g d�}|| j j D ]}|�� |v r||�� = q|S )N)rZ r% z last-modifiedr� zx-amz-version-idzx-amz-delete-markerz accept-rangesZ connectionr& Zserverz x-amz-id-2zx-amz-request-idzcf-rayr$ )rM Zremove_headersr� )rS rL � to_remove�hr4 r4 r5 �_sanitize_headers� s zS3._sanitize_headersc C s� |j dkrtd|j ��|j dkr0td|j ��| jjdu r�z| �|�}W n6 ty� } z |jdkrh|�d}W Y d}~n d}~0 0 d} d} |s�| jjr�d}|r�| �|�} | d } t | d �}| jj �rL| } |du r�| �|�} | d } t | d �}||u �rt j t } n | jjt } || k�rL| du �rD| �|�} | d } t | d �}| } d} | �rj| �| � t| dd �} n tdd �} | jj�r�d | d < | �� | d < | jj�r�d | d< | jj�r�d| d< | jj| d< |�r�| �|� | jj�r�| jj| d< |�s�d| d< nd| d< | �r| �|||| |�}n<td|�� |�� f ddd�| d< | jd|| d�}| �|�}|d �r�t|d �dk�r�d |d < t d � t|��| jjdu �r�|�r�z| � ||� W n6 t�y� } z |jdk�r�|�W Y d}~n d}~0 0 |S ) a Remote copy an object and eventually set metadata Note: A little memo description of the nightmare for performance here: ** FOR AWS, 2 cases: - COPY will copy the metadata of the source to dest, but you can't modify them. Any additional header will be ignored anyway. - REPLACE will set the additional metadata headers that are provided but will not copy any of the source headers. So, to add to existing meta during copy, you have to do an object_info to get original source headers, then modify, then use REPLACE for the copy operation. ** For Minio and maybe other implementations: - if additional headers are sent, they will be set to the destination on top of source original meta in all cases COPY and REPLACE. It is a nice behavior except that it is different of the aws one. As it was still too easy, there is another catch: In all cases, for multipart copies, metadata data are never copied from the source. rK r Nr� FTrL r% rJ r� r� r$ r r r! r" r# ZCOPYzx-amz-metadata-directiveZREPLACEz/%s/%srr zx-amz-copy-sourcer� r( r� �Error�� r� zHServer error during the COPY operation. Overwrite response status to 500) r+ r, rM r� �get_aclr� r� r r> r@ r; r$ ZMAX_CHUNK_SIZE_MBr< Z multipart_copy_chunk_size_mbra r r� r9 r: r� �copy_file_multipartr rb rW r� r� r r �set_acl)rS �src_uri�dst_urir� �src_sizerC � replace_meta�acl�excrE rL Zsrc_infoZ src_headersZ thresholdr� r� r4 r4 r5 � object_copy� s� � �� � zS3.object_copyc C s | j |||||dd�S )NT)rj )rm )rS rg rh r� ri rC r4 r4 r5 � object_modifyH s �zS3.object_modifyc C s^ | � |||||�}td||f � |d r:t|d �dv rP| �|� td|� n td|� |S )NzObject %s copied to %sr� )ZCopyObjectResultZ CompleteMultipartUploadResultzObject '%s' deletedzGObject '%s' NOT deleted because of an unexpected response data content.)rm r r r\ r )rS rg rh r� ri rC Z response_copyr4 r4 r5 � object_moveN s � �� �zS3.object_movec C sl | j d|d�}z| �|�}W nJ tyf } z2|jdkrN|jsNd|_d|_||_|�W Y d }~n d }~0 0 |S )Nr� rH r� Z NoSuchKeyz!The specified key does not exist.)r� r� r� r� �code�messagerN )rS rd r� r� rl r4 r4 r5 r> ] s zS3.object_infoc C sP |� � r | jd|dd id�}n| jd|�� dd id�}| �|�}t|d �}|S )Nr� rk �rd r� r� r� r� )rU r� rb r� r )rS rd r� r� rk r4 r4 r5 rd l s � � z S3.get_aclc C st d| }t d||f � tddidd�}|�� rJ| jd|||dd id �}n | jd |�� ||dd id �}| �|�}|S ) Nr zset_acl(%s): acl-xml: %sr# rY TrJ r� rk r� r� r )r r rU r� rb r� )rS rd rk rQ rL r� r� r4 r4 r5 rf x s � � z S3.set_aclc C sp t dd�}|du rdnd}d}|d| 7 }|d7 }td| � t|�|d < | jd |||d d id �}| �|�}|S ) NTrJ ZEnabledZ SuspendedzI<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">�<Status>%s</Status>z</VersioningConfiguration>zset_versioning(%s)r� r� r� r� r� )rS rd ZenabledrL r� rQ r� r� r4 r4 r5 �set_versioning� s � zS3.set_versioningc C s, | j d|dd id�}| �|�}t|d d�S )Nr� r� rr r� ZStatus)r� r� r r[ r4 r4 r5 r� � s � zS3.get_versioningc C s. | j d|�� dd id�}| �|�}t|d �S )Nr� �policyr� r� �r� rb r� r r[ r4 r4 r5 � get_policy� s � z S3.get_policyc C s\ d}|d| 7 }|d7 }t dd�}d|d< t|�|d< | jd |||d d id �}| �|�}|S ) Nz;<LegalHold xmlns="http://s3.amazonaws.com/doc/2006-03-01/">rs z </LegalHold>TrJ rY r# r� r� z legal-holdr� �r r+ r� r� )rS rd Zlegal_hold_statusrQ rL r� r� r4 r4 r5 �set_object_legal_hold� s � zS3.set_object_legal_holdc C sh d}|d| 7 }|d| 7 }|d7 }t dd�}d|d< t|�|d < | jd |||d d id �}| �|�}|S ) Nz;<Retention xmlns="http://s3.amazonaws.com/doc/2006-03-01/">z<Mode>%s</Mode>z%<RetainUntilDate>%s</RetainUntilDate>z </Retention>TrJ rY r# r� r� Z retentionr� rx )rS rd r Zretain_until_daterQ rL r� r� r4 r4 r5 �set_object_retention� s � zS3.set_object_retentionc C s8 t dd�}d|d< | jd|||dd id�}| �|�}|S )NTrJ zapplication/jsonr# r� ru r� )r r� r� �rS rd ru rL r� r� r4 r4 r5 � set_policy� s � z S3.set_policyc C s. | j d|dd id�}td| � | �|�}|S )Nr� ru rr zdelete_policy(%s)�r� r r� r[ r4 r4 r5 � delete_policy� s � zS3.delete_policyc C s. | j d|�� dd id�}| �|�}t|d �S )Nr� �corsr� r� rv r[ r4 r4 r5 �get_cors� s � z S3.get_corsc C sD t dd�}d|d< t|�|d< | jd|||dd id�}| �|�}|S ) NTrJ rY r# r� r� r r� rx )rS rd r rL r� r� r4 r4 r5 �set_cors� s � z S3.set_corsc C s. | j d|dd id�}td| � | �|�}|S )Nr� r rr zdelete_cors(%s)r} r[ r4 r4 r5 � delete_cors� s � zS3.delete_corsc C sL t dd�}t|�|d< | jd|||dd id�}td||f � | �|�}|S )NTrJ r� r� r� r� z(set_lifecycle_policy(%s): policy-xml: %s)r r+ r� r r� r{ r4 r4 r5 �set_lifecycle_policy� s � zS3.set_lifecycle_policyc C s\ t dd�}d|d< d}| jjr(|d7 }n|d7 }|d7 }| jd ||d d id �}| �|�}|S ) NTrJ rY r# zN<RequestPaymentConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> z<Payer>Requester</Payer> z<Payer>BucketOwner</Payer> z </RequestPaymentConfiguration> r� r� r] )r rM rR r� r� )rS rd rL rQ r� r� r4 r4 r5 � set_payer� s � z S3.set_payerc C sB | j d|�� dd id�}td| � | �|�}td|d � |S )Nr� r� r� zget_lifecycle_policy(%s)z%s: Got Lifecycle Policyr� �r� rb r r� r[ r4 r4 r5 �get_lifecycle_policy s � zS3.get_lifecycle_policyc C s. | j d|dd id�}td| � | �|�}|S )Nr� r� rr zdelete_lifecycle_policy(%s)r} r[ r4 r4 r5 �delete_lifecycle_policy s � zS3.delete_lifecycle_policyc C sP t dd�}| jjrd|d< | jd|||dd id�}td||f � | �|�}|S ) NTrJ �Truez!x-amz-skip-destination-validationr� � notificationr� z+set_notification_policy(%s): policy-xml: %s)r rM Zskip_destination_validationr� r r� r{ r4 r4 r5 �set_notification_policy s � zS3.set_notification_policyc C sB | j d|�� dd id�}td| � | �|�}td|d � |S )Nr� r� r� zget_notification_policy(%s)z%s: Got notification Policyr� r� r[ r4 r4 r5 �get_notification_policy s � zS3.get_notification_policyc C s d}| � ||�S )Nz7<NotificationConfiguration></NotificationConfiguration>)r� )rS rd Z empty_configr4 r4 r5 �delete_notification_policy$ s z S3.delete_notification_policyc C s� |j dkrtd|j ��d}|d7 }|D ]0\}}|d7 }|d| 7 }|d| 7 }|d7 }q(|d 7 }|d 7 }td d �}t|�|d < |�� r�| jd|||dd id�}n | jd|�� ||dd id�}td||f � | �|�}|S )NrK r z9<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">z<TagSet>z<Tag>r� z <Value>%s</Value>z</Tag>z </TagSet>z </Tagging>TrJ r� r� �taggingr� r� r z set_tagging(%s): tagset-xml: %s) r+ r, r r+ rU r� rb r r� ) rS rd �tagsetsrQ r� r� rL r� r� r4 r4 r5 � set_tagging( s2 � � zS3.set_taggingc C sr |� � r | jd|dd id�}n| jd|�� dd id�}td| � | �|�}|d }t|d�}td |d � |S ) Nr� r� rr r� r� zget_tagging(%s)r� ZTagz%s: Got object taggingr� )rU r� rb r r� r )rS rd r� r� �xml_datar� r4 r4 r5 � get_taggingB s � � zS3.get_taggingc C sP |� � r | jd|dd id�}n| jd|�� dd id�}td| � | �|�}|S )Nr� r� rr r� r� zdelete_tagging(%s))rU r� rb r r� r[ r4 r4 r5 �delete_taggingQ s � � zS3.delete_taggingc C s* g }| � |||�D ]\}}|�|� q|S r0 )�get_multipart_streamingr� )rS rd r� r� � upload_listr� �uploadsr4 r4 r5 � get_multipart\ s � zS3.get_multipartc c s4 |r |� � pi }|�� }d}d}|}d |d< |�r0| j|d||d�}|d } t| d�} |t| �7 }||krr|| }t| d�} | r�| �� dkr�d }|�r$|d ks�||k �r| r�t| d �} | s�| d d } | |d < t| d�} | r�| |d< nd|v r�|d= nd | fV �q0td|d � n|| fV �q0|| fV q,d S )NTr r� )r� r� r� r� ZUploadr� r� Fr� Z NextKeyMarkerr� Z KeyMarkerZNextUploadIdMarkerZUploadIdMarkerr� )r� rb r� r rE r r� r )rS rd r� r� rb r� r� r� r� r� r� � xml_truncatedZnext_keyZupload_id_markerr4 r4 r5 r� e sR � � � zS3.get_multipart_streamingc C s, g }| � ||||�D ]\}}|�|� q|S r0 )�list_multipart_streamingr� )rS rd � upload_idr� r� � part_listr� r r4 r4 r5 �list_multipart� s � zS3.list_multipartc c s� |r |� � pi }d}d}|}|r�| �||||�}|d } t| d�} |t| �7 }||kr^|| }t| d�} | rx| �� dkr|d}|r�|dks�||k r�| r�t| d �} | s�| d d } | |d < n d| fV q�td |d � n || fV q�|| fV q d S ) NTr r� ZPartr� r� Fr� ZNextPartNumberMarkerZ PartNumberzpart-number-markerz!Listing continues after Part '%s')r� �list_multipart_noparser rE r r� r ) rS rd r� r� r� r� r� � max_partsr� r� r� r� Znext_part_numberr4 r4 r5 r� � sB � � � zS3.list_multipart_streamingc C sF |d u r i }|dkr t |�|d< ||d< | jd||d�}| �|�}|S )Nr� z max-parts�uploadIdr� rr )rC r� r� )rS rd r� r� r� r� r� r4 r4 r5 r� � s � zS3.list_multipart_noparsec C s" | j d|d|id�}| �|�}|S )Nr� r� rr r� )rS rd �idr� r� r4 r4 r5 �abort_multipart� s � zS3.abort_multipartc C s2 | j d|�� dd id�}| �|�}t|d �}|S )Nr� �loggingr� r� )r� rb r� r )rS rd r� r� � accesslogr4 r4 r5 � get_accesslog� s � zS3.get_accesslogc C s^ | � |�}td|�� |f � |�td�� |�td�� td|�� |f � | �||� d S )NzCurrent ACL(%s): %sZREAD_ACPZWRITEzUpdated ACL(%s): %s)rd r rd Z appendGranteer rf )rS rd rk r4 r4 r5 �set_accesslog_acl� s zS3.set_accesslog_aclc C s� t � }|r |�|� |�|� n|�� d| }td||f � | jd|�� |dd id�}z| �|�}W n^ ty� } zF| j d dkr�t d� | � t d |�� �� | �|�}n� W Y d } ~ n d } ~ 0 0 ||fS ) Nr z$set_accesslog(%s): accesslog-xml: %sr� r� r� �CodeZ InvalidTargetBucketForLoggingz.Setting up log-delivery ACL for target bucket.zs3://%s) r Z enableLoggingZ setAclPublicZdisableLoggingr r� rb r� r� r r� r% ) rS rd �enableZlog_target_prefix_urir� r� rQ r� r� r� r4 r4 r5 � set_accesslog� s& � zS3.set_accesslogc C s� d dd�}|r |s|r t d��|r>|�� }|�� r:|�� p<d }|rZ||d< |rZd| |d< tj�tj| tjd @ �} t| | ||||�} t d|d � | S )Nr� )rb rd z>Both 'uri' and either 'bucket' or 'object' parameters suppliedrb rd r~ z CreateRequest: resource[uri]=%s) r, rb rU rW rx � http_methodsZgetkey� operationsrI r ) rS Z operationrd rb rW rL rQ r� rN rO r� r4 r4 r5 r� s zS3.create_requestc C s | j j| d d S )Nr � )rM � max_retries)rS �retriesr4 r4 r5 � _fail_wait s z S3._fail_waitc O s8 |d � d�}d|v r~t|d �dkr~t|d d�}t|d d�}| �||� td|� |rp|tj|< td|� ||i |��S |jd k�r,|d � d �} | �r"| �d �r�| d d � } n| �d �r�| dd � } t d | �j } |j d }| �|| � td| � |�r|tj|< td|� ||i |��S t d|� t |��d S )NrL zx-amz-bucket-regionr� r z .//Bucketz .//EndpointzRedirected to: %szRedirected to region: %srX r� zhttp://� zhttps://rz rb z�Redirection error: No info provided by the server to where should be forwarded the request (HEAD request). (Hint target region: %s))rg rE r r� r rI rl rO rj r rp rN r r� ) rS r� r� �fn�args�kwargsZ redir_regionZ redir_bucketr� Z location_urlr4 r4 r5 �_http_redirection_handler# s6 z S3._http_redirection_handlerc O s d|v r�t |d �dkr�t|d d�}|dkrvt|d d�}|durl|tj|jd < td|� ||i |��S td � nn|d kr�t|d d �}|d kr�td � d| _||i |��S n2|dkr�|� � s�| j s�td� d| _ ||i |��S n.|� � �s| j �std� d| _ ||i |��S dS )zR Returns None if no handler available for the specific error code r� r r� Z AuthorizationHeaderMalformedZRegionNrb zForwarding request to %szVCould not determine bucket the location. Please consider using the --region parameter.ZInvalidRequest�Messagez\The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.z Endpoint requires signature v4TZInvalidArgument� Falling back to signature v2) rE r rI rl rN r r r r_ ra r` ) rS r� r� r� r� r� � failureCode�regionrq r4 r4 r5 �_http_400_handlerH s2 zS3._http_400_handlerc O sz d|v rnt |d �dkrnt|d d�}|dkrnt|d d�}|dkrn|�� sn| jsntd� d| _||i |��S t|��d S ) Nr� r r� r� r� z=AWS authentication requires a valid Date or x-amz-date headerr� T)rE r ra r` r r� )rS r� r� r� r� r� r� rq r4 r4 r5 �_http_403_handlerr s zS3._http_403_handlerc C s� |j �d�r�|�� s�tj�|j d t� j�dkr�td� zFtd|j d �}| j |dd�}|durr|tj|j d < td|� W n6 t y� } z td |� td � W Y d}~n d}~0 0 dS ) a? Get and update region for the request if needed. Signature v4 needs the region of the bucket or the request will fail with the indication of the correct region. We are trying to avoid this failure by pre-emptively getting the correct region to use, if not provided by the user. rb r� z====== SEND Inner request to determine the bucket region =====zs3://T)r� NzE===== SUCCESS Inner request to determine the bucket region (%r) =====z,getlocation inner request failure reason: %sz?===== FAILED Inner request to determine the bucket region =====) rN rg ra rI rl r" rm r r% r� r? )rS r� Zs3_urir� rl r4 r4 r5 �update_region_inner_request� s& ���� z S3.update_region_inner_requestc C s� |d u r| j j}| �|� t|j�|_|j}|�� \}}}i }td� d }z�t� | � |d ��}| � ||j �}td|||t |jp�d�f � |j�|||j|� |j�� } | j|d< | j|d< t| �� �|d< | �� |d< d |d v r�t|d d �} | |d < t�|� W �n< ttf�y( � Y �n" ttf�y* } z�td t � |� � t | d ��rz| j �rz| j t j t j!t j"fv�s�d t#| �v �s�dt#| �v �r�t$| t%��s�� t�&|� |�rt'd|d | f � t'd| �(|� � t)�*| �(|�� | �+||d �W Y d } ~ S t,d|d ��W Y d } ~ n(d } ~ 0 td t � |� � � Y n0 td t � |� � |d dv �r|| �-||| j+|�S |d dk�r | �.||| j+|�} | �r�| S t/|�} |�r| j0dv �rt'd|d | f � t'd| �(|� � t)�*| �(|�� | �+||d �S | �|d dk�r*| �1||| j+|�S |d dk�r@t/|��|d dk�s\|d dk�r�t/|�} |d dk�rvd }|�r�t'd|d | f � t'd| �(|� � t)�*| �(|�� | �+||d �S | �|d d k �s�|d d k�r�t/|��|S ) Nz"Processing request, please wait...rb zESending request method_string=%r, uri=%r, headers=%r, body=(%i bytes)re r� �reasonrL r� �x-amz-meta-s3cmd-attrs� s3cmd-attrs� Response: �errno� [Errno 104]� [Errno 32]� Retrying failed request: %s (%s)rd �Waiting %d sec...r zRequest failed for: %s�i- i3 � �Z BadDigestZOperationAbortedZTokenRefreshRequiredZRequestTimeout� i� rc � r� r �� �+ )2rM r� r� r rQ rL rt r r& rg rh r� r rE �cr� � getresponser� r� r � getheaders�read�parse_attrs_header�putZ S3SSLErrorZS3SSLCertificateErrorr6 r? �pprint�pformat�hasattrr� �EPIPE� ECONNRESET� ETIMEDOUTrC rB �SocketTimeoutException�closer r� r[ �sleepr� ZS3RequestErrorr� r� r� rp r� )rS r� r� rL rO rN r� �connrd � http_response�attrsr� � handler_fn�errr4 r4 r5 r� � s� � � �� $ zS3.send_requestc C s� | j jstd� | �|�S d|vr*d|d< | j �||�}z| �|�}W n0 tyv } z|�d� � W Y d}~n d}~0 0 |j|d� |�d� |S )zrWrapper around send_request for slow requests. To be able to show progression for small requests z$Sending slow request, please wait...�actionr� �failedN)�current_position�done)rM �progress_meterr r� �progress_classr? r� r� )rS r� rF Zoperation_size�progressr� rl r4 r4 r5 �send_request_with_progress s z S3.send_request_with_progressc ) C s^ |d u r| j j}| �|� | d u r*| j j} | jr8| r8d} |j} t| d � } } |j} | j jrtd|d< | j � || �}n t d| � t � � }|r�t ||| �}n t ||| �}||_| r�| s�d} nd| d< |�� \}}} zdt�| �|d ��}|j�|| �||j�� | �� D ] }|j�t|�t| | �� �q|j�� W n� t�y\ } z� W Y d }~n�d }~0 t �y } z�| j j�r�|� d � |�r�t d |d |f � t d | � |� � t � | � |�� | �!||||||d ||�W Y d }~S t"d|d ��W Y d }~n d }~0 0 |dk�r |�#|� t$� }�z�d }| �r�t%�%|jj&gg g t'�\}}}|�r`|j�(� }n |�s�|�s�t dt'� d| _d} | �r�|�r�|j)tj*k�r�|�r�|�+� tj,|j_-| dk�r�t.| j j/| �}|dk�r�|�+|�} n|} | �s�t0d��| j j1dk�rt � � } |�2| � |j�3| � | j j�r8|j2t4| �d� | t4| �8 } |} | j j1dk�r�t � � | } t5|�| j j1 } t6| | | �} | �r�t � t.| | j j7�� �q�|�8� }!|j�(� }i }"|j)|"d< |j9|"d< t:|�;� �|"d< |�+� |"d< | |"d< t�<|� t=dt>�?|"� � W �n� t�y6 } z� W Y d }~�n�d }~0 t0�yr } z$| j j�rZ|� d � � W Y d }~�n�d }~0 t �y� } �zr| j j�r�|� d � |�r�d}#t@|d��r�|jA�r�|jAtAjBtAjCtAjDfv�s�d tE|�v �s�d tE|�v �rbtF|tG��sbzN|j�(� }i }"|j)|"d< |j9|"d< t:|�;� �|"d< |�+� |"d< | |"d< d}#W n t �y` tHd � Y n0 |#�s�t d |d |f � t d | � |� � t � | � |�� | �!||||||d ||| � W Y d }~S n t=d | |f � t"d|d ��W Y d }~n d }~0 0 t � � }$|$| |"d!< |"d! �r4t5|"d �|"d! �p:t5d"�|"d#< | j j�r\|�2� |� d$� |"d d%v �r�| jI||"| j!||||||| d&� S |"d d'k�r�| jJ||"| j!||||||| d&� }%|%�r�|%S tK|"�}&|&jLd(v�r�|&�|"d d)k�r| jM||"| j!||||||| d&� S |"d d*k�rB|�rBd| _| j!||||||d ||dd+� S d,|"d v�r\d|"d d,< |"d d-k �sx|"d d.k�r�d}'|"d d/k�r�d}'|"d d0k�r�|�r�|d1 �p�d2}n|"d d3k�rd}'nN|"d d4k�r�d}'|�r�|d1 �p�d2}n&|"d d'k�rtK|"�}&|&jLd(v �rd}'tK|"�}&|'�r�|�r�t d |d |&f � |�rNt d5| � t d | � |� � t � | � |�� | �!||||||d ||| � S t d6| � t"d7|& ��|&�t=d8|!|"d �d,d��Nd9�f � |"d �d,d��Nd9�}(d:|(v� rZ|(|�8� k� rZ|"d �d;�d<k� rZt d=� |� rFt d>| � | �!||||||d ||| � S t d6| � t"d?��|"S )@NFr% �uploadr� z!Sending file '%s', please wait...z 100-continue�expectrb r� r� rd r� r zUpload failed for: %sre zQHTTP Expect Continue feature disabled because of no reply of the server in %.2fs.Tr z3File smaller than expected. Was the file truncated?�Zdelta_positionr� r� rL r� rD r� r� r� r� zXCannot retrieve any response status before encountering an EPIPE or ECONNRESET exceptionzUpload failed: %s (%s)zGiving up on '%s' %s�elapsedr� �speedr� r� )�offset� chunk_size�use_expect_continuer� r� r� i� )r� r& r� r� rc � � g{�G�z�?i� r� z(Retrying on lower speed (throttle=%0.2f)�$Too many failures. Giving up on '%s'r z"MD5 sums: computed=%s, received=%sr' r r r! zMD5 Sums don't match!zRetrying upload of %sz0MD5 sums of sent and received files don't match!)OrM r� r� Zuse_http_expectr� rL r@ r2 r� r� r r[ r* r) rQ rt r& rg rh r� � putrequestr� r �keys� putheaderr � endheadersr� r? r� r r� r� rB � S3UploadError�seekr �selectZsock�EXPECT_CONTINUE_TIMEOUTr� r� ZCONTINUEr� Z _CS_REQ_SENTZ_HTTPConnection__state�minrA r5 � limitrater� Zwrapper_send_bodyrE �float�maxZ throttle_max� hexdigestr� r r� r� r r� r� r� r� r� r� r� rC rB r� r r� r� r� rp r� r� ))rS r� rJ rF �bufferZthrottler� r� r� r� rL � size_left� size_totalr r� �timestamp_startZ sha256_hashrO rN r� �headerr� �md5_hashr� �readable�writableZ exceptional�lr� � start_timeZlimitrate_throttle� real_duration�expected_durationZ md5_computedr� Z known_error� timestamp_endr� r� � try_retry� md5_from_s3r4 r4 r5 rB s� (& � � �� �& , �� � � � ", � z S3.send_filec C s� t � � }t| ||||�}|�|� |�� }t � � } | | |d< ||d< |d rbt|d �|d phtd�|d< |d r�t|d �dkr�tt|d d���|S )Nr� rD r� r� r� rb r� )r[ r$ Zupload_all_partsZcomplete_multipart_uploadr� r r� r ) rS rJ rL rd rD rC r r� r� r r4 r4 r5 r= s (zS3.send_file_multipartc C s | � |||||�S r0 )r= )rS rg rh rD rL rC r4 r4 r5 re / s �zS3.copy_file_multipartc ! C sn |d u r| j j}| �|� |�� \}}}|j} | j jrNd|d< | j �|d�} n td| � t�� } d } z�t � | � |d ��} | j � || �|| j�� |�� D ] } | j �t| �t|| �� q�|dkr�td| � | j �dd| � | j �� i }| j �� }|j|d < |j|d < t|�� �|d < d |d v �rDt|d d �}||d < tdt� |� � W �n6 t �y� } z� W Y d }~�nd }~0 t t f�y� } z�| j j�r�| � d� t!|d��r�|j"�r�|j"t"j#t"j$t"j%fv�s�dt&|�v �s�dt&|�v �rt'|t(��s� t �)| � |�rlt*d|d |f � t*d| �+|� � t�,| �+|�� | j-|||||d d�W Y d }~S t.d|d ��W Y d }~n d }~0 0 |d dk �s�|d dk�r�|�/� |d< |d d v �r�| �0||| j-||||�S |d d k�r| �1||| j-||||�}|�r|S t2|��|d d k�r>| �3||| j-||||�S |d dk �sZ|d dk�rd }|d d k�rrd!}n|d d"k�r�d!}|�r�|d }|�r�| �+|�}t*d|t2|�f � t*d| � t�,|� | j-|||||d d�S t*d#| � t.d| ��t2|��|dk�rt4� }t5|d d$ �}|| }|}| j j�rL|| _6|| _7|| _8�z |dk�rv|�/d�}t9|�dk�svJ �||k �r\|| j j:k�r�| j j:�p�|}| j j;dk�r�t�� } |�/|�}t9|�dk�r�t<d%��| j j;dk�rt�� | } t=|�| j j; } | | k�rt�,| | � |�>|� |dk�r2|�?|� |t9|�7 }| j j�rv| j?t9|�d&� �qvt �@| � W �n$ tA�y� � Y �n t t f�y� } z�| j j�r�| � d� t!|d��r�|j"�r�|j"t"j#t"j$t"j%fv�s�dt&|�v �s�dt&|�v �r t'|t(��s � t �)| � |�rjt*d|d |f � t*d| �+|� � t�,| �+|�� | j-|||||d d�W Y d }~S t.d|d ��W Y d }~n d }~0 0 |�B� t�� } | j j�r�| �?� | � d'� |d � d(d)��Cd*�} d+|d v�rz|d d, } W n tD�y Y n0 d-| v�r�|dk�r(|�E� |d,< n^ztF| �|d,< W nL t �y� } z2|j"t"jGk�rht*d.| |f � t*d/� W Y d }~n d }~0 0 |� d,�| k|d0< | | |d1< ||d2< |d1 �r�t=|d2 �|d1 �p�t=d3�|d4< |d2 |t5|d d$ � k�rt*d5|t5|d d$ � |d2 f � td6|� d,� � d-| v�rj|d0 �sj|d � d7�d8k�rjt*d9|� d,�| f � |S ):NZdownloadr� r z#Receiving file '%s', please wait...rb zRequesting Range: %d .. endZRangez bytes=%d-r� r� rL r� r� r� r� r� r� r� r� rd r� r )r� zDownload failed for: %sr� r� r� r� r� r� Fr� Tr� r� r% z EOF from S3!r� r� r& re r' zx-amz-meta-s3tools-gpgencr r zUnable to open file: %s: %sz(Unable to verify MD5. Assume it matches.Zmd5matchr� rD r� r� z4Reported size (%s) does not match received size (%s)z ReceiveFile: Computed MD5 = %sr r! z5MD5 signatures do not match: computed=%s, received=%s)HrM r� r� rt r2 r� r� r r[ r& rg rh r� r� r� r r� r� r r r� r� r� r� r r� r� r� r� r� r6 r? r� r� r� r� r� r� rC rB r� r� r r� r� rI ZS3DownloadErrorr� r� r� r� r� r r@ Z total_sizeZinitial_positionr� rE Z recv_chunkr� r� r� �writer� r� r7 �flushr� �KeyErrorr� r, �ENOENT)!rS r� rJ rF rK r� rO rN rL r r� r r� r r� r� r� r� r� r ro Z retry_delayr r� r r� r� Z this_chunkr r r r r r4 r4 r5 rI 4 sl � � �� �& � � � � � � � �� �& , �( �z S3.recv_file)N)NNNr� )NNNr� )NNNr� )NN)F)N)N)N)N)N)NF)Nre )r re )NNre F)NNre )NNre )Nr� )Nr� )Nr� )Nr� )Nr� )NF)NNNNre N)N)r )re r Nr r� N)re )re )r N)Zru rv rw r! r� �targetsr� ZcodesrT r� rh r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r rG rL rT rR r\ r^ ra rm rn ro r> rd rf rt r� rw ry rz r| r~ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� rB r= re rI r4 r4 r4 r5 rx � s � ��� 5 ! f � � � 6 � + � %* h � � rx c C s. i }| � d�D ]}|� d�\}}|||< q|S )Nr� �:)rD )Z attrs_headerr� �attrr� r� r4 r4 r5 r� s r� )ZZ __future__r r r/ r r[ r� r= r- r� Zxml.saxr Zsocketr r� r� r r r r r3 r r r r r � ImportError� urllib.parser� Z BaseUtilsr r r r r r r ZUtilsr r r r r r r r r r r! r" Z ExceptionsZ MultiPartr$ r% r& ZCryptor' r( r) r* r+ r, r- r. Zctypesr/ r: ZMagicr1 r6 � TypeErrorr. Z MAGIC_MIME�load�AttributeErrorr7 r� rC Z error_strr< r; rG r� r< �__all__rW rI rx �appendr� r4 r4 r4 r5 �<module> s� $ ( P I