OXIESEC PANEL
- Current Dir:
/
/
opt
/
gsutil
/
gslib
/
commands
/
__pycache__
Server IP: 2a02:4780:11:1594:0:ef5:22d7:a
Upload:
Create Dir:
Name
Size
Modified
Perms
📁
..
-
02/11/2025 08:19:49 AM
rwxr-xr-x
📄
__init__.cpython-39.pyc
323 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
acl.cpython-39.pyc
18.76 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
autoclass.cpython-39.pyc
6.01 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
bucketpolicyonly.cpython-39.pyc
6.75 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
cat.cpython-39.pyc
4.17 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
compose.cpython-39.pyc
4.57 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
config.cpython-39.pyc
40.97 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
cors.cpython-39.pyc
6.56 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
cp.cpython-39.pyc
42.21 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
defacl.cpython-39.pyc
11.25 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
defstorageclass.cpython-39.pyc
5.78 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
du.cpython-39.pyc
8.47 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
hash.cpython-39.pyc
7.87 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
help.cpython-39.pyc
6.41 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
hmac.cpython-39.pyc
12.39 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
iam.cpython-39.pyc
24.6 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
kms.cpython-39.pyc
14.88 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
label.cpython-39.pyc
10.46 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
lifecycle.cpython-39.pyc
6.37 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
logging.cpython-39.pyc
9.04 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
ls.cpython-39.pyc
18.81 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
mb.cpython-39.pyc
12.23 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
mv.cpython-39.pyc
5.06 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
notification.cpython-39.pyc
25.47 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
pap.cpython-39.pyc
6.53 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
perfdiag.cpython-39.pyc
62.18 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rb.cpython-39.pyc
3.78 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
requesterpays.cpython-39.pyc
5.53 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
retention.cpython-39.pyc
20.02 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rewrite.cpython-39.pyc
16.12 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rm.cpython-39.pyc
10.86 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rpo.cpython-39.pyc
5.96 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
rsync.cpython-39.pyc
53.86 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
setmeta.cpython-39.pyc
11.25 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
signurl.cpython-39.pyc
20.99 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
stat.cpython-39.pyc
5.04 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
test.cpython-39.pyc
17.73 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
ubla.cpython-39.pyc
6.96 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
update.cpython-39.pyc
12.26 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
version.cpython-39.pyc
5.25 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
versioning.cpython-39.pyc
5.51 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
web.cpython-39.pyc
7.63 KB
02/11/2025 08:19:49 AM
rw-r--r--
Editing: perfdiag.cpython-39.pyc
Close
a +(Wg�u � @ sB d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlmZ ddlm Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZdd lm Z dd lm Z dd lm Z dd lm Z ddl Z dd l m!Z! dd l m"Z" ddl#m$Z$ ddl#m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+ ddl,m-Z- ddl.m/Z/ ddl0m1Z1 ddl2m3Z4 ddl5m6Z6 ddl7m8Z8 ddl7m9Z9 ddl:m;Z; ddl<m=Z= dd l<m>Z> dd l?m@Z@ dd lAmBZB dd lAmCZC dd lAmDZD dd!lAmEZE dd"lAmFZF dd#lGmHZH dd$lGmIZI dd%lGmJZJ dd&lGmKZK dd'lGmLZL d(ZMd)eM d* ZNe d+d,�ZOe d-d.�ZPe d/d0�ZQe d1d2�ZRe d3d4�ZSi ZTG d5d6� d6eU�ZVG d7d8� d8eV�ZWdQd9d:�ZXdRd;d<�ZYdSd=d>�ZZdTd?d@�Z[dUdAdB�Z\dCdD� Z]dEdF� Z^G dGdH� dHe_�Z`dVdKdL�ZaG dMdN� dNe$�ZbdOdP� ZcdS )Wz%Contains the perfdiag gsutil command.� )�absolute_import)�print_function)�division)�unicode_literalsN)� defaultdict)� namedtuple)� cStringIO)� http_client)�range)�NotFoundException)�ServiceException)�Command)�DummyArgChecker)�CommandArgument)�config)� ApiSelector)�CommandException)�FilePart)�StorageUrlFromString)�storage_v1_messages)� text_util)�GetMaxRetryDelay)�ResumableThreshold)�UTF8)�GetCloudApiInstance)� GetDownloadSerializationData)�"CalculateB64EncodedMd5FromContents)�CheckFreeSpace)�GetDiskCounters)� GetFileSize)�IS_LINUX)�IsRunningInCiEnvironment)� DivideAndCeil)�HumanReadableToBytes)�MakeBitsHumanReadable)�MakeHumanReadable)� Percentilez� gsutil perfdiag [-i in.json] gsutil perfdiag [-o out.json] [-n objects] [-c processes] [-k threads] [-p parallelism type] [-y slices] [-s size] [-d directory] [-t tests] [-j ratio] gs://<bucket_name>... z <B>SYNOPSIS</B> a�! <B>DESCRIPTION</B> The ``perfdiag`` command runs a suite of diagnostic tests for a given Cloud Storage bucket. The ``bucket_name`` parameter must name an existing bucket to which the user has write permission. Several test files will be uploaded to and downloaded from this bucket. All test files will be deleted at the completion of the diagnostic if it finishes successfully. For a list of relevant permissions, see `Cloud IAM permissions for gsutil commands <https://cloud.google.com/storage/docs/access-control/iam-gsutil>`_. gsutil performance can be influenced by a number of factors originating at the client, server, or network level. Some examples include the following: + CPU speed + Available memory + The access path to the local disk + Network bandwidth + Contention and error rates along the path between gsutil and Google servers + Operating system buffering configuration + Firewalls and other network elements The `perfdiag` command is provided so that customers can run a known measurement suite when troubleshooting performance problems. <B>PROVIDING DIAGNOSTIC OUTPUT TO THE CLOUD STORAGE TEAM</B> If the Cloud Storage team asks you to run a performance diagnostic please use the following command, and email the output file (output.json) to the @google.com address provided by the Cloud Storage team: gsutil perfdiag -o output.json gs://your-bucket Additional resources for discussing ``perfdiag`` results include the `Stack Overflow tag for Cloud Storage <https://stackoverflow.com/questions/tagged/google-cloud-storage>`_ and the `gsutil GitHub repository <https://github.com/GoogleCloudPlatform/gsutil/issues>`_. <B>OPTIONS</B> -n Sets the number of objects to use when downloading and uploading files during tests. Defaults to 5. -c Sets the number of `processes <https://en.wikipedia.org/wiki/Process_(computing)>`_ to use while running throughput experiments. The default value is 1. -k Sets the number of `threads <https://en.wikipedia.org/wiki/Thread_(computing)>`_ per process to use while running throughput experiments. Each process will receive an equal number of threads. The default value is 1. NOTE: All specified threads and processes will be created, but may not by saturated with work if too few objects (specified with ``-n``) and too few components (specified with ``-y``) are specified. -p Sets the type of parallelism to be used (only applicable when threads or processes are specified and threads * processes > 1). The default is to use ``fan``. Must be one of the following: fan Use one thread per object. This is akin to using gsutil ``-m cp``, with sliced object download / parallel composite upload disabled. slice Use Y (specified with ``-y``) threads for each object, transferring one object at a time. This is akin to using parallel object download / parallel composite upload, without ``-m``. Sliced uploads not supported for s3. both Use Y (specified with ``-y``) threads for each object, transferring multiple objects at a time. This is akin to simultaneously using sliced object download / parallel composite upload and ``gsutil -m cp``. Parallel composite uploads not supported for s3. -y Sets the number of slices to divide each file/object into while transferring data. Only applicable with the slice (or both) parallelism type. The default is 4 slices. -s Sets the size (in bytes) for each of the N (set with ``-n``) objects used in the read and write throughput tests. The default is 1 MiB. This can also be specified using byte suffixes such as 500K or 1M. NOTE: these values are interpreted as multiples of 1024 (K=1024, M=1024*1024, etc.) NOTE: If ``rthru_file`` or ``wthru_file`` are performed, N (set with ``-n``) times as much disk space as specified will be required for the operation. -d Sets the directory to store temporary local files in. If not specified, a default temporary directory will be used. -t Sets the list of diagnostic tests to perform. The default is to run the ``lat``, ``rthru``, and ``wthru`` diagnostic tests. Must be a comma-separated list containing one or more of the following: lat For N (set with ``-n``) objects, write the object, retrieve its metadata, read the object, and finally delete the object. Record the latency of each operation. list Write N (set with ``-n``) objects to the bucket, record how long it takes for the eventually consistent listing call to return the N objects in its result, delete the N objects, then record how long it takes listing to stop returning the N objects. rthru Runs N (set with ``-n``) read operations, with at most C (set with -c) reads outstanding at any given time. rthru_file The same as ``rthru``, but simultaneously writes data to the disk, to gauge the performance impact of the local disk on downloads. wthru Runs N (set with ``-n``) write operations, with at most C (set with ``-c``) writes outstanding at any given time. wthru_file The same as wthru, but simultaneously reads data from the disk, to gauge the performance impact of the local disk on uploads. -m Adds metadata to the result JSON file. Multiple ``-m`` values can be specified. Example: gsutil perfdiag -m "key1:val1" -m "key2:val2" gs://bucketname Each metadata key will be added to the top-level "metadata" dictionary in the output JSON file. -o Writes the results of the diagnostic to an output file. The output is a JSON file containing system information and performance diagnostic results. The file can be read and reported later using the ``-i`` option. -i Reads the JSON output file created using the ``-o`` command and prints a formatted description of the results. -j Applies gzip transport encoding and sets the target compression ratio for the generated test files. This ratio can be an integer between 0 and 100 (inclusive), with 0 generating a file with uniform data, and 100 generating random data. When you specify the ``-j`` option, files being uploaded are compressed in-memory and on-the-wire only. See `cp -j <https://cloud.google.com/storage/docs/gsutil/commands/cp#options>`_ for specific semantics. <B>MEASURING AVAILABILITY</B> The ``perfdiag`` command ignores the boto num_retries configuration parameter. Instead, it always retries on HTTP errors in the 500 range and keeps track of how many 500 errors were encountered during the test. The availability measurement is reported at the end of the test. Note that HTTP responses are only recorded when the request was made in a single process. When using multiple processes or threads, read and write throughput measurements are performed in an external process, so the availability numbers reported won't include the throughput measurements. <B>NOTE</B> The ``perfdiag`` command runs a series of tests that collects system information, such as the following: + Retrieves requester's IP address. + Executes DNS queries to Google servers and collects the results. + Collects network statistics information from the output of ``netstat -s`` and evaluates the BIOS product name string. + If a proxy server is configured, attempts to connect to it to retrieve the location and storage class of the bucket being used for performance testing. None of this information will be sent to Google unless you proactively choose to send it. � FileDataTuplez size md5 data�FanDownloadTuplez6need_to_slice object_name file_name serialization_data�SliceDownloadTuplez<object_name file_name serialization_data start_byte end_byte�FanUploadTuplez9need_to_slice file_name object_name use_file gzip_encoded�SliceUploadTuplez@file_name object_name use_file file_start file_size gzip_encodedc @ s e Zd ZdZdS )�Errorz%Base exception class for this module.N��__name__� __module__� __qualname__�__doc__� r2 r2 �&/opt/gsutil/gslib/commands/perfdiag.pyr, 2 s r, c @ s e Zd ZdZdS )�InvalidArgumentz)Raised on invalid arguments to functions.Nr- r2 r2 r2 r3 r4 7 s r4 c C s@ t | |�}|jr&| �|j|j|j� n| �|j||j|j� dS )a Function argument to apply for performing fanned parallel downloads. Args: cls: The calling PerfDiagCommand class instance. args: A FanDownloadTuple object describing this download. thread_state: gsutil Cloud API instance to use for the operation. N)r � need_to_slice�PerformSlicedDownload� object_name� file_name�serialization_data�Download��cls�args� thread_state� gsutil_apir2 r2 r3 �_DownloadObject< s ��r@ c C s, t | |�}| �|j||j|j|j|j� dS )a Function argument to apply for performing sliced downloads. Args: cls: The calling PerfDiagCommand class instance. args: A SliceDownloadTuple object describing this download. thread_state: gsutil Cloud API instance to use for the operation. N)r r: r7 r8 r9 � start_byte�end_byter; r2 r2 r3 �_DownloadSliceM s �rC c C sN t | |�}|jr.| j|j|j|j||jd� n | j|j|j||j|jd� dS )a Function argument to apply for performing fanned parallel uploads. Args: cls: The calling PerfDiagCommand class instance. args: A FanUploadTuple object describing this upload. thread_state: gsutil Cloud API instance to use for the operation. �� gzip_encodedN)r r5 �PerformSlicedUploadr8 r7 �use_filerE �Uploadr; r2 r2 r3 � _UploadObjectZ s ��rI c C s2 t | |�}| j|j|j||j|j|j|jd� dS )a Function argument to apply for performing sliced parallel uploads. Args: cls: The calling PerfDiagCommand class instance. args: A SliceUploadTuple object describing this upload. thread_state: gsutil Cloud API instance to use for the operation. rD N)r rH r8 r7 rG � file_start� file_sizerE r; r2 r2 r3 � _UploadSliceq s �rL c C s t | |�}| �||� dS )a Function argument to apply for performing parallel object deletions. Args: cls: The calling PerfDiagCommand class instance. object_name: The object name to delete from the test bucket. thread_state: gsutil Cloud API instance to use for the operation. N)r �Delete)r< r7 r> r? r2 r2 r3 �_DeleteWrapper� s rN c C s | j �t|�� dS �z9Simple exception handler to allow post-completion status.N��logger�error�str)r< �er2 r2 r3 �_PerfdiagExceptionHandler� s rU c C s d S �Nr2 )�_r2 r2 r3 �_DummyTrackerCallback� s rX c @ s$ e Zd ZdZdZdd� Zdd� ZdS )� DummyFilezDA dummy, file-like object that throws away everything written to it.Zbwc O s d S rV r2 )�selfr= �kwargsr2 r2 r3 �write� s zDummyFile.writec C s d S rV r2 �rZ r2 r2 r3 �close� s zDummyFile.closeN)r. r/ r0 r1 �moder\ r^ r2 r2 r2 r3 rY � s rY �d � P c C sx |d }t �t||��}d}||k rtt||| �}t|d| �}|| }| �|d|� � | �d| � ||7 }q dS )am Writes data into a file like object. Args: fp: A file like object to write the data to. file_size: The amount of data to write to the file. random_ratio: The percentage of randomly generated data to write. This can be any number between 0 and 100 (inclusive), with 0 producing uniform data, and 100 producing random data. max_unique_random_bytes: The maximum number of bytes to generate pseudo-randomly before beginning to repeat bytes. The default was chosen as the next prime larger than 5 MiB. g Y@r � N� x)�os�urandom�min�intr\ ) �fprK � random_ratioZmax_unique_random_bytesZ random_bytesZtotal_bytes_written� num_bytesZ num_bytes_seqZnum_bytes_randomr2 r2 r3 �_GenerateFileData� s rk c @ s� e Zd ZdZejdg d�eddddddejej gej e � d�gd� Z ej dg d d ei d �Zd Zd ZdZdZdZdZdZdZdZdZeeeeeefZeeefZeeefZ e j � dde j j!j"j#�Z$e j �%ddd�Z&dZ'dZ(d Z)d Z*dZd d �Z+d!d"� Z,d[d%d&�Z-d'd(� Z.d)d*� Z/e0j1d+d,� �Z2d-d.� Z3d/d0� Z4G d1d2� d2e5j6�Z7d3d4� Z8d5d6� Z9d7d8� Z:d\d9d:�Z;d]d;d<�Z<d^d=d>�Z=d_d?d@�Z>dAdB� Z?d`dDdE�Z@dadFdG�ZAdHdI� ZBdJdK� ZCdLdM� ZDdNdO� ZEdPdQ� ZFdRdS� ZGdTdU� ZHdVdW� ZIdXdY� ZJdCS )b�PerfDiagCommandz*Implementation of gsutil perfdiag command.�perfdiag)Zdiag� diagnosticZperfZ performancer rb zn:c:k:p:y:s:d:t:m:i:o:j:F) �command_name_aliases�usage_synopsis�min_args�max_args�supported_sub_args� file_url_ok�provider_url_ok�urls_start_arg�gs_api_support�gs_default_api�argparse_arguments� command_helpzRun performance diagnostic)� help_name�help_name_aliases� help_type�help_one_line_summary� help_text�subcommand_help_text)r i i � � ZrthruZ rthru_fileZwthruZ wthru_file�lat�list�fan�slice�both� Credentials�gs_host�gs_port�P � � i @ g N@Tc C s� | j �d|� |rtjnd}tj|tj|d�}|�� \}}tjrTt|t �rT|� t �}|rv|j rvt d|j d�|�f ��|r~|S |j S )a� Executes a command in a subprocess. Args: cmd: List containing the command to execute. raise_on_error: Whether or not to raise an exception when a process exits with a non-zero return code. return_output: If set to True, the return value of the function is the stdout of the process. mute_stderr: If set to True, the stderr of the process is not printed to the console. Returns: The return code of the process or the stdout if return_output is set. Raises: Exception: If raise_on_error is set to True and any process exits with a non-zero return code. zRunning command: %sN)�stdout�stderrz8Received non-zero return code (%d) from subprocess '%s'.� )rQ �debug� subprocess�PIPE�Popen� communicate�six�PY3� isinstance�bytes�decoder � returncoder �join) rZ �cmd�raise_on_error� return_output� mute_stderrr� �pZ stdoutdatarW r2 r2 r3 �_Exec s �zPerfDiagCommand._Execc C s$ | j | j td�kr | j�d� dS )zBOutputs a warning message if a large amount of data is being used.�2GiBz2This is a large operation, and could take a while.N)� num_objects� thru_filesizer# rQ �infor] r2 r2 r3 �_WarnIfLargeData< s z PerfDiagCommand._WarnIfLargeData�gsutil_test_filer` c C s� t jd|| jdd�\}}t�|d�� }t|||� W d � n1 sF0 Y |sX|r�t|d��P}|rpt|�nd}|r�t|�nd} |r�|� � nd} t || | �t |< W d � n1 s�0 Y | j � |� |S )a� Creates a temporary file of the given size and returns its path. Args: file_size: The size of the temporary file to create. mem_metadata: If true, store md5 and file size in memory at temp_file_dict[fpath].md5, tempfile_data[fpath].file_size. mem_data: If true, store the file data in memory at temp_file_dict[fpath].data prefix: The prefix to use for the temporary file. Defaults to gsutil_test_file. random_ratio: The percentage of randomly generated data to write. This can be any number between 0 and 100 (inclusive), with 0 producing uniform data, and 100 producing random data. Returns: The file path of the created temporary file. z.binF)�suffix�prefix�dir�text�wbN�rb)�tempfile�mkstemp� directoryrd �fdopenrk �openr r �readr' �temp_file_dict�temporary_files�add) rZ rK � mem_metadata�mem_datar� ri �fd�fpathrh �md5�datar2 r2 r3 � _MakeTempFileA s � * . z PerfDiagCommand._MakeTempFilec C s� i | _ t� | _t� | _d| _d| _tt�| _d| _ d| _ | j | j v rrg | _ | jD ] }| j|ddd�}| j �|� qP| j �| j| j| j| jf��r�| jdddd�| _| j �| j| jf�r�| j| jdd| jd�| _tj� | j�| _ | j �| j| jf��r�g | _ g | _ t | j!�}|| j| j" k�r�| j#�$d| j"t%| j�f � | �&� t'| j"�D ]:}| j| jd| jd�}| j �|� | j �tj� |�� �qJn t(d t%| j| j" �t%|�f ��t)� | _*| j#�+| �,� � d S ) z?Performs setup operations needed before diagnostics can be run.r FT)r� r� i P )r� r� ri z) Creating %d local files each of size %s.)r� ri zbNot enough free disk space for throughput files: %s of disk space required, but only %s available.N)-�results�setr� �temporary_objects�total_requests�request_errorsr rg �error_responses_by_code�connection_breaks�teardown_completed�LAT� diag_tests� latency_files�test_lat_file_sizesr� �append� intersection�RTHRU�WTHRU� RTHRU_FILE� WTHRU_FILE�tcp_warmup_filer� �gzip_compression_ratio�mem_thru_file_namerd �path�basename�mem_thru_object_name�thru_file_names�thru_object_namesr r� r� rQ r� r% r� r r rY � discard_sink� addFilter�_PerfdiagFilter)rZ rK r� Zfree_disk_spacerW r8 r2 r2 r3 �_SetUpi sp ��� ��� ���zPerfDiagCommand._SetUpc C sr | j sht�� z,| jD ]}t�|� q| jr8t�| j� W n t yL Y n0 | j D ]}| � || j � qTd| _ dS )zDPerforms operations to clean things up after performing diagnostics.TN) r� r� �clearr� rd �remove�delete_directory�rmdirr� �OSErrorr� rM r? )rZ r� r7 r2 r2 r3 � _TearDown� s zPerfDiagCommand._TearDownc c sH | j �d|� t�� }dV t�� }|| �|| � | j �d|� dS )a� A context manager that measures time. A context manager that prints a status message before and after executing the inner command and times how long the inner command takes. Keeps track of the timing, aggregated by the given key. Args: key: The key to insert the timing value into a dictionary bucket. bucket: A dictionary to place the timing value in. Yields: For the context manager. z%s starting...Nz%s done.)rQ r� �timer� )rZ �key�bucket�t0�t1r2 r2 r3 �_Time� s zPerfDiagCommand._Timec C sD d}d}d}d}d}|�s@t t�� d| d t� �}z |� }| jd7 _d}W q t| j��y< } z�|d7 }|| jkr�| j�d� W Y d}~�q@t |t ��r|j dkr�| j |j d7 < | jd7 _| j d7 _ |d7 }t�|� n� || jk�r(| j�d � W Y d}~�q@n| jd7 _W Y d}~qd}~0 0 q|S ) z�Runs an operation with retry logic. Args: func: The function to run. Returns: True if the operation succeeds, False if aborted. Fr N� rb Tz,Reached maximum total retries. Not retrying.i� z3Reached maximum server error retries. Not retrying.)rf �randomr r� �tuple� exceptions�MAX_TOTAL_RETRIESrQ r� r� r �statusr� r� r� �sleep�MAX_SERVER_ERROR_RETRIESr� ) rZ �func�successZserver_error_retriedZ total_retried�iZ return_val� next_sleeprT r2 r2 r3 � _RunOperation� s> �&z PerfDiagCommand._RunOperationc s� t t��jd< t�j�D ]�}�j�d|d � �jD ]�}t| � �j � � �t j � |��_� j�t��}�j�d||�� t���� ���fdd�}��|� ���fdd�}��|�}t|������fd d �}��|� ���fd d �}��|� q4qd S )zRuns latency tests.�latencyz Running latency iteration %d...rb zN File of size %s located on disk at '%s' being diagnosed in the cloud at '%s'.c s` t �� j�} ��d� �jd ��* �jj| ���jdgd� W d � n1 sR0 Y d S )Nz UPLOAD_%dr� �name)�size�provider�fields)r� �BytesIOr� r� r� r? � UploadObjectr� )Zio_fp)� file_datarK rZ � upload_targetr2 r3 �_Upload; s �z1PerfDiagCommand._RunLatencyTests.<locals>._Uploadc sX �� d� �jd ��. �jj�j�j�jg d�d�W d � S 1 sJ0 Y d S )Nz METADATA_%dr� )r� � contentType� mediaLinkr� �r� r� )r� r� r? �GetObjectMetadata� bucket_namer7 r� r2 �rK rZ �urlr2 r3 � _MetadataF s �z3PerfDiagCommand._RunLatencyTests.<locals>._Metadatac sX �� d� �jd ��. �jj�j�j�j�j�d� W d � n1 sJ0 Y d S )Nz DOWNLOAD_%dr� )r� r9 )r� r� r? �GetObjectMediar r7 r� r� r2 )rK rZ r9 r r2 r3 � _DownloadR s �z3PerfDiagCommand._RunLatencyTests.<locals>._Downloadc sR �� d� �jd ��( �jj�j�j�jd� W d � n1 sD0 Y d S )Nz DELETE_%dr� �r� )r� r� r? � DeleteObjectr r7 r� r2 r r2 r3 �_Delete] s �z1PerfDiagCommand._RunLatencyTests.<locals>._DeleteN)r r� r� r r� rQ r� r� r� � bucket_url�Clonerd r� r� r7 r� r% � StorageUrlToUploadObjectMetadatar� r ) rZ r� r� Zreadable_file_sizer� r Zdownload_metadatar r r2 )r� rK rZ r9 r� r r3 �_RunLatencyTests' s. � z PerfDiagCommand._RunLatencyTestsc @ s e Zd Zdd� ZdS )z PerfDiagCommand._PerfdiagFilterc C s* |� � }d|v p&d|v p&d|v p&d|v S )NzCopying file:///z Copying gs://z Computing CRCzgsutil -m perfdiag)� getMessage)rZ �record�msgr2 r2 r3 �filterg s ��z&PerfDiagCommand._PerfdiagFilter.filterN)r. r/ r0 r r2 r2 r2 r3 r� e s r� c C s | j �t|�� dS rO rP )rZ rT r2 r2 r3 rU m s z)PerfDiagCommand._PerfdiagExceptionHandlerc C sh g }t t|��D ]0}|r || nd}|�t||| ||| �� q| jt|tdt| jj | j | j d� dS )ay Performs a parallel download of multiple objects using the fan strategy. Args: need_to_slice: If True, additionally apply the slice strategy to each object in object_names. object_names: A list of object names to be downloaded. Each object must already exist in the test bucket. file_names: A list, corresponding by index to object_names, of file names for downloaded data. If None, discard downloaded data. serialization_data: A list, corresponding by index to object_names, of serialization data for each object. N�r� r� �� arg_checker� parallel_operations_override� process_count� thread_count) r �lenr� r( �Applyr@ rU r �ParallelOverrideReason�PERFDIAG� processes�threads)rZ r5 � object_names� file_namesr9 r= r� r8 r2 r2 r3 �PerformFannedDownloadq s" ���z%PerfDiagCommand.PerformFannedDownloadc C s� |r:t |d�� }|�| j� W d � n1 s00 Y t| j| j�}g }t| j�D ]>}|| }t|d | d | jd �} |�t||||| �� qV| j t |t dt | j j| j| jd� dS )a3 Performs a download of an object using the slice strategy. Args: object_name: The name of the object to download. file_name: The name of the file to download data to, or None if data should be discarded. serialization_data: The serialization data for the object. �abNrb r r )r� �truncater� r" � num_slicesr rf r� r) r rC rU r r r r r ) rZ r7 r8 r9 rh �component_sizer= r� rA rB r2 r2 r3 r6 � s, * ���z%PerfDiagCommand.PerformSlicedDownloadc C sZ g }t t|��D ]"}|�t||| || ||�� q| jt|tdt| jj | j | j d� dS )aB Performs a parallel upload of multiple files using the fan strategy. The metadata for file_name should be present in temp_file_dict prior to calling. Also, the data for file_name should be present in temp_file_dict if use_file is specified as False. Args: need_to_slice: If True, additionally apply the slice strategy to each file in file_names. file_names: A list of file names to be uploaded. object_names: A list, corresponding by by index to file_names, of object names to upload data to. use_file: If true, use disk I/O, otherwise read upload data from memory. gzip_encoded: Flag for if the file will be uploaded with the gzip transport encoding. If true, a lock is used to limit resource usage. r r N) r r r� r* r rI rU r r r r r )rZ r5 r! r rG rE r= r� r2 r2 r3 �PerformFannedUpload� s ���z#PerfDiagCommand.PerformFannedUploadc s0 t �j�j�}�fdd�t�j�D �}g }t�j�D ]<} | | } t|t| j| �}|�t||| || ||�� q4z��j t |t dt �j j�j�jd� g �t�j�D ] } tjj|| d�} ��| � q�� ���fdd�} ��| � W �j t|t dt �j j�j�jd� n$�j t|t dt �j j�j�jd� 0 dS ) a� Performs a parallel upload of a file using the slice strategy. The metadata for file_name should be present in temp_file_dict prior to calling. Also, the data from for file_name should be present in temp_file_dict if use_file is specified as False. Args: file_name: The name of the file to upload. object_name: The name of the object to upload to. use_file: If true, use disk I/O, otherwise read upload data from memory. gsutil_api: CloudApi instance to use for operations in this thread. gzip_encoded: Flag for if the file will be uploaded with the gzip transport encoding. If true, a semaphore is used to limit resource usage. c s g | ]}� t |� �qS r2 )rS ��.0r� )r7 r2 r3 � <listcomp>� s z7PerfDiagCommand.PerformSlicedUpload.<locals>.<listcomp>r r )r� c s. t �� } �| _�jj| _� j�| �jd� d S �Nr )�apitools_messages�Objectr� r r r� � ComposeObjectr� )�dst_obj_metadata�r? r7 �request_componentsrZ r2 r3 �_Compose s �z5PerfDiagCommand.PerformSlicedUpload.<locals>._ComposeN)r" r� r% r rf r� r� r� r+ r rL rU r r r r r r, �ComposeRequest�SourceObjectsValueListEntryr� rN ) rZ r8 r7 rG r? rE r& Zcomponent_object_namesr= r� Zcomponent_start�src_obj_metadatar2 r2 r0 r3 rF � sf � � ��� �� � �z#PerfDiagCommand.PerformSlicedUploadc C s� |rdnd}|rdnd}| j �d|| jt| j�f � | �� | j| j| j| jd�| j |< |�r| j }| j }g }t | j�D ]�}| j �| j | � | j| jv r�| jj| jj| j | ddg| jjd �}n | �| j | | j | | j|�}t�| j | � t| j | d ��� |�t |�� qznF| j �| j � | �| j | j | j|�}d }| j g| j }t |�g| j }tj � | j!�} | j �| � | �| j!| | j� | �"| | j� t#�#� } | jd k�r�| jd k�r�t | j�D ]2}|�r�|| nd } | �"|| | j| || � �q�nz| j| j$| j%fv �r*| j| j%k} | �&| |||� nH| j| j'k�rrt | j�D ].}|�rT|| nd } | �(|| | || � �qBt#�#� } | | }| j| j }|| }|| j | d < || j | d< || j | d< d S )zRuns read throughput tests.�read_throughput_file�read_throughput� with file I/O� z9 Running read throughput tests %s (%s objects of size %s))rK r r � parallelismr� r �r� r� r# Nrb � time_took�total_bytes_copied�bytes_per_second))rQ r� r� r% r� r� r r �parallel_strategyr� r� r� r r� r� r� r� r? r r r �schemerH rd �unlinkr� r^ r� r r� r� r� r� r� r: r� �FAN�BOTHr" �SLICEr6 )rZ rG � test_name�file_io_stringr! r r9 r� � obj_metadata�warmup_obj_namer� r8 r5 r� r< r= r> r2 r2 r3 �_RunReadThruTests s� ���� � ��� � � � z!PerfDiagCommand._RunReadThruTestsc s� |rdnd}|rdnd}� j �d|� jt� j�f � � �� � j� j� j� j� j � j d�� j |< t j �� j�}� j�|� � �� j|� j� |r�� j}� j}n&� jg� j }� fdd�t� j�D �}|D ]}� j�|� q�t�� }� jd k�r0� jd k�r0t� j�D ]&} � j|| || � j|� j d � �qnx� j� j� jfv �rh� j� jk} � j| |||� j d � n@� j� j k�r�t� j�D ]&} � j || || |� j� j d � �q�t�� } | | } � j� j } | | }| � j | d < | � j | d < |� j | d < dS )z Runs write throughput tests.�write_throughput_file�write_throughputr8 r9 z: Running write throughput tests %s (%s objects of size %s))rK r r r: �gzip_encoded_writesr� c s g | ]}� j t|� �qS r2 )r� rS r( r] r2 r3 r* � s z6PerfDiagCommand._RunWriteThruTests.<locals>.<listcomp>rb rD r< r= r> N) rQ r� r� r% r� r� r r r? rL r� r� rd r� r� r� r� r� rH r? r� r� r� r r� rB rC r' rD rF )rZ rG rE rF rH r! r r7 r� r� r5 r� r< r= r> r2 r] r3 �_RunWriteThruTestss sz ���� � � � � z"PerfDiagCommand._RunWriteThruTestsc sT d�j i�jd< g }g }d�dd� td�D ��}d| d �t�j �D ]N}�jd d d �d �}tj�|�}|�|� |�t d ||d d �� �j � |� qF�j � d �j � �jt|ttd� g �g �t�� }t|��t� ����fdd�� � ���fdd�}�j � d�jj�j � �� �rT��|� �� �rt�� | �jk�r�j �d� �qT�qt�� } t����| | d��jd d< dd� |D �}�j � d�j � �jt|ttd� �j � d�jj�j � g �g �t�� }t|����r(� ��fdd�} ��| � ��r�t�� | �jk�rڈj �d� �q(�q�t�� } t����| | d��jd d < d S ) z0Runs eventual consistency listing latency tests.� num_files�listingr9 c S s g | ]}t �tj��qS r2 )r� �choice�string�ascii_lowercase)r) rW r2 r2 r3 r* � � z1PerfDiagCommand._RunListTests.<locals>.<listcomp>r� zgsutil-perfdiag-list-�-r T)r� r� r� Fz' Writing %s objects for listing test...)r c sp t � � } t�jj�jj�d�jdgd��}t|��jkrH�j � d�j� t � � }� � || � t dd� |D ��S )z>Lists and returns objects in the bucket. Also records latency.�/z items/name)r� � delimiterr� r� z5Listing produced more than the expected %d object(s).c S s g | ] }|j j�qS r2 )r� r� )r) �objr2 r2 r3 r* � rS z@PerfDiagCommand._RunListTests.<locals>._List.<locals>.<listcomp>) r� r� r? � ListObjectsr r r� r r� rQ �warningr� r� )r� �objectsr� )�list_latencies� list_prefixrZ r2 r3 �_List� s" ���z,PerfDiagCommand._RunListTests.<locals>._Listc s& � � } �� | �@ � ��t��� d S rV )�updater� r ��names)r] �expected_objects� files_seen� found_objectsr2 r3 �_ListAfterUpload� s z7PerfDiagCommand._RunListTests.<locals>._ListAfterUploadz5Listing bucket %s waiting for %s objects to appear...z)Maximum time reached waiting for listing.)�num_listing_callsr[ �files_seen_after_listingr< �insertc S s g | ]}|�qS r2 r2 )r) r7 r2 r2 r3 r* rS z'Deleting %s objects for listing test...z8Listing bucket %s waiting for %s objects to disappear...c s" � � } �� | � ��t��� d S rV )�intersection_updater� r r_ )r] rb rc r2 r3 �_ListAfterDelete s z7PerfDiagCommand._RunListTests.<locals>._ListAfterDelete�deleteN)r� r� r� r r� rd r� r� r� r* r� r� rQ r� r rI rU r r� r� r r r� �MAX_LISTING_WAIT_TIMErY r rN ) rZ Z list_objectsr= Z random_idrW r� r7 Ztotal_start_timerd Ztotal_end_timeri r2 )r] ra rb rc r[ r\ rZ r3 � _RunListTests� s� � � �� � ��� � �z PerfDiagCommand._RunListTestsNc s� d��du rt | j��j�� }||_t|��zz|rBt||���n t | j||� � } t� | ��������fdd�� � fdd�} �r�| n� } �� | �W �r��� � S n�r��� � 0 dS )a� Performs an upload to the test bucket. The file is uploaded to the bucket referred to by self.bucket_url, and has name object_name. Args: file_name: The path to the local file, and the key to its entry in temp_file_dict. object_name: The name of the remote object. gsutil_api: Cloud API instance to use for this upload. use_file: If true, use disk I/O, otherwise read everything from memory. file_start: The first byte in the file to upload to the object. (only should be specified for sliced uploads) file_size: The size of the file to upload. (only should be specified for sliced uploads) gzip_encoded: Flag for if the file will be uploaded with the gzip transport encoding. If true, a lock is used to limit resource usage. Returns: Uploaded Object Metadata. Nc sH � t � k r&�j���j� g d��d�S �j���j� g d�t�d�S d S )N)r� r r� )r� r� r� rE )r� r� r� �tracker_callbackrE )r r� r� �UploadObjectResumablerX r2 )rK rh r? rE rZ r� r2 r3 � _InnerUploadX s" ��z,PerfDiagCommand.Upload.<locals>._InnerUploadc s2 t jj� � � W d � S 1 s$0 Y d S rV )�gslib�command�!concurrent_compressed_upload_lockr2 )ro r2 r3 �_InnerUploadLockm s z0PerfDiagCommand.Upload.<locals>._InnerUploadLock) r� r� r r r7 r r r� r� r� r� r^ ) rZ r8 r7 r? rG rJ rK rE � upload_urlr� rs Z upload_funcr2 )ro rK rh r? rE rZ r� r3 rH , s, � �zPerfDiagCommand.Uploadc sn d�zV|dur$t |d������ n�j�� ������fdd�}��|� W �rj��� n�rh��� 0 dS )a� Downloads an object from the test bucket. Args: object_name: The name of the object (in the test bucket) to download. gsutil_api: CloudApi instance to use for the download. file_name: Optional file name to write downloaded data to. If None, downloaded data is discarded immediately. serialization_data: Optional serialization data, used so that we don't have to get the metadata before downloading. start_byte: The first byte in the object to download. (only should be specified for sliced downloads) end_byte: The last byte in the object to download. (only should be specified for sliced downloads) Nzr+bc s" �j �jj���j�� �d� d S )N)r� rA rB r9 )r r r r� r2 �rB rh r? r7 rZ r9 rA r2 r3 �_InnerDownload� s �z0PerfDiagCommand.Download.<locals>._InnerDownload)r� �seekr� r� r^ )rZ r7 r? r8 r9 rA rB rv r2 ru r3 r: w s �zPerfDiagCommand.Downloadc s6 z � ��fdd�}�� |� W n ty0 Y n0 dS )z�Deletes an object from the test bucket. Args: object_name: The name of the object to delete. gsutil_api: Cloud API instance to use for this delete. c s � j �jj��jd� d S r+ )r r r r� r2 �r? r7 rZ r2 r3 � _InnerDelete� s �z,PerfDiagCommand.Delete.<locals>._InnerDeleteN)r� r )rZ r7 r? ry r2 rx r3 rM � s zPerfDiagCommand.Deletec C s. d}g }t dd��P}|�� dd� }|D ]*}|�� \}}}}|d �� r(|�|� q(W d � n1 sh0 Y i }t dd���}|D ]�}|�� dd� } | \ }}}} }} } } }}}||v r�t| �| } t|�| }t| �} t| �} t| �} t|�}| | | || |f||< q�W d � n1 �s 0 Y |S ) a� Retrieves disk I/O statistics for all disks. Adapted from the psutil module's psutil._pslinux.disk_io_counters: http://code.google.com/p/psutil/source/browse/trunk/psutil/_pslinux.py Originally distributed under under a BSD license. Original Copyright (c) 2009, Jay Loden, Dave Daeschler, Giampaolo Rodola. Returns: A dictionary containing disk names mapped to the disk counters from /disk/diskstats. i z/proc/partitions�rr� N���z/proc/diskstats� )r� � readlines�split�isdigitr� rg )rZ � sector_size� partitions�f�lines�linerW r� �retdict�values�reads�rbytes�rtime�writes�wbytes�wtimer2 r2 r3 �_GetDiskCounters� s, * 6z PerfDiagCommand._GetDiskCountersc C s� z| j ddgddd�}W n ty8 | j�d� Y dS 0 |�� �� }d}d}d}d}|�d�D ]�}d |v std |v rxd}|r�|du r�d |v s�d |v s�d |v r�d�dd� |D ��}|r�|du r�d|v s�d|v r�d�dd� |D ��}|r`|du r`d|v �sd|v �sd|v r`d�dd� |D ��}q`i }z(t|�|d< t|�|d< t|�|d< W n0 t t f�yz d|d< d|d< d|d< Y n0 |S )z�Tries to parse out TCP packet information from netstat output. Returns: A dictionary containing TCP information, or None if netstat is not available. �netstat�-sTF)r� r� zGnetstat not found on your system; some measurement data will be missingN� ztcp:ztcp statisticszsegments retransmitedzretransmit timeoutszsegments retransmittedr9 c s s | ]}|t jv r|V qd S rV �rQ �digits�r) �cr2 r2 r3 � <genexpr>� rS z/PerfDiagCommand._GetTcpStats.<locals>.<genexpr>zsegments receivedzpackets receivedc s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� rS zsegments send outz packets sentz segments sentc s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� rS �tcp_retransmit� tcp_received�tcp_sent) r� r� rQ rY �strip�lowerr~ r� rg � ValueError� TypeError)rZ �netstat_outputZ found_tcpr� r� r� r� �resultr2 r2 r3 � _GetTcpStats� s` � ��� �� ��� z PerfDiagCommand._GetTcpStatsc C s� i }t jt jt jt jf}tj�ddd�|d< tj�ddd�}tj�ddd�}t |�|d < tj�dd |rjdnd �r|| j � d � zt � t � � �|d < W n |y� d|d < Y n0 | j|d< t�dt�� �|d< dd| jg}z8| j|ddd�}t�d|�}|�r|�d�nd|d< W n ttf�y0 d|d< Y n0 t�r�z@tdd�� }|�� dk|d< W d � n1 �sl0 Y W n t�y� Y n0 |�dd ��r�t � � } d d d d d | g}z t � } | j|d| d�|d!< W n ttf�y� d|d!< Y n0 | j j | j j d"d#g| j j!d$�} | j"|d%< | j#|d&< z,t�� } t � | j� t�� } | | |d'< W n |�yf Y n0 z t �$| j�\} }}||d(< W n |�y� g }g |d(< Y n0 g |d)< |D ]>}z"t �%|�\} }}|d) �&| � W n |�y� Y n0 �q�z@g d*�}| j|ddd�}t�d+|�}|�r*|�d,�nd|d-< W n ttf�yT d|d-< Y n0 i |d.< |D ]X}z<t � � }t�� } |�'|| j(f� t�� } | | |d. |< W n |�y� Y n0 �qb|�rXd}z*t�� } t � |�}t�� } | | |d/< W n |�y Y n0 z<t � � }t�� } |�'|�p$||f� t�� } | | |d0< W n |�yV Y n0 zt)�*� |d1< W n t+�y� d|d1< Y n0 zt,t-�.� �|d2< W n t/tf�y� d|d2< Y n0 d}d}d}d}z�td3d���}|D ]�}|�0d4��r t1d�2d5d6� |D ���d7 }n||�0d8��r6t1d�2d9d6� |D ���d7 }nR|�0d:��r`t1d�2d;d6� |D ���d7 }n(|�0d<��r�t1d�2d=d6� |D ���d7 }�q�W d � n1 �s�0 Y W n t3t4f�y� Y n0 ||||d>�|d?< i |d@< t5t�D ]>}t6t|�}|�7� �r�t8|t9j:��rdA|v �s�||d@ |< �q�i |dB< g dC�} | D ]l} zPt| d��}|�;� } W d � n1 �sn0 Y | �<� |dB t-j=�>| �< W n t3�y� Y n0 �q>|| j?dD< dS )Ez Collects system information.�Boto� is_secureT�boto_https_enabled�proxyN� proxy_portr � using_proxy� proxy_rdnsFz�DNS lookups are disallowed in this environment, so some information is not included in this perfdiag run. To allow local DNS lookups while using a proxy, set proxy_rdns to False in your boto file.� ip_addressr9 �tempdir�%a, %d %b %Y %H:%M:%S +0000� gmt_timestamp�nslookupz -type=CNAME)r� r� z = (?P<googserv>[^.]+)\.Zgoogserv�googserv_routez /sys/class/dmi/id/product_namerz zGoogle Compute Engine �on_gce�gcloudZcomputeZ instancesr� z --filter=�gce_instance_info�location� storageClassr; �bucket_location�bucket_storageClass�google_host_dns_latency� googserv_ips�googserv_hostnames)r� z -type=TXTzo-o.myaddr.google.com.z text\s+=\s+"(?P<dnsip>[\.\d]+)"Zdnsip� dns_o-o_ip� google_host_connect_latencies�proxy_dns_latency�proxy_host_connect_latency� cpu_count�load_avgz /proc/meminfoZMemTotalc s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� � rS z2PerfDiagCommand._CollectSysInfo.<locals>.<genexpr>�� ZMemFreec s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� � rS ZBuffersc s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� � rS ZCachedc s s | ]}|t jv r|V qd S rV r� r� r2 r2 r3 r� � rS )� mem_total�mem_free� mem_buffers� mem_cached�meminfoZ gsutil_configr� �tcp_proc_values)z /proc/sys/net/core/rmem_defaultz/proc/sys/net/core/rmem_maxz /proc/sys/net/core/wmem_defaultz/proc/sys/net/core/wmem_maxz!/proc/sys/net/ipv4/tcp_timestampsz/proc/sys/net/ipv4/tcp_sackz%/proc/sys/net/ipv4/tcp_window_scaling�sysinfo)@�socketrR �herror�gaierror�timeout�botor �get�getint�boolrQ r� � gethostbyname� gethostnamer� r� �strftime�gmtime� XML_API_HOSTr� �re�search�groupr r� r r� �readliner! r? � GetBucketr r r@ r� r� �gethostbyname_ex� gethostbyaddrr� �connect� XML_API_PORT�multiprocessingr� �NotImplementedErrorr� rd � getloadavg�AttributeError� startswithrg r� �IOErrorr� r� �getattr�isupperr� r� � string_typesr� r� r� r� r� ) rZ r� Z socket_errors� proxy_hostr� r� Znslookup_cname_output�mrh �hostnamer� Z bucket_infor� r� rW Z ipaddrlistZ googserv_ipZnslookup_txt_output�sock�proxy_ipr� r� r� r� r� r� �attr� attr_valueZstats_to_check�fname�valuer2 r2 r3 �_CollectSysInfo sJ � � � � 4� � � � � � �,� � ( z PerfDiagCommand._CollectSysInfoc s� t |�}tt|��| � t�t� fdd�|D ��| �}tjt|��d�ddd� tjd� d �d �ddd� tjd|d �d �ddd� tjdt |d �d �d �ddd� t�dt |d �d �d �d� dS )zAPrints out mean, standard deviation, median, and 90th percentile.c 3 s | ]}|� d V qdS )r� Nr2 )r) �x��meanr2 r3 r� � rS z0PerfDiagCommand._DisplayStats.<locals>.<genexpr>� r9 r� ��endz%.1fr� � � g �?r| g�������?N) r �float�sum�math�sqrtr � print_to_fdrS �rjustr& )rZ �trials�n�stdevr2 r� r3 � _DisplayStats� s ��z PerfDiagCommand._DisplayStatsc 1 C s t �� t �d� t �d�d�� t �d� d| jv �r�t �� t �d� t �d�d�� t �d� t �d� t �d� t| jd �D �]8}t| jd | �}|�d �\}}t|�}|d kr�t jd �d �d dd� t jt|��d �d dd� | � |� |dk�r:t jd�d �d dd� t jt|��d �d dd� | � |� |dk�r~t jd�d �d dd� t jt|��d �d dd� | � |� |dkr�t jd�d �d dd� t jt|��d �d dd� | � |� q�d| jv �r`t �� t �d� t �d�d�� t �d� | jd }t �d| j t|d �t|d �f � t �dt |d d � � d |v �r`t �d |d � d | jv �r�t �� t �d� t �d!�d�� t �d� | jd }t �d| j t|d �t|d �f � t �dt |d d � � d |v �r�t �d |d � d"| jv �r�t �� t �d� t �d#�d�� t �d� | jd" }t �d| j t|d �t|d �f � t �d$t |d d � � d |v �r�t �d |d � d%| jv �r:t �� t �d� t �d&�d�� t �d� | jd% }t �d| j t|d �t|d �f � t �d$t |d d � � d |v �r:t �d |d � d'| jv �rxt �� t �d� t �d(�d�� t �d� | jd' } | d) } | d* } t �d+| d, � t �d-| d. � t �d/| d0 � t �d1d2� d3d4� | d5 D �� � t �d6d2� t t| d7 �� � t �d8| d, � t �d-| d. � t �d/| d0 � t �d1d2� d9d4� | d5 D �� � t �d6d2� t t| d7 �� � d:| jv � r�t �� t �d� t �d;�d�� t �d� | jd: } t �d<| d= � t �d>| d? � t �d@| jdA � t �dB| j�dCdD� � t �dE| j�dFdD� � dG| v �r�| dG } dH}zt�| dI�}W n t�y\ Y n0 |�r�t�|�}tj�|�}t �dJ|�dK� � dL| v �r�t �dM| dL � | dL �r�t �dN| dO �dPdQ� � t �dR| dS � t �dT| dU � t �dV| dW � t �dXdY� | dZ � � t �d[dY� | d\ � � t �d]| d^ � t �d_| d` � t �da| db � zPt �dct| dd de � � t �dft| dd dg | dd dh | dd di � � W n t�y� Y n0 dj| v �rFdk| v �rF| dj }| dk }dlD ]J}z.|dm| |dm| }t �dn||f � W n t�y> Y n0 �q�n t �do� dp| v � rjdq| v � rjt jdrdd� | dp }| dq }t jd ds�dt�dd� duD ]}t j|�d �dd� �q�t �� t|�D ]�}|| }|| }|\}} } } } } |\}!}"}#}$}%}&t jd |�dt�dd� |!| |"| |#| |$| |%| |&| g}'|'D ] }t jt|��d �dd� � q>t �� �q�dv| v � r�t jdwdd� t�| dv �D ]}(t �dx|( � � q�dy| v � r�t �dz| dy � d{| v � r�t �d|| d{ � d}| v � r�t �d~| d} d � d�| v � rBt �d�� t�| d� �D ] \})}*t �d�|)|*d f � � q d�| v � rbt �d�| d� d � d�| v � r�t �d�| d� d � d�| jv � r�d�| jv � r�t �� t �d� t �d��d�� t �d� t| jd� �}+t| jd� �},t| jd� �}-|+d�k� r|+|, t |+� d� nd�}.t �d�|+ � t �d�|, � t �d�|- � t �d�|. � d�| jv � r�tt�| jd� ��}/|/� r�t �d�� t �dP� d�d4� |/D ��� | j � r�t | j d���"}0t j | j|0d�d�� W dH � n1 � s�0 Y t �� t �d�| j � t �� dHS )�z/Displays results collected from diagnostic run.zN==============================================================================zDIAGNOSTIC RESULTS�N r� zN------------------------------------------------------------------------------ZLatencyzOOperation Size Trials Mean (ms) Std Dev (ms) Median (ms) 90th % (ms)zO========= ========= ====== ========= ============ =========== ===========rW ZMETADATA�Metadatar� r9 r� r� �DOWNLOADr: �UPLOADrH �DELETErM rK zWrite Throughputz5Copied %s %s file(s) for a total transfer size of %s.rK r= zWrite throughput: %s/s.r> � r: zParallelism strategy: %srJ z Write Throughput With File I/Or7 zRead ThroughputzRead throughput: %s/s.r6 z Read Throughput With File I/OrO ZListingrg rj zAfter inserting %s objects:rN z0 Total time for objects to appear: %.2g secondsr< z" Number of listing calls made: %sre z) Individual listing call latencies: [%s]�, c s s | ]}d | V qdS �z%.2gsNr2 �r) r� r2 r2 r3 r� � rS z2PerfDiagCommand._DisplayResults.<locals>.<genexpr>r[ z' Files reflected after each call: [%s]rf zAfter deleting %s objects:c s s | ]}d | V qdS r r2 r r2 r2 r3 r� � rS r� zSystem InformationzIP Address: %sr� zTemporary Directory: %sr� zBucket URI: %s� bucket_urizgsutil Version: %s�gsutil_version�Unknownzboto Version: %s� boto_versionr� Nr� zMeasurement time: %sz%Y-%m-%d %I:%M:%S %p %Zr� zRunning on GCE: %szGCE Instance: %sr� r� z zBucket location: %sr� zBucket storage class: %sr� zGoogle Server: %sr� z!Google Server IP Addresses: %sz r� z Google Server Hostnames: %sr� z#Google DNS thinks your IP is: %sr� zCPU Count: %sr� zCPU Load Average: %sr� zTotal Memory: %sr� r� zFree Memory: %sr� r� r� � netstat_end� netstat_start)�sent�receivedZ retransmitztcp_%sz!TCP segments %s during test: %dzQTCP segment counts not available because "netstat" was not found during test runs�disk_counters_end�disk_counters_startzDisk Counter Deltas: Zdiskr� )r� r� r� r� r� r� r� zTCP /proc values: z %s = %sr� zBoto HTTPS Enabled: %sr� z$Requests routed through proxy: %sr� zALatency of the DNS lookup for Google Storage server (ms): %.1fg @�@r� z7Latencies connecting to Google Storage server IPs (ms):z %s = %.1fr� z@Latency of the DNS lookup for the configured proxy (ms): %.1fr� z8Latency connecting to the configured proxy (ms): %.1fr� r� zIn-Process HTTP Statisticsr� r r` z Total HTTP requests made: %dzHTTP 5xx errors: %dzHTTP connections broken: %dzAvailability: %.7g%%r� zError responses by code:c s s | ]}d | V qdS )z %s: %sNr2 r� r2 r2 r3 r� 2 rS �wr� )�indentz Output file written to '%s'.)!r r� �centerr� �sortedr~ rg r� r% r� r� r$ r� �maprS r� r� �strptimer� �calendar�timegm�datetime� fromtimestampr� �replacer� r� � iteritemsr� � output_filer� �json�dump)1rZ r� r� �op�numbytesZ write_thruZwrite_thru_fileZ read_thruZread_thru_filerO rg rj r� Z ts_string� timetuple� localtimeZlocaldtZ netstat_afterZnetstat_beforeZtcp_type�deltaZ disk_afterZ disk_beforeZcolnameZdiskname�before�afterZreads1Zwrites1Zrbytes1Zwbytes1Zrtime1Zwtime1Zreads2Zwrites2Zrbytes2Zwbytes2Zrtime2Zwtime2Zdeltas�item�ipr� �totalZ numerrorsZ numbreaksZ availabilityZ sorted_codesr� r2 r2 r3 �_DisplayResults s� ��� � � � ����� � ����� � ����� � ����� � ��������������� � � � �� � ��� � �� �� � � � � �� � �� �� ��� � 2z PerfDiagCommand._DisplayResultsc C s> z t |�}|dk rt|��|W S ty8 t|��Y n0 dS )a^ Tries to convert val argument to a positive integer. Args: val: The value (as a string) to convert to a positive integer. msg: The error message to place in the CommandException on an error. Returns: A valid positive integer. Raises: CommandException: If the supplied value is not a valid positive integer. rb N)rg r r� )rZ �valr r2 r2 r3 �_ParsePositiveInteger< s z%PerfDiagCommand._ParsePositiveIntegerc C sh d| _ d| _d| _d| _d| _d| _t�� | _d| _ t | j �| _ d| _ d| _i | _d| _d| _| j�r| jD �]�\}}|dkr�| �|d �| _ |d kr�| �|d �| _|d kr�| �|d �| _|dkr�|�� | jv r�|�� | _n td| ��|dkr�| �|d�| _|dk�r2zt|�| _W n t�y0 td��Y n0 |dk�rd|| _tj�| j��sdd| _ t� | j� |dk�r�t � | _ |� � � d�D ].}|�� | j v�r�td| ��| j � |� �q�|dk�r�|� d�}t!|�dk�r�td | ��|\}}|| j|< |d k�r tj�"|�| _ |d k�r�tj�"|�| _tj�#| j��s@td | ��zNt$| jd ��,}t%�&|�| _'| j(�)d!| j� W d � n1 �s�0 Y W n t�y� td"| ��Y n0 dS |d#krld| _zt*|�| _W n t�y� d$| _Y n0 | jd%k �s| jdkrltd&��ql| jdk�s&| jdk�r8| j�s8| j+| _n(| jdk�r`| jdk�r`| j�r`td'��| j,�sp| �-� t.| j,d% �| _/| j/j0| _1| j/�2� �s�| j/�3� �r�td(| j,d% ��| jtd)�k�r�| j4| j v �s�| j5| j v �r�td*��| j| j6| j7fv }| j1d+k�o| j �8| j5| j9�} |�r,| �r,td,��| j:j;| j/j<| j/j0d-gd.� t=j>t?j@t?jAt?jBt=jCtDg| _EdS )/z&Parses arguments for perfdiag command.r� rb N� r� Fr` z-nz,The -n parameter must be a positive integer.z-cz,The -c parameter must be a positive integer.z-kz,The -k parameter must be a positive integer.z-pz)'%s' is not a valid parallelism strategy.z-yz,The -y parameter must be a positive integer.r� zInvalid -s parameter.z-dTz-t�,z8List of test names (-t) contains invalid test name '%s'.z-m�:r� z,Invalid metadata key-value combination '%s'.z-oz-iz Invalid input file (-i): '%s'.rz zRead input file: '%s'.z'Could not decode input file (-i): '%s'.z-jr{ r z7The -j parameter must be between 0 and 100 (inclusive).zrCannot specify parallelism strategy (-p) without also specifying multiple threads and/or processes (-c and/or -k).zOThe perfdiag command requires a URL that specifies a bucket. "%s" is not valid.r� z~For in-memory tests maximum file size is 2GiB. For larger file sizes, specify rthru_file and/or wthru_file with the -t option.�s3zQSliced uploads are not available for s3. Use -p fan or sequential uploads for s3.�idr )Fr� r r r? r% r� r� � gettempdirr� r� r� �DEFAULT_DIAG_TESTSr� r � input_file� metadata_keysrL r� �sub_optsr. r� �PARALLEL_STRATEGIESr r# r� rd r� �exists�makedirsr� r~ �ALL_DIAG_TESTSr� r �abspath�isfiler� r �loadr� rQ r� rg rB r= �$RaiseWrongNumberOfArgumentsExceptionr r r@ r� � IsCloudUrl�IsBucketr� r� rD rC r� r� r? r� r r � HTTPExceptionr� rR r� r� � BadStatusLiner r� ) rZ �o�arE �piecesr� r� r� Z perform_sliceZslice_not_availabler2 r2 r3 � _ParseArgsQ s ��� �� � � 4� �� � � � �� �� � �zPerfDiagCommand._ParseArgsc C s� | � � | jr| �� dS tj�ddd� | j�d| j| j | j | j | j t | j�d�| j�| j| j� �z�| �� | �� | �� }|r�|| jd d< tr�t� | jd d < t| j �| jd < d | jd < | j| jd < | j| jv r�| � � | j | jv r�| � � | j | jv �r| j dd� | j!| jv �r(| j dd� | j"| jv �r>| � � | j#| jv �rT| �$� | �� }|�rp|| jd d< t�r�t� | jd d< | j%| jd< | j&| jd< | j'| jd< | j(| jd< t)j*| jd< tj+| jd< | �,� | �� W | �,� n | �,� 0 dS )z3Called by gsutil when the command is being invoked.r r� � num_retries�0z�Number of iterations to run: %d Base bucket URI: %s Number of processes: %d Number of threads: %d Parallelism strategy: %s Throughput file size: %s Diagnostics to run: %s Gzip compression ratio: %s Gzip transport encoding writes: %sr r� r r r rm Z json_format�metadataT)rG r r r� r� r� r� r r )-rH r6 r, r� r r� rQ r� r� r r r r? r% r� r� r� r� rL r� r� r� r� r r rS r7 r� r r� rI r� rM r� r� �LISTrl r� r� r� r� rp �VERSION� __version__r� )rZ r� r2 r2 r3 � RunCommand� sd � zPerfDiagCommand.RunCommand)TFF)r FFr� r` )F)F)F)F)Fr NF)NNr N)Kr. r/ r0 r1 r �CreateCommandSpec� _SYNOPSISr �XML�JSONr � MakeNCloudBucketURLsArgument� command_spec�HelpSpec�_DETAILED_HELP_TEXT� help_specr� r� r� r� r� r� rL rB rD rC r< r5 r9 r� r r� �gs� connection� GSConnection� DefaultHostr� r� r� r� r� ZKEY_BUFFER_SIZErk r� r� r� r� r� � contextlib�contextmanagerr� r� r �logging�Filterr� rU r" r6 r' rF rI rM rl rH r: rM r� r� r� r� r, r. rH rO r2 r2 r2 r3 rl � s� �� � � # � (P* .> " � * � K X Hu � N � +)9 ` : rl c C s< | � � r,| �� r,t�� }| j|_| j|_|S td| ��d S )NzENon-cloud URL upload target %s was created in perfdiag implemenation.) rA �IsObjectr, r- r7 r� r r� r )� storage_urlr� r2 r2 r3 r 0 s �r )N)N)N)N)N)r r` ra )dr1 � __future__r r r r r � collectionsr r r] r r r_ r� r� rd r� r� r� rQ r� r� r� r� �boto.gs.connectionr� � six.movesr r r rp �gslib.cloud_apir r � gslib.commandr r �gslib.command_argumentr �gslib.commandsr �gslib.cs_api_mapr �gslib.exceptionr �gslib.file_partr �gslib.storage_urlr �"gslib.third_party.storage_apitoolsr r, Z gslib.utilsr Zgslib.utils.boto_utilr r �gslib.utils.constantsr � gslib.utils.cloud_api_helperr r �gslib.utils.hashing_helperr Zgslib.utils.system_utilr r r r r! �gslib.utils.unit_utilr" r# r$ r% r& rQ rW r' r( r) r* r+ r� � Exceptionr, r4 r@ rC rI rL rN rU rX �objectrY rk rl r r2 r2 r2 r3 �<module> s� �� < ���� � $ t