OXIESEC PANEL
- Current Dir:
/
/
opt
/
gsutil
/
gslib
/
__pycache__
Server IP: 2a02:4780:11:1594:0:ef5:22d7:a
Upload:
Create Dir:
Name
Size
Modified
Perms
📁
..
-
02/11/2025 08:19:48 AM
rwxr-xr-x
📄
__init__.cpython-39.pyc
2.94 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
__main__.cpython-39.pyc
20.78 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
boto_resumable_upload.cpython-39.pyc
14.86 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
boto_translation.cpython-39.pyc
47.41 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
bucket_listing_ref.cpython-39.pyc
4.23 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
cloud_api.cpython-39.pyc
41.26 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
cloud_api_delegator.cpython-39.pyc
21.69 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
command.cpython-39.pyc
63.61 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
command_argument.cpython-39.pyc
4.25 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
command_runner.cpython-39.pyc
15.75 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
context_config.cpython-39.pyc
6.98 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
cred_types.cpython-39.pyc
851 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
cs_api_map.cpython-39.pyc
3.25 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
daisy_chain_wrapper.cpython-39.pyc
8.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
devshell_auth_plugin.cpython-39.pyc
1.24 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
discard_messages_queue.cpython-39.pyc
989 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
exception.cpython-39.pyc
5.84 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
file_part.cpython-39.pyc
4.04 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
gcs_json_api.cpython-39.pyc
53.37 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
gcs_json_credentials.cpython-39.pyc
13.84 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
gcs_json_media.cpython-39.pyc
19.41 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
help_provider.cpython-39.pyc
2.36 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
iamcredentials_api.cpython-39.pyc
7.62 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
impersonation_credentials.cpython-39.pyc
1.79 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
kms_api.cpython-39.pyc
9.34 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
lazy_wrapper.cpython-39.pyc
2.27 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
metrics.cpython-39.pyc
26.95 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
metrics_tuple.cpython-39.pyc
491 bytes
02/11/2025 08:19:48 AM
rw-r--r--
📄
name_expansion.cpython-39.pyc
20.03 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
no_op_credentials.cpython-39.pyc
936 bytes
02/11/2025 08:19:49 AM
rw-r--r--
📄
parallel_tracker_file.cpython-39.pyc
8.89 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
plurality_checkable_iterator.cpython-39.pyc
3.2 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
progress_callback.cpython-39.pyc
4.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
project_id.cpython-39.pyc
841 bytes
02/11/2025 08:19:48 AM
rw-r--r--
📄
pubsub_api.cpython-39.pyc
7.69 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
resumable_streaming_upload.cpython-39.pyc
5.23 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
seek_ahead_thread.cpython-39.pyc
4.09 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
sig_handling.cpython-39.pyc
4.62 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
storage_url.cpython-39.pyc
16.48 KB
02/11/2025 08:19:48 AM
rw-r--r--
📄
tab_complete.cpython-39.pyc
10.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
thread_message.cpython-39.pyc
12.17 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
tracker_file.cpython-39.pyc
17.01 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
tz_utc.cpython-39.pyc
2.05 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
ui_controller.cpython-39.pyc
30.35 KB
02/11/2025 08:19:49 AM
rw-r--r--
📄
wildcard_iterator.cpython-39.pyc
23.94 KB
02/11/2025 08:19:48 AM
rw-r--r--
Editing: metrics.cpython-39.pyc
Close
a +(Wg� � @ s d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlmZ ddl m Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZdd lmZ dd lmZ ddlZdd l m Z dd l m Z dd l m!Z! ddl"m#Z# ddl"m$Z$ dZ%dZ&dZ'dZ(dZ)dZ*dZ+e�,d�Z-e j.�/e j.�0dd��Z1dZ2dd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7� Z3G d8d9� d9e4�Z5d:d;� Z6d<d=� Z7ej8e6d>d?� ��Z9e6dLd@dA��Z:e6dBdC� �Z;e6dDdE� �Z<e6dFdG� �Z=e6dHdI� �Z>dMdJdK�Z?dS )Nz:Static data and helper functions for collecting user data.� )�absolute_import)�print_function)�division)�unicode_literalsN)� defaultdict��wraps)�input)�urllib)�VERSION)�Metric)� system_util)�CalculateThroughput)�HumanReadableToBytesz(https://ssl.google-analytics.com/collectzUA-36037335-16zUA-36037335-17ZCommandZRetryableErrorZ FatalErrorZPerformanceSummaryz.*google\.com$�~z.gsutil/analytics-uuidZDISABLED�ec�ea�el�evZcd1Zcd2Zcd3Zcd4Zcd5Zcd6Zcd7Zcd8Zcd9Zcd10Zcm1Zcm2Zcm3Zcm4Zcm5Zcm6Zcm7Zcm8Zcm9Zcm10Zcm11Zcm12Zcm13Zcm14) zEvent Categoryz Event Actionz Event Labelz Event Value� Command Name�Global Options�Command-Level Options�Config� Command Alias� Fatal Error�Parallelism Strategy�Source URL Type�Provider Types� Timestamp�Execution Time�Retryable Errors�Is Google Corp User� Num Processes� Num Threads�#Number of Files/Objects Transferred�!Size of Files/Objects Transferred�Average Overall Throughput� Num Retryable Service Errors� Num Retryable Network Errors�Thread Idle Time Percent�Slowest Thread Throughput�Fastest Thread Throughput� Disk I/O Timec @ s� e Zd ZdZeefdd�ZdZdZdd� Z e efdd��Z e d d � �Z e d d � �Ze d)dd��Ze d*dd��Ze dd� �Zdd� Zdd� Zeddfdd�ZG d d � d e�Zd d � Zd d!� Zd"d#� Zd$d%� Zd+d'd(�Z dS ),�MetricsCollectorz�A singleton class to handle metrics reporting to Google Analytics (GA). This class is not thread or process-safe, and logging directly to the MetricsCollector instance can only be done by a single thread. c C s� t � | _|| _t�� | _t�� }tj � dd�}|r6t }| � � }d}t �t�� �rTd}ddd|d|d d td |td |i| _d jt�� t�� d�| _g | _tt�| _d| _ dS )a Initialize a new MetricsCollector. This should only be invoked through the GetCollector or StartTestCollector functions. Args: ga_tid: The Google Analytics tracking ID to use for metrics collection. endpoint: The URL to send requests to. �GSUtilZuse_test_GA_propertyr � �v�1�tid�cid�t�eventr r! z{system}/{release})�system�releaseN) �_GetTimeInMillis� start_time�endpoint�logging� getLogger�loggerr- �_GetCID�boto�config�getbool�_GA_TID_TESTING�_ValidateAndGetConfigValues�_GOOGLE_CORP_HOST_RE�match�socket� gethostname� _GA_LABEL_MAP� ga_params�format�platformr6 r7 � user_agent�_metricsr �int�retryable_errors�perf_sum_params)�self�ga_tidr: r3 Zuse_test_property� config_valuesZ is_corp_user� rT � /opt/gsutil/gslib/metrics.py�__init__x s0 � � zMetricsCollector.__init__Nc sL g � d�� �fdd�}dD ]\}}|||dd� d� qd�d D ] \}}|||�fd d�d� q>d D ]\}}|||d d� d� qbd D ]V\}}t j�||�}|r�zt|�}� �||f� W q� ty� � �|�f� Y q�0 q�|dddd� d� |dddd� d� |dddd� d� |dddd� d� |dddd� d� d�tdd � � D ���S ) z�Parses the user's config file to aggregate non-PII config values. Returns: A comma-delimited string of config values explicitly set by the user in key:value pairs, sorted alphabetically by key. �INVALIDc s` z@t j�| |�}|r,||�r,� �||f� n|r>� �|�f� W n � �|�f� Y n0 d S �N)r? r@ � get_value�append)�section�category� validation_fn� config_value)rS �invalid_value_stringrT rU �GetAndValidateConfigValue� s zOMetricsCollector._ValidateAndGetConfigValues.<locals>.GetAndValidateConfigValue))�Boto�https_validate_certificates)r. �disable_analytics_prompt)r. Z use_magicfile)r. Ztab_completion_time_logsc S s t | ��� dv S )N)�true�false)�str�lower��valrT rT rU �<lambda>� s �z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>)r[ r\ r] i� ) )ra �debug)ra Zhttp_socket_timeout)ra Z num_retries)ra Zmax_retry_delay)r. Zdefault_api_version)r. Z%sliced_object_download_max_components)r. Zparallel_process_count)r. Zparallel_thread_count)r. Z software_update_check_period)r. Ztab_completion_timeout)�OAuth2Zoauth2_refresh_retriesc s t | ��� ot| �� k S rX )rf �isdigitrN rh )�small_int_thresholdrT rU rj � s ))r. Zresumable_threshold)r. Zrsync_buffer_lines)r. Ztask_estimation_thresholdc S s t | ��� S rX )rf rm rh rT rT rU rj � � ))r. Z(parallel_composite_upload_component_size)r. Z#parallel_composite_upload_threshold)r. Z%sliced_object_download_component_size)r. Z sliced_object_download_thresholdr. Z check_hashesc S s | dv S )N)Zif_fast_else_failZif_fast_else_skip�always�neverrT rh rT rT rU rj s �Zcontent_languagec S s | � � ot| �dkS )N� )�isalpha�lenrh rT rT rU rj ro Zjson_api_versionc S s | d � � dko | dd � �� S )Nr r0 r/ )rg rm rh rT rT rU rj ro Z prefer_apic S s | dv S )N)�json�xmlrT rh rT rT rU rj ro rl Z token_cachec S s | dv S )N)Z file_systemZ in_memoryrT rh rT rT rU rj s ��,c S s g | ]}d � |d |d ��qS )z{0}:{1}r r/ )rJ )�.0r@ rT rT rU � <listcomp> s z@MetricsCollector._ValidateAndGetConfigValues.<locals>.<listcomp>)r? r@ rY r rZ � ValueError�join�sorted) rQ r` r[ Z bool_categoryZsmall_int_categoryZlarge_int_categoryZdata_size_categoryr^ Z size_in_bytesrT )rS r_ rn rU rC � sn � � � �������z,MetricsCollector._ValidateAndGetConfigValuesc C s" t �� r dS t js t | �t _t jS )zDReturns the singleton MetricsCollector instance or None if disabled.N)r- � IsDisabled� _instance)rR rT rT rU � GetCollector# s z MetricsCollector.GetCollectorc C s t jdu rt �� t jS )z6Returns whether metrics collection should be disabled.N)r- �_disabled_cache�_CheckAndSetDisabledCacherT rT rT rU r} - s zMetricsCollector.IsDisabledc C s� t j�d�dkrd| _n�t j�d�dkr8d| _| �� n|tj�ddd�rPd| _ndt� � rjt j�d� | _nJt j � t �r�t t �� }|�� tk| _W d � q�1 s�0 Y nd| _d S ) z1Sets _disabled_cache based on user opt-in or out.�GSUTIL_TEST_ANALYTICSr1 T�2Fr. Zuse_gcloud_storage�GA_CIDN)�os�environ�getr� �StartTestCollectorr? r@ rA r �InvokedViaCloudSdk�path�exists�_UUID_FILE_PATH�open�read�_DISABLED_TEXT)�cls�frT rT rU r� 4 s .z*MetricsCollector._CheckAndSetDisabledCache�https://example.com�user-agent-007c C s` | � � rdtjd< d| _| t|�| _|du r6ddd�}|| j_|| j_tjd dkr\d | j_dS ) aN Reset the singleton MetricsCollector with testing parameters. Should only be used for tests, where we want to change the default parameters. Args: endpoint: str, URL to post to user_agent: str, User-Agent string for header. ga_params: A list of two-dimensional string tuples to send as parameters. �0r� FN�b�d)�a�cr� r ) r} r� r� r� rB r~ rI rL r9 )r� r: rL rI rT rT rU r� L s z#MetricsCollector.StartTestCollectorc C s dt jd< d| _|| _dS )z�Reset the MetricsCollector with default parameters after testing. Args: original_instance: The original instance of the MetricsCollector so we can set the collector back to its original state. r1 r� N)r� r� r� r~ )r� Zoriginal_instancerT rT rU �StopTestCollectorg s z"MetricsCollector.StopTestCollectorc C sP t j�t�rDtt��} | �� }W d � n1 s20 Y |rD|S t j�d�S )z�Gets the client id from the UUID file or the SDK opt-in, or returns None. Returns: str, The hex string of the client id. Nr� )r� r� r� r� r� r� r� r� )r� r3 rT rT rU r> s s &zMetricsCollector._GetCIDc C s | j �|� dS )a Extends self.ga_params to include new parameters. This is only used to record parameters that are sent with every event type, such as global and command-level options. Args: new_params: A dictionary of key-value parameters to send. N)rI �update)rQ � new_paramsrT rT rU �ExtendGAParams� s z MetricsCollector.ExtendGAParamsc C s | j �t| �S )z�Convenience function for getting a ga_param of the collector. Args: param_name: The descriptive name of the param (e.g. 'Command Name'). Must be a key in _GA_LABEL_MAP. Returns: The GA parameter specified, or None. )rI r� rH )rQ � param_namerT rT rU � GetGAParam� s zMetricsCollector.GetGAParamr c K s� d|fd|fd|fd|ft d t� fg}|�dd� t�|�D �� |�dd� t�| j�D �� |d u rrt� | j }|�t d |f� tj � t |��}| j �t | jd || jd �� d S ) a� Adds a GA metric with the given parameters to the metrics queue. Args: category: str, the GA Event category. action: str, the GA Event action. label: str, the GA Event label. value: int, the GA Event value. execution_time: int, the execution time to record in ms. **custom_params: A dictionary of key, value pairs containing custom metrics and dimensions to send with the GA Event. r r r r r c S s g | ]\}}|d ur||f�qS rX rT �rx �kr0 rT rT rU ry � s z4MetricsCollector.CollectGAMetric.<locals>.<listcomp>c S s g | ]\}}|d ur||f�qS rX rT r� rT rT rU ry � s Nr �POST)r: �method�bodyrL )rH r8 �extend�six� iteritemsrI r9 rZ r �parse� urlencoder| rM r r: rL ) rQ r\ �action�label�value�execution_time� custom_params�params�datarT rT rU �CollectGAMetric� s( � � � ��z MetricsCollector.CollectGAMetricc @ s( e Zd ZdZdd� ZG dd� de�ZdS )z)MetricsCollector._PeformanceSummaryParamszEThis class contains information to create a PerformanceSummary event.c C s� d| _ d| _d| _d| _t� | _tjr0t�� | _ d| _ d| _ d| _ d| _ t| j�| _d | _d | _d | _d| _d| _d| _d| _d| _d| _d S )Nr F)� num_processes� num_threads� num_retryable_service_errors� num_retryable_network_errors�set�provider_typesr �IS_LINUX�GetDiskCounters�disk_counters_start�uses_fan� uses_slice�thread_idle_time�thread_execution_timer � _ThreadThroughputInformation�thread_throughputs�avg_throughput�total_elapsed_time�total_bytes_transferred�num_objects_transferred�is_daisy_chain� has_file_dst� has_cloud_dst� has_file_src� has_cloud_src�rQ rT rT rU rV � s* z2MetricsCollector._PeformanceSummaryParams.__init__c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd S ) zFMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformationzDA class to keep track of throughput information for a single thread.c C s d| _ d| _d | _d | _d S )Nr )r� r� �task_start_time� task_sizer� rT rT rU rV � s zOMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.__init__c C s || _ || _d S rX )r� r� )rQ r9 Zbytes_to_transferrT rT rU � LogTaskStart� s zSMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.LogTaskStartc C s4 | j || j 7 _ | j| j7 _d | _d | _d S rX )r� r� r� r� )rQ Zend_timerT rT rU � LogTaskEnd s zQMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.LogTaskEndc C s t | j| j�S rX )r r� r� r� rT rT rU � GetThroughput s �zTMetricsCollector._PeformanceSummaryParams._ThreadThroughputInformation.GetThroughputN)�__name__� __module__� __qualname__�__doc__rV r� r� r� rT rT rT rU r� � s r� N)r� r� r� r� rV �objectr� rT rT rT rU �_PeformanceSummaryParams� s 'r� c C s� | � d�dvrdS | jdu r&| �� | _d|v rB| j|d d� dS t�|�D ]�\}}|dv rzt| j|�}|szt| j||� |dv r�t| j|�}t| j||| � |dv r�t| j|�}||k r�t| j||� |d krL| jj� |� qLdS ) ag Updates the _PeformanceSummaryParams object. Args: params: A dictionary of keyword arguments. - uses_fan: True if the command uses fan parallelism. - uses_slice: True if the command uses slice parallelism. - avg_throughput: The average throughput of the data transfer. - is_daisy_chain: True if the transfer uses the daisy-chain method. - has_file_dst: True if the transfer's destination is a file URL. - has_cloud_dst: True if the transfer's destination is in the cloud. - has_file_src: True if the transfer has a file URL as a source. - has_cloud_src: True if the transfer has a cloud URL as a source. - total_elapsed_time: The total amount of time spent on Apply. - total_bytes_transferred: The total number of bytes transferred. - thread_idle_time: The additional amount of time that threads spent idle in Apply. - thread_execution_time: The additional amount of time that threads spent executing in Apply. - num_retryable_service_errors: The additional number of retryable service errors that occurred. - num_retryable_network_errors: The additional number of retryable network errors that occurred. - num_processes: The number of processes used in a call to Apply. - num_threads: The number of threads used in a call to Apply. - num_objects_transferred: The total number of objects transferred, as specified by a ProducerThreadMessage. - provider_types: A list of additional provider types used. - file_message: A FileMessage used to calculate thread throughput and number of objects transferred in the non-parallel case. r )�cp�rsyncN� file_message)r� ) r� r� r� r� r� r� r� r� r� r� r� )r� r� r� r� )r� r� r� ) r� rP r� �_ProcessFileMessager� r� �getattr�setattrr� r� )rQ r� r� �paramZ cur_valuerT rT rU � UpdatePerformanceSummaryParams s* z/MetricsCollector.UpdatePerformanceSummaryParamsc C s\ | j j|j|jf }|jrH| j js:| j js:| j jd7 _|�|j � n|� |j |j � dS )aT Processes FileMessages for thread throughput calculations. Update a thread's throughput based on the FileMessage, which marks the start or end of a file or component transfer. The FileMessage provides the number of bytes transferred as well as start and end time. Args: file_message: The FileMessage to process. r/ N) rP r� Z process_idZ thread_id�finishedr� r� r� r� �timer� �size)rQ r� � thread_inforT rT rU r� R s �z$MetricsCollector._ProcessFileMessagec C s� | � d�}|r6| jf t|d�td t| j�� �i�� t�| j�D ]&\}}| jf t |d�td |i�� qB| � d�}|r�| jt |d� dS )zDAggregates command and error info and adds them to the metrics list.r )r\ r� r r N) r� r� �_GA_COMMANDS_CATEGORYrH �sumrO �valuesr� r� �_GA_ERRORRETRY_CATEGORY�_GA_ERRORFATAL_CATEGORY)rQ � command_name� error_typeZ num_errorsZfatal_error_typerT rT rU � _CollectCommandAndErrorMetricsh s( � ��� � �z/MetricsCollector._CollectCommandAndErrorMetricsc C s� | j du rdS i }dD ] \}}t| j |�|t| < qtjr~| j j}t�� }tdd� |�� D ��tdd� |�� D �� |td < | j j r�| j j r�dnd}nd }||td < | j j r�| j j r�dnd }n| j j r�d nd }||td < | j j | j j }|�rt| j j �t|� |td < | j j�rPdd� | j j�� D �} t| �|td < t| �|td < d�t| j j��|td < | j j �oz| j j| j j �o�| j j| j j| j j �o�| j j| j j �o�| j jd�} d�tdd� t�| �D ���} t| j j �} | j f t | | d�|�� dS )zCAggregates PerformanceSummary info and adds the metric to the list.N))r� r" )r� r# )r� r' )r� r( )r� r& )r� r$ )r� r% c S s g | ]}|d |d �qS �� � rT �rx �statrT rT rU ry � ro zEMetricsCollector._CollectPerformanceSummaryMetric.<locals>.<listcomp>c S s g | ]}|d |d �qS r� rT r� rT rT rU ry � ro r, ZbothZcloud�filer Zfan�sliceZnoner r) c S s g | ] }|� � �qS rT )r� )rx �threadrT rT rU ry � s �r* r+ rw r )Z CloudToCloudZ CloudToFileZ DaisyChainZ FileToCloudZ FileToFilec S s g | ]\}}|r|�qS rT rT )rx Z transfer_typeZcondrT rT rU ry � s �)r\ r� r� ) rP r� rH r r� r� r� r� r� r� r� r� r� r� r� �floatr� �min�maxr{ r| r� r� r� r� r� r� r8 r� r� �_GA_PERFSUM_CATEGORY) rQ r� � attr_namer� Z disk_startZdisk_endZ src_url_typeZstrategyZ total_timeZ throughputsZtransfer_typesr� Zapply_execution_timerT rT rU � _CollectPerformanceSummaryMetric� s� ��� �� �� � � ��� ���z1MetricsCollector._CollectPerformanceSummaryMetricFc C s� | � � | �� | jsdS |s(| j�� }tj�d�dkr>tj }t j dd�}t � |j�}|� t�| j|� W d � n1 s~0 Y t�| j� g | _|dur�t � d| �}t � d�|||��}tjd|g}tj�� }tj�tj�|d < t� } t �|�D ] \} } t � | �| t � | �< q�zDtj || t j �o0t j d �} | j�d � |�r\| � � | j�d � W n> t!�y� | j�d � zt�"|j� W n Y n0 Y n0 dS )aZ Reports the collected metrics using a separate async process. Args: wait_for_report: bool, True if the main process should wait for the subprocess to exit for testing purposes. log_level: int, The subprocess logger's level of debugging for testing purposes. log_file_path: str, The file that the metrics_reporter module should write its logs to. If not supplied, the metrics_reporter module will use a predetermined default path. This parameter is intended for use by tests that need to evaluate the contents of the file at this path. Nr� r� F)�deletezr"%s"z_from gslib.metrics_reporter import ReportMetrics; ReportMetrics(r"{0}", {1}, log_file_path={2})z-cZ PYTHONPATH)�env�shellz$Metrics reporting process started...z#Metrics reporting process finished.z*Metrics reporting process failed to start.)#r� r� rM r= �getEffectiveLevelr� r� r� r; �WARN�tempfile�NamedTemporaryFiler� � ensure_str�name�pickle�dumprk rJ �sys� executable�copy�pathsepr{ r� �dictr� � subprocess�Popen�PY3r � IS_WINDOWS� communicate�OSError�unlink) rQ Zwait_for_reportZ log_levelZ log_file_pathZtemp_metrics_fileZtemp_metrics_file_nameZreporting_codeZexecution_argsZexec_envZsm_envr� r0 �prT rT rU � ReportMetrics� sT , �� � z MetricsCollector.ReportMetrics)r� r� N)N)FNN) r� r� r� r� �_GA_TID� _GA_ENDPOINTrV r~ r� rC � staticmethodr r} � classmethodr� r� r� r> r� r� r r� r� r� r� r� r� r� r rT rT rT rU r- q sF 8o � � )FHi �r- c s t � �� fdd��}|S )z�Function decorator to capture and log any exceptions. This is extra insurance that analytics collection will not hinder the command being run upon an error. Args: func: The function to wrap. Returns: The wrapped function. c sT z� | i |��W S t yN } z&t�d�}|�d� j|� W Y d }~n d }~0 0 d S )N�metricsz6Exception captured in %s during metrics collection: %s)� Exceptionr; r< rk r� )�args�kwds�er= ��funcrT rU �Wrapper? s �z'CaptureAndLogException.<locals>.Wrapperr �r r rT r rU �CaptureAndLogException2 s r c s t � �� fdd��}|S )a\ Function decorator to ignore an exception on collecting thread stats. An exception can happen if the thread_stats dictionary's manager gets shutdown before the thread's process is successfully killed. See _ThreadStat.AggregateStat for how we handle that case. Args: func: The function to wrap. Returns: The wrapped function. c s" z� | i |��W S Y n0 d S rX rT )r r r rT rU r Y s z+CaptureThreadStatException.<locals>.Wrapperr r rT r rU �CaptureThreadStatExceptionK s r c C s t �� } | r| �� dS )z9Reports the metrics that were collected upon termination.N)r- r r )� collectorrT rT rU �Shutdownf s r c C s� t �� }|sdS | r0|�d�s0|�td | i� |rh|�d�shd�tdd� |D ���}|�td |i� |�d�} | szdS |r�d�| d�|��}|�td |i� |r�|�d �s�d�td d� |D ���}|�td |i� |r�|�d �s�|�td |i� dS ) a� Logs info about the gsutil command being run. This only updates the collector's ga_params. The actual command metric will be collected once ReportMetrics() is called at shutdown. Args: command_name: str, The official command name (e.g. version instead of ver). subcommands: A list of subcommands as strings already validated by RunCommand. We do not log subcommands for the help or test commands. global_opts: A list of string tuples already parsed by __main__. sub_opts: A list of command-level options as string tuples already parsed by RunCommand. command_alias: str, The supported alias that the user inputed. Nr r rw c S s g | ]}|d � d��qS �r �-��strip�rx �optrT rT rU ry � ro z$LogCommandParams.<locals>.<listcomp>z{0} {1}� r c S s g | ]}|d � d��qS r! r# r% rT rT rU ry � ro r )r- r r� r� rH r{ r| rJ ) r� Z subcommands� global_optsZsub_optsZ command_aliasr Zglobal_opts_stringZfull_command_nameZsub_opts_stringrT rT rU �LogCommandParamso s2 � � �r) c C s@ t �� }|r<|j| j d7 < | jr2tdd� n tdd� dS )z�Logs that a retryable error was caught for a gsutil command. Args: message: The RetryableErrorMessage posted to the global status queue. r/ )r� )r� N)r- r rO r� Zis_service_error�LogPerformanceSummaryParams)�messager rT rT rU �LogRetryableError� s r, c C s& t �� }|r"|�td | jji� dS )z}Logs that a fatal error was caught for a gsutil command. Args: exception: The exception that the command failed on. r N)r- r r� rH � __class__r� )� exceptionr rT rT rU � LogFatalError� s �r/ c K s t �� }|r|�| � dS )a� Logs parameters necessary for a PerformanceSummary. gsutil periodically monitors its own threads; at the end of the execution of each cp/rsync command, it will present a performance summary of the command run. Args: **params_to_update: A dictionary. See UpdatePerformanceSummaryParams for details. The argument ambiguity at this level allows for flexibility in dealing with arguments that are processed similarly. N)r- r r� )Zparams_to_updater rT rT rU r* � s r* c C s� t j�dd�} tj�t�s�| s�t�� s�t dt � d� d �}t }|� � d dkrZt�� j}t�tj�t�� ttd��}|�|� W d � n1 s�0 Y d S ) z�Asks a user to opt-in to data collection if a UUID file does not exist. If the user agrees, generates a UUID file. Will not prompt if part of SDK. r. rc � z�gsutil developers rely on user feedback to make improvements to the tool. Would you like to send anonymous usage statistics to help improve gsutil? [y/N]r' r �y�wN)r? r@ rY r� r� r� r� r r� r �textwrap�fillr� rg �uuidZuuid4�hex�CreateDirIfNeeded�dirnamer� �write)Zdisable_promptZenable_analyticsZ text_to_writer� rT rT rU �'CheckAndMaybePromptForAnalyticsEnabling� s ��� r: c C s | d u rt � � } t| d �S )Ni� )r� rN )Z time_in_secrT rT rU r8 � s r8 )NNNNN)N)@r� � __future__r r r r �atexit� collectionsr � functoolsr r; r� r rK �rerF r r r� r3 r� r5 r� � six.movesr r r? �gslibr Zgslib.metrics_tupler � gslib.utilsr �gslib.utils.unit_utilr r r r rB r� r� r� r� �compilerD r� � expanduserr{ r� r� rH r� r- r r �registerr r) r, r/ r* r: r8 rT rT rT rU �<module> s� �� F � 1