File: //lib/python3/dist-packages/ocrmypdf/__pycache__/_sync.cpython-38.pyc
U
��Z^$: � @ s� d dl Z d dlZ d dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
d dl m Z d dl
Z
d dlmZ ddlmZ ddlmZmZmZ dd lmZmZmZmZmZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z, dd
l-m.Z.m/Z/m0Z0 dd l1m2Z2m3Z3 dd l4m5Z5 dd
l6m7Z7 ddl8m9Z9 edd�Z:dd� Z;dd� Z<dd� Z=dd� Z>dd� Z?dd � Z@d d � ZAG d d � d eB�ZCd!d"� ZDd)d$d%�ZEd*d'd(�ZFdS )+� N)�
namedtuple)�Path)�mkdtemp)�tqdm� )�
OcrGrafter)�
PDFContext�cleanup_working_files� make_logger)�convert_to_pdfa�
copy_final�create_ocr_image�create_pdf_page_from_image�create_visible_page_jpg�generate_postscript_stub�get_orientation_correction� get_pdfinfo�is_ocr_required�merge_sidecars�metadata_fixup�ocr_tesseract_hocr�ocr_tesseract_textonly_pdf� optimize_pdf�preprocess_clean�preprocess_deskew� preprocess_remove_background� rasterize�rasterize_preview�render_hocr_page�!should_visible_page_image_use_jpg�triage�validate_pdfinfo_options)�check_requested_output_file�create_input_file�report_output_file_size)�ExitCode�ExitCodeException)�qpdf)�available_cpu_count)�file_claims_pdfa�
PageResultz>pageno, pdf_page_from_image, ocr, text, orientation_correctionc C s. |rt || �}|r t|| �}|r*t|| �}|S �N)r r r )� page_contextZimage�remove_background�deskew�clean� r0 �0/usr/lib/python3/dist-packages/ocrmypdf/_sync.py�
preprocessJ s
r2 c C s` | j }d}d }d }d }t| ��rL|jr<t| j| �}t|| �}t| j| |dd�}t|j|j |j
g�s~t | ||j |j
dd� }} nV|js�t | ||j |j
|j d�} |j
r�t| j| |ddd�}
n|}
t | |
|j |j
|jd�}t|| �} d }|j�s| } t| j��rt| | �} t| | �}|jdk�r2t| | �\}
}t|
| �}|jd k�rLt| | �\}}t| j||||d
�S ) Nr F)�
correction�remove_vectors)r/ TZ_ocr)r3 r4 Z
output_tagZhocrZsandwich)�pagenoZpdf_page_from_imageZocr�text�orientation_correction)�optionsr Z rotate_pagesr �originr r �anyr/ Z clean_finalr4 r2 r- r. Zlossless_reconstructionr
r Zpageinfor r Z pdf_rendererr r r r* r5 )r, r8 r7 Zpdf_page_from_image_outZocr_outZtext_outZrasterize_preview_outZ
rasterize_outZ ocr_imageZpreprocess_outZrasterize_ocr_outZ
ocr_image_outZvisible_image_outZhocr_outr0 r0 r1 �exec_page_syncT s�
��� ���
� �
��r; c C s: | }|j j�d�r&t|�}t|||�}t||�}t||�S )N�pdfa)r8 � output_type�
startswithr r r r )Zpdf_file�contextZpdf_outZ ps_stub_outr0 r0 r1 � post_process� s
r@ c C s@ t � t jt j� tj�| �}t�� }g |_|�|� |tj _
dS )z Initialize a process pool workerN) �signal�SIGINT�SIG_IGN�loggingZhandlersZ QueueHandler� getLogger�
addHandler�PIL�Image�MAX_IMAGE_PIXELS)�queue�
max_pixels�h�rootr0 r0 r1 � worker_init� s
rN c C s |t j_d S r+ )rG rH rI )Z_queuerK r0 r0 r1 �worker_thread_init� s rO c C sn z.| � � }|dkrW qjt�|j�}|�|� W q tk
rf ddl}tdtj d� |j
tj d� Y q X q dS )a� Listen to the worker processes and forward the messages to logging
For simplicity this is a thread rather than a process. Only one process
should actually write to sys.stderr or whatever we're using, so if this is
made into a process the main application needs to be directed to it.
See https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
Nr zLogging problem)�file) �getrD rE �nameZhandle� Exception� traceback�print�sys�stderr� print_exc)rJ �recordZloggerrT r0 r0 r1 � log_listener� s rZ c
C sN t t| j�| jj�}|dkr*| j�d|� t d| jj| �}| jjdkrVtj �
� | j_| jj� dt |�� |dkr�| j�d|� | jj
r�ddlm} t}n
tj}t}dgt| j� }t| �}t�d �}tjt|fd
�}|�� td t| j� d d
d| jj d���} ||||tj j fd�}
z�zh|
�!t"| �#� �} z2| �$� } | j%|| j&< | �'� |�(| � | �'� W n t)k
�rz Y �q�Y nX �q,W nN t*k
�r� |
�+� � Y n0 t,k
�r� tj �-dd��s�|
�+� � Y nX W 5 |� d� |
� � |
� � X W 5 Q R X |� � | jj.�r(t/|| �}
t0|
| jj.| � |�1� }t2|| �}t0|| jj3| � dS )z!Execute the pipeline concurrentlyr z&Start processing %d pages concurrently� NZOMP_THREAD_LIMITz&Using Tesseract OpenMP thread limit %dr )�Pool���)�target�args� ZOCRZpageg �?)ZtotalZdescZunitZ
unit_scale�disable)Z processes� initializerZinitargs�PYTEST_CURRENT_TEST� )4�min�len�pdfinfor8 �jobs�log�infoZ
tesseract_env�os�environ�copy�
setdefault�strZ use_threadsZmultiprocessing.dummyr\ rO �multiprocessingrN r ZQueue� threadingZThreadrZ �startr � progress_barrG rH rI Z
put_nowait�close�joinZimap_unorderedr; Zget_page_contexts�nextr6 r5 �updateZ
graft_page�
StopIteration�KeyboardInterruptZ terminaterS rQ Zsidecarr r �finalizer@ � output_file)r? Z max_workersZ tess_threadsr\ rb ZsidecarsZocrgraftZ log_queueZlistenerZpbarZpoolZresultsZ page_resultr6 Zpdfr0 r0 r1 �exec_concurrent� sv
�
�
r| c @ s e Zd ZdZdS )�
NeverRaisez!An exception that is never raisedN)�__name__�
__module__� __qualname__�__doc__r0 r0 r0 r1 r} F s r} c C s$ t jdkr| |kS t j�| |�S d S )N�nt)rk rR �path�samefile)�f1�f2r0 r0 r1 r� L s
r� rd c C sB t j| dd�}|�t j� t �d�}|�|� t �|��|� |S )NT)Zdelayz7[%(asctime)s] - %(name)s - %(levelname)7s - %(message)s)rD Z FileHandlerZsetLevel�DEBUGZ FormatterZ setFormatterrE rF )Z log_filename�prefixZlog_file_handlerZ formatterr0 r0 r1 �configure_debug_loggingS s �
r� Fc
C s� t | t�}| jst� | _tdd�}| js2| jdkrPtj� dd�sPt
t |�d � �z,�zt
| � t| |�\}}t||tj�|d�| |�}t|| j| jd�}t| |||�}t|� t|� | jd kr�|�d
� n�t| jtj�r�n�| j � d ��r0t | j�} | d �r|�d
| d � n |� d| d � t j!W W ��RS t"�#| j|��sX|� d� t j$W W ��*S t%| || j� W �n |�sxt&nt'k
�r� }
z4| jdk�r�|�(d� n
|�)d� t j* W Y �W ��S d }
~
X Y n� |�s�t+nt'k
�r6 }
zFt,|
��r|�)dt-|
�jt,|
�� n|�)t-|
�j� |
j. W Y �W �XS d }
~
X Y nF |�sBt/nt'k
�rz }
z |�(d� t j0 W Y �W �S d }
~
X Y nX W 5 t || � X t j1S )Nzcom.github.ocrmypdf.)r� r rc rd z debug.logz
origin.pdf)Zdetailed_page_analysisZprogbar�-zOutput sent to stdoutr< �passz!Output file is a %s (as expected)Z conformancez5Output file is okay but is not PDF/A (seems to be %s)z)Output file: The generated PDF is INVALIDry z%s: %sz2An exception occurred while executing the pipeline)2r
r~ rh r( r Zkeep_temporary_files�verboserk rl rQ r� r r r" r# r r� ru r Zredo_ocrrs r r! r| r{ rj r� �devnullr= r> r) Zwarningr% Zpdfa_conversion_failedr' ZcheckZinvalid_output_pdfr$ ry r} Z exception�errorZctrl_cr& ro �typeZ exit_coderS Z other_error�ok) r8 Zapiri Z work_folderZstart_input_fileZoriginal_filenameZ
origin_pdfrg r? Z pdfa_info�er0 r0 r1 � run_pipeline^ s|
� � �
��
$ r� )rd )F)GrD Zlogging.handlersrp rk rA rV rq � collectionsr �pathlibr Ztempfiler rG r Z_graftr Z _jobcontextr r r
Z _pipeliner r r
r r r r r r r r r r r r r r r r r r r r! Z _validationr" r# r$ �
exceptionsr% r& �execr' Zhelpersr( r< r) r* r2 r; r@ rN rO rZ r| rS r} r� r� r� r0 r0 r0 r1 �<module> sD d �
W
b