File: //lib/python3/dist-packages/ocrmypdf/__pycache__/_pipeline.cpython-38.pyc
U
��Z^Av � @ s d dl Z d dlZd dlZd dlmZmZ d dlmZ d dlmZ d dl Z d dl
Z
d dl m Z d dl
mZ ddlmZ dd lmZ dd
lmZ dd lmZmZmZmZmZ dd lm Z m Z dd
l m Z ddl m!Z! ddl"m"Z" ddl#m$Z$ ddl%m&Z&m'Z'm(Z( dZ)dd� Z*dTdd�Z+dd� Z,dUdd �Z-d d � Z.d d � Z/d!d"� Z0d#d$� Z1d%d&� Z2d'd(� Z3d)d*� Z4d+d,� Z5dVd.d/�Z6d0d1� Z7d2d3� Z8d4d5� Z9d6d7� Z:d8d9� Z;d:d;� Z<d<d=� Z=d>d?� Z>d@dA� Z?dBdC� Z@dDdE� ZAdFdG� ZBdHdI� ZCdJdK� ZDdLdM� ZEdNdO� ZFdPdQ� ZGdRdS� ZHdS )W� N)�datetime�timezone)�Path)� copyfileobj)�encode_pdf_date)�Image� )� leptonica)� PROGRAM_NAME)� __version__)�DpiError�EncryptedPdfError�InputFileError�PriorOcrFoundError�UnsupportedImageFormatError)� ghostscript� tesseract)� safe_symlink)�
HocrTransform)�optimize)�generate_pdfa_ps)�
Colorspace�Encoding�PdfInfoi� c
C s� |� d� zt�| �}W nB tk
rZ } z$|�t|��| |j�� t� |�W 5 d }~X Y nX |�� |� d� d|j kr�|j d dkr�|j s�|� d|j
� |� d|j d � |�d� t � �n&|j s�|� d|j
� |�d� t � �|j d k�r|�d
� t� �d |j k�rB|j d k�r&|� d
� n |j dk�rB|�d� t� �W 5 Q R X z`|� d� t
j}|j �rxt
�|j |j f�}t|d��}t
j| |d|d� W 5 Q R X |� d� W n8 t
jk
�r� } z|�|� t� |�W 5 d }~X Y nX d S )Nz6Input file is not a PDF, checking if it is an image...zInput file is an image�dpi)�` r zImage size: (%d, %d)zImage resolution: (%d, %d)z�Input file is an image, but the resolution (DPI) is not credible. Estimate the resolution at which the image was scanned and specify it using --image-dpi.z�Input file is an image, but has no resolution (DPI) in its metadata. Estimate the resolution at which image was scanned and specify it using --image-dpi.)ZRGBAZLAzEThe input image has an alpha channel. Remove the alpha channel first.Z
iccprofileZRGBz-Input image has no ICC profile, assuming sRGBZCMYKz/Input CMYK image has no ICC profile, not usablez+Image seems valid. Try converting to PDF...�wbF)�
layout_fun�
with_pdfrw� outputstreamz,Successfully converted to PDF, processing...)�infor �open�EnvironmentError�error�str�replace�
input_filer � image_dpi�sizer �mode�img2pdfZdefault_layout_fun�get_fixed_dpi_layout_fun�convertZImageOpenError)r& � output_file�options�log�im�er Zoutf� r2 �4/usr/lib/python3/dist-packages/ocrmypdf/_pipeline.py�triage_image_file2 sf
�� �
� �
r4 � c C s> t | d��}|�|�}W 5 Q R X t�d|�}|r:|�d�S dS )z�Try to find version signature at start of file.
Not robust enough to deal with appended files.
Returns empty string if not found, indicating file is probably not PDF.
�rbs
%PDF-(\d\.\d)r � )r! �read�re�search�group)r& Z
search_window�fZ signature�mr2 r2 r3 �_pdf_guess_versiono s
r> c
C s� z,t |�r*|jr|�d� t||� |W S W nL tk
rx } z.|�d|� �� t|��|| �}t|�|�W 5 d }~X Y nX t ||||� |S )NzTArgument --image-dpi is being ignored because the input file is a PDF, not an image.zTemporary file was at: )
r> r' �warningr r" �debugr$ r% r r4 )Zoriginal_filenamer& r- r. r/ r1 �msgr2 r2 r3 �triage s �
rB Fc C sL zt | ||d�W S tjk
r, t� �Y n tjk
rF t� �Y nX d S )N)�detailed_page_analysis�progbar)r �pikepdfZ
PasswordErrorr
ZPdfErrorr )r& rC rD r2 r2 r3 � get_pdfinfo� s �
rF c C s� | j }| j}| j}|jr(|�d� t� �|jrJ|j�d�rJ|�d� t� �|j r�|j
rh|�d� t� �n|� d� |j s�|�
d� d S )Nz~This PDF contains dynamic XFA forms created by Adobe LiveCycle Designer and can only be read by Adobe Acrobat or Adobe Reader.�pdfaa( This input file uses a PDF feature that is not supported by Ghostscript, so you cannot use --output-type=pdfa for this file. (Specifically, it uses the PDF-1.6 /UserUnit feature to support very large or small page sizes, and Ghostscript cannot output these files.) Use --output-type=pdf instead.zVThis PDF has a user fillable form. --redo-ocr is not currently possible on such files.z_This PDF has a fillable form. Chances are it is a pure digital document that does not need OCR.z�Use the option --force-ocr to produce an image of the form and all filled form fields. The output PDF will be 'flattened' and will no longer be fillable.)r/ �pdfinfor. Zneeds_renderingr# r Z has_userunit� output_type�
startswithZ has_acroform�redo_ocrr? � force_ocrr )�contextr/ rH r. r2 r2 r3 �validate_pdfinfo_options� s4 �����rN c C sT t | jp
t|jpd| jr tnd�}t | jp,t|jp4d| jr>tnd�}t|�t|�fS )z+Get the DPI when nonsquare DPI is tolerabler )�max�xres�VECTOR_PAGE_DPI�
oversample�
has_vector�yres�float)�pageinfor. rP rT r2 r2 r3 � get_page_dpi� s � �rW c C sP | j pd}| jpd}| jp d}tt|| p,t|| p6t| jr@tnd|jpJd��S )zBGet the DPI when we require xres == yres, scaled to physical unitsr r )rP rT �userunitrU rO rQ rS rR )rV r. rP rT rX r2 r2 r3 �get_page_square_dpi� s
��rY c C s. t t| jp t| jpt| jr tnd|jp(d��S )z=Get the DPI when we require xres == yres, in Postscript unitsr )rU rO rP rQ rT rS rR )rV r. r2 r2 r3 �get_canvas_square_dpi� s ��rZ c C sv | j }| j}| j}d}|jrH|j|jkrH|�d|j� d|j� �� d}n�|jr�|jsj|jsj|j sjt
d��nR|jr�|� d� d}n<|j r�|j r�|�
d� n
|� d� d}n|jr�|� d � d}n\|j�s|j�s|jr�|jr�|� d
|j� d �� n*|j�r |�
d t� d
�� n|� d� d}|�rr|j�rr|j�rr|j|j }||jd k�rrd}|�
d|d d�d|jd�d�� |S )NTzskipped z as requested by --pages Fz@page already has text! - aborting (use --force-ocr to force OCR)z@page already has text! - rasterizing text and running OCR anywayzYsome text on this page cannot be mapped to characters: consider using --force-ocr insteadz redoing OCRz$skipping all processing on this pagez$page has no images - rasterizing at z3 DPI because --force-ocr --oversample was specifiedz>page has no images - all vector content will be rasterized at za DPI, losing some resolution and likely increasing file size. Use --oversample to adjust the DPI.z�page has no images - skipping all processing on this page to avoid losing detail. Use --force-ocr if you wish to perform OCR on pages that have vector content.�@B z page too big, skipping OCR (z.1fz MPixels > z MPixels --skip-big))rV r. r/ Zpages�pagenor@ Zhas_textrL Z skip_textrK r r Zhas_corrupt_text�warn�imagesZlossless_reconstructionrR rQ Zskip_bigZ width_pixelsZ
height_pixels)� page_contextrV r. r/ Z ocr_requiredZ pixel_countr2 r2 r3 �is_ocr_required� s\ �
�
�
�� �r` c
C sR |� d�}t|j|j�}t|j|j�}tj| |||d|j||f|jjd d� |S )Nzrasterize_preview.jpgZjpeggrayr )rP rT �
raster_devicer/ �page_dpir\ ) �get_pathrZ rV r. rY r �
rasterize_pdfr/ r\ )r& r_ r- �
canvas_dpirb r2 r2 r3 �rasterize_preview; s
�
rf c C s� ddddd�}dddd d�}| j j}d
}|j| jjkrR|d krLd || }qdd
}n|d kr`d}nd}d
}|d kr�d|�|d�� d�}|d|�|jd�� �7 }|� d|jd�d|� �S )z=
Describe the page rotation we are going to perform.
u ⇧u ⇨u ⇩u ⇦)r �Z � i � u ⬏u ↻u ⬑r7 r z will rotate zrotation appears correctz confidence too low to rotatez no changezwith existing rotation �?z, zpage is facing z
, confidence z.2fz - )rV �rotation�
confidencer. �rotate_pages_threshold�get�angle)r_ � orient_conf�
correction� directionZturnsZexisting_rotation�actionZfacingr2 r2 r3 �describe_rotationL s rt c C s^ t j| |jj|jj|j|jjd�}|jd }|j�t |||�� |j
|jj krZ|dkrZ|S dS )a�
Work out orientation correct for each page.
We ask Ghostscript to draw a preview page, which will rasterize with the
current /Rotate applied, and then ask Tesseract which way the page is
oriented. If the value of /Rotate is correct (e.g., a user already
manually fixed rotation), then Tesseract will say the page is pointing
up and the correction is zero. Otherwise, the orientation found by
Tesseract represents the clockwise rotation, or the counterclockwise
correction to rotation.
When we draw the real page for OCR, we rotate it by the CCW correction,
which points it (hopefully) upright. _graft.py takes care of the orienting
the image and text layers.
)� engine_mode�timeoutr/ �
tesseract_envih r ) r Zget_orientationr. �
tesseract_oem�tesseract_timeoutr/ rw ro r rt rl rm )Zpreviewr_ rp rq r2 r2 r3 �get_orientation_correctioni s �
��rz r7 c s
ddddg� d�|d kr |j j}|�d|� d��}|j}� �fdd �}|jD ]N}|jd
kr\qL|jd krL|jtj kr||d��qL|jtj
kr�|d��qL|d��qL|j r�|d��� � } |j �
d | � �� t||j �}
t||j �} tj| ||
|
| |j | | f|jd ||d
�
|S )NZpngmonoZpnggrayZpng256Zpng16mr � rasterizez.pngc s t �� �| ��S �N)rO �index)Zcs�Z colorspacesZ
device_idxr2 r3 �at_least� s zrasterize.<locals>.at_least�imager zRasterize with )rP rT ra r/ rb r\ rk Z
filter_vector)r. �remove_vectorsrc rV r^ Ztype_�bpcZcolorr r} ZgrayrS r/ r@ rZ rY r rd r\ ) r& r_ rq Z
output_tagr� r- rV r r� Zdevicere rb r2 r~ r3 r{ � sF
� r{ c C sD t dd� |jjD ��r0|�d�}t�| |� |S |j�d� | S d S )Nc s s | ]}|j d kV qdS )r N)r� )�.0r� r2 r2 r3 � <genexpr>� s z/preprocess_remove_background.<locals>.<genexpr>z pp_rm_bg.pngz'background removal skipped on mono page)�anyrV r^ rc r Zremove_backgroundr/ r )r& r_ r- r2 r2 r3 � preprocess_remove_background� s
r� c C s* |� d�}t|j|j�}t�| ||� |S )Nz
pp_deskew.png)rc rY rV r. r Zdeskew)r& r_ r- r r2 r2 r3 �preprocess_deskew� s
r� c C s@ ddl m} |�d�}t|j|j�}|�| |||j|jj� |S )Nr )�unpaperz pp_clean.png) �execr� rc rY rV r. Zcleanr/ Z unpaper_args)r& r_ r� r- r r2 r2 r3 �preprocess_clean� s
�r� c C sx |� d�}|j}t�| ���R}ddlm} ddlm} |�d|j�}|�|�}|j d \} }
|j
� d| |
f � |j �s&d} |j
r�d } |jj| dd
�D ]�} d d � | D �}
t| �d
t|
�d
}}|
d | |j|
d | |
d | |j|
d | g}dd � |D �}|j
� d|� |j||d� q�|j�rJtj�|�}|�� }|�� }~t| �t|
�f}|j||d� W 5 Q R X |S )z�Create the image we send for OCR. May not be the same as the display
image depending on preprocessing. This image will never be shown to the
user.zocr.pngr )�
ImageColor)� ImageDrawz#ffffffr zresolution %r %rNT)ZvisibleZcorruptc S s g | ] }t |��qS r2 )rU )r� �vr2 r2 r3 �
<listcomp> s z$create_ocr_image.<locals>.<listcomp>g R@� � r c S s g | ]}t t|���qS r2 )�int�round)r� �cr2 r2 r3 r�
s z blanking %r)Zfill)r )rc r. r r! �PILr� r� Zgetcolorr) r r/ r@ rL rK rV Z
get_textareasrU ZheightZ rectangleZ thresholdr ZPixZfrompilZ#masked_threshold_on_background_normZtopilr� �save)r� r_ r- r. r0 r� r� ZwhiteZdrawrP rT �maskZtextareaZbboxZxscaleZyscaleZ pixcoordsZpixr r2 r2 r3 �create_ocr_image� sF
�
� r� c
C sX |� d�}|� d�}|j}tj| ||g|j|j|j|j|j|j |j
|j |j d� ||fS )Nz
ocr_hocr.hocrz ocr_hocr.txt) r& Z output_files�languageru �
tessconfigrv � pagesegmode�
user_words�
user_patternsrw r/ )
rc r. r Z
generate_hocrr� rx �tesseract_configry �tesseract_pagesegmoder� r� rw r/ )r& r_ Zhocr_outZ
hocr_text_outr. r2 r2 r3 �ocr_tesseract_hocr s"
�
r� c C s | j otdd� | j D ��S )Nc s s | ]}|j tjkV qd S r| )�encr Zjpeg)r� r0 r2 r2 r3 r� 1 s z4should_visible_page_image_use_jpg.<locals>.<genexpr>)r^ �all)rV r2 r2 r3 �!should_visible_page_image_use_jpg/ s r� c C sl |� d�}t�| ��N}t|j|j�}|j�d||f�}t|d �t|d �f}|j |d|d� W 5 Q R X |S )Nz visible.jpgr r r ZJPEG)�formatr )
rc r r! rY rV r. r rn r� r� )r� r_ r- r0 Z fallback_dpir r2 r2 r3 �create_visible_page_jpg4 s
r� c
C s� |� d�}t|j|j�}t�||f�}t| d��F}t|d��0}|j�d� tj |d||d� |j�d� W 5 Q R X W 5 Q R X |S )Nz visible.pdfr6 r r, F)r r r z convert done)
rc rY rV r. r* r+ r! r/ r@ r, )r� r_ r- r r Zimfile�pdfr2 r2 r3 �create_pdf_page_from_imageE s
� r� c C s: |� d�}t|j|j�}t| |�}|j|d dddd� |S )Nz ocr_hocr.pdfFT)Z
imageFileNameZshowBoundingboxesZ
invisibleTextZinterwordSpaces)rc rY rV r. r Zto_pdf)Zhocrr_ r- r �
hocrtransformr2 r2 r3 �render_hocr_pageZ s
�r� c C sZ |� d�}|� d�}|j}tj| d |||j|jd|j|j|j|j |j
|j |j d� ||fS )Nz ocr_tess.pdfz ocr_tess.txtT)� input_imageZskip_pdf�
output_pdf� output_textr� ru Z text_onlyr� rv r� r� r� rw r/ )
rc r. r Z generate_pdfr� rx r� ry r� r� r� rw r/ )r� r_ r� r� r. r2 r2 r3 �ocr_tesseract_textonly_pdfh s(
�r� c s� � fdd���fdd�dD �}d}|d k rx|j r:|j |d< |jrJ|j|d< |jrZ|j|d < |jrj|j|d
< |jd krxd }t� d
t� d|� d
t�� � �|d< dt j
� �|d< dt j kr�t j d |d< dt j kr�t j d |d< t
t�tj��|d< |S )Nc s4 z� j | }t|�W S ttfk
r. Y dS X d S )Nr7 )�docinfor$ �KeyError� TypeError)�key�s)�base_pdfr2 r3 �from_document_info� s
z'get_docinfo.<locals>.from_document_infoc s i | ]}|� |��qS r2 r2 )r� �k)r� r2 r3 �
<dictcomp>� s � z get_docinfo.<locals>.<dictcomp>)�/Title�/Author� /Keywords�/Subjectz
/CreationDateZOCRr� r� r� r� ZsandwichzOCR-PDFri z
/ Tesseract z/Creatorzpikepdf z /ProducerZOCRMYPDF_CREATORZOCRMYPDF_PRODUCERz/ModDate)�titleZauthor�keywordsZsubjectZ pdf_rendererr
�VERSIONr �versionrE r �os�environr r Znowr Zutc)r� r. ZpdfmarkZ renderer_tagr2 )r� r� r3 � get_docinfo s2
�
�
r� c C s | � d�}t|� |S )Nzpdfa.ps)rc r )rM r- r2 r2 r3 �generate_postscript_stub� s
r� c C s� |j }|j}|�d�}|�d�}d}t�| ��`}|jrp|j�� D ].\} }
dt|
�kr@t|
��dd�|j| < d}q@|r�|� |� n
t
| |� W 5 Q R X t j |j
||g||j|j|jd d� |S ) Nzfix_docinfo.pdfzpdfa.pdfF� � T���)Z pdf_versionZ pdf_pagesr- Z compressionr/ Z pdfa_part)r. rH rc rE r! r� �items�bytesr% r� r r Z
generate_pdfaZ min_versionZpdfa_image_compressionr/ rI ) Z input_pdfZ
input_ps_stubrM r. Z
input_pdfinfoZfix_docinfo_filer- ZmodifiedZpdf_filer� r� r2 r2 r3 �convert_to_pdfa� s.
� r� c C s$ t �| �j}||jjd kr dS dS )Nr[ TF)r� �stat�st_sizer. Z
fast_web_view)� working_filerM Zfilesizer2 r2 r3 �should_linearize� s r� c
s� � � d�}� j�� �fdd�}t�� j���}t�| ���}t|��}|�� �V}|j|ddd� d|krt|�dd�|d< |�� }t |�
� �t |�
� � } || � W 5 Q R X |j |d d tj j
�jd
kr�t| � �ndd � W 5 Q R X W 5 Q R X |S ) Nz metafix.pdfc sN | sd S �j �d�r0� j�d� � j�d| � n� j�d� � j�d| � d S )NrG z�Some input metadata could not be copied because it is not permitted in PDF/A. You may wish to examine the output PDF's XMP metadata.z1The following metadata fields were not copied: %rz^Some input metadata could not be copied.You may wish to examine the output PDF's XMP metadata.)rI rJ r/ r? r@ r# r )�missing�rM r. r2 r3 �report_on_metadata� s" � �� �z*metadata_fixup.<locals>.report_on_metadataF)Zdelete_missingZ
raise_failurezxmp:CreateDatezxmp:ModifyDater7 Tr �Zcompress_streamsZ
preserve_pdfaZobject_stream_modeZ linearize)rc r. rE r! �originr� Z
open_metadataZload_from_docinforn �set�keysr� �ObjectStreamMode�generater r� )
r� rM r- r� Zoriginalr� r� �metaZ
meta_originalr� r2 r� r3 �metadata_fixup� s,
� � r� c C s6 |� d�}tddtjjt| |�d�}t| |||� |S )Nz optimize.pdfTr� )rc �dictrE r� r� r� r )r& rM r- Z
save_settingsr2 r2 r3 � optimize_pdf s
�r� c C s� |� d�}t|ddd���}t| �D ]�\}}|dkr<|�d� |r�t|ddd��6}|�� }|�d�rv|�|d d� � n
|�|� W 5 Q R X q"|�d |d
� d �� q"W 5 Q R X |S ) Nz sidecar.txt�wzutf-8)�encodingr � �rr� z[OCR skipped on page r �])rc r! � enumerate�writer8 �endswith)Z txt_filesrM r- �streamZpage_numZtxt_fileZin_Ztxtr2 r2 r3 �merge_sidecars s
"r� c
C sl |j �d| |� t| d��H}|dkr>t|tjj� tj�� n t|d��}t||� W 5 Q R X W 5 Q R X d S )Nz%s -> %sr6 �-r )r/ r@ r! r �sys�stdout�buffer�flush)r& r- rM Z input_streamZ
output_streamr2 r2 r3 �
copy_final. s r� )r5 )FF)r r7 N)Ir� r9 r� r r �pathlibr Zshutilr r* rE Zpikepdf.models.metadatar r� r r7 r Z_versionr
r r� �
exceptionsr r
r r r r� r r Zhelpersr r� r r rG r rH r r r rQ r4 r> rB rF rN rW rY rZ r` rf rt rz r{ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r2 r2 r2 r3 �<module> sl =
) J & �
5
8'#5