%$}ge(dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZddlmZmZejjd kr,eejd rejd ejjd kr,eejd rejd d e de fdZde de fdZde de fdZddZde ddfdZ e!dkr edSdS)z. Implementing a basic command-line interface. N)version)python_version)Any) cli_crawler cli_discoveryexaminefile_processing_pipelineload_blacklistload_input_dictprobe_homepageurl_processing_pipeline write_result)PARALLEL_CORESSUPPORTED_FMT_CLIzUTF-8 reconfigurezutf-8)encodingparserreturnc  |dd}|}|dd}|dd}|}|dd}|d d }|}|d d d t|ddt|dddt|ddtt |dddt|ddd|ddd t|d!d"t|d#d$d|d%d&d'd(d)*|d+d,d'd(d)*|d-d.d'd(d)*|d/d0d'd(d)*|d1d2d'd(d)*|d3d4d|d5d6d7t8|d9d:d;d|dd?d|d@dAd|dBdCdD|dEdFdD|dGdHd|dIdJd|dKdLt|dMdNd|dOdPt|dQdRd|dSdTd|dUdVt dWX|dYdZd|d[d\d|d]d^d|d_d`d|dadbd|dcddd|dedfd|dgdhdidjdkl|dmdndodptdqdrts|S)tz,Add argument groups and arguments to parser.Inputz%URLs, files or directories to processOutputz+Determines if and how files will be written NavigationzLink discovery and web crawling Extractionz-Customization of text and metadata processingFormatzSelection of the output formatz-iz --input-filez'name of input file for batch processing)helptypez --input-dirz5read files from a specified directory (relative path)z-uz--URLzcustom URL downloadz --parallelzAspecify a number of cores/threads for downloads and/or processing)rrdefaultz-bz --blacklistz:file containing unwanted URLs to discard during processingz--listz/display a list of URLs without downloading them store_true)ractionz-oz --output-dirz6write results in a specified directory (relative path)z --backup-dirz9preserve a copy of downloaded files in a backup directoryz --keep-dirsz-keep input directory structure and file namesz--feedz.look for feeds and/or pass a feed URL as input?TF)rnargsconstrz --sitemapzBlook for sitemaps for the given website and/or enter a sitemap URLz--crawlzJcrawl a fixed number of pages within a website starting from the given URLz --explorez=explore the given websites (combination of sitemap and crawl)z--probez?probe for extractable content (works best with target language)z --archivedz=try to fetch URLs from the Internet Archive if downloads failz --url-filterzLonly process/output URLs containing these patterns (space-separated strings)+)rr"rz-fz--fastz!fast (without fallback detection)z --formattingz,include text formatting (bold, italic, etc.)z--linksz5include links along with their targets (experimental)z--imagesz.include image sources in output (experimental)z --no-commentszdon't output any comments store_falsez --no-tableszdon't output any table elementsz--only-with-metadataz4only output those documents with title, URL and datez--with-metadataz&extract and add metadata to the outputz--target-languagez*select a target language (ISO 639-1 codes)z --deduplicatez+filter out duplicate documents and sectionsz --config-filezAoverride standard extraction parameters with a custom config filez --precisionz;favor extraction precision (less noise, possibly less text)z--recallz8favor extraction recall (more text, possibly more noise)z--output-formatzdetermine output formattxt)rchoicesrz--csvzshorthand for CSV outputz--htmlzshorthand for HTML outputz--jsonzshorthand for JSON outputz --markdownzshorthand for MD outputz--xmlzshorthand for XML outputz--xmlteizshorthand for XML TEI outputz--validate-teizvalidate XML TEI outputz-vz --verbosecountrz&increase logging verbosity (-v or -vv))r rrz --versionz!show version information and exitrz Trafilatura trafilaturaz - Python )rr r) add_argument_groupadd_mutually_exclusive_group add_argumentstrintrrrr) rgroup1 group1_exgroup2group3 group3_exgroup4group5 group5_exs O/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/cli.pyadd_argsr8s & &w0W X XF3355I  & &x1^ _ _F  & &|5V W WF3355I  & &|5d e eF  & &x1Q R RF3355I 4F """=T """42 """  ` .::: mY """ N+--- nU """ X """  L+---8M!u>>>;a!u>>>9i!u>>>;\!u>>>9^!u>>>  \+--- k!--- h@+--- K+---  T+---  M+--- 8,...  >,... .S+--- )E+--- +I """ J+--- ` """  Z+---  W+--- ,6 1 %'''77+---88+---88+---<6+---77+---:;+--- (6+--- k'1E  0Sw}55SSAQAQSS  Margsctjd}t|}t|S)z(Define parser for command-line argumentsz&Command-line interface for Trafilatura) description)argparseArgumentParserr8map_args parse_args)r:rs r7r@r@s=  $1Y Z Z ZF f  F F%%'' ( ((r9cBdD]}t||r ||_n|S)z2Map existing options to format and output choices.)csvhtmljsonmarkdownxmlxmltei)getattr output_format)r:otypes r7r?r?s?F 4   !&D  E  Kr9cfttjdd}t|dS)z Run as a command-line utility. rN)r@sysargv process_args)r:s r7mainrOs- chqrrl # #Dr9cd}|jdkr+tjtjtjn5|jdkr*tjtjtj|jrt|j|_|j s|j s|j rt|}n|j rt|n|jrt!|n|jrt%|nv|js|jr t+|}t-||}nHt/tjj||j}t7|||dkrtj|dSdS)z8Perform the actual processing according to the argumentsrr)streamlevel)urlN)verboselogging basicConfigrLstdoutWARNINGDEBUG blacklistr explorefeedsitemaprcrawlrprober input_dirr input_fileURLr rr stdinbufferreadrexit)r: exit_code url_storeresults r7rNrNsI |q3:W_EEEEE   3:W]CCCC ~8'77  |#ty#DL#!$''  #D #t  # &&&& #DH##D)) +D)<< )..00$DHEEEVT"""A~~ ~r9__main__)rN)"__doc__r=rVrLimportlib.metadatarplatformrtypingr cli_utilsrrr r r r r rrsettingsrrrXrhasattrrstderrr8r@r?rOrN__name__r9r7rvs, &&&&&&######??????????????????????87777777:'!!ggcj-&H&H!JG,,,:'!!ggcj-&H&H!JG,,,PSPSPPPPf)S)S))))33 *s*t****Z zDFFFFFr9