%$}gH80dZddlZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z mZmZmZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ej5e6Z7ddhZ8de"de&de9fdZ:dededede&de e e9e;ff dZddddddddddddddddddddddde%dfded ee9d7ee9d!e=d"e=d#e=d$e=d%e=d&e9d8e=d'ee9d(e=d)e=d*e=d+e=d,e=d-eee9efd.e=d/e=d0ee;d1ee e9d2ee e9d9ee9d4eed5edee&dee9f6d:Z?dS);z4 Extraction configuration and processing functions. N)copydeepcopy)AnyDictOptionalSetTupleUnion)_ElementElementXPath strip_tags) HtmlElement)baseline)content_fingerprintduplicate_test)compare_extraction)build_html_output convert_tagsprune_unwanted_nodes tree_cleaning)extract_commentsextract_content)Documentextract_metadata)DEFAULT_CONFIG Extractor use_config) LANGID_FLAGcheck_html_langlanguage_filter load_htmlnormalize_unicode)build_json_outputcontrol_xml_outputxmltotxtxmltocsv)REMOVE_COMMENTS_XPATHmarkdowntxtdocumentoptionsreturnc ^d|jvr|jdD]d}|jdkrWt |dkrD|js=|js6|}| |jdkr||et||}n|jdkrt||j }n|jdkrt||j }n|jd krt||j }n|j rCd }d D]8}t||r&||d t!t||d z }9|d z }nd}|t#|j|j }|j1|d t#|j|j }t)|S)zMConvert XML tree to chosen format, clean the result and output it as a stringxml*graphicrNcodecsvjsonhtmlz--- ) titleauthorurlhostname descriptionsitenamedate categoriestags fingerprintidlicensez:  )formatbodyitertaglentexttail getparentremover&r( formattingr% with_metadatargetattrstrr' commentsbodystripr$)r,r-elementparent returnstringheaderattrs P/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/core.pydetermine_returnstringrZ,s }))#.. + +G y((LLA%% & &!**,,%&**>*>MM'***)(G<< 5 '*<== 6 ! !(73HII 6 ! !(73HII    F  J J8T**JIIWXt-D-D)E)EIIIIF g FFF O(8=':L"M"MOO  ,*cchx7LgN`.a.acciikkL \ * ** cleaned_treecleaned_tree_backup tree_backupc.t||\}}}|js%t|t|||||\}}}||jkrF|jdks;t t|\}}}td||||fS)z?Execute the standard cascade of extractors used by Trafilatura. precisionz+non-clean extracted length: %s (extraction)) rfastrrmin_extracted_sizefocusrLOGGERdebug)r\r]r^r-postbody temp_textlen_texts rYtrafilatura_sequenceries%4L'$J$J!Hi < (:  [ ! !     ) ) %)X',,,W]k5Q5Q(0+1F1F(G(G%)X BHMMM Y ((r[FTpython filecontentr9ra no_fallbackfavor_precision favor_recallinclude_comments output_formattarget_languageinclude_tablesinclude_imagesinclude_formatting include_links deduplicatedate_extraction_paramsrOonly_with_metadata max_tree_size url_blacklistauthor_blacklistas_dict prune_xpathconfigcH |r|}tjdt|rtjdt|rtd|rt |t s@t d*id|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d|d|d|} t |}|"td|t|j rL|j sts>t||j dur'td|j t|jrt||j|j|j |j}|j|jvr'td|jt|jr<|jr|jr|js'td|j tnt1}|2t |t2r|g}t5|d|D}t7t9||}t9|}t;|||jp|j}|jrt?||\}}}}ntAddd}}}|j!dkrt5|tD}tG||||\}} }!|j$rtK||j$kr8t&d tK|tO|d!tK||j$kr5t&d"tK||j t|jr+||j(kr t&d#|j |!|j)kr4||j*kr)t&d$|!||j t|j+r9tY||d%ur't&d&|j t|j rEt[| ||j |\}"}|"d%ur't&d'|j tn8#t\tf$r$td(|j YdSwxYw|j/d)krOta||j1|_2|jr!ta||j1|_||_3|j2|_4n| |c|_4|_3||_5|s|n|6S)+alInternal function for text extraction returning bare Python variables. Args: filecontent: HTML code as string. url: URL of the webpage. fast: Use faster heuristics and skip backup extraction. no_fallback: Will be deprecated, use "fast" instead. favor_precision: prefer less text but correct extraction. favor_recall: prefer more text even when unsure. include_comments: Extract comments along with the main text. output_format: Define an output format, Python being the default and the interest of this internal function. Other values: "csv", "html", "json", "markdown", "txt", "xml", and "xmltei". target_language: Define a language to discard invalid documents (ISO 639-1 format). include_tables: Take into account information within the HTML element. include_images: Take images into account (experimental). include_formatting: Keep structural elements related to formatting (present in XML format, converted to markdown otherwise). include_links: Keep links along with their targets (experimental). deduplicate: Remove duplicate segments and documents. date_extraction_params: Provide extraction parameters to htmldate as dict(). with_metadata: Extract metadata fields and add them to the output. only_with_metadata: Only keep documents featuring all essential metadata (date, title, url). url_blacklist: Provide a blacklist of URLs as set() to filter out documents. author_blacklist: Provide a blacklist of Author Names as set() to filter out authors. as_dict: Will be deprecated, use the .as_dict() method of the document class. prune_xpath: Provide an XPath expression to prune the tree before extraction. can be str or list of str. config: Directly provide a configparser configuration. options: Directly provide a whole extractor configuration. Returns: A Python dict() containing all the extracted information or None. Raises: ValueError: Extraction problem. H"no_fallback" will be deprecated in a future version, use "fast" insteadzR"as_dict" will be deprecated, use the .as_dict() method on bare_extraction results:max_tree_size is deprecated, use settings.cfg file insteadr~rprar`recallcommentsrNlinksimagestablesdeduplangr9rOrxr{rz date_paramsNzempty HTML tree: %sFzwrong HTML meta language: %szblacklisted URL: %szno metadata: %sc,g|]}t|S)r ).0xs rY z#bare_extraction..s.M.M.MAuQxx.M.M.Mr[rFrDrzoutput tree too long: %shiz'output tree too long: %s, discarding %sznot enough comments: %sz+text and comments not long enough: %s %s %sTz!discarding duplicate document: %szwrong language: %szdiscarding data: %srjr)7warningswarnPendingDeprecationWarning ValueError isinstancerr#rderrorrrar r!sourcerOrr9rr{rzwarningrxr=r7rrQrrrrrrr rcr)riryrIrermin_extracted_comm_sizemin_output_sizemin_output_comm_sizerrr" TypeErrorrEr'rNrJrRraw_textrFr|)#rkr9rarlrmrnrorprqrrrsrtrurvrwrOrxryrzr{r|r}r~r-treer,r\r]rR temp_comments len_commentsrfrgrhis_not_target_langs# rYbare_extractionrsD  V %     ` %   WUVVV  *Wi88    6 '-  &o   <  &%  *)  - "> "> + !  (-  21 .-! "(-# $/.% *p%% < LL. 4 4 4  < !W\ ! !tW\22e;; ;W^LLL     "' # ( H|w4444hlCCC  ) !  !"*. !5=\ ! .???   zzH  "+s++ ,*m '.M.M.M.M.MNND%T$ZZ99 "<00$L'7;;V(,WW    OFVgGG CL-||9@Q-L =K ' '/ >STTL(< -tW) ) %)X  !8}}w444 7XGGG8T***8}}w444 =MMN !   D w/N N N LL2GN C C C w. . .w;;; LL=       = ^Hg>>$FF LL "> + !  (-  21 *>! ".-# $(-% &/.' . H :h99t~[(( >X % %K     (#6HN##c)C0A,B,BB$$H "(G 4 44r[)@__doc__loggingrrrtypingrrrrr r lxml.etreer r r r lxml.htmlrr deduplicationrrexternalrhtmlprocessingrrrrmain_extractorrrmetadatarrsettingsrrrutilsr r!r"r#r$r0r%r&r'r(xpathsr) getLogger__name__rdrrQrZintriboolrrrr[rYrs9999999999999999;;;;;;;;;;;;!!!!!!>>>>>>>>(((((( >=======00000000;;;;;;;;;;KJJJJJJJJJJJ))))))  8 $ $5! 6+X6+ 6+c6+6+6+6+r))$)) )  8S#  ))))>!!!%) $7;$#'(,+/!% #'1d;d;d; #d; d; d;  d;  d;d;d;c]d;d;d;d;d;d;%T#s(^4d; !d;"#d;$C=%d;&CH%'d;(s3x()d;*+d;,#-d;. /d;0i 1d;2eHd38n,-.3d;d;d;d;R#!! %) $7;$#'(,+/"&!% #'5555 #5}5  5  5  55555c]55555 !5"%T#s(^4#5$%5&'5(C=)5*CH%+5,s3x(-5.3-/50#152 354i 556c]7555555r[