!}gvdZdZdZdZdZddlmZddlZdd lm Z ej e Z Gd d e ZdS) zI Output formatting to text via lxml xpath nodes abstracted in this file. newspaperz Lucas Ou-YangMITzCopyright 2014, Lucas Ou-Yang)unescapeN) innerTrimc\eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdS)OutputFormattercd|_||_|j|_|j|_|j|_dSN)top_nodeconfig get_parserparserlanguagestopwords_class)selfr s Z/var/www/py-google-trends/myenv/lib/python3.11/site-packages/newspaper/outputformatters.py__init__zOutputFormatter.__init__s?  k,,..  %5cZ|r(||_|j||_dSdS)zRequired to be called before the extraction process in some cases because the stopwords_class has to set incase the lang is not latin based N)rr get_stopwords_classr)r meta_langs rupdate_languagezOutputFormatter.update_languages>  ;%DM // ::   ; ;rc|jSr )r rs r get_top_nodezOutputFormatter.get_top_node&s }rc||_d\}}||jjr|}||||| | | }||fS)zReturns the body text of an article, and also the body article html if specified. Returns in (text, html) form )r) r remove_negativescores_nodesr keep_article_htmlconvert_to_html links_to_textadd_newline_to_bradd_newline_to_lireplace_with_textremove_empty_tagsremove_trailing_media_divconvert_to_text)rr htmltexts r get_formattedzOutputFormatter.get_formatted)s!  d ((*** ; ( *''))D                  &&(((##%%d|rcg}t|D]} |j|}n:#t$r-}t dt|d}Yd}~nd}~wwxYw|rRt|}t| d}d|D}| |d |S)Nz%s ignoring lxml node error: %s\nc8g|]}|dS) )strip).0ns r z3OutputFormatter.convert_to_text..Ks"999A1773<<999rz ) listrrgetText ValueErrorloginfo __title__rrsplitextendjoin)rtxtsnodetxterrtxt_liss rr(zOutputFormatter.convert_to_text?s**,,-- % %D k))$//   :IsKKK  %smm#C....u5599999 G$$${{4   sA A9 #A44A9c|j|}|j|Sr )rclean_article_htmlr nodeToString)r cleaned_nodes rr!zOutputFormatter.convert_to_htmlOs9{55d6G6G6I6IJJ {'' 555rc\|j|jdD] }d|_ dS)Nbrtagr-)rgetElementsByTagr r*)res rr#z!OutputFormatter.add_newline_to_brSs;--dm-FF  AAFF  rcR|j|jdD]}|j|d}|ddD][}|j|dz|_|j|D]}|j|\dS)NulrHlir-)rrJr r5r* getChildrenremove)rrKli_listrNcs rr$z!OutputFormatter.add_newline_to_liWs--dm-FF * *Ak221$2??Gcrcl * *+--b11E90044**AK&&q))))* * * *rc`|j|ddS)z[Cleans up and converts any nodes that should be considered text into text. aNr stripTagsrrs rr"zOutputFormatter.links_to_text_s. d//11377777rc|j|jd}|D]]}|j|d}|rt |nd}|dkr'||^dS)zvIf there are elements inside our top node that have a negative gravity score, let's give em the boot. z*[gravityScore] gravityScorerrN)r css_selectr getAttributefloat getparentrQ)r gravity_itemsitemscores rrz+OutputFormatter.remove_negativescores_nodeses .. M,.. ! . .DK,,T>BBE$)0E%LLLqEqyy  ''---  . .rch|j|ddddddS)a Replace common tags with just text so we don't have any crazy formatting issues so replace
, , , etc.... With whatever text is inside them. code : http://lxml.de/api/lxml.etree-module.html#strip_tags bstrongirGsupNrVrs rr%z!OutputFormatter.replace_with_textqsF      hT5 B B B B Brc|j|dg}||D]}|j|}|j|}|dks|dkrv|stt |j|ddkrGt |j|ddkr|j|dS) zIt's common in top_node to exit tags that are filled with data within properties but not within the tags themselves, delete them *rGz\robjectrHrembedN) rgetElementsByTagsrreversegetTagr5lenrJrQ)r all_nodeselrIr*s rr&z!OutputFormatter.remove_empty_tags{s#K11     #((  ' 'B+$$R((C;&&r**Dt tu}}  -DK889**++./00DK889))**-.// ""2&&& ' 'rcdfd j}t|dkrdS|d}|dkrj|dSdS)zPunish the *last top level* node in the top_node if it's DOM depth is too deep. Many media non-content links are eliminated: "related", "loading gallery", etc rcj|}|s|Sd}|D]}||dz}||kr|}|S)zComputes depth of an lxml element via BFS, this would be in parser if it were used anywhere else besides this method rr)rrP)r>depthchildren max_depthrSe_depth get_depthrs rrvz.get_depthsg{..t44H  I ( (#)Auqy11Y&& 'I rNrO)r)rrPrrmrQ)rtop_level_nodes last_nodervs` @rr'z)OutputFormatter.remove_trailing_media_divs       +11$2C2C2E2EFF   ! # # F#B' 9Y  1 $ $ K  y ) ) ) ) ) % $rN)__name__ __module__ __qualname__rrrr+r(r!r#r$r"rr%r&r'rrr r s666;;;,!!! 666***888 . . .BBB'''$*****rr )__doc__r9 __author__ __license__ __copyright__r)rloggingr*r getLoggerr{r7rhr r~rrrs    / g!!T*T*T*T*T*fT*T*T*T*T*r