!}gdZdZdZdZdZddlZddlZddlZddlZddlm Z ddl m Z dd l m Z dd lmZmZmZd d lmZd d lmZmZejeZeddZeddZeddZedZedZedZedZ edZ!edZ"edZ#e$Z%dZ&dZ'dZ(gdZ)gdZ*gd Z+Gd!d"e,Z-dS)#a Newspaper uses much of python-goose's extraction code. View their license: https://github.com/codelucas/newspaper/blob/master/GOOSE-LICENSE.txt Keep all html page extraction code within this file. Abstract any lxml or soup parsing code in the parsers.py file! newspaperz Lucas Ou-YangMITzCopyright 2014, Lucas Ou-YangN) defaultdict)parse) tldextract)urljoinurlparse urlunparse)urls)StringReplacementStringSplitterz�z#!z?_escaped_fragment_=z»»z\|z - _/ » : z a[rel=tag]zMa[href*='/tag/'], a[href*='/tags/'], a[href*='/topic/'], a[href*='?keyword=']z ^[A-Za-z]{2}$) storyarticlefeaturefeaturedslides slideshowgallerynewsvideomediavradiopress)careerscontactaboutfaqtermsprivacyadvert preferencesfeedbackinfobrowsehowtoaccount subscribedonateshopadmin)amazon doubleclicktwitterceZdZdZdZdZdZdZd(dZdZ d Z d Z d Z d Z d ZdZdZdZdZdZdZdZd)dZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'dS)*ContentExtractorc||_|j|_|j|_|j|_dSN)config get_parserparserlanguagestopwords_class)selfr;s T/var/www/py-google-trends/myenv/lib/python3.11/site-packages/newspaper/extractors.py__init__zContentExtractor.__init__8s8 k,,..  %5cZ|r(||_|j||_dSdS)zRequired to be called before the extraction process in some cases because the stopwords_class has to set incase the lang is not latin based N)r>r;get_stopwords_classr?)r@ meta_langs rAupdate_languagez ContentExtractor.update_language>s>  ;%DM // ::   ; ;rCctjdfdd}fd}gd}gd}g}g}|D]9}|D]4} |j||| } || 5:|D]z} d} | jd kr1| d } t| d kr| d } n | jpd} t| d kr||| {||S) zcFetch the authors of the article, return as a list Only works for english articles z\dcHt|Sr:)boolsearch)d_digitss rAcontains_digitsz5ContentExtractor.get_authors..contains_digitsNsq))** *rCci}g}|D]W}||vrd||<||X|S)zRemove duplicates from provided list but maintain original order. Derived from http://www.peterbe.com/plog/uniqifiers-benchmark r )lowerappendtitle)lstseenresultitems rA uniqify_listz2ContentExtractor.get_authors..uniqify_listQsgDF , ,::<<4''%&TZZ\\" djjll++++MrCc"tjdd|}tjdd|}|}tjd|}d|D}g}g}gd}|D]d}||vr>t |dkr*|d|g}D|s||et |d k}|r(|d||S) a  Takes a candidate line of html or text and extracts out the name(s) in list form: >>> parse_byline('
By: Lucas Ou-Yang,Alex Smith
') ['Lucas Ou-Yang', 'Alex Smith'] <[^<]+?>rz[bB][yY][\:\s]|[fF]rom[\:\s]z [^\w'\-\.]c6g|]}|Sstrip).0ss rA zFContentExtractor.get_authors..parse_byline..q :::17799:::rC)and,rrr)resubr]splitlenrQjoin) search_str name_tokens_authorscurname delimiterstoken valid_namerNs rA parse_bylinez2ContentExtractor.get_authors..parse_byline^s+ B ;;J >JOOJ#))++J (=*==K::k:::KHG)))J$ * *J&&7||a'' (9(9:::"$(//*NN5)))g,,!+J 3 1 1222OrC)namerelitempropclassid)authorbylinez dc.creatorbylattrvaluermetaz@contentr) recompiler=getElementsByTagextendtagxpathrhtext)r@docrWrqATTRSVALSmatchesauthorsr{valfoundmatchcontentmmrMrNs @@rA get_authorszContentExtractor.get_authorsHsd*T"" + + + + +   ( ( ( ( ( X;::888 & &D & & 44St34OOu%%%% &  6 6EGyF""[[,,r77Q;; eG**7||a||G44555|G$$$rCc d}tjtj|}|r$|d}||}|r|Sdddddddddd ddd d d ddd ddddddddddddddddddddd dg }|D]e}|j||d|d} | r8|j| d|d}||}|r|cSfdS)a,3 strategies for publishing date extraction. The strategies are descending in accuracy and the next strategy is only attempted if a preferred one fails. 1. Pubdate from URL 2. Pubdate from metadata 3. Raw regex searches in the HTML + added heuristics cr|r4 t|S#ttttf$rYdSwxYwdSr:) date_parser ValueError OverflowErrorAttributeError TypeError)date_strs rAparse_date_strz.parse_date_strsT  &x000"M>9M    44   s 44rpropertyzrnews:datePublishedr) attributer|rzarticle:published_timerrOriginalPublicationDatert datePublisheddatetimezog:published_timearticle_date_originalpublication_datez sailthru.date PublishDatepubdaterr|rzN)rerKr STRICT_DATE_REGEXgroupr=r getAttribute) r@urlrr date_matchr datetime_objPUBLISH_DATE_TAGSknown_meta_tag meta_tagss rAget_publishing_datez$ContentExtractor.get_publishing_dates   Yt5s;;  $!''**H)>(33L $##%/D! # #$/G! # # +D! # #$" $ $$/B! # # +B! # # +=! # # ?! # # =! # ##i" $ $' ,0 ( (N 44#K0$W-5//I (;33aL"9-// .~h77 (''''trCcPd}j|d}|t|dkr|Sj|d}d}d}j|dpg}fd|D}|r}|td |d}t|d d krd}d d |D}|dp|dpd} tj d} | d| } | d| } | d|  } ||krd }n_| r | | kr|}d }nR| r/| | vr+| r)| | vr%t|t| kr|}d }n!| r| | kr| | r| }d }|s"d|vr |t|}d }|s"d|vr |t|}d }|s"d|vr |t |}d }|s"d|vr |t"|}d }|s"d|vr |t$|}d }t&|}| d| }| |kr|}|S)aFetch the article title and analyze it Assumptions: - title tag is the most reliable (inherited from Goose) - h1, if properly detected, is the best (visible to users) - og:title and h1 can help improve the title extraction - python == is too strict, often we need to compare filtered versions, i.e. lowercase and ignoring special chars Explicit rules: 1. title == h1, no need to split 2. h1 similar to og:title, use h1 3. title contains h1, title contains og:title, len(h1) > len(og:title), use h1 4. title starts with og:title, use og:title 5. use title, after splitting rrRrNrFh1cDg|]}j|Sr[)r=getText)r^rr@s rAr`z.ContentExtractor.get_title.. s55553dk11#66555rCT)keyreverserrdcg|]}||Sr[r[r^xs rAr`z.ContentExtractor.get_title..s%L%L%LA!%La%L%L%LrCzmeta[property="og:title"]zmeta[name="og:title"]z[^\u4e00-\u9fa5a-zA-Z0-9\ ]|-rrr)r=rrhrsortrgriget_meta_contentrer~rfrP startswith split_title PIPE_SPLITTER DASH_SPLITTERUNDERSCORE_SPLITTERSLASH_SPLITTERARROWS_SPLITTERMOTLEY_REPLACEMENT replaceAll)r@rrR title_element title_textused_delimeter title_text_h1title_element_h1_listtitle_text_h1_list title_text_fb filter_regexfilter_title_textfilter_title_text_h1filter_title_text_fb filter_titles` rA get_titlezContentExtractor.get_titles" 44Sg4FF  C $6$6!$;$;L[((q)9::   $ < ?? B c#:;; B?A  z"@AA (,,R<<BBDD+//MBBHHJJ+//MBBHHJJ J & &!NN ! "&:>R&R&R&J!NN ! "&:>O&O&O('P-AEV-V-V &&]););;;&J!NN ! "&:>O&O&O%001EFF'P&J!N "#"3"3))*m*799J!N "#"3"3))*m*799J!N "#"3"3))*6I*799J!N "#"3"3))*n*799J!N "&J"6"6))*o*799J!N"--j99 $''E2288:: < / /!E rCNcd}d}||}|r.s1OOO1vOvOOOrCN2c:g|]}tj|Sr[r prepare_url)r^f source_urls rAr`z2ContentExtractor.get_feed_urls..s6555  +Az::555rC)r=rrrlistset)r@r categoriestotal_feed_urlscategorykwargs feed_elements feed_urlss ` rA get_feed_urlszContentExtractor.get_feed_urlsws" . .H$/FGGF8DK8 (( &((MOO OOOI  " "9 - - - -)#2#.5555$3555s?3344rCcdddd}|jj|fi|}|r#|j|dd}|SdS)zExtract the favicon from a website http://en.wikipedia.org/wiki/Favicon linkrsiconrr{r|rrr)r=rr)r@rrr}favicons rA get_faviconzContentExtractor.get_favicons\  @@+t{+C::6::  k..tAw??GNrrCcJ|j|d}|Jddddddddg}|D];}|jj|fi|}|r$|j|d d }n<|r8|dd }tjt |r|SdS) z+Extract content language from meta lang)r{Nr}z http-equivzcontent-languagerrrrrrd)r=rrrerKRE_LANGrP)r@rr{itemsrVr}r|s rA get_meta_langzContentExtractor.get_meta_langs{''&'99 < ,..@@E   3t{3C@@4@@;33Qi411DE  %!HEy%(( %{{}}$trCc|j||}d}|4t|dkr!|j|dd}|r|SdS)zExtract a given meta content form document. Example metaNames: "meta[name=description]" "meta[name=keywords]" "meta[property=og:type]" Nrrr)r= css_selectrhrr])r@rmetanamer}rs rArz!ContentExtractor.get_meta_contentsj{%%c844  D A k..tAw BBG  #==?? "rrCcdgdz\}}}}}||d}|sdddd}|jj|fdd i|} | r| d d nd}|sP||d }|s8ddd d} |jj|fi| } | r| d d nd}|p|p|p|}|rt ||SdS)z:Returns the 'top img' as specified by the website Nzmeta[property="og:image"]rrszimg_src|image_srcr use_regexTrrzmeta[name="og:image"]rr)rr=rrr) r@ article_urlrtop_meta_imagetry_onetry_two try_threetry_fourlink_img_src_kwargselemslink_icon_kwargss rAget_meta_img_urlz!ContentExtractor.get_meta_img_urls4BF =)X''-HII G8KLL 0DK0\\\H[\\E.3=eAhll6***G G 11#7NOO  G/5uv'V'V$8DK8QQ@PQQE7<FuQx||F333$H DGDyDH  8;77 7rrCc.||dS)z:Returns meta type of article, open graph protocol zmeta[property="og:type"]rr@rs rA get_meta_typezContentExtractor.get_meta_types$$S*DEEErCc.||dS)zHIf the article has meta description set in the source, use that zmeta[name=description]r rs rAget_meta_descriptionz%ContentExtractor.get_meta_descriptions$$S*BCCCrCc.||dS)zEIf the article has meta keywords set in the source, use that zmeta[name=keywords]r rs rAget_meta_keywordsz"ContentExtractor.get_meta_keywordss$$S*?@@@rCcHtt}|j|d}|D]}|jdp|jd}|jdp|jd}|r|sp||}}|rt|}d|vr|||<| d}| d}||}t|tst|tr||i||<||}t|D]\} } | t|dz kr||| <n|| st|| <n]t|| ts(t|| tr d || i|| <|| }|S) Nr}rrrrr|rrr identifier)rdictr=rattribrr]isdigitintrgpop isinstancestrrrh) r@rdata propertiesproprr|key_headrefidxparts rA get_meta_datazContentExtractor.get_meta_datas4  [++C88 ! ! D+//*--H1H1HCKOOI..J$+//'2J2JE e ekkmmC}} #E #~~!S ))C..CwwqzzHx.C#s## %z#s';'; %"*CX8n&s^^ T#c((Q,&& %CIEwwt}}: $CII s33:z#''$--QT7U7U:".s4y 9CI$i rCc8|j|ddd}|r!|j|ddnd}||d}|p|pd}|r|}t |}|jst |}tjd |j|j } | d } n#t$r |j } YnwxYwt|j|j| dddf}|S) z Return the article's canonical URL Gets the first available value of: 1. The rel=canonical tag 2. The og:url tag rrs canonicalrrrrzmeta[property="og:url"]z.*{}(?=/)/(.*)r )r=rrrr]r hostnamererformatpathrrr scheme) r@rrlinksr&og_urlmeta_urlparsed_meta_urlparsed_article_urlstrip_hostname_in_meta_path true_paths rAget_canonical_linkz#ContentExtractor.get_canonical_link sX ,,Sf53>-@@CHODK,,U1Xv>>>R &&s,EFF,,"  4~~''H&x00O"+ 4&.k%:%:".0* &!3!<==)./0/0+5 ; A A! D DII%555 / 4III5&'9'@'9'BI')2r'344sC$$C87C8cddi}|jj|fi|}d|D}tfd|D}|S)z.7sD???7;;u+=+=? E""???rCc0g|]}t|Sr[)r)r^rrs rAr`z1ContentExtractor.get_img_urls..9s3*** !c22***rC)r=rr)r@rr img_kwargsimg_tagsr img_linkss ` rA get_img_urlszContentExtractor.get_img_urls2sU^ /4;/BBzBB??'???****$(***++ rCc|||}t|}|rt||dSdS)zRetrieves the first image in the 'top_node' The top node is essentially the HTML markdown where the main article lies and the first image in that area is probably signifigcant. rr)r<rr)r@rtop_node node_imagess rAget_first_img_urlz"ContentExtractor.get_first_img_url=sG '' X>> ;''  8; A77 7rrCcl|gSddi}|jj|fi|}|r d|DSd|DS)zZReturn a list of urls or a list of (url, title_text) tuples if specified. Nracpg|]3}|d|d|jf4Sr)rrr^rBs rAr`z.ContentExtractor._get_urls..Us8MMMquuV}}MQUU6]]AF+MMMrCcbg|],}|d|d-SrrrDs rAr`z.ContentExtractor._get_urls..Vs1???!v?f ???rCr=r)r@rtitlesa_kwargsa_tagss rA _get_urlszContentExtractor._get_urlsHsf ;I3<--c>>X>>  NMMVMMM M??v????rCFcb|tdgS|rHtjddt |}tjd|}d|D}|pgSt |tr|j|}n|}| ||S)z`doc_or_html`s html page or doc and returns list of urls, the regex flag indicates we don't parse via lxml and just search the html. Nz0Must extract urls from either html, text or doc!rYrzNhttp[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+c6g|]}|Sr[r\)r^rs rAr`z-ContentExtractor.get_urls..erarC) logcriticalrerfrfindallrr= fromstringrJ)r@ doc_or_htmlrGregexrs rAget_urlszContentExtractor.get_urlsXs   LLK L L LI  %&S#k2B2BCCK*01<>>K;:k:::K$" $ k3 ' ' +((55CCC~~c6***rCc  ||}g}|D]|}tj|d}tj|d}tj|d}|s!|s|jjrtd|zh|r4|dr|jjrtd|z|r+|dkr%|dkr|jjrtd|z|rtj |} tj } | j d } d} | D]Q} | | j krD|jjr4td t| d t| j d } nR| s2| j | j kr"|jjrtd |z| j dvr |jjrtd|z||dz|zd| dD}d|vr|dt#|dkr3t#|ddkr|||z^|jjrtd|z~gd}g}|D]}tj|}tj |j }|dz|z}d}|D]L}||vr"|jjrtd|zd }nM|s|||dt'|D]h\}}|dr d|z}|||<n|dr d|z}|||<|dr|dd}|||<it+t-|}fd|D}d |D}|S)!zInputs source lxml root and source url, extracts domain and finds all of the top level urls, we are assuming that these are the category urls. cnn.com --> [cnn.com/latest, world.cnn.com, cnn.com/asia] F)allow_fragmentsz+elim category url %s for no domain and path#z'elim category url %s path starts with #httphttpsz7elim category url %s for bad scheme, not http nor https.zsubdomain contains at z and Tz(elim category url %s for domain mismatch)mrz)elim category url %s for mobile subdomainz://c8g|]}t|dk|S)r)rhrs rAr`z6ContentExtractor.get_category_urls..s#HHHQSVVaZZqZZZrCrz index.htmlr rz;elim category url %s for >1 path chunks or size path chunks)Br%helpr(legalr+sitemapprofiler/mobiler_facebookmyspacer6linkedinbebo friendster stumbleuponyoutubevimeostoremailr*mapspasswordimgurflickrrK subscriptionitunes siteindexeventsstopjobsr# newsletterr0academyshoppingpurchasezsite-mapr2r1rvproductr)r,ticketscouponsforumboardarchiver-r.zhow tor&r'chartsservicesr$plusr3loginsignupregister developerproxyrz4elim category url %s for subdomain contain stopword!z//zhttp:Nc:g|]}tj|Sr[r)r^p_urlrs rAr`z6ContentExtractor.get_category_urls..s6999")%<<999rCcg|]}||Sr:r[)r^cs rAr`z6ContentExtractor.get_category_urls..sCCCqQ]]]]rC)rSr get_scheme get_domainget_pathr;verboseprintrrextract subdomainrgdomainrrQremoverhrPrendswithrr)r@rr page_urlsvalid_categoriesrr*rr) child_tld domain_tldchild_subdomain_partssubdomain_containsr# path_chunks stopwords_valid_categoriesr conjunctionbadbadwordr category_urlss ` rAget_category_urlsz"ContentExtractor.get_category_urlsns: MM#&& > ?> ?E_UEBBBF_UEBBBF=>>>D $ ;&#G!"### ,, ;&MCeKLLL 6V++'0A0A;&:0278:::* ?&.u55 '/ ;; (1(;(A(A#(F(F%%*"1Dz000;.I!E$'IIIIs:3D/E/E/E$GIII-1* 1* E")Z->>>{*! *,1 2444 !(J66{*5 +-2 3555$++FUNV,CDDDD IH$**S//HHH ;..&&|444{##q((SQ-@-@2-E-E$++FTM::::{*? 57< =??? " " " & 0 0E=''D"*511;I*y0KC$  ==??k&7&7&9&999{*= 35: ;===CE :  0!((///  %%%!"344 - -HAu&& -',!!$$!!$'' -%',!!$~~c"" -crc ',!!$ %6!7!7889999&7999 CCMCCC rCcttt|dkrtS|j|t }|s)|j|t }|stSg}|D]3}|j|}|r||4t|SNr) rhr NO_STRINGSr=rA_REL_TAG_SELECTORA_HREF_TAG_SELECTORrrQr)r@relementstagselrs rA extract_tagszContentExtractor.extract_tagss tCyy>>Q   ;)) #%% "{--(**H "!! ! !B+%%b))C ! C   4yyrCc d}||}td}d}d}g}g}|D]} |j| } ||j| } || } | dkr| s|| t|} d}t| dz}|D]} td}| | r |dkrtd|z dz}|dz }| dkrp| |z |krgt|| |z z }tt|td }t||z}|d krtd }|j| } ||j| } t| |z}|j| }|||||d||vr|||j|}|H||d|||dz ||vr|||dz }|dz }d}|D]%}||}||kr|}|}||}&|S) N?rr>rdg?rr (r)nodes_to_checkfloatr=rr?r>get_stopword_countis_highlink_densityrQrh is_boostablepowabsr getParent update_scoreupdate_node_count get_score)r@rr>rstarting_boostcntr parent_nodesnodes_with_textnode text_node word_statshigh_link_density nodes_numbernegative_scoringbottom_negativescore_nodes boost_scoreboosternegscoreupscore parent_nodeparent_parent_nodetop_node_scorerscores rAcalculate_best_nodez$ContentExtractor.calculate_best_nodesX,,S11s  " - -D ++D11I--t}-EE""9--  $ 8 8 > > ,,..22;L2&&t,,,?++ %*<%8%84%?"#% % D((K  && (!88"'~)=(C"D"DK"a'Nb   1$)CCC#2lQ6FGIIG"'WeAhh)?)?(?"@"@K";//2BBH"}}&+Ahh  ++D11I--t}-EE""9-- *7799KGHHG+//55K   k7 3 3 3  " "; 2 2 2,..##K000"&!6!6{!C!C !-&&'91===!!"4gkBBB%\99 ''(:;;; 1HC FAA  ANN1%%E~%%!&rCcld}d}d}d}||}|D]}|j|}||krq||krdS|j|} ||j| } | |krdS|dz }dS) auA lot of times the first paragraph might be the caption under an image so we'll want to make sure if we're going to boost a parent node that it should be connected to other paragraphs, at least for the first n paragraphs so we'll want to make sure that the next sibling is a paragraph and has at least some substantial weight to it. prrFrTr ) walk_siblingsr=getTagrr?r>r) r@rpara steps_awayminimum_stopword_countmax_stepsaway_from_nodenodes current_nodecurrent_node_tagparagraph_textrs rArzContentExtractor.is_boostableFs !""#""4((! L#{11,?? 4''!888 55!%!4!4\!B!B!114=1II&&~6600225KKK44a urCc6|j|Sr:)r=previousSiblingsr@rs rArzContentExtractor.walk_siblingsas{++D111rCc||}||}|D]3}|||}|D]}|d|4|Sr)get_siblings_scorerget_siblings_contentinsert)r@r>baseline_score_siblings_pararesultsrpsrs rA add_siblingszContentExtractor.add_siblingsds'+'>'>x'H'H$$$X..# & &L**:<rrr createElementrQ)r@current_siblingre0potential_paragraphsrfirst_paragraphrrparagraph_scoresibling_baseline_scorerrrs rArz%ContentExtractor.get_siblings_contentns  # % %DK//@@AAAEE Bw ]2&&4K#';#?#?S$@$*$* #+t';))O;..??D4yy1}}%)%9%9%)]&:&4&4..t44#+5*G*G*I*I16s.,0,D,D+----) %&B&<'=!>!> ?22;L2 $ 9 9$'d!:!?!?AIIaLLL rCcd}d}d}|j|d}|D]}|j|}||j|}||} |dkr| s|dz }||z }|dkr||z }|S)aWe could have long articles that have tons of paragraphs so if we tried to calculate the base score against the total text score of those paragraphs it would be unfair. So we need to normalize the score based on the average scoring of the paragraphs within the top node. For example if our total score of 10 paragraphs was 1000 but each had an average value of 100 then 100 should be our base. irrrrrdr )r=rrr?r>rr) r@r>baseparagraphs_numberparagraphs_scorerrrrrs rArz#ContentExtractor.get_siblings_scores55hC5HH" D DD ++D11I--t}-EE""9--  $ 8 8 > > ,,..22;L2!Q&! J$A$A$C$CC q #&77D rCcd}|j|d}|rt|}||z}|j|dt |dS)zAdds a score to the gravityScore Attribute we put on divs we'll get the current score then add the score we're passing in to the current. r gravityScoreN)r=rr setAttributer)r@r add_to_score current_score score_string new_scores rArzContentExtractor.update_scorese  {//nEE  0!,//M!L0    ~s9~~FFFFFrCcd}|j|d}|rt|}||z}|j|dt |dS)z=Stores how many decent nodes are under a parent node r gravityNodesN)r=rrrr)r@r add_to_countr count_stringrs rArz"ContentExtractor.update_node_countse {//nEE  . --M!L0    ~s9~~FFFFFrCcx|j|d}|sdS|j|}d|D}|sdSt t |}g}|D]/}||j|0d|}|} t t | } t t |} t | |z } t | | z} | dkrdSdS)zChecks the density of links within a node, if there is a high link to text ratio, then the text is less likely to be relevant rBrFc:g|]}||Sr[)isalnum)r^words rAr`z8ContentExtractor.is_highlink_density..s%AAA$$,,..AAAArCTrr)r=rrrgrrhrQri)r@rr+rwords words_numbersbr link_text link_wordsnum_link_words num_links link_divisorrs rArz$ContentExtractor.is_highlink_densitys/ ,,QC,88 5{""1%%AA$**,,AAA 4SZZ((  1 1D IIdk))$// 0 0 0 0GGBKK __&& s://#e**%% ^l:;; lY.// C<<4urCc0||pdS)z>Returns the gravityScore as an integer from this node r)get_node_gravity_scorers rArzContentExtractor.get_scores**4005A5rCc^|j|d}|sdSt|S)Nr)r=rr)r@r gravity_scores rArz'ContentExtractor.get_node_gravity_scores4 00~FF  4]###rCcVg}dD]#}|j||}||z }$|S)zXReturns a list of nodes we want to search on like paragraphs and tables )rpretdrrF)r@rrrrs rArzContentExtractor.nodes_to_checksD% $ $CK00#0>>E e #NNrCcN|j|d}|D]I}|j|}t|dkr|j|J|j|d}t|dkr |jdkrdSdS)NrrrrTF)r=rrrhrr)r@rsub_paragraphsrtxtsub_paragraphs_2s rAis_table_and_no_para_existz+ContentExtractor.is_table_and_no_para_exists55aS5AA & &A+%%a((C3xx"}} ""1%%%;77s7CC  A % %!%4--4urCc||}||}t|dz}||kr |jdkrdSdS)Ng{Gz?rFT)rrr)r@rrrcurrent_node_score thresholds rAis_nodescore_threshold_metz+ContentExtractor.is_nodescore_threshold_metsV--!^^A...3.//  * * 5trCc||}|j|D]Q}|j|}|dkr/||r|j|R|S)zRemove any divs that looks like non-content, clusters of links, or paras with no gusto; add adjacent nodes which look contenty r)rr= getChildrenrrr)r@r>rre_tags rA post_cleanupzContentExtractor.post_cleanup s  **((.. * *AK&&q))E||++A..*K&&q))) rCr:)FF)(__name__ __module__ __qualname__rBrGrrrrrrrrr rrrr$r2r<r@rJrSrrrrrrrrrrrrrrrr r$r[rCrAr8r87so666 ;;;X%X%X%H===~qqqf<<<<2$   2   2FFF DDD AAA %%%N$$$L      @@@ ++++,||||$FFFP6222"""H8 G G G G G G8666 $$$         rCr8).__doc__ __title__ __author__ __license__ __copyright__rloggingre collectionsrdateutil.parserrrr urllib.parserr r rr utilsr r getLoggerr%rMrESCAPED_FRAGMENT_REPLACEMENTrrrrrrCOLON_SPLITTERSPACE_SPLITTERrrrrr good_paths bad_chunks bad_domainsobjectr8r[rCrAr:s    /  ######000000!!!!!!666666666644444444g!!&&z26600  " "&&y$77u%% u%% $nS))$$ .(($$$$ SUU !B %%% AAA 322 _____v_____rC