!}g'dZdZdZdZdZddlZddlZddlmZm Z m Z m Z m Z ddl m Z ejeZd Zd Zd ZeezZgd Zgd ZgdZgdZddZdZddZddZdZdZdZdZ dZ!dS)zt Newspaper treats urls for news articles as critical components. Hence, we have an entire module dedicated to them. newspaperz Lucas Ou-YangMITzCopyright 2014, Lucas Ou-YangN)parse_qsurljoinurlparseurlsplit urlunsplit) tldextracti Nz(?<=\W)zs([\./\-_]{0,1}(19|20)\d{2})[\./\-_]{0,1}(([0-3]{0,1}[0-9][\./\-_])|(\w{3,5}[\./\-_]))([0-3]{0,1}[0-9][\./\-]{0,1})?) htmlhtmmdrstaspxjsprhtmlcgixhtmljhtmlaspshtml) storyarticlefeaturefeaturedslides slideshowgallerynewsvideomediavradiopress)careerscontactaboutfaqtermsprivacyadvert preferencesfeedbackinfobrowsehowtoaccount subscribedonateshopadmin)amazon doubleclicktwitterFct|}dfd|jdD}|r |dd}nd}t |dd|fz|zS)z0 Remove all param arguments from a url. &c3FK|]}||VdSN) startswith).0qry_item keep_paramss N/var/www/py-google-trends/myenv/lib/python3.11/site-packages/newspaper/urls.py zremove_args...sK   { + +N))rjoinquerysplitr )urlr@fragsparsedfiltered_queryfrags ` rA remove_argsrO)sc]]FXX!'!3!3C!8!8N abbz fRaRjN#44t; < <???  s9< A?/A::A?c|rt|}|t|dkr|rtd|zdSd|v}d|vod|v}|s|r|rtd|zdSt|j}|d sdS|d r |dd }d |d D}t|d krit|}|r|tvr|rtd |zdS|d d}t|dkr |d|d <d|vr| dtj |} | j } | j} |r|d nd} | t vr|rtd|zdSt|d krd\} }n*| d} | d}| r| dks|dkrz| |kr7| d| dDvr|rtd|zdS|| kr7| d| dDvr|rtd|zdSt|dkr|rtd|zdSt$D]#}||vs|| kr|rtd|zdS$t'jt*|}||rtd|zdSt,D]9}|d|Dvr|rtd |zdS:|rtd!|zdS)"a3 Is this URL a valid news-article url? Perform a regex check on an absolute url. First, perform a few basic checks like making sure the format of the url is right, (scheme, domain, tld). Second, make sure that the url isn't some static resource, check the file type. Then, search of a YYYY/MM/DD pattern in the url. News sites love to use this pattern, this is a very safe bet. Separators can be [\.-/_]. Years can be 2 or 4 digits, must have proper digits 1900-2099. Months and days can be ambiguous 2 digit numbers, one is even optional, some sites are liberal with their formatting also matches snippets of GET queries with keywords inside them. ex: asdf.php?topic_id=blahlbah We permit alphanumeric, _ and -. Our next check makes sure that a keyword is within one of the separators in a url (subdomain or early path separator). cnn.com/story/blah-blah-blah would pass due to "story". We filter out articles in this stage by aggressively checking to see if any resemblance of the source& domain's name or tld is present within the article title. If it is, that's bad. It must be a company link, like 'cnn is hiring new interns'. We also filter out articles with a subdomain or first degree path on a registered bad keyword. N z/ %s rejected because len of url is less than 11Fzmailto:zhttp://zhttps://z) %s rejected because len of url structure/c8g|]}t|dk|Srlenr>xs rA zvalid_url..#<<<Q!1rCrz %s rejected due to bad filetype.indexrEz%s caught for a bad tld)rr-_rDc6g|]}|Sr8lowerris rArkzvalid_url.. BBB17799BBBrCz%s verified for being a slugTc6g|]}|Sr8rtris rArkzvalid_url..rvrCz#%s caught for path chunks too smallz%s caught for bad chunksz%s verified for datec6g|]}|Sr8rt)r>ps rArkzvalid_url..s ;;;!AGGII;;;rCz%s verified for good pathz%s caught for default false)r`rhprintrpathr=endswithrIurl_to_filetype ALLOWED_TYPESremover extract subdomainrUru BAD_DOMAINScount BAD_CHUNKSresearch DATE_REGEX GOOD_PATHS)rJverbosetestr1r2r{ path_chunks file_type last_chunktld_datsubdtldurl_slug dash_countunderscore_countb match_dateGOODs rA valid_urlrfs@H # {c#hhmm SELsRSSSu s B 3  ;Zs%:B R MEFLMMMu C== D ??3  u }}SCRCy=>#.. Z!^^'7!';'; ) ) )BBhnnS.A.ABBBBBGE"@3"FGGGt j ( (BBhnnS.A.ABBBBBGE"@3"FGGGt ;1 FE?#EFFFu  qDyy ?83>???55 ):s++J 7E036777t ::<<;;{;;; ; ; @9C?@@@44 <:3c9::: 5rCct|j}|dr |dd}d|dD}|dd}t |dkrdS|d}t |dks|t vr|SdS)z Input a URL and output the filetype of the file specified by the url. Returns None for no filetype. 'http://blahblah/images/car.jpg' -> 'jpg' 'http://yahoo.com' -> None rcNrdc8g|]}t|dk|Srfrgris rArkz#url_to_filetype..rlrCrm)rr{r|rIrhrur~)abs_urlr{rrrs rAr}r}s G   !D }}SCRCy<rs    /  JJJJJJJJJJJJJJ!!!!!!g!! &D - :333 %%% AAA 322 ====".*IIIIX,......,,,-----rC