%$}gG ddZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z m Z mZmZddlmZejdZejdZejd Zejd ejZ d#d ed ede e de eefffdZ d$d edeeed edeefdZd edefdZd edefdZd ede eeffdZd ede eeeffdZded edefdZ deedeedeefdZ!d%d edededefdZ"d ed!eedefd"Z#dS)&zD Functions related to URL manipulation and extraction of URL parts. N)unescape)AnyListOptionalSetTupleUnion)urljoinurlsplit urlunsplit SplitResult)get_tldz{(?:(?:f|ht)tp)s?://(?:[^/?#]{,63}\.)?([^/?#.]{4,63}\.[^/?#]{2,63}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[0-9a-f:]{16,})(?:/|$)z (?<=\D):\d+z ^www[0-9]*\.z(?:feed(?:burner|proxy))FurlfastreturnNNc|rt|tsdS|rwt|}|r[td|ddd}|dd}|r||fSt|dd }|dS|jtd|j fS) z0Cached function to extract top-level domain infor@.rT) as_object fail_silently) isinstancestr DOMAIN_REGEXmatchSTRIP_PORT_REGEXsubsplitrdomainCLEAN_FLD_REGEXfld)rr domain_match full_domain clean_matchtldinfos P/var/www/py-google-trends/myenv/lib/python3.11/site-packages/courlan/urlutils.py get_tldinfor*s jc**z 0#))#..  0*..r<?3H3H3M3Mb3QRRK%++C003K 0"K//cT>>>Gz >?..r7;?? ?? blacklistcf|t}t||\}}|r ||vr||vr|ndS)z;Extract domain name information using top-level domain infoNr)setr*)rr,rr"r&s r)extract_domainr01sXEE %c555FK  !22{)7S7S  r+ct|trtt|}n5t|tr|}nt dt ||S)z3Parse a string or use urllib.parse object directly.zwrong input type:)rrr rr TypeErrortype)r parsed_urls r)_parser5As_#s8hsmm,, C % %8 +T#YY777 r+c\t|}|jr |jdz}nd}||jzS)ziStrip URL of some of its parts to get base URL. Accepts strings and urllib.parse ParseResult objects.z://r)r5schemenetloc)rr4r7s r) get_base_urlr9Ls<J"U* J% %%r+ct|}t|}tdd|j|j|jg}|dkrd}|r|st d|||fS)zvDecompose URL in two parts: protocol + host/domain and path. Accepts strings and urllib.parse ParseResult objects.r/zincomplete URL: )r5r9r pathqueryfragment ValueError)rr4hostnamepathvals r)get_host_and_pathrBWsJJ''H R*"2J4GHG"}} 3731C11222 W r+cJt|d}t|}||fS)zXConvenience function returning domain and host info (protocol + host/domain) from a URL.Tr.)r0r9)r domainnamebase_urls r) get_hostinforFgs,$///JC  H x r+baseurlc|dr|St|j}t|}|j|dfvr,|jr|St |dSt ||S)z8Prepend protocol and host information to relative links.{rhttp)r7) startswithr r8r7r _replacer )rGr base_netloc split_urls r)fix_relative_urlsrOns ~~c 7##*K I R000   J),,F,;;<<< 7C  r+ link_list urlfilterctt|Sfd|D}|s d|D}tt|S)zDReturn a list of links corresponding to the given substring pattern.Ncg|]}|v| SrT).0lrQs r) zfilter_urls..s<<<1Y!^^Q^^^r+cFg|]}t||SrT)FEED_WHITELIST_REGEXsearch)rUrVs r)rWzfilter_urls..s,PPPq1E1L1LQ1O1OPPPPr+)sortedr/)rPrQ filtered_lists ` r) filter_urlsr]~shc)nn%%%<<<< <<rxs 9999999999999999CCCCCCCCCCCCrz 2:n--"*_--!rz"=rtDD!@@ @@ 5 eCHo -.@@@@.BG    !#c(+ :>  c]      &c&c&&&& 3 5c?      c eHSM3$67     !s ! ! ! ! ! ! &49 &# &49 & & & &SS#c(tr+