$$}gtJ*UddlmZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZmZmZddlmZmZdd lmZmZdd lmZmZmZmZdd lmZdd lm Z dd l!m"Z"dZ#dZ$dZ%dZ&ia'e e(e"fe)d<de e(e"ffdZ*de(de"ddfdZ+de(ddfdZ,edd d7de-de(de-fdZ.de/fd Z0 d8de-d!e/d"eede e(e"ffd#Z1Gd$d%eZ2Gd&d'e2Z3Gd(d)e2Z4dddde3fd*ee(efde-d+e-d,e-d-e-d"eedeee e(e/efeddefffd.Z5 d9d*ee(efde-d+e-d,e-d-e-d"eedee(fd/Z6 d:d*ee(efde-d0e-d+e-d,e-d-e-d"eedeee(e ffd1Z7 d9d*ee(efde-d+e-d,e-d-e-d"eedeed2ee(e(e(fffd3Z8 d;d4ee(efd,e-d-e-d"eede-f d5Z9dtj|d}|r&|jr|||nAtjD]-\}}|r&|jr|||.t |S)zUUpdate TLD names. :param fail_silently: :param parser_uid: :return: Nr8)appendrget source_urlr&itemsall)r8r9resultsresults_append parser_clss r.r&r&UsG^N \*d33  */  N++-+HH   '/n&6&6   "J  j3 //m/LL w<<r-c:tjd}|dddd|dd d d d |tjdd}|j}|j}tt|| S)zCLI wrapper for update_tld_names. Since update_tld_names returns True on success, we need to negate the result to match CLI semantics. zUpdate TLD names) descriptionr9?Nz*UID of the parser to update TLD names for.)nargsdefaulthelpz--fail-silentlyr8F store_truez Fail silently)destrHactionrIr)r9r8) argparseArgumentParser add_argument parse_argssysargvr9r8intr&)parserargsr9r8s r.r'r'qs  $1C D D DF  9         SXabb\ * *DJ&M   - P P PP  r- retry_count parser_classcB|st}|||S)aBuild the ``tlds`` list if empty. Recursive. :param fail_silently: If set to True, no exceptions are raised and None is returned on failure. :param retry_count: If greater than 1, we raise an exception in order to avoid infinite loops. :param parser_class: :type fail_silently: bool :type retry_count: int :type parser_class: BaseTLDSourceParser :return: List of TLD names :rtype: obj:`tld.utils.Trie` r8rV)rr)r8rVrWs r.rrs3$ .-  % %# &  r-c ReZdZe ddededeeee ffdZ dS) rFrr8rVr)c|dkr |rdStt}|j|vr||j|S t|jr|j}nt |j}t |dd}t }|j}d}|j} |D]W} d| vrd}|r| snJd | vr| d} | d d vr7|| | Xt|j|| n#t$rZ|| |dz }|||cY | S#t $rYSwxYwt $r5} |r,Yd} ~ | dS#t $rYdSwxYw| d} ~ wwxYw | n:#t $rYn.wxYw# | w#t $rYwwxYwxYw|S)z[Parse. :param fail_silently: :param retry_count: :return: rNrutf8)encodingFz===BEGIN PRIVATE DOMAINS===Tz// xn--r)/ )privater;rY)rr% local_pathrr codecs_openraddinclude_privatesplitstripr(closeIOErrorr&r Exception) clsr8rV _tld_namesrb local_filetrietrie_addprivate_sectionrelineerrs r.rz(BaseMozillaTLDSourceParser.get_tld_namess ?? !t   Nj ( (3>*6 2 S^$$ 9 ^ (88 $ZvFFFJ66DxH#O!1O" E E0D88&*O"?E$$::<<?D7k))DJJLL*ODDDDD &s~t < < <            } = = = 1 K$$+%     """"        ttt   """""            """"       """"    sC#DG!NN N7:N $sDy/ "NNN[NNNr-rc>eZdZUdZdZeed<dZeed<dZeed<dS) rMozilla TLD source.mozillauidz4https://publicsuffix.org/list/public_suffix_list.datr>zres/effective_tld_names.dat.txtrbN) rsrtru__doc__r|rx__annotations__r>rbr,r-r.rrsHCLJLLL7J77777r-rcLeZdZUdZdZeed<dZeed<dZeed<dZ e ed <d S) r rzmozilla_public_onlyr|z?https://publicsuffix.org/list/public_suffix_list.dat?publiconlyr>z+res/effective_tld_names_public_only.dat.txtrbFreN) rsrtrur}r|rxr~r>rbrerwr,r-r.r r sd$C$$$IDJCCC!OT!!!!!r-r url fix_protocol search_publicsearch_privatec|s|stdt||}t|tsM|r|dsd|} t |}n"#t $r}|r dd|fcYd}~S|d}~wwxYw|}|j} | s|rdd|fSt|| } | dr| d} | d} |j } || j} d} d}d}t| }t!|d z d d D]l}| |}| jnZ|| jkrnM| j|}|| jd }|n| d z } |} | jr| }| }m||jr|s|jr |s|jr|rdd|fSt-| ||krd }nt/d ||z }| ||fS) zProcess URL. :param parser_class: :param url: :param fail_silently: :param fix_protocol: :param search_public: :param search_private: :return: zJEither `search_public` or `search_private` (or both) shall be set to True.)r8rW)z//zhttp://https://rN)r.rr*) domain_name)rr isinstancer startswithr ValueErrorhostnamerlowerendswithrstriprfrbrootlenrangechildren exceptionr=leafrarmax)rr8rrrrWrl parsed_urler domain_partsr/nodecurrent_length tld_lengthmatchlen_domain_partsipartchild non_zero_is r.r#r#s$  ^ %    #,J c; ' '   #/L M M #"S""C !#JJ    T3&&&&&&    %K %  %z) )$$$ $##%%KC  .!((-- $$S))L'2 * + 0DNJ E<(( #a'R 0 0A = E 4> ! ! E !!$'' =M%%c**E = E ! 9 'JE    &+m  $)=   =z) )# <<< <:%% ,z9:: Z //s$A'' B1A?7B=A??Bc d|vrtd|s|rtnt}t||||||\}}} |dS|J|dkr| jSd||dz dS)a?Extract the first level domain. Extract the top level domain based on the mozilla's effective TLD names dat file. Returns a string. May throw ``TldBadUrl`` or ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD match found respectively. :param url: URL to get top level domain from. :param fail_silently: If set to True, no exceptions are raised and None is returned on failure. :param fix_protocol: If set to True, missing or wrong protocol is ignored (https is appended instead). :param search_public: If set to True, search in public domains. :param search_private: If set to True, search in private domains. :param parser_class: :type url: str | SplitResult :type fail_silently: bool :type fix_protocol: bool :type search_public: bool :type search_private: bool :return: String with top level domain (if ``as_object`` argument is set to False) or a ``tld.utils.Result`` object (if ``as_object`` argument is set to True); returns None on failure. :rtype: str as_objectzH`as_object` argument is deprecated for `get_fld`. Use `get_tld` instead.rr8rrrrWNrrr)rrr r#rjoin) rr8rrrrWkwargsrrrs r.rrsDf%      2 " "1  ,7 #!#%! ,,,(L*jt  ! ! !A~~"" 88La!1!12 3 33r-rc|s|rtnt}t||||||\}}} |dS|J|s*|dkr| jSd||dS|dkrd} d} | j Jd| j} n]d|d|dz } d||dz |} d||d} t | | | | S) aExtract the top level domain. Extract the top level domain based on the mozilla's effective TLD names dat file. Returns a string. May throw ``TldBadUrl`` or ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD match found respectively. :param url: URL to get top level domain from. :param fail_silently: If set to True, no exceptions are raised and None is returned on failure. :param as_object: If set to True, ``tld.utils.Result`` object is returned, ``domain``, ``suffix`` and ``tld`` properties. :param fix_protocol: If set to True, missing or wrong protocol is ignored (https is appended instead). :param search_public: If set to True, search in public domains. :param search_private: If set to True, search in private domains. :param parser_class: :type url: str | SplitResult :type fail_silently: bool :type as_object: bool :type fix_protocol: bool :type search_public: bool :type search_private: bool :return: String with top level domain (if ``as_object`` argument is set to False) or a ``tld.utils.Result`` object (if ``as_object`` argument is set to True); returns None on failure. :rtype: str rNrrzNo hostname in URLr) subdomaindomaintldr)rr r#rrr) rr8rrrrrWrrrrr_tlds r.rrsTJ   2 " "1  ,7 #!#%! ,,,(L*jt  ! ! ! 3 >>& &xx Z[[1222A~~ "..0D..."HH\*:JN*:;<< ,zA~ 'BCDDxx Z[[122 F   r-NNNc |s|rtnt} t||d||||}|dS|j|j|jfS#t tttf$rYnwxYwdS)zParse TLD into parts. :param url: :param fail_silently: :param fix_protocol: :param search_public: :param search_private: :param parser_class: :return: Tuple (tld, domain, subdomain) :rtype: tuple T)r8rrrrrWNr) rr rrrrrrrr)rr8rrrrWobjs r.r!r!&s&   2 " "1    '%')%    ;##w CM11 (*A: N       sAAA"!A"valuec\|s|rtnt}t|dd|||}||kS)agCheck if given URL is tld. :param value: URL to get top level domain from. :param search_public: If set to True, search in public domains. :param search_private: If set to True, search in private domains. :param parser_class: :type value: str :type search_public: bool :type search_private: bool :return: :rtype: bool Tr)rr r)rrrrWrs r.rrUsX$   2 " "1   #%!    D D=r-c0|rt|dSiadS)zReset the ``tld_names`` to empty value. If ``tld_names_local_path`` is given, removes specified entry from ``tld_names`` instead. :param tld_names_local_path: :type tld_names_local_path: str :return: N)r"r%r4s r.r$r$ys, 455555 r-)FN)FrN)FFTTN)FFFTTN)TTN)N); __future__rrMrQcodecsrrc functoolsros.pathrtypingrrr r r r urllib.parser rbaserr exceptionsrrrrhelpersrresultrrnr __author__ __copyright__ __license____all__r%rxr~rr(r"rwr&rSr'rrrr r#rrr!rr$r,r-r.rs1''''''' &&&&&&;;;;;;;;;;;;;;;;........//////// !   , <  ( 4T ?c4i / /)- /  / / / /.#.$.... 3d###37-0 $#6c: .2*+ #t)_ >PPPPP!4PPPf888887888"""""'A"""$ .D s0s0 sK s0s0s0 s0  s0 *+ s0 5cC, -uT45L/M MNs0s0s0s0p .2 B4B4 sK B4B4B4 B4  B4 *+ B4c]B4B4B4B4N .2QQ sK QQQ Q  Q  Q*+QeCK !QQQQl .2 ,, sK ,,, ,  , *+ , 5! "E#sC-$8 89,,,,b.2 !! k! "!!!*+ !  !!!!H#r-