%$}g|" * U d Z ddlZddlZddlmZ ddlmZmZmZmZm Z ddl
Z
ddl m Z
n
# e$ r dZ
Y nw xY wddlmZ ddlmZmZmZ dd lmZ ej e Zd
d hZee e d < e
j e
j j e
j ! d
dg d Z" e
j# e" Z$ eddd
d Z% ej& dej' Z( ej& dej' Z) G d d Z*dede+fdZ,de-de+fdZ.d e-dee fd Z/d e e-ef defd Z0d edefd!Z1d"edee fd#Z2d$ede+fd%Z3d&ed$edefd'Z4d(edee fd)Z5d(e e-eef dee fd*Z6d+ed,ee defd-Z7d.edefd/Z8dS )0z7
Module bundling functions related to HTML processing.
N)datetime)AnyListOptionalSetUnion)detect)
from_bytes) HtmlElement
HTMLParser
fromstring )
MAX_FILE_SIZEutf-8utf_8UNICODE_ALIASES )i i i i i )totalconnectstatus_forcelist)retriesFT) collect_idsdefault_doctypeencoding
remove_pisz^< ?! ?DOCTYPE.+?/ ?>z(c : e Zd ZdZg dZdedededededd f d
Zd S ) Extractorz0Defines a class to store all extraction options. extensiveformatmaxminoriginalextensive_searchmax_datemin_date
original_date outputformatreturnNc L || _ || _ || _ || _ || _ d S )Nr )selfr$ r% r& r' r( s N/var/www/py-google-trends/myenv/lib/python3.11/site-packages/htmldate/utils.py__init__zExtractor.__init__4 s+ 0' % % +
) __name__
__module__ __qualname____doc__ __slots__boolr strr- r. r, r r / ss 66AAAI , , , ,
,
,
, , , , , ,r. r datar) c > | rt | t k rdS dS )z6Check if the input object is suitable to be processed.TF)lenr r7 s r, is_wrong_documentr; C s$ 3t99},,t 5r. c V | d n# t $ r Y dS w xY wdS )zLSimple heuristic to determine if a bytestring uses standard unicode encodingzUTF-8FT)decodeUnicodeDecodeErrorr: s r, isutf8r? J sC G uu 4s
&& bytesobjectc V t | rdgS g }t >t | d }|'| | t | dd pt | }| d |D d |D S )zz#detect_encoding..c s :::1AJ:::r. c $ g | ]
}|t v |S r6 )r )rC gs r, rE z#detect_encoding..f s" ; ; ;!!?":":A":":":r. )r? cchardet_detectappendlowerr
extend)r@ guessescchardet_guessdetection_resultss r, detect_encodingrO S s k yG"(55jA % NN>//11 2 2 2 #;vv#677R:k;R;R NN::(9:::;;; < ;w ; ; ;;r. filecontentc t | t r| S d}t | D ]M} | | } n4# t t
f$ r t d| d}Y Jw xY w|pt | dd S )znGuess bytestring encoding and try to decode to Unicode string.
Resort to destructive conversion otherwise.Nzwrong encoding detected: %sr replace)r errors)
isinstancer5 rO r= LookupErrorr> LOGGERwarning)rP htmltextguessed_encodings r, decode_filerZ i s +s## H +K 8 8 "))*:;;H
E /0 NN8:J K K KHHH Ks;KKKKs A.A43A4responsec t | t j j st | d r| j }n| }t
| S )zRead the urllib3 object corresponding to the server response, then
try to guess its encoding and decode it to return a unicode stringr7 )rT urllib3r[ HTTPResponsehasattrr7 rZ )r[ resp_contents r, decode_responsera } sJ (G , 9:: ghPV>W>W } | $ $$r. urlc t d| d }|j dk r"t d|j | nwt |j r t d| nGt |j S # t $ r&}t d| | Y d}~nd}~ww xY wdS ) a, Fetches page using urllib3 and decodes the response.
Args:
url: URL of the page to fetch.
Returns:
HTML code as string, or Urllib3 response object (headers + body), or empty string in case
the result is invalid, or None if there was a problem with the network.
GET )timeout z!not a 200 response: %s for URL %sz incorrect input data for URL %szdownload error: %s %sN) HTTP_POOLrequeststatusrV errorr; r7 ra Exception)rb r[ errs r, fetch_urlrn s
2 $$UC$<<
?c ! ! LLRepair faulty HTML strings to make then palatable for libxml2.doctype
r )countzz\1> ) partition DOCTYPE_TAGsub enumerateiter
splitlinesendswith FAULTY_HTML)rs ro firstline_restilines r, repair_faulty_htmlr s I ' 1 1$ 7 7 1d __R!_<r s 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2222222 OOO ) ) ) ) ) ) 9 9 9 9 9 9 9 9 9 9 # # # # # #
8 $ $ $g .S . . . +B C C C ##
... $
G 7 7 7 juw4 bj0"$77 bj+RT22 , , , , , , , ,(C D 4 < <49 < < < <,LU5#: . L3 L L L L(%c %c % % % %3 8C= :#s #t # # # #
3
3
3
+)> 1%sK 78 1Xk=R 1 1 1 1h[ DI + ,c ,c , , , , , ,s , 66