!}gdZdZdZdZdZddlZddlZddlmZdd l m Z dd l m Z ej eZd Zd Zdd ZddZdZGddeZddZdS)zb All code involving requests and responses over the http network must be abstracted in this file. newspaperz Lucas Ou-YangMITzCopyright 2014, Lucas Ou-YangN) Configuration) ThreadPoolcjz ISO-8859-1c6|r|nd|it|d|dS)zThis Wrapper method exists b/c some values in req_kwargs dict are methods which need to be called every time we make a request z User-AgentT)headerscookiestimeoutallow_redirectsproxiesr)r useragentrr s Q/var/www/py-google-trends/myenv/lib/python3.11/site-packages/newspaper/network.pyget_request_kwargsrs4 &D77L)+D44   c t|||S#tjj$r+}td|d|Yd}~dSd}~wwxYw)z HTTP response code agnostic zget_html() error. z on URL: N)get_html_2XX_onlyrequests exceptionsRequestExceptionlogdebug)urlconfigresponsees rget_htmlr %sj fh777   / qqq##>???rrrrrsA A  Ac |p t}|j}|j}|j}|j}|t |St jdd|it||||}t |}|j r| |S)zConsolidated logic for http requests from newspaper. We handle error cases: - Attempt to find encoding of the html by using HTTP header. Fallback to 'ISO-8859-1' if not provided. - Error out if a non 2XX HTTP response code is returned. Nr) rbrowser_user_agentrequest_timeoutrr _get_html_from_responsergetrhttp_success_onlyraise_for_status)rrrrr rr htmls rrr/s  &}F)I$GnGnG&x000|MM M%gy'7KKMMH #8 , ,D $!!### Krc|jtkr|j}nn|j}d|jdvrKt j|j}t|dkr|d|_|j}|pdS)Ncharsetz content-typerr) encoding FAIL_ENCODINGtextcontentr r&rutilsget_encodings_from_contentlen)rr) encodingss rr%r%JsM))} H,00@@ @ @ AA(-PPI9~~!!$-aL!} :2rc eZdZdZddZdZdS)MRequestaWrapper for request object for multithreading. If the domain we are crawling is under heavy load, the self.resp will be left as None. If this is the case, we still want to report the url which has failed so (perhaps) we can try again later. Nc||_||_|p t}|j|_|j|_|j|_|j|_d|_ dSN) rrrr#rr$r rr resp)selfrrs r__init__zMRequest.__init___sO *=??2- ~ ~  rc n tj|jfit|j|j|j|j|_|j j r|j dSdS#tj j $r5}tdt!|zYd}~dSd}~wwxYw)Nz[REQUEST FAILED] )rr&rrr rrr r8rr'r(rrrcriticalstr)r9rs rsendz MRequest.sendis 7 TXKK1C dndlDL2J2JKKDI{, - **,,,,, - -"3 7 7 7 LL,s1vv5 6 6 6 6 6 6 6 6 6 7sA%A++B4?*B//B4r7)__name__ __module__ __qualname____doc__r:r>r"rrr5r5YsA 77777rr5c|p t}|j}|j}t||}g}|D]%}|t ||&|D]}||j||S)zRequest multiple urls via mthreading, order of urls & requests is stable returns same requests but with response variables filled. ) rnumber_threadsthread_timeout_secondsrappendr5add_taskr>wait_completion)urlsr num_threadsr pool m_requestsrreqs rmultithread_requestrNss &}F'K+G k7 + +DJ11(3//0000   ch r)NNr7)rB __title__ __author__ __license__ __copyright__loggingr configurationr mthreadingrsettingsr getLoggerr?rr-rr rr%objectr5rNr"rrrYs    / ((((((""""""g!!    6   77777v7774r