%$}gd/dZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z mZmZmZmZmZ ddlZn #e$rYnwxYwddlmZmZd d lmZmZd d lmZmZmZd d lm Z d d l!m"Z"m#Z#m$Z$ej%e&Z'e ddZ(dZ)dZ*dZ+GddZ,de-de-de ee-ee-ffdZ.de-de ee-ee-ee-ffdZ/de-de-dee fdZ0de-dee fdZ1de-dee-de2fd Z3d!ee-de2fd"Z4 d6de-d$e,d%ee-ddfd&Z5d'eed$e,ddfd(Z6 d7d)e-d*ee-d+ee d!eee-d,eee-d-ee-de,fd.Z7 d8d$e,d/e2de,fd0Z8e*e+ddde ddfde-d1e9d2e9d!eee-d3eee-d*ee-d4ed+ee d-ee-de ee-ee-ffd5Z:dS)9zC Functions dedicated to website navigation and crawling/spidering. N) ConfigParser)sleep)ListOptionalTuple)RobotFileParser)UrlStore extract_linksfix_relative_urls get_base_urlis_navigation_pageis_not_crawlable)XPathtostring)baselineprune_unwanted_nodes)Responsefetch_response fetch_url)DEFAULT_CONFIG) LANGID_FLAG decode_file load_htmlF) compressedstrictz /robots.txt ic eZdZdZgdZ ddedeedeedeeddf d Zdedefd Z dedefd Z d e ddfd Z dee ede efdZdedefdZdS)CrawlParametersz6Store necessary information to manage a focused crawl.) startbaselangrulesrefi known_numis_on prune_xpathNr r"r#r(returnc||_|||_|||_||_|pt |j|_d|_d|_ d|_ ||_ dS)NrT) r _get_base_urlr!_get_referencer$r" get_rulesr#r%r&r'r()selfr r"r#r(s R/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/spider.py__init__zCrawlParameters.__init__0su  ++E22 ++E22#' 050M499M9M  *5cLt|}|std||S)z#Set reference domain for the crawl.zcannot start crawl: )r ValueError)r.r r!s r/r+zCrawlParameters._get_base_urlAs3 '' =;E;;<< < r1cp|ddkr|dddn|S)zDetermine the reference URL./rr)countrsplit)r.r s r/r,zCrawlParameters._get_referenceHs6*/++c*:*:a*?*?u||C##A&&UJr1 url_storect||j|_t ||j|_dS)z*Adjust crawl data based on URL store info.N)boolfind_unvisited_urlsr!r'lenfind_known_urlsr&)r.r9s r/update_metadatazCrawlParameters.update_metadataLsD)77 BBCC Y66tyAABBr1todoc(|sgSfd|DS)z.Prepare the todo list, excluding invalid URLs.c<g|]}|jk j|v|S)r r$).0ur.s r/ z/CrawlParameters.filter_list..Us*EEEa1 ??tx1}}}}}r1rC)r.r@s` r/ filter_listzCrawlParameters.filter_listQs* IEEEE4EEEEr1linkcz|j s|jd|o|j|vot| S)z9Run checks: robots.txt rules, URL type and crawl breadth.*)r# can_fetchr$r)r.rHs r/ is_valid_linkzCrawlParameters.is_valid_linkWsI^ >tz33C>> +D  +$T*** r1NNN)__name__ __module__ __qualname____doc__ __slots__strrrr0r+r,r r?rrGr;rLrCr1r/rr,sO<<cccI #+/%) 666sm6( 6 c] 6  6666"33KCKCKKKKCCdCCCC Fc 3FS FFFF  # $      r1r htmlstringhomepager)cTd|vrd|vr||fSt|}|||fS|d}|r|dnd}|rd|vrtjd|||fS|dd d d}|d st|}t||}t|}|tj d |d Stjd|||fS)z:Check if there could be a redirection by meta-refresh tag.z "refresh"z "REFRESH"Nz@.//meta[@http-equiv="refresh" or @http-equiv="REFRESH"]/@contentr;zno redirect found: %srzurl=httpzfailed redirect: %s)NNzsuccessful redirect: %s) rxpathlogginginfosplitstriplowerreplace startswithr r rwarning)rTrU html_treeresultsresulturl2base_url newhtmlstrings r/refresh_detectionri`sS*$$J)F)F8##*%%I8##ooJG# *WQZZF $S&& ,h7778## <<  Q  % % ' ' - - / / 7 7 C CD ??6 " "1%% 400dOOM-t444z L*D111 $ r1c6t|d}|r|jsdS|j|dfvr!tjd|j|j}t |j}t ||\}}|dStjd|||t|fS)zBCheck if the homepage is redirected and return appropriate values.FdecoderMr5zfollowed homepage redirect: %sNzfetching homepage OK: %s) rdataurlr[r\rridebugr )rUresponserTnew_htmlstring new_homepages r/probe_alternative_homepagersshu555H  8= |Hc?** 5x|DDD<X]++J$5Z#J#J NL M,l;;; <l)C)C CCr1 robots_urlrmct}|| ||n3#t$r&}t d|Yd}~dSd}~wwxYw|S)zEParse a robots.txt file with the standard library urllib.robotparser.zcannot read robots.txt: %sN)rset_urlparse splitlines ExceptionLOGGERerror)rtrmr#excs r/ parse_robotsr}s   E MM* DOO%%&&&&  13777ttttt Ls'A A=A88A=rgc\|tz}t|}|rt||ndS)z?Attempt to fetch and parse robots.txt file for a given website.N)ROBOTS_TXT_URLrr})rgrtrms r/r-r-s3N*J Z D-1 ;< D ) ) )t;r1languagec|rF|rDtr=t|\}}}tj|\}}t ||kSdS)zRun a baseline extraction and use a language detector to check if the content matches the target language. Return True if language checks are bypassed.T)rr py3langidclassifyr;)rTr_textres r/is_target_languagersY(h(;(j)) 4&t,, Fh&''' 4r1r@c4td|DS)z6Probe if there are still navigation URLs in the queue.c34K|]}t|VdS)N)r )rDrns r/ z&is_still_navigation..s+773!#&&777777r1)any)r@s r/is_still_navigationrs 77$777 7 77r1rWparamsrnc^t||jsdS|r|jxt|jtr |jg|_t |}|@t |d|jD}t|}gg}}t||p|j d|jddD]R}| |st|r| |=| |St||dS)zExamine the HTML code and process the retrieved internal links. Extract and filter new internal links after an optional language check. Store the links in todo-list while prioritizing the navigation ones.Nc,g|]}t|SrC)r)rDxs r/rFz!process_links..s.T.T.TAuQxx.T.T.Tr1FT) pagecontentrn external_boolrwith_navr)urls appendleft)rr"r( isinstancerSrrrrlr r!rLr append URL_STOREadd_urls)rTrrntreelinkslinks_priorityrHs r/ process_linksrsS j&+ 6 61f(4 f(# . . 6"("4!5F $$  '.T.TAS.T.T.TUUD!$..00J>E  6;   ##D))   d # #   ! !$ ' ' ' ' LL     En=====r1rpc||jsdSt|jgdt t |j||jdS)z2Convert urllib3 response object and extract links.NT)visited)rmrrrnrrr!)rprs r/process_responsersY x}  ~t444+hm,,ffkBBBBBr1r r"r#knownr(ct||||}t|pgdt||t|j|j|s4t|jgdt|d}n| t|S)zInitialize crawl by setting variables, copying values to the URL store and retrieving the initial page if the crawl starts.T)rr)rF)initial) rrrrG store_rulesr!r#r crawl_pager?)r r"r#r@rr(rs r/ init_crawlrsUD% = =FEKR666 F..t44555 &+v|444 *>>>FD111y))) Mr1rct|j}|s:d|_t t|j|_|S|xjdz c_|rGt|\}}}|r1|r/|r-t |gt|||n!t|d}t||| t|S)z6Examine a webpage, extract navigation links and links.Fr)rnrk)rget_urlr!r'r=r>r&r%rsrrrrr?)rrrnrTrU new_base_urlrps r/rrs   FK ( (C  y88EEFF  HHMHH +-G-L-L* Hl  7( 7| 7   z * * * *f# 6 6 6 6!#e4446*** 9%%% Mr1 max_seen_urlsmax_known_urls known_linksconfigc \t||||||} t| j|dd} | jrQ| j|krF| j|kr;t| } t| | jr| j|kr | j|k;tt t | j}tt t| j}||fS)aBasic crawler targeting pages of interest within a website. Args: homepage: URL of the page to first page to fetch, preferably the homepage of a website. max_seen_urls: maximum number of pages to visit, stop iterations at this number or at the exhaustion of pages on the website, whichever comes first. max_known_urls: stop if the total number of pages "known" exceeds this number. todo: provide a previously generated list of pages to visit / crawl frontier. known_links: provide a list of previously known pages. lang: try to target links according to language heuristics. config: use a different configuration (configparser format). rules: provide politeness rules (urllib.robotparser.RobotFileParser() format). prune_xpath: remove unwanted elements from the HTML pages using XPath. Returns: List of pages to visit, deque format, possibly empty if there are no further pages to visit. Set of known links. DEFAULT SLEEP_TIME)default)rrget_crawl_delayr!getfloatr'r%r&rrlistdictfromkeysr<r>) rUrrr@rr"rr#r(r sleep_times r/focused_crawlerr2s:$t[+ N NF** V__Y EE+J  M11f6F6W6WF## j  M11f6F6W6W  i;;FKHHII J JDt}}Y%>%>v{%K%KLLMMK  r1)rW)NNNNN)F);rQr[ configparserrtimertypingrrrurllib.robotparserrcourlanr r r r r rr ImportError lxml.etreerrcorerr downloadsrrrsettingsrutilsrrr getLoggerrNrzrr MAX_SEEN_URLSMAX_KNOWN_URLSrrSrirsr}r-r;rrrrrrintrrCr1r/rs%%%%%%((((((((((......    D '&&&&&&&00000000::::::::::$$$$$$6666666666  8 $ $ He 4 4 4  1 1 1 1 1 1 1 1 h##"# 8C=(3- '(####LDD 8C=(3-# 67DDDD2 S  0I    << 9<<<<3(3-D8d3i8D8888#>#>#> #> ##> #>#>#>#>L Cx  C  C  C C C C '+ $!%!%   3- O $ 49   DI   # :  B'( $'+)'+!%..... 49  . $s)$ . 3- . . O $.#. 49d3i  ......s9AA