%$}gI dZddlZddlZddlmZddlmZddlmZmZm Z m Z ddl m Z ddl mZddlmZdd lmZdd lmZd d lmZd d lmZmZd dlmZmZmZeje Z!e dddddeDdddddZ"dZ#dZ$dZ%ee#dzZ&dZ'dZ(edZ)dZ*dZ+dZ,ej-dZ.ej-d e,d!e+d"e*d#e*d$e+d% Z/ej-d e,d!e+d&e+d'e,d( Z0d)Z1ej-d*e1d+e*d,e,d-e*d.e1d/e,d0 2d1d2ej3Z4ej-d3e,d4e+d4e*d5Z5ej-d6e,d7e+d7e*d0ej3Z6ej-d8e,d7e+d7e*d0ej3Z7ej-d9e,d7e+d7e*d:Z8gd;Z9d<e:e9d =DZ;ej-d>Z<ej-d?Z=ej-d@ej3Z>ej-d9e*dAe+dAe,d0Z?ej-d9e*dBe+dCe*dDe+dE Z@ej-d9e+dAe,d0ZAej-dFe,d0ZBej-dGe,dHe,dIZCej-dJZDej-dKZEej-dLZFej-dMZGej-dNZHej-d9e,dOZIej-dPe,d0ZJej-dQZKej-d9e,dRZLej-dSZMej-dTZNej-dUZOej-d9e,dVZPej-dWZQej-d9e,dOZRej-dXe,dIZSdYedZe eeeffd[ZTd\eeUd]edZeeUfd^ZVd_eWdZeWfd`ZXdaeWdbeWdZe eWeWffdcZYddeUdZeefdeZZddeUdfeUdgedhedZeeUf diZ[ddeUdfeUdZeeUfdjZ\eekddeeUdfeUdle]dgedhedZeeUf dmZ^dYed]edZeeUfdnZ_doeUdpe eUd]edZeeUfdqZ`dYed]edZeeUfdrZadseUd]edZeeUfdtZbdS)uz: Custom parsers and XPath expressions for date extraction N)datetime) lru_cache)ListOptionalPatternTuple)DateDataParser)default_parsers)parse)XPath) HtmlElement) CACHE_SIZE) Extractor trim_text) convert_date is_valid_datevalidate_and_convertTcg|]}|dv| S))zno-spaces-timez relative-time timestamp).0ps S/var/www/py-google-trends/myenv/lib/python3.11/site-packages/htmldate/extractors.py r$s.   HHH HHHpastF) NORMALIZEPARSERSPREFER_DATES_FROMPREFER_LOCALE_DATE_ORDERRETURN_AS_TIMEZONE_AWARESTRICT_PARSING) languageslocalesregionsettingszr.//*[self::div or self::h2 or self::h3 or self::h4 or self::li or self::p or self::span or self::time or self::ul]z.//*a [ contains(translate(@id|@class|@itemprop, "D", "d"), 'date') or contains(translate(@id|@class|@itemprop, "D", "d"), 'datum') or contains(translate(@id|@class, "M", "m"), 'meta') or contains(@id|@class, 'time') or contains(@id|@class, 'publish') or contains(@id|@class, 'footer') or contains(@class, 'info') or contains(@class, 'post_detail') or contains(@class, 'block-content') or contains(@class, 'byline') or contains(@class, 'subline') or contains(@class, 'posted') or contains(@class, 'submitted') or contains(@class, 'created-post') or contains(@class, 'publication') or contains(@class, 'author') or contains(@class, 'autor') or contains(@class, 'field-content') or contains(@class, 'fa-clock-o') or contains(@class, 'fa-calendar') or contains(@class, 'fecha') or contains(@class, 'parution') or contains(@id, 'footer-info-lastmod') ] | .//footer | .//small z/text()4z).//div[@id="wm-ipp-base" or @id="wm-ipp"]z [0-3]?[0-9]z [0-1]?[0-9]z199[0-9]|20[0-3][0-9]z \b(\d{8})\bz(?:\D|^)(?:(?Pz)[\-/.](?Pz)[\-/.](?Pz )|(?Pz)[\-/.](?Pz")[\-/.](?P\d{2,4}))(?:\D|$)z )|(?Pz)[\-/.](?Pz ))(?:\D|$)u January?|February?|March|A[pv]ril|Ma[iy]|Jun[ei]|Jul[iy]|August|September|O[ck]tober|November|De[csz]ember| Jan|Feb|M[aä]r|Apr|Jun|Jul|Aug|Sep|O[ck]t|Nov|De[cz]| Januari|Februari|Maret|Mei|Agustus| Jänner|Feber|März| janvier|février|mars|juin|juillet|aout|septembre|octobre|novembre|décembre| Ocak|Şubat|Mart|Nisan|Mayıs|Haziran|Temmuz|Ağustos|Eylül|Ekim|Kasım|Aralık| Oca|Şub|Mar|Nis|Haz|Tem|Ağu|Eyl|Eki|Kas|Ara z (?Pz )\s (?Pz)(?:st|nd|rd|th)?,? (?Pz )| (?Pz))(?:st|nd|rd|th|\.)? (?:of )? (?Pz)[,.]? (?P) z\D(z)[/_-](z )(?:\D|$)z"dateModified": ?"(-z"datePublished": ?"((z).[0-9]{2}:[0-9]{2}:[0-9]{2}) )janjanuarujännerjanuaryjanuarijanvierocakoca)febfebruarfeberfebruaryfebruariufévrieruşubatuşub)marumärumärzmarchmaretmartmars)apraprilavrilnisannis)maymaimeiumayıs)junjunijunejuinhaziranhaz)juljulijulyjuillettemmuztem)augaugustagustusuağustosuağuaout)sep september septembreueylüleyl)octoktoberoctoberoctobreoktekimeki)novnovemberukasımkasnovembre)decdezdezemberdecemberdesemberu décembreuaralıkarac$i|] \}}|D]}||Srr)rmnummlistmonths r rqs?D%5BGE4r)startz[.:,_/ -]|^\d+$u^\d{2}:\d{2}(?: |:|$)|^\D*\d{4}\D*$|[$€¥Ұ£¢₽₱฿#₹]|[A-Z]{3}[^A-Z]|(?:^|\D)(?:\+\d{2}|\d{3}|\d{5})\D|ftps?|https?|sftp|\.(?:com|net|org|info|gov|edu|de|fr|io)\b|IBAN|[A-Z]{2}[0-9]{2}|®u(?:date[^0-9"]{,20}|updated|last-modified|published|posted|on)(?:[ :])*?([0-9]{1,4})[./]([0-9]{1,2})[./]([0-9]{2,4})|(?:Datum|Stand|Veröffentlicht am):? ?([0-9]{1,2})\.([0-9]{1,2})\.([0-9]{2,4})|(?:güncellen?me|yayı(?:m|n)lan?ma) *?(?:tarihi)? *?:? *?([0-9]{1,2})[./]([0-9]{1,2})[./]([0-9]{2,4})|([0-9]{1,2})[./]([0-9]{1,2})[./]([0-9]{2,4}) *?(?:'de|'da|'te|'ta|’de|’da|’te|’ta|tarihinde) *(?:güncellendi|yayı(?:m|n)landı)z)[/.-](z)/(z)/([0-9]{2})|(z)[.-](z)[.-]([0-9]{2})z^\D?(u$(?:©|\©|Copyright|\(c\))\D*(?:z)?-?(z)\Dz"/([0-9]{4}/[0-9]{2}/[0-9]{2})[01/]z ([0-9]{4})/([0-9]{2})/([0-9]{2})z(\D([0-9]{4}[/.-][0-9]{2}[/.-][0-9]{2})\Dz(([0-9]{4})[/.-]([0-9]{2})[/.-]([0-9]{2})z-\D([0-3]?[0-9][/.-][01]?[0-9][/.-][0-9]{4})\Dz)\D?$z^(zE(\D19[0-9]{2}[01][0-9][0-3][0-9]\D|\D20[0-9]{2}[01][0-9][0-3][0-9]\D)z)([01][0-9])([0-3][0-9])zK\D([0-3]?[0-9]/[01]?[0-9]/[0129][0-9]|[0-3][0-9]\.[01][0-9]\.[0129][0-9])\Dz ([0-9]{2})$z(\D([12][0-9]{3}[/.-](?:1[0-2]|0[1-9]))\Dz)[/.-](1[0-2]|0[1-9]|)z!\D([01]?[0-9][/.-][12][0-9]{3})\Dz(?}||||?||fS)zFDelete unwanted sections of an HTML document and return them as a list)DISCARD_EXPRESSIONSappend getparentremove)rs my_discardedsubtrees rdiscard_unwantedr|s`L&t,,,,G$$$""7++++  rtesturloptionsc|t|}|rtd|d t t |dt |dt |d}t ||j|j|j r| |jSn9#t$r,}td|d|Yd}~nd}~wwxYwdS) zEExtract the date out of an URL string complying with the Y-M-D formatNzfound date in URL: %srrearliestlatestzconversion error: %s %s) COMPLETE_URLsearchLOGGERdebugrintrformatminmaxstrftime ValueError)r}r~match dateobjecterrs rextract_url_daters  ##G,,  G LL0%( ; ; ; G%c%(mmSq]]CaMMRR  W[?&..w~>>>? G G G 6a#FFFFFFFF G 4sBC C>"C99C>yearc,|dkr ||dkrdndz }|S)z!Adapt year from YY to YYYY formatdZilir)rs r correct_yearrs& czz  , Krdayrpc*|dkr |dkr||fn||fS)z/Swap day and month values if it seems feasible. r)rrps rtry_swap_valuesrs$ 2::#))E3<<#uErstringcrt|}|sdS|jdkrdnd} t||dtt ||ddt||d}}}t|}t||\}}t|||}n#t$rYdSwxYwt d ||S) zTry full-text parse for date elements using a series of regular expressions with particular emphasis on English, French, German and TurkishNr)rrpr)day2month2year2rr.rzmultilingual text found: %s)LONG_TEXT_PATTERNr lastgrouprgroup TEXT_MONTHSlowerstriprrrrrr)rrgroupsrrprrs r regex_parsers3  $ $V , ,E t ?f $ $ ! (   F1I&& ' '  EKKq 2288::@@EEF G G  F1I&& ' 'U D!!$S%00 UdE3// tt LL. ;;; sCD DD outputformatmin_datemax_datec P td||ddrtd}|ddr tt |ddt |ddt |dd}n#t $r&td|ddYnwxYw tj|}nv#t $ritd| t|d }n8#ttt f$rtd |YnwxYwYnwxYw|Ct|||| r0td || |St |}|r t |d ddt |d ddt |d dd}}}t|||}t|d|| r0td|| |Sn1#t $r$td|dYnwxYwt |}|r |jdkrgt |dt |dt |d}}}nt |dt |dt |d}}}t#|}t%||\}}t|||}t|d|| r0td|| |Sn1#t $r$td|dYnwxYwt& |}|r$ |jdkrRtt |dt |dd }nQtt |dt |dd }t|d|| r0td|| |Sn1#t $r$td|dYnwxYwt)|} t+| ||| S)z!Try to bypass the slow dateparserzcustom parse test: %sNr(z8-digit error: %sznot an ISO date string: %sF)fuzzyzdateutil parsing error: %srzparsing result: %sr%Y-%m-%dzYYYYMMDD match: %szYYYYMMDD value error: %srrrrprrrzregex match: %szregex value error: %sz Y-M match: %szY-M value error: %s)rrisdigitrrr fromisoformatdateutil_parse OverflowError TypeErrorrrYMD_NO_SEP_PATTERNr YMD_PATTERNrrrr YM_PATTERNrr) rrrr candidaterrrprrs r custom_parser s LL(&111bqbz4 !A#;    G >$rr OOS!%5%5s6!A#;7G7G  > > > 0&!*===== > G$26::  G G G 96BBBG .vU C C CII%y*=GGGLL!=vFFFFFG  G  )\HX V V V ! LL-y 9 9 9%%l33 3  % %f - -E 8 8"58BQB<00#eAhqsm2D2Dc%PQ(STUVSV-FXFX%D uc22IY XhWWW 8 19=== )),777 8 ? ? ? LL3U1X > > > > > ?   v & &E 8 8%'' F++,, G,,-- E**++!e F++,, H--.. G,,--!U $D)),S%88 U uc22IY XhWWW 8 . ::: )),777 8 < < < LL0%( ; ; ; ; ; <   f % %E 8 8'))$ F++,,c%++g2F2F.G.G % G,,--s5;;x3H3H/I/I1 Y XhWWW 8 _i888 )),777 8 : : : LL.a 9 9 9 9 9 :V$$J L8H   sAB))-CCC22%E%D*)E%*2EE%EE%$E% A(I66+J$#J$D P+QQ#B.U+VVc&td| t|d}n<#tt f$r(}d}td||Yd}~nd}~wwxYw|rtj||ndS)zEUse dateutil parser or dateparser module according to system settingszsend to external parser: %sdate_objNzexternal parser error: %s %s) rrEXTERNAL_PARSER get_date_datarrerrorrr)rrtargetrs rexternal_date_parserrs LL.777B ..v66zB : &BBB 3VSAAAAAAAAB7= F8 V\ 2 2 2$Fs >A7A22A7)maxsizeextensive_searchc|sdSt|dt}|r4dtttj|cxkrdksndSt |rdSt||||}||S|r?t|r%t||}t||||r|SdS)zIUse a series of heuristics and rules to parse a potential date expressionNrr) rMAX_SEGMENT_LENsummapstrrDISCARD_PATTERNSrrTEXT_DATE_PATTERNrr)rrrrr customresultdateparser_results r try_date_exprrs tv  // 0F c#ck6":":;;AAAArAAAAtv&&t  hIIL%-44V<<%0FF  |hx    %% $ 4rcz|d}|#t|d|SdS)zSkim through image elementsz'.//meta[@property="og:image"][@content]Ncontent)findrget)rsr~elements r img_searchrsG iiABBG KK " "     4rtext date_patternc||}|rat|dd|j|jr>td||dt |dd|jSdS)zILook for date expressions using a regular expression on a string of text.rrrzregex found: %s %srN)rrrrrrrr)rrr~rs rpattern_searchrs    % %E B a*w{7;B  )<qBBBE!Hj'.AAA 4rc|jrtnt}|dD]*}|jr d|jvrt |j||cSdS)z8Look for JSON time patterns in JSON sections of the treezK.//script[@type="application/ld+json" or @type="application/settings+json"]z"dateN)originalJSON_PUBLISHED JSON_MODIFIEDxpathrr)rsr~ json_patternelems r json_searchrsq &-%5H>>=L U@@y G4944 diw????? 4r htmlstringc t|}|rpttd|} t |ddkrKt t|dt|dt|d}nltt|dt|d\}}tt|d}t |||}t|d|j |j r| |jSn8#tt f$r$t"d|dYnwxYwdS) z5Look for author-written dates throughout the web pageNrrrrrrz!cannot process idiosyncrasies: %s) TEXT_PATTERNSrlistfilterrlenrrrrrrrrr IndexErrorrrr)rr~rpartsrrrprs ridiosyncrasies_searchrsg   , ,E HVD%,,..1122 H58}}!!$Sq]]CaMM3uQx==QQ ,Sq]]CaMMJJ U#CaMM22$T5#66 : GK :!))'.999 :J' H H H LLrs  111111111111&%%%%%444444333333!!!!!!! ''''''''IIIIIIIIII  8 $ $ .    $   $$($)   &D  BlY677eGHH   ! RZ//bjZGZZhZZvZZZZ+3ZZZ RZBGBBhBBBB.5BBB  BJ8L88 88.588 88 88,38889@ b99DrzSSSSS&SSSTT  P'PPHPPvPPPRTRVWW :G::h:::::BDBJHHH8HHfHHH   #,9V1#=#=#= BJ1222:    QD   RZ OV O OH O OW O O OPPRZWWWHWWfWWHWWW<<<'<<<==rz-7---.. BJFGFF'FFF @AA bj<==  bj!LMMBJJKKRZ PQQ"*0'00011 2:&G&&& ' ' bjLBJEGEEEFF"*Rrz.)) GHHrz>w>>>?? @AAbj,g,,,-- :g:::;;;5d;>O1O+P c] c](ssFFSFU38_FFFF  2:a a"a.6aBJa c]aaaaH G GC GHSM G G G G :% SM%%% %  % c] %%%%P     c]       #,  c]      c]" c]r