%$}g% dZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZdd lmZd d lmZd d lmZejd ZeeddZedd*dededede fdZ!d+dededeefdZ"d,dedede#fdZ$GddZ%d edefd!Z&d"\Z'Z(Z)Z*Gd#d$Z+e+eZ,d%eddfd&Z-d'ed(ede fd)Z.dS)-z>Code parts dedicated to duplicate removal and text similarity.N)SequenceMatcher) lru_cache)blake2b)add)RLock)AnyDictListOptionalUnion)_Element)LRU_SIZE)trimz\.[^/?#]{2,63}$ bit_countcFt|dS)N1)bincount)xs Y/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/deduplication.pyrsSVV\\#5F5Fimaxsize? reference new_string thresholdreturnctd|}td|}td|||kS)zIReturn the similarity ratio between two short strings, here domain names.N)STRIP_EXTENSIONsubrratio)rrrs ris_similar_domainr&sQ ##B 22I $$R44J 4J 7 7 = = ? ?9 LLr@ inputstringlengthcFg}|D]J}|tj}|r||Kg}t dddD]*fd|D}t||dz kr|cS+|S)zbSplit input into list of tokens and adjust length threshold to make sure there is enough data.c:g|]}t|k|S)len).0tis r z!sample_tokens..-s$222s1vvzz!zzzr)splitstripstring punctuationisalnumappendranger/)r(r)tokenstokensampler2s @r sample_tokensr?#sF""$$!! F.// ==?? ! MM% F 1b"  2222V222 v;;&1* $ $MMM % Mrcdt|}t||S)z=Create a bag of words and generate a hash for a given string.  digest_size)joinr?r6rencodedigest)r(r) teststrings rgenerate_bow_hashrI3sR- 4455;;==J :$$&&F ; ; ; B B D DDrc &eZdZdZddgZ ddededeed dfd Zded efd Z e d ded e efdZ ded efdZ d efdZded eefdZdeeeefd eefdZded efdZded efdZdS)SimhashzAImplement a basic Charikar hashing approach of string similarity.hashr)r"r'Nr( existing_hashr cr||_||p|||_dS)z&Store length and existing or new hash.N)r)validate create_hashrL)selfr(r)rMs r__init__zSimhash.__init__>s5 MM-00QD4D4D[4Q4Q rctt|ddS)z&Return a numerical hash of the string.rCbig)int from_bytesrrFrG)rQr(s r_hashz Simhash._hashHsB~~ K&&((a 8 8 8 ? ? A A5   ri@rr=cHfdtjDS)z2Create vector to add to the existing string vectorcNg|]!}d|zzrdnd"S)rr,)rX)r0r2rQr=s rr3z*Simhash._vector_to_add..]s7VVVaTZZ&&!q&19rVVVr)r;r))rQr=s``r_vector_to_addzSimhash._vector_to_addZs.WVVVV5CUCUVVVVrc dg|jzt||jD]8}ttt||9t fdt|jDS)zCalculates a Charikar simhash. References used: https://github.com/vilda/shash/ https://github.com/sean-public/python-hashes/blob/master/hashes/simhash.py Optimized for Python by @adbar. rc3:K|]}|dkd|zVdS)rrNr.)r0r2vectors r z&Simhash.create_hash..js/HHaa16HHr)r)r?listmaprr[sumr;)rQr(r=r^s @rrPzSimhash.create_hash_s t{""; << H HE#c64+>+>u+E+EFFGGFFHHHH5#5#5HHHHHHrc:t|jddS)z3Convert the numerical hash to a hexadecimal string.r4N)hexrLrQs rto_hexzSimhash.to_hexls49~~abb!!r inputhashcT t|dS#ttf$rYdSwxYw)z2Convert the hexadecimal hash to a numerical value.N)rV TypeError ValueErrorrQrgs r _hash_to_intzSimhash._hash_to_intps> y"%% %:&   44 s ''c\t|tr,dtt|cxkrdkrnn|St|trU|r,dt|cxkrdkrnnt|S||SdS)z9Validate the input hash and return it, or None otherwise.N) isinstancerVr/strisdigitrmrls rrOzSimhash.validatews i % % "C NN0C0C*I*I*I*Ir*I*I*I*I*I  i % % 0  "" &rS^^'A'A'A'Ar'A'A'A'A'A9~~%$$Y// /tr other_hashc:t|j|jz S)zJReturn distance between two hashes of equal length using the XOR operator.)BIN_COUNT_FUNCrLrQrts rhamming_distancezSimhash.hamming_distancesdi*/9:::rcL|j||z |jz S)zjCalculate how similar this hash is from another simhash. Returns a float from 0.0 to 1.0. )r)rxrws r similarityzSimhash.similaritys& d33J???4;NNr)r"r'N)__name__ __module__ __qualname____doc__ __slots__rrrVr rRrXrr r[rPrfrmr rOrrxfloatrzr.rrrKrK:sGG"I'+ RRRR } R  RRRR      $YuWCWDIWWWW Is Is I I I I"""""chsm (5c?";     ;3;3;;;;OSOUOOOOOOrrKcontentcDt|S)zACalculate a simhash hex value for meaningful bits of the content.)rKrf)rs rcontent_fingerprintrs 7   " " $ $$r)rrr4c`eZdZdZddeddfdZdedefdZd edefd Zd e d eddfd Z dd Z dS)LRUCachea Pure-Python Least Recently Used (LRU) cache using a circular doubly linked list Adapted from CPython functools.py lru_cache decorator implementation https://github.com/python/cpython/blob/3.9/Lib/functools.py#L524 First adapted by https://github.com/vbarbaresi rr Nct|_||_i|_g|_|j|jddg|jdd<d|_dS)NF)rlockrcacherootfull)rQrs rrRzLRUCache.__init__sIGG  +- !  49dD9 !!!  rlinkc|\}}}}||c|t<|t<|jt}|x|t<|jt<||t<|j|t<|S)N)NEXTPREVr)rQr link_prev link_next_keyresultlasts r _move_linkzLRUCache._move_links\-1* 9dF+4i( $4y'++T TYt_T YT  rkeyc|j5|j|}|r!||cdddS dddn #1swxYwYdS)zgTests if the key that is asked for is in the cache and retrieve its value from the linked list.Nr,)rrgetr)rQrrs rrz LRUCache.getsY - -:>>#&&D -t,, - - - - - - - - - - - - - - - - - - - - - - - -rs1AAAvaluec|j5|j|}|r+||||j|t<n|jrq|j}||c|t<|t<|t|_|jt}dx|jt<|jt<|j|=||j|<nb|jt}||j||g}|x|t<x|jt<|j|<t|j|j k|_ddddS#1swxYwYdS)z Stores a given key in the cache.N) rrrrRESULTrrKEYrrr/r)rQrrroldrootoldkeyrs rputz LRUCache.putsY @ @:>>#&&D @%%%*/ 3''9@"iG471GCL'&/!( DI!Ys^F9==DIcNTYv%6 6*'.DJsOO 9T?D $)S%8DEIIDJI44:c?!$DJ4< ?DIA @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @sD"D77D;>D;c|j5|j|j|jddg|jdd<d|_ddddS#1swxYwYdS)zDelete all cache content.NF)rrclearrrres rrzLRUCache.clears Y   J      Ity$=DIaaaLDI                  s;AAA)r)r N) r{r|r}r~rVrRrrrrrrrr.rrrrs   d    ssss#@s#@3#@4#@#@#@#@JrrrHct|}|dkr|dznd}t||dS)zImplement LRU cache.r,rN)LRU_TESTrr)rHcachevalrs r put_in_cachersB||J''H$NNHqLLE LLU#####relementoptionscFtd|}t||jkrEt |}||jkr t ||dzdSt|dS)z(Check for duplicate text with LRU cache.rBrTF) rrEitertextr/min_duplcheck_sizerrmax_repetitionsrr)rrrHrs rduplicate_testrschhw//112233J :333<< ++ g- - - LLX\ 2 2 24 5r)r)r')r@)/r~rer7difflibr functoolsrhashlibroperatorr threadingrtypingrr r r r lxml.etreer settingsrutilsrcompiler#getattrrVrvrrrboolr&r?bytesrIrKrrrrrrrrrr.rrrs@@  ######33333333333333"*/00k+F+FGG 4MMM#M%MRVMMMM  s C c     EE3EEUEEEEPOPOPOPOPOPOPOPOf%%%%%% %dCPPPPPPPPf 8H % % %$S$T$$$$ H s t      r