%$}g^K d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
m Z ddl m
Z
ddlmZmZ dd lmZmZ ej e Z ej d
Zd ed efd
Zh dZddhZ h dZ h dZ h dZ h dZ ej dej! ej dej! ej dej! ej dej! ej dej! ej dej! dZ"d d hZ#d d hZ$d ed e%fd!Z& G d" d# Z' G d$ d% Z( ej d&ej! ej d'ej! d(Z) ej d)ej! Z*d*ed e+fd+Z,i fd ed,ed e+fd-Z-dS ).a Minimalistic fork of readability-lxml code
This is a python port of a ruby port of arc90's readability project
http://lab.arc90.com/experiments/readability/
Given a html document, it pulls out the main body text and cleans it up.
Ruby port by starrhorne and iterationlabs
Python port by gfxmonk
For list of contributors see
https://github.com/timbertson/python-readability
https://github.com/buriy/python-readability
License of forked code: Apache-2.0.
N)sqrt)
attrgetter)AnyDictOptionalSet)tostring) HtmlElementfragment_fromstring ) load_htmltrimz\.( |$)stringreturnc 0 t | t d S )Nxml)encodingmethod)r str)r s \/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/readability_lxml.py _tostringr & s FS 7 7 77 >
apdloluldivimgpretable
blockquoter article> tdr r" > ddr dtlir r formasideaddress>
h1h2h3h4h5h6thnavfooterheader> r r r' r embedinputzcombx|comment|community|disqus|extra|foot|header|menu|remark|rss|shoutbox|sidebar|sponsor|ad-break|agegate|pagination|pager|popup|tweet|twitterz#and|article|body|column|main|shadowzKarticle|body|content|entry|hentry|main|page|pagination|post|text|blog|storyzbutton|combx|comment|com-|contact|figure|foot|footer|footnote|form|input|masthead|media|meta|outbrain|promo|related|scroll|shoutbox|sidebar|sponsor|shopping|tags|tool|widgetz.<(?:a|blockquote|dl|div|img|ol|p|pre|table|ul)z+https?:\/\/(?:www\.)?(?:youtube|vimeo)\.com)unlikelyCandidatesReokMaybeItsACandidateRe
positiveRe
negativeRedivToPElementsRevideoRebodyhtmlr r elemc ^ t t | S )z7Return the length of the element with all its contents.)lenr text_content)r? s r text_lengthrC W s$ tD%%''(( ) ))r c . e Zd ZdZddgZdededdfdZdS ) Candidatez,Defines a class to score candidate elements.scorer? r Nc " || _ || _ d S N)rF r? )selfrF r? s r __init__zCandidate.__init__a s !
!% r )__name__
__module__ __qualname____doc__ __slots__floatr
rJ r r rE rE \ sM 22&!I&e &; &4 & & & & & &r rE c e Zd ZdZg dZddedededd fd
Zdefd Z d e
ee f d
e defdZ d e
ee f de
e fdZdedefdZde
ee f fdZdedefdZdede fdZddZddZded e
ee f defdZd S )Documentz,Class to build a etree document out of html.docmin_text_length retry_length rU rV rW r Nc 0 || _ || _ || _ dS )a Generate the document
:param doc: string of the html content.
:param min_text_length: Set to a higher value for more precise detection of longer texts.
:param retry_length: Set to a lower value for better detection of very small texts.
The Document class is not re-enterable.
It is designed to create a new Document() for each HTML file to process it.
API method:
.summary() -- cleaned up content
NrT )rI rU rV rW s r rJ zDocument.__init__k s! . (r c x | j dd D ]}| d} |r| | | }| | }|r| || }n`|du r d}t d t d | j
d }||n| j }| || }t |pd }|r|| j
k rd}|S )
z
Given a HTML file, extracts the text of the article.
Warning: It mutates internal DOM representation of the HTML document,
so it is better to call other API methods before this one.
scriptstyleTFz5Ended up stripping too much - going for a safer parsez=Ruthless and lenient parsing did not work. Returning raw htmlr= N )rU iter drop_treeremove_unlikely_candidates&transform_misused_divs_into_paragraphsscore_paragraphsselect_best_candidate get_articleLOGGERdebugfindsanitizerA rW ) rI r? ruthless
candidatesbest_candidater# r= cleaned_articlearticle_lengths r summaryzDocument.summary| sd HMM(G44 D NN #
2//111 7 7 9 9 9 ..00J ! 7 7
C CN
A **:~FFt## $HLLO S x}}V,,"&"2$$ "mmGZ @ @O !6B 7 7N
NT-> > > " "r rk rl c t d|j dz }t d }|j }|t | n|j g}|D ]}d}||j k s||v r|| j |k rd}nl|j dk ra| | } |j pd}
t |
} | dk r| d k s&| dk r"| d
k r t |
rd}|r| | |S ) N
皙?zFTr r^ P g ?r )
maxrF r r? getparentlisttagget_link_densitytextrA DOT_SPACEsearchappend) rI rk rl sibling_score_thresholdoutputparentsiblingssiblingr| link_density node_content node_lengths r re zDocument.get_article sG #&b.*>*D"E"E $X.. $..00#)#54<<z,Document.get_link_density.. s* MM+d++MMMMMMr z.//a)rC sumfindall)rI r? total_length link_lengths r rx zDocument.get_link_density sG "4((-A MM V8L8LMMMMM \))r c i }| j ddd D ]}| }|| }t | }t | }|| j k ri||fD ] }| ||vr| | ||< !dt | d z t |dz d z }|| xj
|z
c_
||| xj
|dz z
c_
| D ]+\ }} | xj
d| | z
z c_
,|S ) Nr r r$ r ,d )
rU r_ ru r rB rA rV
score_nodesplitminrF itemsrx )
rI rk r? parent_nodegrand_parent_node elem_text
elem_text_lennoderF r s
r rc zDocument.score_paragraphs s
HMM#ud33 A AD ..**K" + 5 5 7 7 T . . 0 011I NNM t 333$&78
=
=#J(>(>'+t'<'#> > >OOOr c d}t d | d | d f D ]L}t d | r|dz }t d | r|dz
}M|S )Nr classidr: rX r9 )filtergetREGEXESr{ )rI r? weight attributes r class_weightzDocument.class_weight s txx'8'8$((4..&IJJ I|$++I66
" |$++I66
"
r c | | }t |j }| }|t v r|dz
}n,|t
v r|dz
}n |t v r|dz }n|t v r|dz }t || S )Nr r ) r r rw lower
DIV_SCORES BLOCK_SCORESBAD_ELEM_SCORESSTRUCTURE_SCORESrE )rI r? rF rw names r r zDocument.score_node s !!$''$(mmyy{{ : QJEE
\
!
! QJEE
_
$
$ QJEE
%
%
% QJE%%%r c
| j d D ]}d t d | d | d f }t | dk ra|j t vrTt d | r4t d | s|
d S )Nz.//* r r r r7 r8 ) rU r joinr r rA rw
FRAME_TAGSr r{ r` )rI r? attrss r ra z#Document.remove_unlikely_candidates s H$$V,,
!
!DHHVD488G+<+r T)r r br)rU r r r{ r mapr rv rw ry stripr insertr enumeratetailr` )rI r? p_elemposchilds r rb z/Document.transform_misused_divs_into_paragraphs) s H$$X.. D -.55ItDzz2233
H$$X.. & &Dy
'TY__ . .
',V44)-D& TY Av&&& $Yt__d C C C
&
&
U: 1%*"2"2"4"4 1 0 8 8F.3j$+FKKKa0009$$OO%%%
&
& &r r c P | dddddd D ]H}| | dk s| | dk r| I| d d
D ] | d D ]Rd j v r3t
d
j d rd_ > St }t |
d D ]a|v r| }|v r
| j nd}||z dk r7t
dj || p d dk rd}fdt" D }|dxx dz cc< |dxx t% d z cc< t) } | }
} | | |v r
|| j nd}|d r&|d d|d d z z k rd |d d } n|d |d k rj t, vrd } n|d |d d z k rd!} n}| | j k r|d dk rd"| d#} n^| | j k r|d d$k rd"| d%} n?|d&k r|
d'k r d(|
d)d*| } n(|d&k r|
d+k r d(|
d)d*| } n|d, dk r| d-k s |d, dk rd.} n| sd/} g }
D ]*}t) | }|r|
| n+t% |
dz } d0 D ]=}t) | }|r*|
| t% |
|k r n>|
r@t5 |
d1k r-d2}| d3d4d5d6 nd2}|r9 t
d7|j || pd8 c|| _ t; | j S )9Nr+ r, r- r. r/ r0 r g Q ?r( textareaiframesrcr< VIDEOz6//table|//ul|//div|//aside|//header|//footer|//sectionz+Removed %s with score %6.3f and weight %-3sr rq Tc \ i | ](}|t d | )S )z.//)rA r )r kindr? s r
z%Document.sanitize..g sB >BD#dll<<< 8 899 r r' r r6 z.//input[@type="hidden"]r r r g?ztoo many images ()zmore
s than
sr z less than 3x
s than sztoo short content length z without a single imager z and too many imagesrX rr ztoo many links z.3fz for its weight g ?r5 K z<