%$}g% d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
ddl m Z m
Z
mZmZmZmZ ddlmZ dd lmZ dd
lmZ dd lmZ ej e Z h d Z ej d
Z ej d Z ej d Z! ej d Z" ej d Z# ej d Z$ G d d Z%de&de'fdZ(de e& de%de e& fdZ)de&de%de e& fdZ*de&de%de e& fd Z+d e&de%de e& fd Z,de%d e
e& de e& fd Z- d*d#e&d$e
e& d%e'd&e.de e& f
d'Z/d(e&d$e
e& de e& fd)Z0dS )+z>
Examining feeds and extracting links for further processing.
N)islice)sleep)ListOptional) check_url clean_url filter_urlsfix_relative_urls get_hostinfo is_valid_url )is_similar_domain) fetch_url) MAX_LINKS) load_html> text/rdftext/rsstext/xml text/atom
text/plain text/rdf+xml text/rss+xml
text/atom+xmlapplication/rdfapplication/rssapplication/xmlapplication/atomapplication/jsonapplication/rdf+xmlapplication/rss+xmlapplication/atom+xmlapplication/feed+jsonapplication/x-atom+xmlapplication/x.atom+xmlz<(feed|rss|\?xml)z(?:\s*)(?:)?(?:\s*)z \bcomments\bzn\.(?:atom|rdf|rss|xml)$|\b(?:atom|rss)\b|\?type=100$|feeds/posts/default/?$|\?feed=(?:atom|rdf|rss|rss2)|feed$c
L e Zd ZdZg dZ d dedededed ee d
df d ZdS )
FeedParametersz.Store necessary information to proceed a feed.basedomainextlangrefFNbaseurlr) referenceexternal target_langreturnc L || _ || _ || _ || _ || _ d S Nr' )selfr- r) r. r/ r0 s Q/var/www/py-google-trends/myenv/lib/python3.11/site-packages/trafilatura/feeds.py__init__zFeedParameters.__init__M s+ ! ! !#. ! )FN) __name__
__module__ __qualname____doc__ __slots__strboolr r6 r7 r5 r& r& I s 44888I %)
" " " " "
" c]
"
" " " " " "r7 r& feed_stringr1 c ^ t | rdS | dd }d|v pd|v S )z$Check if the string could be a feed.TNd zz find_links.. sA HHUOO5txx~~ r7 itemsc g | ]}||S r3 r? )r^ cs r5 r_ z find_links.. s ???a r7 zJSON decoding error: %szPossibly invalid feed: %sz. sO
%%,d*B*B
T " "1 % +C*B*Br7 c 3 & K | ] }|d V
dS )r Nr? r^ ms r5 z find_links.. s7 ! r7 zc B g | ] }|d S )r
)striprh s r5 r_ z find_links.. s4
aDJJLL
r7 )rF
startswithjsonloadsr] decoderJSONDecodeErrorrP debugr) r
LINK_ATTRSfinditerr
LINK_ELEMENTSreDOTALL)r@ rH
candidatess r5
find_linksry s [ ) ) ! !# & & E
G $
; 7 7 ; ;GR H H
@?:????</
G
G
G 6
FFFFF
G
LL4fm D D D ;
$Z%8%8%E%Ey Q Q
;
M 2 2; J JIVV
Is =A$ $4BBc ~ | s"t dj g S t | }fdt | D }|r7t dt
| t
| n t dj |S )z7Extract and refine links from Atom, RSS and JSON feeds.zEmpty feed: %sc \ g | ](}|j k
| d dk &|)S )/ )r, count)r^ rU rH s r5 r_ z!extract_links.. sD 6: $**S//A"5"5
"5"5"5r7 z!Links found: %s of which %s validzInvalid feed for %s)rP rr r) ry rl rW len)r@ rH
feed_linksrS s ` r5
extract_linksr s %v}555 K - - / /88J $Z88 L ; /Z #lBSBS
*FM::: r7
htmlstringc t t | }|"t d|j g S d | d D }|s d | d D }g }t
| D ]q}t |j | }t | }|rI||j k r>t | r/t | s|
| rt dt | t | |S )zxParse the HTML and try to extract feed URLs from the home page.
Adapted from http://www.aaronsw.com/2002/feedfinder/NzInvalid HTML/Feed page: %sc g | ]b}| d t v s.t | dd L| dd cS )typehref )r]
FEED_TYPESLINK_VALIDATION_REre rf s r5 r_ z"determine_feed.. sk 88F z ) ) $ $TXXfb%9%9 : : *
) ) )r7 z //link[@rel="alternate"][@href]c g | ]F}t | d d 0| d d GS )r r )r re r] rf s r5 r_ z"determine_feed.. sX
!((&")=)=>>
HHVR
r7 z
//a[@href]z%Feed URLs found: %s of which %s valid)r rP rr r( xpathdictfromkeysr
r r, r BLACKLISTre rR r )r rH tree feed_urls output_urlsrU s r5 determine_feedr sV Z D | 16;??? JJ @AA I
<00
K
i(( % % d33 %
"" T"" # $$T** #
t $ $ $ LL/Y [AQAQ r7 urlfilterc | j rrt d| j d| j d }|rPt || }t || }t
dt | | j |S g S )z2Alternative way to gather feed links: Google News.z*https://news.google.com/rss/search?q=site:z&hl=z&scoring=n&num=100z!%s Google news links found for %s)r+ r r) r r rP rr r )rH r
downloadedr s r5 probe_gnewsr s
{
k k kFK k k k
&z6::J$Z;;J LL3S__fm
Ir7 F @r[ r0 r/
sleep_timec $ t | \ }}| t d| g S t ||| || }d}t | }|t || } | slt
|| D ]6}
t |
} | r#| t | | 7t | t | dz k r| }| r;t | | } t
dt | | | S t
d| nSt d| | d |k r t | t || S t || S )a Try to find feed URLs.
Args:
url: Webpage or feed URL as string.
Triggers URL-based filter if the webpage isn't a homepage.
target_lang: Define a language to filter URLs based on heuristics
(two-letter string, ISO 639-1 format).
external: Similar hosts only or external URLs
(boolean, defaults to False).
sleep_time: Wait between requests on the same website.
Returns:
The extracted links as a list (sorted list of unique links).
NzInvalid URL: %sr} z%s feed links found for %sz No usable feed links found: %sz Could not download web page: %sr| )r rP rQ r& r r r extendr r rr errorrl r try_homepager ) r[ r0 r/ r r) r- rH r r r rK r@ s r5 find_feed_urlsr s * #3''OFG
~(#...
GVS(K
H
HFI3 J ":v66
&z6::
J
J 'oo J %%mK&H&HIII3xx#g,,*** $Z;;J LL5s: O O O 5s;;;; 6<<< 99S>>W $ $ * 55 5 vy ) ))r7 r- c X t d| t | | S )zhShift into reverse and try the homepage instead of the particular feed
page that was given as input.z&Probing homepage for feeds instead: %s)rP rr r )r- r0 s r5 r r 4 s) LL97CCC '; / //r7 )NFr )1r; rn loggingrv itertoolsr timer typingr r courlanr r r r
r r
deduplicationr downloadsr settingsr utilsr getLoggerr8 rP r compilerC rs rd ru r r r& r= r> rF rW ry r r r floatr r r? r7 r5 r si ! ! ! ! ! ! ! ! - , , , , , 8 $ $
, rz.//
RZ.
/
/
BJ ' '
A
BJ ' ' RZ
" " " " " " " "&73 74 7 7 7 7tCy . T#Y 6%C % %DI % % % %Ps N tCy 0)s )N )tCy ) ) ) )X
8C=
T#Y
$ "& 6* 6* 6*# 6* 6* 6*
#Y 6* 6* 6* 6*r0# 0HSM 0d3i 0 0 0 0 0 0r7