!}gdZdZdZdZdZddlZddlZddlZddlm Z dd l m Z ej e ZGd d eZGd d e ZGddZGddeZdS)z Anything that has to do with threading in this library must be abstracted in this file. If we decide to do gevent also, it will deserve its own gevent file. newspaperz Lucas Ou-YangMITzCopyright 2014, Lucas Ou-YangN)Thread) ConfigurationceZdZdS)ConcurrencyExceptionN)__name__ __module__ __qualname__T/var/www/py-google-trends/myenv/lib/python3.11/site-packages/newspaper/mthreading.pyr r sDrr ceZdZdZdZdZdS)Workerz: Thread executing tasks from a given tasks queue. ctj|||_||_d|_|dS)NT)r__init__taskstimeoutdaemonstart)selfrtimeout_secondss rrzWorker.__init__s8 &   rc  |j|j\}}}n#tj$rYdSwxYw ||i|n##t $rt jYnwxYw|j)NT)r) rgetrqueueEmpty Exception traceback print_exc task_donerfuncargskargss rrunz Worker.run&s # $(JNN4s rN)r r r rr2r6r rrr)r)5sA000 ,,,rr)c$eZdZddZdZddZdS)NewsPoolNc>d|_|p t|_dS)aJ Abstraction of a threadpool. A newspool can accept any number of source OR article objects together in a list. It allocates one thread to every source and then joins. We allocate one thread per source to avoid rate limiting. 5 sources = 5 threads, one per source. >>> import newspaper >>> from newspaper import news_pool >>> cnn_paper = newspaper.build('http://cnn.com') >>> tc_paper = newspaper.build('http://techcrunch.com') >>> espn_paper = newspaper.build('http://espn.com') >>> papers = [cnn_paper, tc_paper, espn_paper] >>> news_pool.set(papers) >>> news_pool.join() # All of your papers should have their articles html all populated now. >>> cnn_paper.articles[50].html u'blahblah ... ' N)poolrconfig)rr;s rrzNewsPool.__init__Ds0 /  rcr|jtd|jd|_dS)zf Runs the mtheading and returns when all threads have joined resets the task. NzCCall set(..) with a list of source objects before calling .join(..))r:r r6r5s rr4z NewsPool.join_sF 9 &(BCC C !!### rrchddlm||}n0tfd|Dr|t|z}nd}|jj}t |||_|D]Q}t|r |j |j 2|j |j RdS)a news_list can be a list of `Article`, `Source`, or both. If caller wants to decide how many threads to use, they can use `override_threads` which takes precedence over all. Otherwise, this api infers that if the input is all `Source` objects, to allocate one thread per `Source` to not spam the host. If both of the above conditions are not true, default to 1 thread. r)SourceNc0g|]}t|Sr ) isinstance).0nr>s r z NewsPool.set..ys#;;;A*Q'';;;r) sourcer>alllenr;thread_timeout_secondsr)r:r@r2download_articlesdownload)r news_listthreads_per_sourceoverride_threadsr.r news_objectr>s @rsetz NewsPool.setjs #"""""  '*KK ;;;;;;; < < ,s9~~=KKK+4{G44 $ 9 9K+v.. 9 "";#@AAAA "";#78888  9 9rr+)rN)r r r rr4rNr rrr8r8BsK00006   999999rr8)r' __title__ __author__ __license__ __copyright__loggingrr threadingr configurationr getLoggerr logrr rr)objectr8r rrrYs3    /  ((((((g!!     9   #####V###4        C9C9C9C9C9vC9C9C9C9C9r