Google
Edit File: pool.pyc
� ۜSec @ sT d g Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z m Z d d l m Z m Z d Z d Z d Z e j � Z d � Z d e f d � � YZ e f e d � Z d e f d � � YZ d e f d � � YZ e Z d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d S( t Pooli����N( t Processt cpu_countt TimeoutError( t Finalizet debugi i i c C s t | � S( N( t map( t args( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt mapstar@ s t MaybeEncodingErrorc B s) e Z d Z d � Z d � Z d � Z RS( sV Wraps possible unpickleable errors, so they can be safely sent through the socket.c C sA t | � | _ t | � | _ t t | � j | j | j � d S( N( t reprt exct valuet superR t __init__( t selfR R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR K s c C s d | j | j f S( Ns( Error sending result: '%s'. Reason: '%s'( R R ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __str__P s c C s d t | � S( Ns <MaybeEncodingError: %s>( t str( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __repr__T s ( t __name__t __module__t __doc__R R R ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR G s c C s� | d k s0 t | � t k r* | d k s0 t � | j } | j } t | d � rn | j j � | j j � n | d k r� | | � n d } x/| d k s� | r�| | k r�y | � } Wn" t t f k r� t d � Pn X| d k r� t d � Pn | \ } } } } } y t | | | � f } Wn t k rI} t | f } n Xy | | | | f � WnM t k r�} t | | d � } t d | � | | | t | f f � n X| d 7} q� Wt d | � d S( Ni t _writers) worker got EOFError or IOError -- exitings worker got sentinel -- exitingi s0 Possible encoding error while sending result: %ss worker exiting after %d tasks( t Nonet typet intt AssertionErrort putt gett hasattrR t closet _readert EOFErrort IOErrorR t Truet Exceptiont FalseR ( t inqueuet outqueuet initializert initargst maxtasksR R t completedt taskt jobt it funcR t kwdst resultt et wrapped( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt workerX s@ 0 ! c B s e Z d Z e Z d d d d d � Z d � Z d � Z d � Z d � Z d i d � Z d d � Z d d � Z d d � Z d i d d � Z d d d � Z e d � � Z e d � � Z e d � � Z e d � � Z d � Z d � Z d � Z d � Z e d � � Z e d � � Z RS( sH Class which supports an async version of the `apply()` builtin c C s) | j � t j � | _ i | _ t | _ | | _ | | _ | | _ | d k r| y t � } Wq| t k rx d } q| Xn | d k r� t d � � n | d k r� t | d � r� t d � � n | | _ g | _ | j � t j d t j d | f � | _ t | j _ t | j _ | j j � t j d t j d | j | j | j | j f � | _ t | j _ t | j _ | j j � t j d t j d | j | j | j f � | _ t | j _ t | j _ | j j � t! | | j" d | j | j# | j | j | j | j | j | j f d d �| _$ d S( Ni s& Number of processes must be at least 1t __call__s initializer must be a callablet targetR t exitpriorityi (% t _setup_queuest Queuet _taskqueuet _cachet RUNt _statet _maxtasksperchildt _initializert _initargsR R t NotImplementedErrort ValueErrorR t TypeErrort _processest _poolt _repopulate_poolt threadingt ThreadR t _handle_workerst _worker_handlerR" t daemont startt _handle_taskst _quick_putt _outqueuet _task_handlert _handle_resultst _quick_gett _result_handlerR t _terminate_poolt _inqueuet _terminate( R t processesR' R( t maxtasksperchild( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � sV $ c C sw t } xj t t t | j � � � D]M } | j | } | j d k r" t d | � | j � t } | j | =q" q" W| S( s� Cleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. s cleaning up worker %dN( R$ t reversedt ranget lenRD t exitcodeR R t joinR" ( R t cleanedR- R3 ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _join_exited_workers� s " c C s� x� t | j t | j � � D]� } | j d t d | j | j | j | j | j f � } | j j | � | j j d d � | _ t | _ | j � t d � q Wd S( s� Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. R5 R R t PoolWorkers added workerN( RY RC RZ RD R R3 RT RN R> R? R= t appendt namet replaceR" RJ RK R ( R R- t w( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRE � s # c C s | j � r | j � n d S( sE Clean up any exited workers and start replacements for them. N( R^ RE ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _maintain_pool� s c C sP d d l m } | � | _ | � | _ | j j j | _ | j j j | _ d S( Ni ( t SimpleQueue( t queuesRe RT RN R t sendRM R t recvRQ ( R Re ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR7 � s c C s. | j t k s t � | j | | | � j � S( s1 Equivalent of `apply()` builtin ( R< R; R t apply_asyncR ( R R. R R/ ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt apply� s c C s. | j t k s t � | j | | | � j � S( s/ Equivalent of `map()` builtin ( R< R; R t map_asyncR ( R R. t iterablet chunksize( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s i c s� | j t k s t � | d k rf t | j � � | j j � � f d � t | � D� � j f � � S| d k sx t � t j � | | � } t | j � � | j j � f d � t | � D� � j f � d � � D� Sd S( sZ Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()` i c 3 s0 | ]& \ } } � j | � | f i f Vq d S( N( t _job( t .0R- t x( R. R0 ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rn R ( Ro R- Rp ( R0 ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c s s" | ] } | D] } | Vq q d S( N( ( Ro t chunkt item( ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s N( R< R; R t IMapIteratorR: R9 R t enumeratet _set_lengthR t _get_tasks( R R. Rl Rm t task_batches( ( R. R0 s, /usr/lib64/python2.7/multiprocessing/pool.pyt imap� s c s� | j t k s t � | d k rf t | j � � | j j � � f d � t | � D� � j f � � S| d k sx t � t j � | | � } t | j � � | j j � f d � t | � D� � j f � d � � D� Sd S( sK Like `imap()` method but ordering of results is arbitrary i c 3 s0 | ]& \ } } � j | � | f i f Vq d S( N( Rn ( Ro R- Rp ( R. R0 ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rn R ( Ro R- Rp ( R0 ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c s s" | ] } | D] } | Vq q d S( N( ( Ro Rq Rr ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s N( R< R; R t IMapUnorderedIteratorR: R9 R Rt Ru R Rv ( R R. Rl Rm Rw ( ( R. R0 s, /usr/lib64/python2.7/multiprocessing/pool.pyt imap_unordered s c C sV | j t k s t � t | j | � } | j j | j d | | | f g d f � | S( s> Asynchronous equivalent of `apply()` builtin N( R< R; R t ApplyResultR: R9 R Rn R ( R R. R R/ t callbackR0 ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRi s +c s� | j t k s t � t | d � s3 t | � } n | d k r} t t | � t | j � d � \ } } | r} | d 7} q} n t | � d k r� d } n t j | | | � } t | j | t | � | � � | j j � f d � t | � D� d f � � S( s< Asynchronous equivalent of `map()` builtin t __len__i i i c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rn R ( Ro R- Rp ( R0 ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr>: s N( R< R; R R t listR t divmodRZ RD R Rv t MapResultR: R9 R Rt ( R R. Rl Rm R| t extraRw ( ( R0 s, /usr/lib64/python2.7/multiprocessing/pool.pyRk ) s ( c C so t j � } xB | j t k s6 | j rP | j t k rP | j � t j d � q W| j j d � t d � d S( Ng�������?s worker handler exiting( RF t current_threadR< R; R: t TERMINATERd t timet sleepR9 R R R ( t poolt thread( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRH > s * c C s1 t j � } x� t | j d � D]� \ } } d } x� t | � D]P \ } } | j ra t d � Pn y | | � Wq> t k r� t d � Pq> Xq> W| r t d � | | d � q q Pq Wt d � y@ t d � | j d � t d � x | D] } | d � q� WWn t k r"t d � n Xt d � d S( Ni����s' task handler found thread._state != RUNs could not put task on queues doing set_length()i s task handler got sentinels/ task handler sending sentinel to result handlers( task handler sending sentinel to workerss/ task handler got IOError when sending sentinelss task handler exiting( RF R� t iterR R Rt R< R R! R ( t taskqueueR R& R� R� t taskseqt set_lengthR- R+ t p( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRL K s6 c C s� t j � } x� y | � } Wn" t t f k r@ t d � d SX| j rm | j t k s_ t � t d � Pn | d k r� t d � Pn | \ } } } y | | j | | � Wq t k r� q Xq x� | ri| j t k riy | � } Wn" t t f k rt d � d SX| d k r+t d � q� n | \ } } } y | | j | | � Wq� t k req� Xq� Wt | d � r�t d � y5 x. t d � D] } | j j � s�Pn | � q�WWq�t t f k r�q�Xn t d t | � | j � d S( Ns. result handler got EOFError/IOError -- exitings, result handler found thread._state=TERMINATEs result handler got sentinels& result handler ignoring extra sentinelR s" ensuring that outqueue is not fulli s7 result handler exiting: len(cache)=%s, thread._state=%s( RF R� R! R R R< R� R R t _sett KeyErrorR RY R t pollRZ ( R& R t cacheR� R+ R, R- t obj( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRP r sX c c sC t | � } x0 t t j | | � � } | s1 d S| | f Vq d S( N( R� t tuplet itertoolst islice( R. t itt sizeRp ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRv � s c C s t d � � d S( Ns: pool objects cannot be passed between processes or pickled( R@ ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __reduce__� s c C s5 t d � | j t k r1 t | _ t | j _ n d S( Ns closing pool( R R< R; t CLOSERI ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s c C s- t d � t | _ t | j _ | j � d S( Ns terminating pool( R R� R< RI RU ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt terminate� s c C sn t d � | j t t f k s% t � | j j � | j j � | j j � x | j D] } | j � qV Wd S( Ns joining pool( R R< R� R� R RI R\ RO RR RD ( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR\ � s c C sW t d � | j j � x9 | j � rR | j j � rR | j j � t j d � q Wd S( Ns7 removing tasks from inqueue until task handler finishedi ( R t _rlockt acquiret is_aliveR R� Rh R� R� ( R% t task_handlerR� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _help_stuff_finish� s c C s� t d � t | _ t | _ t d � | j | | t | � � | j � sc t | � d k sc t � t | _ | j d � t d � t j � | k r� | j d � n | r� t | d d � r� t d � x- | D]" } | j d k r� | j � q� q� Wn t d � t j � | k r$| j d � n t d � t j � | k rP| j d � n | r�t | d d � r�t d � x; | D]0 } | j � rzt d | j � | j � qzqzWn d S( Ns finalizing pools& helping task handler/workers to finishi s joining worker handlerg}Ô%�I�TR� s terminating workerss joining task handlers joining result handlers joining pool workerss cleaning up worker %d( R R� R< R� RZ R� R R R RF R� R\ R R[ R� t pid( t clsR� R% R& R� t worker_handlerR� t result_handlerR� R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRS � s8 $ N( ( ( ( R R R R R R R^ RE Rd R7 Rj R Rx Rz Ri Rk t staticmethodRH RL RP Rv R� R R� R\ R� t classmethodRS ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s0 9 ': R{ c B sD e Z d � Z d � Z d � Z d d � Z d d � Z d � Z RS( c C sS t j t j � � | _ t j � | _ | | _ t | _ | | _ | | | j <d S( N( RF t Conditiont Lockt _condt job_countert nextRn R: R$ t _readyt _callback( R R� R| ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR s c C s | j S( N( R� ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt ready s c C s | j s t � | j S( N( R� R t _success( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt successful s c C sB | j j � z | j s, | j j | � n Wd | j j � Xd S( N( R� R� R� t waitt release( R t timeout( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� s c C s< | j | � | j s t � n | j r/ | j S| j � d S( N( R� R� R R� t _value( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR # s c C s� | \ | _ | _ | j r7 | j r7 | j | j � n | j j � z t | _ | j j � Wd | j j � X| j | j =d S( N( R� R� R� R� R� R" R� t notifyR� R: Rn ( R R- R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� , s N( R R R R� R� R R� R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR{ s R� c B s e Z d � Z d � Z RS( c C s t j | | | � t | _ d g | | _ | | _ | d k r` d | _ t | _ | | j =n | | t | | � | _ d S( Ni ( R{ R R" R� R R� t _chunksizet _number_leftR� Rn t bool( R R� Rm t lengthR| ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR @ s c C s | \ } } | r� | | j | | j | d | j +| j d 8_ | j d k r| j rn | j | j � n | j | j =| j j � z t | _ | j j � Wd | j j � XqnW t | _ | | _ | j | j =| j j � z t | _ | j j � Wd | j j � Xd S( Ni i ( R� R� R� R� R: Rn R� R� R" R� R� R� R$ R� ( R R- t success_resultt successR0 ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� L s* " ( R R R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� > s Rs c B s> e Z d � Z d � Z d d � Z e Z d � Z d � Z RS( c C sk t j t j � � | _ t j � | _ | | _ t j � | _ d | _ d | _ i | _ | | | j <d S( Ni ( RF R� R� R� R� R� Rn R: t collectionst dequet _itemst _indexR t _lengtht _unsorted( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR m s c C s | S( N( ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __iter__w s c C s� | j j � z� y | j j � } Wn� t k r� | j | j k rN t � n | j j | � y | j j � } Wq� t k r� | j | j k r� t � n t � q� Xn XWd | j j � X| \ } } | r� | S| � d S( N( R� R� R� t popleftt IndexErrorR� R� t StopIterationR� R R� ( R R� Rr R� R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� z s& c C s� | j j � z� | j | k r� | j j | � | j d 7_ xJ | j | j k r� | j j | j � } | j j | � | j d 7_ qA W| j j � n | | j | <| j | j k r� | j | j =n Wd | j j � Xd S( Ni ( R� R� R� R� R` R� t popR� R� R: Rn R� ( R R- R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s c C s^ | j j � z< | | _ | j | j k rH | j j � | j | j =n Wd | j j � Xd S( N( R� R� R� R� R� R: Rn R� ( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRu � s N( R R R R� R R� t __next__R� Ru ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRs k s Ry c B s e Z d � Z RS( c C st | j j � zR | j j | � | j d 7_ | j j � | j | j k r^ | j | j =n Wd | j j � Xd S( Ni ( R� R� R� R` R� R� R� R: Rn R� ( R R- R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s ( R R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRy � s t ThreadPoolc B sB e Z d d l m Z d d d d � Z d � Z e d � � Z RS( i ( R c C s t j | | | | � d S( N( R R ( R RV R' R( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s c C s@ t j � | _ t j � | _ | j j | _ | j j | _ d S( N( R8 RT RN R RM R RQ ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR7 � s c C sW | j j � z5 | j j � | j j d g | � | j j � Wd | j j � Xd S( N( t not_emptyR� t queuet cleart extendR t notify_allR� ( R% R� R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s N( ( R R t dummyR R R R7 R� R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s ( t __all__RF R8 R� R� R� t multiprocessingR R R t multiprocessing.utilR R R; R� R� t countR� R R# R R R3 t objectR R{ t AsyncResultR� Rs Ry R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt <module># s, *� �.-I