Google
Edit File: mirror.pyc
� y]Rc @ s� d Z d d l Z d d l Z d d l Z d d l m Z m Z m Z m Z d d l m Z m Z d d l m Z d d l m Z d � Z d f d � � YZ d f d � � YZ d e f d � � YZ d e f d � � YZ e d k r� n d S( s( Module for downloading files from a pool of mirrors DESCRIPTION This module provides support for downloading files from a pool of mirrors with configurable failover policies. To a large extent, the failover policy is chosen by using different classes derived from the main class, MirrorGroup. Instances of MirrorGroup (and cousins) act very much like URLGrabber instances in that they have urlread, urlgrab, and urlopen methods. They can therefore, be used in very similar ways. from urlgrabber.grabber import URLGrabber from urlgrabber.mirror import MirrorGroup gr = URLGrabber() mg = MirrorGroup(gr, ['http://foo.com/some/directory/', 'http://bar.org/maybe/somewhere/else/', 'ftp://baz.net/some/other/place/entirely/'] mg.urlgrab('relative/path.zip') The assumption is that all mirrors are identical AFTER the base urls specified, so that any mirror can be used to fetch any file. FAILOVER The failover mechanism is designed to be customized by subclassing from MirrorGroup to change the details of the behavior. In general, the classes maintain a master mirror list and a "current mirror" index. When a download is initiated, a copy of this list and index is created for that download only. The specific failover policy depends on the class used, and so is documented in the class documentation. Note that ANY behavior of the class can be overridden, so any failover policy at all is possible (although you may need to change the interface in extreme cases). CUSTOMIZATION Most customization of a MirrorGroup object is done at instantiation time (or via subclassing). There are four major types of customization: 1) Pass in a custom urlgrabber - The passed in urlgrabber will be used (by default... see #2) for the grabs, so options to it apply for the url-fetching 2) Custom mirror list - Mirror lists can simply be a list of stings mirrors (as shown in the example above) but each can also be a dict, allowing for more options. For example, the first mirror in the list above could also have been: {'mirror': 'http://foo.com/some/directory/', 'grabber': <a custom grabber to be used for this mirror>, 'kwargs': { <a dict of arguments passed to the grabber> }} All mirrors are converted to this format internally. If 'grabber' is omitted, the default grabber will be used. If kwargs are omitted, then (duh) they will not be used. kwarg 'max_connections' limits the number of concurrent connections to this mirror. When omitted or set to zero, the default limit (2) will be used. 3) Pass keyword arguments when instantiating the mirror group. See, for example, the failure_callback argument. 4) Finally, any kwargs passed in for the specific file (to the urlgrab method, for example) will be folded in. The options passed into the grabber's urlXXX methods will override any options specified in a custom mirror dict. i����N( t URLGrabErrort CallbackObjectt DEBUGt _to_utf8( t _run_callbackt _do_raise( t exception2msg( t _THc C s | S( N( ( t st( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _g s t GrabRequestc B s e Z d Z RS( s This is a dummy class used to hold information about the specific request. For example, a single file. By maintaining this information separately, we can accomplish two things: 1) make it a little easier to be threadsafe 2) have request-specific parameters ( t __name__t __module__t __doc__( ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR j s t MirrorGroupc B s� e Z d Z d � Z d d g Z d � Z d � Z d � Z d � Z d � Z i d � Z d � Z d � Z d d � Z d � Z d d � Z RS( s| Base Mirror class Instances of this class are built with a grabber object and a list of mirrors. Then all calls to urlXXX should be passed relative urls. The requested file will be searched for on the first mirror. If the grabber raises an exception (possibly after some retries) then that mirror will be removed from the list, and the next will be attempted. If all mirrors are exhausted, then an exception will be raised. MirrorGroup has the following failover policy: * downloads begin with the first mirror * by default (see default_action below) a failure (after retries) causes it to increment the local AND master indices. Also, the current mirror is removed from the local list (but NOT the master list - the mirror can potentially be used for other files) * if the local list is ever exhausted, a URLGrabError will be raised (errno=256, No more mirrors). The 'errors' attribute holds a list of (full_url, errmsg) tuples. This contains all URLs tried and the corresponding error messages. OPTIONS In addition to the required arguments "grabber" and "mirrors", MirrorGroup also takes the following optional arguments: default_action A dict that describes the actions to be taken upon failure (after retries). default_action can contain any of the following keys (shown here with their default values): default_action = {'increment': 1, 'increment_master': 1, 'remove': 1, 'remove_master': 0, 'fail': 0} In this context, 'increment' means "use the next mirror" and 'remove' means "never use this mirror again". The two 'master' values refer to the instance-level mirror list (used for all files), whereas the non-master values refer to the current download only. The 'fail' option will cause immediate failure by re-raising the exception and no further attempts to get the current download. As in the "No more mirrors" case, the 'errors' attribute is set in the exception object. This dict can be set at instantiation time, mg = MirrorGroup(grabber, mirrors, default_action={'fail':1}) at method-execution time (only applies to current fetch), filename = mg.urlgrab(url, default_action={'increment': 0}) or by returning an action dict from the failure_callback return {'fail':0} in increasing precedence. If all three of these were done, the net result would be: {'increment': 0, # set in method 'increment_master': 1, # class default 'remove': 1, # class default 'remove_master': 0, # class default 'fail': 0} # set at instantiation, reset # from callback failure_callback this is a callback that will be called when a mirror "fails", meaning the grabber raises some URLGrabError. If this is a tuple, it is interpreted to be of the form (cb, args, kwargs) where cb is the actual callable object (function, method, etc). Otherwise, it is assumed to be the callable object itself. The callback will be passed a grabber.CallbackObject instance along with args and kwargs (if present). The following attributes are defined within the instance: obj.exception = < exception that was raised > obj.mirror = < the mirror that was tried > obj.tries = < the number of mirror tries so far > obj.relative_url = < url relative to the mirror > obj.url = < full url that failed > # .url is just the combination of .mirror # and .relative_url The failure callback can return an action dict, as described above. Like default_action, the failure_callback can be set at instantiation time or when the urlXXX method is called. In the latter case, it applies only for that fetch. The callback can re-raise the exception quite easily. For example, this is a perfectly adequate callback function: def callback(obj): raise obj.exception WARNING: do not save the exception object (or the CallbackObject instance). As they contain stack frame references, they can lead to circular references. Notes: * The behavior can be customized by deriving and overriding the 'CONFIGURATION METHODS' * The 'grabber' instance is kept as a reference, not copied. Therefore, the grabber instance can be modified externally and changes will take effect immediately. c K so | | _ | j | � | _ d | _ t j � | _ d | _ | j | � d � } | j j d | d t � d S( s� Initialize the MirrorGroup object. REQUIRED ARGUMENTS grabber - URLGrabber instance mirrors - a list of mirrors OPTIONAL ARGUMENTS failure_callback - callback to be used when a mirror fails default_action - dict of failure actions See the module-level and class level documentation for more details. i c S sH t j | d � \ } } | o; | j d i � j d t � } | | f S( Nt mirrort kwargst private( R t estimatet gett False( t mt speedt failR ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR s %t keyt reverseN( t grabbert _parse_mirrorst mirrorst _nextt threadt allocate_lockt _lockt Nonet default_actiont _process_kwargst sortt True( t selfR R R R ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt __init__� s R" t failure_callbackc C s( | j d � | _ | j d � | _ d S( NR( R" ( R R( R" ( R& R ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR# s c C sM g } x@ | D]8 } t | t � r8 i t | � d 6} n | j | � q W| S( NR ( t isinstancet basestringR t append( R& R t parsed_mirrorsR ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR s c C s< | j j � t | j � | _ | j | _ | j j � d S( N( R t acquiret listR R t release( R& t gr( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _load_gr& s c C sA | j s3 t d t d � � } | j | _ | � n | j | j S( Ni s No more mirrors to try.( R R R t errorsR ( R& R0 t e( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _get_mirror. s c C s | j j d � p | j } | rs t | � t d � k rK | \ } } } n d i } } | | | | � pm i } n i } t | j p� i � } | j | j j d i � � | j | � | } | j | | � | r| j d d � r| j t j � d _ � n d S( NR( R" R i i ( ( ( t kwR R( t typet dictR" t updatet increment_mirrorR2 t syst exc_info( R& R0 t cb_objt cbt argsR t actiont a( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _failure8 s c C s� | j | j } | j j � y | j j | � } Wn t k rF nw X| j d d � rf | j | =n3 | j | k r� | j d d � r� | j d 7_ n | j t | j � k r� d | _ n | j j � | j d d � r� | j | j =n$ | j d d � r| j d 7_ n | j t | j � k r4d | _ n t r�g | j D] } | d ^ qD} t j d d j | � | j � g | j D] } | d ^ q�} t j d d j | � | j � n d S( s� Tell the mirror object increment the mirror index This increments the mirror index, which amounts to telling the mirror object to use a different mirror (for this and future downloads). This is a SEMI-public method. It will be called internally, and you may never need to call it. However, it is provided (and is made public) so that the calling program can increment the mirror choice for methods like urlopen. For example, with urlopen, there's no good way for the mirror group to know that an error occurs mid-download (it's already returned and given you the file object). remove --- can have several values 0 do not remove the mirror from the list 1 remove the mirror for this download only 2 remove the mirror permanently beware of remove=0 as it can lead to infinite loops t remove_masteri t increment_masteri t removet incrementR s GR mirrors: [%s] %it s MAIN mirrors: [%s] %iN( R R R R- t indext ValueErrorR t lenR/ R t infot join( R& R0 R? t badmirrort indR t grmt selfm( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR9 T s0 ! c C s6 | j d � s | j d � r& | | S| d | Sd S( Nt /( t endswitht startswith( R& t base_urlt rel_url( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _join_url� s c C s� t � } | | _ | | _ t | � | _ | j | � g | _ x0 | j D]% } y | | =WqJ t k rn qJ XqJ Wd } xE| d 7} | j | � } | j | d | j � } | j d � p� | j } | j j | j d i � � } t | | � } t rt j d | | � n y | d | | f | � SWq| t k r�} t rQt j d � n | j j | t | � f � t � } | | _ | d | _ | j | _ | | _ | | _ | j | | � q| Xq| d S( Ni i R R R s MIRROR: trying %s -> %st optss MIRROR: failed( R t funct urlR7 R5 R1 R2 t optionst KeyErrorR4 RU R R RV t derivet getattrR RJ R R+ R R t exceptionR t relative_urlt triesRA ( R& RW RX R5 R0 t kR_ t mirrorchoicet fullurlR RV t func_refR3 t obj( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt _mirror_try� sD c K s� t | � } | | d <| j d � rK | g i t � f | d <| | d <n | j d d � d } y | j | | | � SWnG t k r� } t d | d | d | | � } t | j d t � | � SXd S( Nt filenamet asynct mirror_groupR^ t failfunct urlgrabRX R] ( R7 R t sett popR! Re R R R R ( R& RX Rf R R5 RW R3 Rd ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyRj � s c K s% t | � } d } | j | | | � S( Nt urlopen( R7 Re ( R& RX R R5 RW ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyRm � s c K s/ t | � } | | d <d } | j | | | � S( Nt limitt urlread( R7 Re ( R& RX Rn R R5 RW ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyRo � s N( R R R R' RY R# R R1 R4 RA R9 RU Re R! Rj Rm Ro ( ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR t s n ( 8 # t MGRandomStartc B s e Z d Z d � Z RS( s� A mirror group that starts at a random mirror in the list. This behavior of this class is identical to MirrorGroup, except that it starts at a random location in the mirror list. c K s2 t j | | | | � t j t | � � | _ d S( sg Initialize the object The arguments for intialization are the same as for MirrorGroup N( R R' t randomt randrangeRI R ( R& R R R ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR' � s ( R R R R' ( ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyRp � s t MGRandomOrderc B s e Z d Z d � Z RS( s= A mirror group that uses mirrors in a random order. This behavior of this class is identical to MirrorGroup, except that it uses the mirrors in a random order. Note that the order is set at initialization time and fixed thereafter. That is, it does not pick a random mirror after each failure. c K s* t j | | | | � t j | j � d S( sg Initialize the object The arguments for intialization are the same as for MirrorGroup N( R R' Rq t shuffleR ( R& R R R ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyR' � s ( R R R R' ( ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyRs � s t __main__( R R: Rq R R R R R R R R R R R R R Rp Rs R ( ( ( s5 /usr/lib/python2.7/site-packages/urlgrabber/mirror.pyt <module>[ s " � ^