Your IP : 216.73.217.17
�
�}Iec @ s d Z d d l m Z d d l Z d d l Z d d l Z d d l Z y d d l Z Wn e k
rx d d l j
Z n Xd d g Z e j Z
e e
d � j � � Z e j d d k r� e j d
� e j � e _ n d e j f d � � YZ d
e f d � � YZ d e f d � � YZ d S( s Handle digesting the messages.i����( t print_functionNi i i< t i i t utf8t HTMLStripperc B s2 e Z d Z d � Z d � Z d � Z d � Z RS( s Strip all tags from the HTML.c C s0 t j j | � | j � | | _ t | _ d S( N( t
HTMLParsert __init__t resett collectort Truet collect( t selfR ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR s
c C s2 | j � } | r. | j r. | j j | � n d S( s Keep track of the data.N( t stripR R t append( R
t data( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyt handle_data" s c C s8 t j j | | | � | j � d k r4 t | _ n d S( Nt scriptt style( s scripts style( R t handle_starttagt lowert FalseR ( R
t tagt attrs( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR ( s c C s5 t j j | | � | j � d k r1 t | _ n d S( NR R ( s scripts style( R t
handle_endtagR R R ( R
R ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR - s ( t __name__t
__module__t __doc__R R R R ( ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR s
t DataDigesterc B s� e Z d Z d d g Z d Z d Z e j d � Z e j d e j � Z
e j d � Z e j d � Z d Z
d d
� Z d � Z d � Z d
� Z e d � � Z e d � � Z e d � � Z e d � � Z RS( s The major workhouse class.t valuet digesti i s \S+@\S+s
[a-z]+:\S+s \S{10,}s \sR c C s | d k r t } n d | _ t � | _ g } x� | j | � D]r } xi | j � D][ } | j | � } | j | � rS y | j | j
d d � � Wq� t k
r� qS q� XqS qS Wq@ Wt | � | j
k r� | j | � n | j | | � | j j � | _ t | j � t k st � d S( NR t ignore( t Nonet digest_specR t HASHR t digest_payloadst
splitlinest normalizet should_handle_lineR t encodet UnicodeErrort lent atomic_num_linest
handle_atomict
handle_piecedt hexdigestt HASH_SIZEt AssertionError( R
t msgt spect linest payloadt linet norm( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR P s$
c C s" x | D] } | j | � q Wd S( s We digest everything.N( t handle_line( R
R0 R2 ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR) j s
c C sy xr | D]j \ } } x[ t | � D]M } y& | t | t | � d � | } Wn t k
r_ q X| j | � q Wq Wd S( s# Digest stuff according to the spec.id N( t ranget intR' t
IndexErrorR4 ( R
R0 R/ t offsett lengtht iR2 ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR* o s &
c C s | j j | j � � d S( N( R t updatet rstrip( R
R2 ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR4 z s c C ss | j d d � } | j } | j j | | � } | j j | | � } | j j | | � } | j j d | � j � S( Nt R ( t replacet unwanted_txt_replt longstr_ptrnt subt
email_ptrnt url_ptrnt ws_ptrnR ( t clst st repl( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR# } s c C sM g } t | � } y | j | � Wn t t j f k
r? n Xd j | � S( Nt ( R t feedt UnicodeDecodeErrorR t HTMLParseErrort join( RF R
t stripper( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyt normalize_html_part� s c C s t | � o | j t | � k S( N( R' t min_line_length( RE RF ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR$ � s c c sG x@| j � D]2} | j � d k r%| j d t � } | j � } | r^ | j d d � } n d } | ss d } n' | j � j d d � d k r� d
} n y | j | | � } WnG t t t
f k
r� y | j d d � } Wq� t k
r� q
q� Xn X| j � d k r | j | � Vq?| Vq
| j
� r4q
| j � Vq
Wd S( Nt textt decodeR= R R t asciit _t -s quopri-codect quopris quoted-printablet quotedprintablet strictt html( s quopri-codecs quopris quoted-printables quotedprintable( t walkt get_content_maintypet get_payloadR t get_content_charsetR> R RQ t LookupErrorR& R- t get_content_subtypeRN t is_multipart( RE R. t partR1 t charsett errors( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR! � s2
N( R R R t __slots__RO R( t ret compileRB t
IGNORECASERC R@ RD R? R R R) R* R4 t classmethodR# t staticmethodRN R$ R! ( ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR 3 s" t PrintingDataDigesterc B s e Z d Z d � Z RS( s6 Extends DataDigester: prints out what we're digesting.c C s- t | j d � � t t | � j | � d S( NR ( t printRQ t superRi R4 ( R
R2 ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyR4 � s ( R R R R4 ( ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyRi � s ( i i ( i< i ( R t
__future__R Rd t syst codecst hashlibR t ImportErrort html.parsert parserR t sha1R R' R+ R, t version_infot getwritert stdoutR t objectR Ri ( ( ( s0 /usr/lib/python2.7/site-packages/pyzor/digest.pyt <module> s"
�