U yc,@sLddlZddlZddlmZmZddlmZmZedZ GdddZ dS)N)OptionalUnion)LanguageFilter ProbingStates%[a-zA-Z]*[-]+[a-zA-Z]*[^a-zA-Z-]?c@seZdZdZejfeddddZddddZee e dd d Z ee e dd d Z e eefed ddZeedddZedddZee eefedddZee eefedddZee eefedddZdS) CharSetProbergffffff?N) lang_filterreturncCs$tj|_d|_||_tt|_dS)NT) r DETECTING_stateactiverlogging getLogger__name__logger)selfrr@/tmp/pip-build-0we0fmvo/pip/pip/_vendor/chardet/charsetprober.py__init__,szCharSetProber.__init__)r cCs tj|_dSN)rr r rrrrreset2szCharSetProber.resetcCsdSrrrrrr charset_name5szCharSetProber.charset_namecCstdSrNotImplementedErrorrrrrlanguage9szCharSetProber.language)byte_strr cCstdSrr)rrrrrfeed=szCharSetProber.feedcCs|jSr)r rrrrstate@szCharSetProber.statecCsdS)Ngrrrrrget_confidenceDszCharSetProber.get_confidence)bufr cCstdd|}|S)Ns([-])+ )resub)r rrrfilter_high_byte_onlyGsz#CharSetProber.filter_high_byte_onlycCsZt}t|}|D]@}||dd|dd}|sJ|dkrJd}||q|S)u7 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [€-ÿ] marker: everything else [^a-zA-Z€-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. Nr!) bytearrayINTERNATIONAL_WORDS_PATTERNfindallextendisalpha)r filteredwordsword last_charrrrfilter_international_wordsLs    z(CharSetProber.filter_international_wordscCst}d}d}t|d}t|D]R\}}|dkrB|d}d}q$|dkr$||krr|sr|||||dd}q$|s|||d |S) a[ Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Frc>rs