One Approach to Solving Tokenization Problem for Analysis of Large-Scale Collections of User-Defined Passwords

Abstract

This paper performs an analysis of the algorithm of password tokenization introduced by R. Veras et al. We show main limitations of this approach and propose a new tokenization algorithm - RGramToken, based on frequency dictionaries of English words, bigrams and trigrams. Our approach allows better utilization of information about probabilitiy distribution of words and word combinations in a natural language. The results of comparison analysis of these two algorithms on specially prepared tests with warped phrases demonstrate higher efficiency of RGramToken and its robustness on low quality dictionaries.

Keywords

• passwords
• leaks
• tokenization
• words isolation