n-grams in python, four, five, six grams?
Great native python based answers given by other users. But here's the nltk
approach (just in case, the OP gets penalized for reinventing what's already existing in the nltk
library).
There is an ngram module that people seldom use in nltk
. It's not because it's hard to read ngrams, but training a model base on ngrams where n > 3 will result in much data sparsity.
from nltk import ngrams
sentence = 'this is a foo bar sentences and i want to ngramize it'
n = 6
sixgrams = ngrams(sentence.split(), n)
for grams in sixgrams:
print grams
N-grams: Explanation + 2 applications
Word n-grams will generally be more useful for most text analysis applications you mention with the possible exception of language detection, where something like character trigrams might give better results. Effectively, you would create n-gram vector for a corpus of text in each language you are interested in detecting and then compare the frequencies of trigrams in each corpus to the trigrams in the document you are classifying. For example, the trigram the
probably appears much more frequently in English than in German and would provide some level of statistical correlation. Once you have your documents in n-gram format, you have a choice of many algorithms for further analysis, Baysian Filters, N- Nearest Neighbor, Support Vector Machines, etc..
Of the applications you mention, machine translation is probably the most farfetched, as n-grams alone will not bring you very far down the path. Converting an input file to an n-gram representation is just a way to put the data into a format for further feature analysis, but as you lose a lot of contextual information, it may not be useful for translation.
One thing to watch out for, is that it isn't enough to create a vector [1,1,1,2,1] for one document and a vector [2,1,2,4] for another document, if the dimensions don't match. That is, the first entry in the vector can not be the
in one document and is
in another or the algorithms won't work. You will wind up with vectors like [0,0,0,0,1,1,0,0,2,0,0,1] as most documents will not contain most n-grams you are interested in. This 'lining up' of features is essential, and it requires you to decide 'in advance' what ngrams you will be including in your analysis. Often, this is implemented as a two pass algorithm, to first decide the statistical significance of various n-grams to decide what to keep. Google 'feature selection' for more information.
Word based n-grams plus Support Vector Machines in an excellent way to perform topic spotting, but you need a large corpus of text pre classified into 'on topic' and 'off topic' to train the classifier. You will find a large number of research papers explaining various approaches to this problem on a site like citeseerx. I would not recommend the euclidean distance approach to this problem, as it does not weight individual n-grams based on statistical significance, so two documents that both include the
, a
, is
, and of
would be considered a better match than two documents that both included Baysian
. Removing stop-words from your n-grams of interest would improve this somewhat.
Implementing n-grams for next word prediction
Here`s one way as a starter:
f <- function(queryHistoryTab, query, n = 2) {
require(tau)
trigrams <- sort(textcnt(rep(tolower(names(queryHistoryTab)), queryHistoryTab), method = "string", n = length(scan(text = query, what = "character", quiet = TRUE)) + 1))
query <- tolower(query)
idx <- which(substr(names(trigrams), 0, nchar(query)) == query)
res <- head(names(sort(trigrams[idx], decreasing = TRUE)), n)
res <- substr(res, nchar(query) + 2, nchar(res))
return(res)
}
f(c("Can of beer" = 3, "can of Soda" = 2, "A can of water" = 1, "Buy me a can of soda, please" = 2), "Can of")
# [1] "soda" "beer"
Related Topics
How to Make a Matrix from a List of Vectors in R
Reading Hdf Files into R and Converting Them to Geotiff Rasters
R List Files with Multiple Conditions
Reshaping an Array to Data.Frame
How to Remove Empty Data Frames from a List
How to Create a Different Report for Each Subset of a Data Frame with R Markdown
Get Selected Rows of Rhandsontable
Interpolate Zoo Object with Missing Dates
Replace Character at Certain Location Within String
Compare Two Character Vectors in R
How to Install R Package from Private Repo Using Devtools Install_Github
Marking Specific Tiles in Geom_Tile()/Geom_Raster()
Multiply Many Columns by a Specific Other Column in R with Data.Table
Suppress Messages Displayed by "Print" Instead of "Message" or "Warning" in R
Function for Retrieving Own Ip Address from Within R
Join Data.Table on Exact Date or If Not the Case on the Nearest Less Than Date