Extract Tokens

Access tokens of documents as a vector.

get_tokens(document)

# S3 method for document
get_tokens(document)

# S3 method for documents
get_tokens(document)

# S3 method for corpus
get_tokens(document)

Arguments

document	A document as returned by the *_document family of functions, i.e.: string_document.

Examples

# NOT RUN {
init_textanalysis()

# build document
doc <- string_document("This is a document.")

# extract tokens
get_tokens(doc)
# }