Tokenizers
Tokenizers identify the Tokens/Words in a text. gatenlp allows the use of tokenizers from other NLP libraries like NLTK, Spacy or Stanza and provides the tools to implement your own.
Tokenization is often the first step in an annotation pipeline, it creates the initial set of annotations to work on. Later steps usually process existing annotations and add new features to them or create new annotations from them (e.g. add part of speech (POS) features to existing Tokens or create noun phrase (NP) annotations from Token annotations).
import os
from gatenlp import Document
Use NLTK or own classes/methods for tokenization
The NLTK tokenizers can be used from gatenlp via the gatenlp NLTKTokenizer annotator.
This annotator can take any NLTK tokenizer or any object that has the span_tokenize(str) or tokenize(str) method. The objects that support span_tokenize(str) are preferred, as this method directly returns the spans of Tokens, not a list of tokens like tokenize(str) or a passed tokenize function. With tokenize(str) the spans have to be determined by aligning them to the original text. For this reason, the tokenize/function methods must not token strings which are modified in any way from the original text (e.g. the default NLTK word tokenizer converts beginning double quotes to 2 backquotes and cannot be used for this reason).
Some tokenize methods need to run on sentences instead of full documents, for this it is possible to specify an object/function that splits the document into sentences first. If a sentence tokenizer is specified, then the tokenize method will always be used, even if a span_tokenize method exists.
from gatenlp.processing.tokenizer import NLTKTokenizer
# Text used for the examples below
text = """Barack Obama was the 44th president of the US and he followed George W. Bush and
was followed by Donald Trump. Before Bush, Bill Clinton was president.
Also, lets include a sentence about South Korea which is called 대한민국 in Korean.
And a sentence with the full name of Iran in Farsi: جمهوری اسلامی ایران and also with
just the word "Iran" in Farsi: ایران
Also barack obama in all lower case and SOUTH KOREA in all upper case
"""
doc0 = Document(text)
doc0
Use the NLTK Whitespace Tokenizer
from nltk.tokenize.regexp import WhitespaceTokenizer
tok1 = NLTKTokenizer(nltk_tokenizer=WhitespaceTokenizer())
doc1 = Document(text)
doc1 = tok1(doc1)
doc1
Use the NLTK WordPunctTokenizer
from nltk.tokenize.regexp import WordPunctTokenizer
tok2 = NLTKTokenizer(nltk_tokenizer=WordPunctTokenizer())
doc2 = Document(text)
doc2 = tok2(doc2)
doc2
Use Spacy for tokenization
The AnnSpacy annotator can be used to run Spacy on a document and convert Spacy annotations to gatenlpannotations (see the section on lib_spacy)
If the Spacy pipeline only includes the tokenizer, this can be used for just performing tokenization as well. The following example only uses the tokenizer and adds the sentencizer to also create sentence annotations.
from gatenlp.lib_spacy import AnnSpacy
from spacy.tokenizer import Tokenizer
from spacy.lang.en import English
nlp_spacy = English()
nlp_spacy.add_pipe('sentencizer')
tok3 = AnnSpacy(nlp_spacy, add_nounchunks=False, add_deps=False, add_entities=False)
doc3 = Document(text)
doc3 = tok3(doc3)
doc3
Use Stanza for Tokenization
Similar to Spacy the Stanza library can be used for tokenization (see the lib_stanza documentation) by using a Stanza pipeline that only includes the tokenizer.
from gatenlp.lib_stanza import AnnStanza
import stanza
nlp_stanza = stanza.Pipeline("en", processors="tokenize")
doc4 = Document(text)
tok4 = AnnStanza(nlp_stanza)
doc4 = tok4(doc4)
doc4
2022-11-09 22:03:09,169|INFO|stanza|Loading these models for language: en (English):
========================
| Processor | Package |
------------------------
| tokenize | combined |
========================
2022-11-09 22:03:09,170|INFO|stanza|Use device: gpu
2022-11-09 22:03:09,170|INFO|stanza|Loading: tokenize
2022-11-09 22:03:12,013|INFO|stanza|Done loading processors!
Use Java GATE for Tokenization
The gatenlp GateWorker can be used to run arbitrary Java GATE pipelines on documents, see the GateWorker documentation for how to do this
Notebook last updated
import gatenlp
print("NB last updated with gatenlp version", gatenlp.__version__)
NB last updated with gatenlp version 1.0.8a1