Eduzan / AI & Machine Learning
Basics of Natural Language Processing (NLP)
Computer Science / AI & Machine Learning tutorial chapter - Published 2025-12-17 - AI & Machine Learning
1. Tokenization:
- Definition: The process of breaking down text into smaller units, typically words or subwords, called tokens.
- Purpose: Helps in analyzing the structure of sentences and understanding the semantics of the text.
- Example:
- Input: “Artificial Intelligence is the future.”
- Tokens: [“Artificial”, “Intelligence”, “is”, “the”, “future”, “.”]
2. Stemming:
- Definition: The process of reducing words to their base or root form by removing suffixes.
- Purpose: Helps in grouping similar words together for analysis, though it might result in non-standard word forms.
- Example:
- Input: “running”, “runner”, “ran”
- Stemmed: “run”, “run”, “ran”
3. Lemmatization:
- Definition: Similar to stemming, but lemmatization reduces words to their dictionary form (lemma), ensuring that the word remains valid.
- Purpose: Provides a more accurate representation of the word’s meaning by considering context.
- Example:
- Input: “running”, “runner”, “ran”
- Lemmatized: “run”, “runner”, “run”
1. Bag of Words (BoW):
- Definition: A text representation technique where a text is converted into a set of words (features) with their corresponding frequencies.
- Purpose: Simplifies the text into numerical data, making it easier for machine learning models to process.
- Example:
- Sentences: “I love NLP.”, “NLP is fascinating.”
- BoW Representation: {“I”: 1, “love”: 1, “NLP”: 2, “is”: 1, “fascinating”: 1}
2. TF-IDF (Term Frequency-Inverse Document Frequency):
- Definition: A numerical statistic that reflects how important a word is to a document in a collection or corpus. It’s a product of term frequency and inverse document frequency.
- Purpose: Helps in identifying significant words in a document by downplaying common words and emphasizing unique words.
- Example:
- If “NLP” appears frequently in a document but rarely in others, its TF-IDF score will be high.
3. Word Embeddings:
- Definition: Dense vector representations of words that capture semantic meanings, relationships, and contexts. Common methods include Word2Vec, GloVe, and FastText.
- Purpose: Helps in capturing the meaning and context of words, allowing for better performance in NLP tasks.
- Example:
- The words “king” and “queen” might have embeddings close to each other, reflecting their similar meanings.
1. Recurrent Neural Networks (RNNs):
- Definition: A type of neural network designed for sequence data, where the output from one step is fed as input to the next step.
- Purpose: RNNs are used for tasks where context or sequence order matters, such as language modeling and sequence prediction.
- Example: Predicting the next word in a sentence based on previous words.
2. Long Short-Term Memory Networks (LSTMs):
- Definition: A special type of RNN designed to overcome the limitations of traditional RNNs, particularly in handling long-term dependencies.
- Purpose: LSTMs are used in tasks where it’s important to remember information over longer sequences, like text generation and machine translation.
- Example: Generating text where the context of several previous sentences affects the current word choice.
3. Transformers:
- Definition: A type of deep learning model that relies on self-attention mechanisms to process input data in parallel, rather than sequentially as in RNNs.
- Purpose: Transformers are used in a wide range of NLP tasks, including language translation, text summarization, and sentiment analysis.
- Example: Models like BERT, GPT, and T5 are based on the transformer architecture.
Sentiment Analysis:
- Definition: The process of determining the sentiment (positive, negative, neutral) expressed in a piece of text.
- Use Case: Analyzing customer reviews to determine the overall sentiment toward a product or service.
- Example:
from textblob import TextBlob
text = "I love using this product! It's fantastic."
analysis = TextBlob(text)
sentiment = analysis.sentiment.polarity
print("Sentiment:", "Positive" if sentiment > 0 else "Negative" if sentiment < 0 else "Neutral")End of lesson.