How to Use Texthero to Prep a Text-based Dataset for Your NLP Project

Text Source:

#PROJECT #NLP #YOUR #FOR #DATASET #TEXT-BASED #PREP #TEXTHERO #USE #HOW

How to Use Texthero to Prep a Text-based Dataset for Your NLP Project

Natural Language Processing (NLP) is one of the most important fields of study and research in today’s world. It has many applications in the business sector such as chatbots, sentiment analysis, and document classification.

Preprocessing and representing text is one of the trickiest and most annoying parts of working on an NLP project. Text-based datasets can be incredibly thorny and difficult to preprocess. But fortunately, the latest Python package called Texthero can help you solve these challenges.

Texthero is a simple Python toolkit that helps you work with a text-based dataset. It provides quick and easy functionalities that let you preprocess, represent, map into vectors and visualize text data in just a couple of lines of code.

Texthero is designed to be used on top of pandas, so it makes it easier to preprocess and analyze text-based Pandas Series or Dataframes.

If you are working on an NLP project, Texthero can help you get things done faster than before and gives you more time to focus on important tasks.

NOTE: The Texthero library is still in the beta version. You might face some bugs and pipelines might change. A faster and better version will be released and it will bring some major changes.

Texthero has four useful modules that handle different functionalities that you can apply in your text-based dataset.

This module allows for the efficient pre-processing of text-based Pandas Series or DataFrames. It has different methods to clean your text dataset such as lowercase(), remove_html_tags() and remove_urls().

This module has different algorithms to map words into vectors such as TF-IDF, GloVe, Principal Component Analysis(PCA), and term_frequency.

The last module has three different methods to visualize the insights and statistics of a text-based Pandas DataFrame. It can plot a scatter plot and word cloud.

Texthero is free, open-source, and well documented. To install it open a terminal and execute the following command:

pip install texthero The package uses a lot of other libraries on the back-end such as Gensim, SpaCy, scikit-learn, and NLTK. You don't need to install them all separately, pip will take care of that.

In this article I will use a news dataset to show you how you can use different methods provided by texthero's modules in your own NLP project.

import pandas as pd Then we'll load a dataset from the data directory. The dataset for this article focuses on news in the Swahili Language.

As you can see, in our dataset we have three columns (id, content, and category). For this article we will focus on the content feature.

news_content.head() We have created a new dataframe focused on content only, and then we'll show the top 5 rows.

news_content = hero.clean(news_content) The clean() method runs seven functions when you pass a pandas series. These seven functions are:

If the default pipeline from the clean() method does not fit your needs, you can create a custom pipeline with the list of functions that you want to apply in your dataset.

news_content = news_content.pipe(hero.clean, custom_pipeline) You can see the clean dataset we have created by using custom pipeline .

Here are some other useful functions from preprocessing modules that you can try to clean you text-based dataset.

Tokenize each row of the given Pandas Series by using the tokenize() method and return a Pandas Series where each row contains a list of tokens.

Texthero contains different method to visualize insights and statistics of a text-based Pandas DataFrame.

If you want to know the top words in your text-based dataset, you can use the top_words() method from the visualization module. This method is useful if you want see additional words that you can add to the stop words lists.

This method does not return a bar graph, so I will use matplotlib to visualize the top words in a bar graph.

The wordcloud() method from the visualization module plots an image using WordCloud from the word_cloud package.

hero.wordcloud(news_content.clean_content, max_words=100,) We passed the dataframe series and number of maximum words (for this example, it is 100 words) in the wordcloud() method.

Texthero contains different methods from the representation module that help you map words into vectors using different algorithms such as TF-IDF, word2vec or GloVe. In this section I will show you how you can use these methods.

You can represent a text-based Pandas Series using TF-IDF. I created a new pandas series with two pieces of news content and represented them in TF_IDF features by using the tfidf() method.

We will use only the first 30 pieces of cleaned content from our news_content dataframe and cluster them into groups by using the kmeans() method.

) In the above source code, in the pipeline of the k-means method we passed the numb…

Видео How to Use Texthero to Prep a Text-based Dataset for Your NLP Project канала How4Pc mac10