Data Preprocessing for Machine Learning Models

To prepare data for a machine learning model, we convert organic data into machine-readable, floating-point values. This process involves various methods and aims at maximizing the meaningfulness of the data. We initially examined an algorithm that identified the presence or absence of characters based on an alphabet.

Enhancing this process, we assigned tokens to each letter and added them into an array. This approach gave us additional data and greater resolution, but also produced many zeros. Consequently, this required a larger model and more neurons to generate the data we needed.

Despite these limitations, the method could provide some usable accuracy when determining if a sentence was good or bad. To improve this, we sought to produce a vector map that assigns each letter a floating-point number, ranging from negative two to positive two. This gives each letter a meaningful gradient of values, creating a more refined model that includes the positions of the letters.

With some minor adjustments to our range vector map definition, we achieved an acceptable gradient range from negative two to positive two. The future aim is to write a function using our new vector map to produce a fresh vector output. In our input text, we can ignore or assign a default value to any character not in our alphabet without disrupting the process.

We have now developed a range of data, including exclamation points, which we can feed into our neural network. Moving forward, we can enhance this model by incorporating correlation coefficients and pre-trained embedding models. most AI types, that identify, label, and complete tasks, won't require such enhancements.

Видео Data Preprocessing for Machine Learning Models канала Stephen Blum