A book is more than what it is! – Part II

August 12, 2022 The Editorial Board- Teamware Solutions

Recap

Readers who have landed here directly from the search engines / forums; let us set you context really quick. We aspire to see if there is a writing pattern that can be expressed using Part of Speech (PoS) tags for writers (book authors). Our underlying goal by undergoing this exercise is also to explore the structure of experiments in data science. Organisation for real world projects that could be scaled.

Towards that journey we set up the initial folder structure and added the code to read in a page from the PDF file.

Setting up feedback mechanism

We next take up the printing this to the console. There are two things that crossed our mind. The simplest one is to add a print statement to the function itself. Another is to have separate script that performs such operation. In other languages like C# or java we will have Main that does it for us. For now let us do it inside the function itself.

Now let us take this a step further and extract the entire page’s text.

You must have noticed the challenge. There are two function and we will need only one entry point for the application. That makes us sway towards another approach that we mentioned earlier. So, let us refactor code. Off the bat many calls this fixing the code. In data science and we urge in every manner this is refactor. Come to think about it we developers always refactor code through the entire day.

Refactoring here is simple we replace the last line of both the functions to like this –

Small things like keeping the name of the value a function returns can speed up such exploratory analysis related coding lot easier.

Now we introduce the conductor for choosing the mode of exploration we want. Let us say, we call the conductor Experiemnts.py. Isn’t it apt?

This gives us a place to compile the path of experiments. We can call as many as we want. The next challenge is when we want to change it to something else; i.e., a different set of experiments. We then will miss the current experiment. Having all of them together in this file also clutters the file. We propose to use Git and tagged commits to manage such check-ins. Though, it appears to be kind of inspector gadget without the manual at present, it becomes easier with practice.

We could opt the best-in off the market IDE with workbench. It works as well. But starting from basics always helps to have root go deeper.

Coming back to the task at hand; we have a conductor for the experiment and have a file that reads the file as raw text.

Next, we take up the inferring the Part of Speech from this text. For that we will need some libraries – nltk. This is a basic library but will suffice our need for now. If you have worked with this earlier you will know that it gives PoS tags with abbreviations drawn straight from the Penn bank. Let us first have a summary of them so that we need not remember it. We made it a dictionary as it is finite.

This function we will we consume after generating the PoS tag in our next dispatch.