Overview

Market research is an invaluable practice for all businesses, regardless of industry. It provides data that enables effective pricing strategies, helps to better understand competitors, informs on current trends, and more. In this practice, I programmed a bot that aids in simplifying the process by automating data collection, database management, and more.

What it Does

Shopify is one of the largest ecommerce platforms, with over 5.2 million active digital stores. This makes it a gold mine for information, no matter the niche you are in. The Shopify market research bot understands the schema of most Shopify stores, which means it works on most Shopify sites. It will pull data on every product on the site, parse it, and upload the data to a MySQL database.

The bot handles table creation and design, with the user simply needing to enter their database credentials and desired database name for the tables to be stored. There are currently 3 tables that are created and managed:

• products: A table with all unique products and their details
• variants: A table with all unique variants and their details
• log: A table with pricing, availability, and date data

With these 3 tables, you are able to effectively store data on store products and keep track of changes over time. Analysis of this data can provide a wealth of information on things like pricing, what sells fast, and more.

Set up is simple, and all instructions are included in the readme.txt file included in the project folder.

Logging

Included in the bot is a built-in logging system, keeping track of a variety of information. This includes:

• New products added
• New variants added
• New data added to the log
• Errors / when a URL is skipped
• How well the bot performed

The log is added to the app.log file included in the project and requires no tinkering to get working.

Use Cases

To showcase why this bot is useful, imagine you want to open your very own niche clothing store. Creating an effective pricing strategy can be difficult, as you have no idea how the market will react to your product. Using the bot, you can find Shopify stores (there are several databases to get this information) similar to your niche. All you have to do is copy and paste those URLs into the bot and start collecting data. This can then be analyzed to help you decide where to start, and handle things like holidays, sales, etc.

Case Study

Over the course of the next 3 months, I will be conducting a case study similar to the situation above. I will pick a niche market, find relevant stores, and run the script daily to collect data. After 3 months of collection, I will analyze the data and use it to create a strategy for my hypothetical store. When the case study is finished, I will publish the results to my portfolio.

Conclusion

This project covers a wide variety of challenges: Python programming, data scraping, data model design, automated database management, SQL syntax, data pipelines, logical flow, and more. It showcases the power of knowing how to get and work with the never-ending supply of data accessible on the internet.