Create a crawler (spider) bot that can be used for specific purposes, related to data collection.
We developed a lightweight web crawler based on Python with a simple web interface that can be used to automatically build search index for any particular website.
- Low hardware requirements
- Built-in web server
- Built-in web interface
Python, asyncio, aiohttp