SUPER SPIDER
URL
spider and domain extractor script
You are going to love this script! A simple, yet
powerful cgi script with mysql database interface that will allow you to spider the
internet and collect unique urls and domain names at the speed of light! 5 minutes
to configure, set your starting url and the spider will start probing the internet from
the starting url collecting every single link and unique domain name it finds along the
way. A great way to construct targeted marketing databases or start your own search
engine. Script allows you to select spidering depth or just leave it until your hard
drive explodes with data. This script is a masterpiece.
System Requirements
Features
-
Sorts out all non unique urls and domain names
automatically
-
Able to stop and restart spider from where you
left off
-
Mysql data storage for maximum efficiency and
speed
-
Customizable starting (seed) url target
-
Customizable spidering depth
