Future Skills Fiesta:
 Get up to 30% OFF on Career Booster Combos
AVAIL NOW
D H M S

Information Gathering using Spiderfoot: A Practical Walkthrough

Author by: Ruchi Bisht
Apr 21, 2025 1064

Information gathering is the first and most crucial phase in ethical hacking, penetration testing, and cybersecurity assessments. One of the most powerful tools for automating Open Source Intelligence (OSINT) reconnaissance is Spiderfoot. This tool is designed to collect a vast amount of information about a target by leveraging various data sources and APIs.

Information Gathering using Spiderfoot

In this walkthrough, we will explore how to set up and use Spiderfoot effectively for information gathering.

What is Spiderfoot?

Spiderfoot is an open-source tool for OSINT reconnaissance. It scans targets and gathers intelligence from over 200 different sources, including DNS records, Whois records, IP addresses, social media, data breaches, open ports and vulnerabilities, and more. It provides a comprehensive profile of the target with minimal manual effort.

Features of Spiderfoot

  • Web-based and CLI Interface: Run it via a browser or terminal
  • Modular Architecture: Allows customization of scans
  • API Integration: Connects with external services like Shodan, Have I Been Pwned, VirusTotal, and more
  • Automated Data Correlation: Cross-references gathered data to identify vulnerabilities
  • Customizable Reports: Outputs in multiple formats for analysis

Step 1: Installing Spiderfoot

Spiderfoot can be installed on Linux, Windows, and macOS. It also offers a web-based GUI and a Command-Line Interface (CLI) for flexibility.

On Linux

Spiderfoot comes pre-installed in Kali Linux, but if it’s missing, you can install it manually:

sudo apt update && sudo apt install python3 python3-pip

On Linux

You can also install it from the source:

apt install spiderfoot

apt install spiderfoot

Running Spiderfoot

To start Spiderfoot’s web interface, run:

spiderfoot -l 127.0.0.1:5001

Running Spiderfoot

By default, the web interface runs on http://127.0.0.1:5001. Open it in a browser to access the dashboard.

web interface runs

Step 2: Setting Up a Target

After launching Spiderfoot’s web UI:

1. Navigate to New Scan.
2. Enter the target domain, IP, or organization name.
3. Choose a scan profile

  • All: Runs all modules to gather every possible detail (slow but thorough).
  • Footprint: Maps public-facing info about the target’s network and identity.
  • Investigate: Checks for malicious indicators along with basic footprinting.
  • Passive: Gathers intel without directly interacting with the target.

 4. Click Run Scan to initiate information gathering.

Setting Up a Target

Step 3: Selecting Modules

Spiderfoot offers over 200 modules categorized under different data types. Key modules include:

1. Network and Infrastructure Intelligence

  • sfp_dnsresolve: Resolves domain names to IP addresses.
  • sfp_whois: Retrieves WHOIS registration data.
  • sfp_ports: Identifies open ports.

 2. Social Media & Public Profiles

  • sfp_twitter: Extracts Twitter mentions.
  • sfp_facebook: Finds Facebook pages related to the target.

 3. Dark Web & Data Breaches

  • sfp_pwned: Checks if the target appears in data breaches.
  • sfp_darkweb: Searches for references to the target in dark web marketplaces and forums.

 4. Email and Contact Information

  • sfp_email: Finds email addresses linked to the domain.
  • sfp_phone: Extracts phone numbers if available.

 5. Threat Intelligence

  • sfp_virustotal: Checks if the target appears in VirusTotal reports.
  • sfp_shodan: Gathers exposed services and vulnerabilities from Shodan.

Threat Intelligence

You can enable or disable modules based on your reconnaissance scope.

Step 4: Running the Scan

Once modules are selected, Spiderfoot starts scanning in the background. The progress can be monitored in Scan Status.

  • Active modules show real-time data collection.
  • The system automatically correlates results from multiple sources.
  • Depending on the scope, scans can take minutes to hours.

Running the Scan

Step 5: Analyzing the Results

Viewing Data in the Web UI

Spiderfoot provides structured visualization:

  • Graph View: Shows relationships between entities like domains, emails, and IPs.
  • Raw Data: Displays detailed logs from each module.
  • Dashboard: Summarizes key findings.

Analyzing the Results

Exporting Results

Spiderfoot allows exporting scan results in multiple formats:

  1. After the scan completes, go to the Scans tab
  2. Select the scan
  3. On the top right of Scans, click “Export” and select “JSON” format

Spiderfoot allows exporting

Alternatively, use the web UI to download results as:

  • CSV for spreadsheets
  • GEXF – For graph analysis (e.g. in Gephi).

Advanced Usage of Spiderfoot

1. API Integration

Spiderfoot can integrate with external APIs to enhance OSINT collection. Supported APIs include Shodan, VirusTotal, and Have I Been Pwned.

To configure an API:

  • Navigate to Settings > API Keys.
  • Enter the API key from the respective service.
  • Save changes and rerun scans for enriched data.

2. CLI Mode for Automation

For those preferring command-line usage, Spiderfoot can be run in CLI mode:

spiderfoot -s testfire.net -u all -o csv > results.csv

CLI Mode for Automation

This command scans testfire.net using all modules and saves results in results.csv.

Mitigation Steps

  • Secure exposed services (e.g., disable unnecessary open ports).
  • Change compromised credentials found in data leaks.
  • Implement monitoring for threats mentioned on the dark web.

Best Practices for Using Spiderfoot

  • Define Clear Objectives: Avoid unnecessary modules to keep scans efficient
  • Respect Legal Boundaries: Use Spiderfoot responsibly and follow ethical guidelines
  • Leverage API Keys: Unlock extended data sources for comprehensive results
  • Automate Scans: Utilize CLI mode for scheduled reconnaissance tasks

Conclusion

Spiderfoot is a valuable OSINT tool for Ethical Hackers. By leveraging its extensive module library, you can uncover crucial intelligence about a target organization. Use the data responsibly and in compliance with legal frameworks.

Check out other related articles:

How Can InfosecTrain Help?

InfosecTrain offers Certified Ethical Hacker (CEH) training and Advanced Penetration Testing (APT) courses, providing hands-on experience in information gathering using tools like SpiderFoot. These courses help cybersecurity professionals develop skills in reconnaissance, vulnerability assessment, and ethical hacking techniques. SpiderFoot automates data collection from open sources, aiding in threat intelligence and security analysis. By learning these tools, participants enhance their ability to identify risks, analyze security threats, and conduct penetration testing effectively.

CEH v13 AI Certification Training

TRAINING CALENDAR of Upcoming Batches For CEH v13

Start Date End Date Start - End Time Batch Type Training Mode Batch Status
11-May-2025 28-Jun-2025 09:00 - 13:00 IST Weekend Online [ Close ]
31-May-2025 06-Jul-2025 19:00 - 23:00 IST Weekend Online [ Open ]
07-Jun-2025 13-Jul-2025 09:00 - 13:00 IST Weekend Online [ Open ]
21-Jun-2025 27-Jul-2025 19:00 - 23:00 IST Weekend Online [ Open ]
06-Jul-2025 16-Aug-2025 09:00 - 23:00 IST Weekend Online [ Open ]
CEH Exam Sprint- Strategy & Practice
TOP