February 2019

Incredibly fast crawler designed for OSINT.

demo

Key Features

Data Extraction

Photon can extract the following data while crawling:
  • URLs (in-scope & out-of-scope)
  • URLs with parameters (example.com/gallery.php?id=2)
  • Intel (emails, social media accounts, amazon buckets etc.)
  • Files (pdf, png, xml etc.)
  • Secret keys (auth/API keys & hashes)
  • JavaScript files & Endpoints present in them
  • Strings matching custom regex pattern
  • Subdomains & DNS related data
The extracted information is saved in an organized manner or can be exported as json.
save demo

Flexible

Control timeout, delay, add seeds, exclude URLs matching a regex pattern and other cool stuff. The extensive range of optionsprovided by Photon lets you crawl the web exactly the way you want.

Genius

Photon's smart thread management & refined logic gives you top notch performance.
Still, crawling can be resource intensive but Photon has some tricks up it's sleeves. You can fetch URLs archived by archive.org to be used as seeds by using --wayback option.
In Ninja Mode which can be accessed by --ninja, 4 online services are used to make requests to the target on your behalf.
So basically, now you have 4 clients making requests to the same server simultaneously which gives you a speed boost if you have a slow connection, minimizes the risk of connection reset as well as delays requests from a single client.

Plugins

Docker

Photon can be launched using a lightweight Python-Alpine (103 MB) Docker image.
$ git clone https://github.com/s0md3v/Photon.git
$ cd Photon
$ docker build -t photon .
$ docker run -it --name photon photon:latest -u google.com
To view results, you can either head over to the local docker volume, which you can find by running docker inspect photonor by mounting the target loot folder:
$ docker run -it --name photon -v "$PWD:/Photon/google.com" photon:latest -u google.com

Frequent & Seamless Updates

Photon is under heavy development and updates for fixing bugs. optimizing performance & new features are being rolled regularly.
If you would like to see features and issues that are being worked on, you can do that on Development project board.
Updates can be installed & checked for with the --update option. Photon has seamless update capabilities which means you can update Photon without losing any of your saved data.

Contribution & License

You can contribute in following ways:
  • Report bugs
  • Develop plugins
  • Add more "APIs" for ninja mode
  • Give suggestions to make it better
  • Fix issues & submit a pull request
Please read the guidelines before submitting a pull request or issue.
Do you want to have a conversation in private? Hit me up on my twitter, inbox is open :)
Photon is licensed under GPL v3.0 license

Use the damn terminal

This guide was written for Debian and Ubuntu but should work with any linux distribution and with OSX.
What we, hackers, need is a portable easy and fast to install setup requiring minimal configuration. It’s also important the setup to work servers might you need it.
Here’s some tips and advice to help you out.

Keyboard typing

To be honest, that part sucks. It’s a hard skill to acquire but it’s the most important one. If you can type fast, without looking at your keyboard and without typos, you’ll be like those Hollywood hackers. No software can beat that.
Klavaro is your friend. Just “apt-get install klavaro” and here you go. Check this guide for more info.

Terminal emulator

Don’t use the default terminal emulator. It suck.
What we’re going to use here is Terminator. It’s not that great but it works everywhere. What we need it for is split windows. Download it. As you get accommodated to using the terminal you’ll need to multi task.
Make sure you change the font to something that suits you. Smaller is better as it will allow you to cram more window in your term.
At some point you’ll probably want to customize your options. Check the terminator config manual.

Term browsing

Yep… It’s a thing. And it’s very useful. W3M is your friend. “apt-get install w3m” and “w3m duckduckgo.com”.

Multiple sessions over multiple servers

Byobu is a neat software to help you manage multiple terminal sessions. It keep them alive on your local and remote machines. Once installed use the F1 key to configure, access help and use the F2, F3, F4 to create and more between windows.
Just type “apt-get install byobu”. To enable by default on remote servers use “byobu-enable”.
My favorite trick is keyboard copy/paste. Press F7 and move around, then press spacebar to select your text, press enter to return in normal mode. Paste with F12 and then CTRL plus ].

Bash the shell

Bash is great and all but ZSH is greater.
The first thing you need to learn about is auto-complete. It’s what happen with you start typing a command or a path and hit the TAB key. ZSH auto-complete is freaking awesome.
Then there’s OhMyZSH. One command curl install and you’ll have a complete setup and you’ll be ready to roll. It’s a bliss. Be sure to check included themes and plug-ins.

Vim (and not Emacs)

Vim is a great code editor but… vim.spf13 made it awesome. Perfect even. Vim is hard to learn at first but it’s on all systems. You’ll be glad to know about it when you’ll start navigating in those weirds Russian servers ;)

Color schemes

At some point you’ll want to choose and get used to a color scheme. It’s really important when you spend hours in the terminal so take your time to try some out.
Solarized is the most popular one. I don’t like it but it’s everywhere. You’ll always be able to use it whatever app your on.
If you need help choosing a color scheme check Vim Colors.

Powerline fonts

Some themes and softwares like Vim can take advantage of patched fonts and provide you with advanced feedback. Installing them is easy.
cd ~/Downloads ; git clone https://github.com/powerline/fonts ; cd fonts ; ./install.sh ; cd .. ; rm -fr fonts

With ZSH

To take advantage of powerline fonts with ZSH use the agnoster theme. You’ll need to edit your ~/.zshrc file.

With SPF13 VIM

You need to create ~/.vimrc.before.local and add the following line “let g:airline_powerline_fonts=1”.

Footprinting & gain information Technology Brief


Footprinting phase allows the attacker to gather the inf ormation regarding internal and external security architecture; he has to face a target. Collection of inf ormation also helps to identif y the vulnerabilities within a system, which exploits, to gain access. Getting deep inf ormation about target reduces the focus area & bring attacker closer to the target. The attacker focuses the target by mean of the range of IP address he has to go through, to hack target or regarding domain inf ormation or else.
Footprinting  Concepts
The first step to ethical hacking is Footprinting. Footprinting is the collection of every possible inf ormation regarding the target and target network. This collection of inf ormation helps in identif ying diff erent possible ways to enter into the target network. This collection of inf ormation may  have gathered through publicly- available personal inf ormation and sensitive inf ormation from  any  secret  source.  Typically,  footprinting   &   reconnaissance   is perf orming social engineering attacks, system or network attack, or through any other technique. Active and passive methods of reconnaissance are also popular for gaining inf ormation of target directly or indirectly. The overall purpose of this phase is to keep interaction with the target to gain inf ormation without any detection or alerting.
anonymous Footprinting
anonymous footprinting includes footprinting through online sources. In anonymous footprinting, inf ormation about a target is shared by posting with an assumed name. This type inf ormation is shared with the real credential to avoid trace to an actual source of inf ormation.
Internet Footprinting
Internet Footprinting includes the Footprinting and reconnaissance methods for gaining inf ormation through the internet. In Internet Footprinting, processes such as Google Hacking, Google Search, Google Application including search engines other than Google as well.
Objectives of Footprinting
The major objectives of Footprinting are: -
    1.  To know security posture
    2. To reduce focus area
    3. Identif y vulnerabilities
    4. Draw network map
Footprinting  Methodology
It is not a big deal to get inf ormation regarding anyone as the internet, social media, of ficial websites and other resources have much inf ormation about their users which are not sensitive, but a collection of inf ormation may fulfill the requirements of an attacker and attacker can gather enough inf ormation by a little eff ort. Below are more often techniques used by hackers: -
    • Footprinting through Search Engines
    • Footprinting through Advance Google Hacking Techniques
    • Footprinting through Social Networking Sites
    • Footprinting through Websites
    • Footprinting through Email
    • Footprinting through Competitive Intelligence
    • Footprinting through WHOIS
    • Footprinting through DNS
    • Footprinting through Network
    • Footprinting through Social Engineering

Footprinting through Search Engines
The most basic option that is very responsive as well is Footprinting through search engines. Search engines extract the inf ormation about an entity you have searched for from internet. You can open a web browser and through any search engine like Google or Bing, search for any  organization.  The result collects every available inf ormation on the internet.

Information Gathering Using Groups, Forums, and Biogs
Groups, Forums, Blogs, and Communities can be a great source of sensitive inf ormation. Joining with fake ID on these platf orms and reaching closest to the target organization's group is not a big deal for anyone. Any of ficial and non-of ficial group can leak sensitive inf ormation.

Footprinting using Advanced Google Hacking Techniques

Google Advanced Search Operators
Some advanced options can be used to search for a specific topic using search engines. These Advance search operators made the searching more appropriate and focused on a certain topic. Advanced search operators by google are: -
Advanced Search Operators
Description
site :
Search for the result in the given domain
related :
Search for Similar web pages
cache :
Display the web pages stored in Cache
link :
List the websites having a link to a specific web page
allintext :
Search for websites containing a specific keyword
intext :
Search for documents containing a specific keyword
allintitle :
Search for websites containing a specific keyword in the title
intitle :
Search for documents containing a specific keyword in the title
allinurl :
Search for websites containing a specific keyword in URL
inurl :
Search for documents containing a specific keyword in URL
Google Advanced Search Operators
For Google Advanced Search, you can also go to the following URL:

I https://www.google.com/advanced_search

Google Hacking Database (GHDB)
Google hacking database provide the updated inf ormation that is usef ul for exploitation such as footholds, sensitive directories, vulnerable files, error messages and much more.
Google hacking is used to speed up searches. As shown in the figure, through www.exploit-db.com, you can search GHDB or browse the category of GHDB. Similarly, www.hackersforcharity.org is also an online platf orm for GHDB.
Enter the following URL:

Footprinting through Social Networking Sites
Social Engineering
Social Engineering in Inf ormation Security ref ers to the technique of psychological manipulation. This trick is used to  gather  inf ormation  from diff erent social networking and other plat£arms from people for fraud, hacking and getting inf ormation for being close to the target.
Footprinting using Social Engineering  on Social Networking  Sites
Social Networking is one of the best inf ormation sources among other sources. Diff erent popular and most widely used social networking site has made quite easy to find someone, get to know about someone, including its basic personal inf ormation as well as some sensitive inf ormation as well. Advanced features on these social networking sites also provide up-to-date inf ormation. An Example of footprinting through social networking sites can be  finding  someone  on FacebookTwitterLinkedlnInstagram  and  much
What Users Do
Information
What attacker gets
People maintain their profile
  • Photo of the target
  • Contact numbers
  • Email Addresses
  • Date of birth
  • Location
  • Work details
  • Personal Information about a target including personal information,  photo, etc.
  • Social engineering
People updates their Most recent personal information
  • Platform & Technology related information.
  • Target Location .
  • List of Employees I Friends I
Family .
  • Nature of business
status
  • Most recent location
  • Family & Friends information
  • Activities & Interest
  • Technology related information
  • Upcoming events information
                                                                      Social Engineering
Website Footprinting
Website Footprinting includes monitoring and investigating about the target organization's of ficial website for gaining inf ormation such as Software running, versions of these software's, operating systems, Sub-directories, database, scripting inf ormation, and other details. This inf ormation can be gathered by online service as defined earlier like netcraf t.com or by using software such as Burp Suite, Zaproxy, Website Inf ormer, Firebug, and others. These tools can bring inf ormation like connection type and status and last modification inf ormation. By getting these type of inf ormation, an attacker can examine source code, developer's details, file system structure and scripting.
Determining the Operating System
Using websites such as Netcraf t.com can also help in searching for Operating systems that are in use by the targeted organizations. Go to the website www.netcraf t.com and enter the target organization's of ficial URL. Results in the figure below are hidden to avoid legal issues.

websites is Shodan, i.e. www.shodan.io . SHODAN search engine lets you find connected devices such as router, servers, IoT & other devices by using a variety of filters.