Example:

$ cat ip.txt | python ipsearch.py -b bingkey.txt
208.201.239.101 :
    craftzine.com: CRAFT: Dedicated to the renaissance in the world of ... : 
        http://craftzine.com/
    Web 2.0 Summit 2009 - Co-produced by TechWeb & O'Reilly Conferences ... : 
        http://www.web2summit.com/web2009
    Perl.com Home Page : 
        http://www.perl.com/
    craftzine.com: CRAFT Projects : 
        http://craftzine.com/projects/
    tim.oreilly.com -- Various Thing I've Written: Tim O'Reilly's Archive : 
        http://tim.oreilly.com/
    Web 2.0 Expo San Francisco 2009 - Co-produced by TechWeb & O'Reilly ... : 
        http://www.web2expo.com/webexsf2009
    Google free proxy! : 
        http://www.oreillynet.com/pub/h/4807
    makezine.com: MAKE: Magazine : 
        http://makezine.com/magazine/
    Web 2.0 Summit 2010 - Co-produced by UBM TechWeb & O'Reilly ... : 
        http://www.web2summit.com/web2010
    O'Reilly Tools of Change for Publishing Conference 2010 - O'Reilly ... : 
        http://www.toccon.com/toc2010
    A total of 66200 results found, do manual test to find the rest : 
 
67.222.96.148 :
    O'Reilly Radar - Insight, analysis, and research about emerging ... : 
        http://radar.oreilly.com/
    Tools of Change for Publishing : 
        http://toc.oreilly.com/
    Best iPhone Apps - O'Reilly Media : 
        http://iphoneapps.oreilly.com/
    InsideRIA - Community for Rich Internet Application Developers and ... : 
        http://insideria.com/
    Google Wave: What Might Email Look Like If It Were Invented Today? - O ... : 
        http://radar.oreilly.com/2009/05/google-wave-what-might-email-l.html
    Best iPhone Apps: Best App Archives - O'Reilly Media : 
        http://iphoneapps.oreilly.com/archive.html
    Build a $21 Portable Vocal Booth - O'Reilly Digital Media Blog : 
        http://blogs.oreilly.com/digitalmedia/2008/02/build-a-portable-vocal-booth.html
    The Definition of Insanity is, perhaps, using that quote. - O'Reilly ... : 
        http://blogs.oreilly.com/digitalmedia/2006/10/the-definition-of-insanity-is.html
    Low End Linux Netbook Prices Continue To Drop - O'Reilly Broadcast : 
        http://broadcast.oreilly.com/2009/06/low-end-linux-netbook-prices-c.html
    David Pogue's Top 10 Tips for the iPhone 3GS and iPhone 3.0 Software ... : 
        http://broadcast.oreilly.com/2009/07/david-pogues-top-10-tips-for-t.html
    A total of 43800 results found, do manual test to find the rest : 
 
72.34.60.160 :
    Ignite : 
        http://ignite.oreilly.com/
    El primer Ignite Madrid tendrá lugar el 3 de Marzo del 2010, en el ... : 
        http://ignite.oreilly.com/2010/02/post-4.html

Download or clone ( hg clone http://martin.swende.se/hgwebdir.cgi/ipsearch/ )

It does this:

• Searches each search engine with exclusions for already found domains, such as “site:example.com -site:a.example.com -site:b.example.com” until all search results are exhausted and no more are found.
• Except for google, which only allows 40 keywords. If the search is not exhausted by then, it tries to use google ’slowsearch’, i.e it uses the last working query and clicks “next page” until it does not find any more results.

Usage:

$ ./subsearch.py -d oreilly.com -b bingkey.txt -y yahoo_app_id.txt 
Using domain oreilly.com
Using yahoo search API as well
Using bing search API as well
--------------------------------------------
Yahoo:
www.oreilly.com
radar.oreilly.com
ignite.oreilly.com
tim.oreilly.com
wethemedia.oreilly.com
oreilly.com
en.oreilly.com
--------------------------------------------
Bing:
oreilly.com
radar.oreilly.com
toc.oreilly.com
answers.oreilly.com
digitalmedia.oreilly.com
iphoneapps.oreilly.com
tim.oreilly.com
en.oreilly.com
--------------------------------------------
Google:
oreilly.com
answers.oreilly.com
microsoftpress.oreilly.com
digitalmedia.oreilly.com
iphoneapps.oreilly.com
tim.oreilly.com
radar.oreilly.com
--------------------------------------------
Performing google slow-search
Done
--------------------------------------------
Done: 12 subdomains found:
digitalmedia.oreilly.com
en.oreilly.com
www.oreilly.com
tim.oreilly.com
ignite.oreilly.com
toc.oreilly.com
oreilly.com
microsoftpress.oreilly.com
answers.oreilly.com
iphoneapps.oreilly.com
wethemedia.oreilly.com
radar.oreilly.com

Download or clone ( hg clone http://martin.swende.se/hgwebdir.cgi/subsearch/ )

To use the tool it, you do this :

python googlyhack.py -domain foobar.com

And then it goes through the entire GHDB (which is an XML-file, just drop in a new replacement if you find a newer one. Please also send it to me!). One feature which I missed in w3af and therefore implemented was that If google blocks you, GooglyHack pause until you tell it to continue:

Aborted after 1200 requests 
Stopped after 1200 requests (of 1466). Press any key to continue, or ctrl-c to quit

If you choose to abort, you can start it later (or on another computer) to continue again where it left off:

python googlyhack.py -domain foobar.com -i 1200

Here is a snippet of an example run against domain microsoft.com :

List loaded, length 1466
Starting at index 0
0:0 hits
1:0 hits
2:0 hits
3:0 hits
4:0 hits
5:0 hits
6:0 hits
7:2 hits
Example urls:
-Title: http://social.msdn.microsoft.com/Forums/en-US/sqlexpress/thread/30d3b3c9-9bf8-4265-82bb-d192f232cf24
-Url: Need to convert the following mysql to sql server
-Title: http://social.msdn.microsoft.com/forums/pt-BR/520/thread/0e964ff4-0b2e-4a1b-80f6-1151e05aabb5/
-Url: SQL Server Management Studio Express - Exportar com dados?
Query used : site:microsoft.com "# Dumping data for table"
GHDB Description: SQL database dumps. LOTS of data in these. So much data, infact, I'm pressed to think of what else an ev1l hax0r would like to know about a target database.. What's that? Usernames and passwords you say? Patience, grasshopper.....

I thought that google-blocking would be an issue, but using the ajax-api I had no such problems. So no additional steps are taken to bypass blocking, such as using alterating useragents or randomizing the ghdb-list. Apparantly the ajax-api is pretty nice against robotic behaviour.

Complete source can be found at http://martin.swende.se/hgwebdir.cgi/GooglyHack/
Get it either by :

hg clone http://martin.swende.se/hgwebdir.cgi/GooglyHack/

-OR-

wget http://martin.swende.se/hgwebdir.cgi/GooglyHack/raw-file/tip/src/googlyhack.py
wget http://martin.swende.se/hgwebdir.cgi/GooglyHack/raw-file/tip/src/GHDB.xml

Enjoy! (and remember : don’t be evil!)

The idea I had in mind was that I wanted the computer to
- Appear as having lots and lots of open ports
- Each open port should appear to run some service
- Each service should be recognised by nmap
…
- But actually, the incoming packets should be dropped

I got the inspiration for this when reading up on the Sockstress implementation. What we want to avoid is to have any state on the server side, since this is supposed to run on a low-end VPS-machine without affecting the other part of the system. So, how can this be achieved ? This is the basic idea:

1. Using iptables, set up so that packets to the desired ports are put into a netfilter queue.
2. Using python netfilter module, connect the netfilter queue to a python program.
3. In the python program, create a suitable response packet, and send this over a raw socket.
3a. On a SYN-packet, respond with a SYN-ACK. (This makes the port look open). The other part will finish the three-part handshake with an ACK packet.
3b. On the ACK-packet, respond with two piggybacked packets. First, an ACK/PSH with some data that looks like some valid service, e.g “Windows FTP server” or “A-311 Death welcome”. Secondly, close the connection with a FIN/ACK packet.
4. For extra credits, try to answer prudently to any other packets that may come in. Also, try to avoid endless repeating by e.g checking sequence numbers.
5. Remember that the netfilter queue from iptables still waits for a judgement on what to do with this packet? Respond to the kernel with a “DROP” signal. The OS now drops the packet and forgets all about it.

Iptables

To set up so we can fiddle the packets in python, we do this :

#Create a new iptable chain called fuzzywall
iptables -N fuzzywall
#
# Fuzzywall
# We only run it on port 1337 to start off
# - Optionally : send to logging facilties (can be found in syslog)
#iptables -A fuzzywall -p tcp --dport 1337 -j LOG --log-prefix "Sending to queue"
#Add rule to 'fuzzywall', if dest port is 1337, send to netfilter queue
iptables -A fuzzywall -p tcp --dport 1337 -j QUEUE
 
#Add our chain to INPUT
iptables -A INPUT -j fuzzywall

Python netfilter

We need to get the kernel netfilter queue into our python program. To do this, I used nfqueue library from Pollux (http://www.wzdftpd.net/blog/index.php?2008/06/01/22-nfqueue-bindings ). It is very easy to set up :

    q = nfqueue.queue()
    q.open()
    q.bind(AF_INET)
    q.set_callback(callback)
    q.create_queue(0)
    q.set_queue_maxlen(5000)
    print "starting"
    try:
        q.try_run()
    except KeyboardInterrupt, e:
        print "interrupted"
 
    print "unbinding"
    q.unbind(AF_INET)
    print "closing"
    q.close()

In addition, a handler needs to be specified which can return the verdict back to iptables :

def callback(dummy, payload):
    data = payload.get_data()
 
    handler.handlePacket(data)
 
    payload.set_verdict(nfqueue.NF_DROP)
    return

And, of course, it is in handler.handlePacket where the packetcrafting is peformed.

Python Packetcrafting

In order to play with packets, I found two libraries that could be used; impacket (http://pypi.python.org/pypi/Impacket/0.9.5) and dpkt (http://code.google.com/p/dpkt/). They have a few differences, but both can be used to parse and create all kinds of packets. In the end, I used both of them (which maybe was a bit suboptimal – but worth it just to test how they worked). This example uses a dpkt-packet parsed from raw data and generates a default response packet using impacket:

def createReturnPacket(self,packet,seqOffset = 0):
 
        dst = inet_ntoa(packet.dst)
        src = inet_ntoa(packet.src)
        sport = packet.tcp.sport
        dport = packet.tcp.dport
 
        #set the IPs
        ip = ImpactPacket.IP()
        ip.set_ip_src(dst)
        ip.set_ip_dst(src)
        #set the ports
        tcp = ImpactPacket.TCP()
        tcp.set_th_sport(dport)
        tcp.set_th_dport(sport)
        #Set a window size
        tcp.set_th_win(10)
        #Bundle the tcp packet inside the ip-packet
        ip.contains(tcp)
        return (ip,tcp,src)

I can now use this packet to create a SYN/ACK in response to a SYN :

        #Is this a SYN or an ACK?
        if (in_packet.tcp.flags & dpkt.tcp.TH_SYN) != 0:
            out_tcp_packet.set_SYN()
            out_tcp_packet.set_ACK()
            out_tcp_packet.set_th_seq(0)
            out_tcp_packet.set_th_ack(in_packet.tcp.seq+1);
            self._send(dst, out_ip_packet)

Once an appropriate return packet is created, it can be sent raw on the socket :

    def _send(self, dst, ip):
        s = socket(AF_INET, SOCK_RAW, IPPROTO_TCP)
        s.setsockopt(IPPROTO_IP, IP_HDRINCL, 1)
        s.sendto(ip.get_packet(), (dst, 0))
        return  

Fooling nmap

The desired functinality was that if an nmap service version scan was performed, such as:

nmap -p 1337 -sV <host>

Nmap should score a match. When a service version in nmap is performed, nmap sends so called ‘probes’ to the port and checks what is returned. It has a list of fingerprints for known services. For example, one service probe called “TCP GetRequest” sends “GET / HTTP/1.0\r\n\r\n”, and if the response matches “HTTP/1\.[01] \d\d\d.*\r\nDate: .*\r\nServer: Apache\r\n”, it is identified as an Apache server. If it does NOT match, nmap goes through the rest of the probes – which we definitely want to avoid, as we want as little overhead as possible.

First of all, nmap has the so called ‘Null probe’, which means that it connects to the port and waits for a few seconds to check if the service will volunteer some info without the client sending any data. It is in this segment I wanted to have my matches, since it is the first test. All service fingerprints are located in a file called nmap-service-probes, and look something like this :

 
match acap m|^\* ACAP \(IMPLEMENTATION \"CommuniGate Pro ACAP (\d[-.\w]+)\"\) | p/CommuniGate Pro ACAP server/ i/for mail client preference sharing/ v/$1/
 

To match ‘acap’, we must send some data that matches the expression after between the first pipes. It is trivial to write such a match : \* ACAP (IMPLEMENTATION “CommuniGate Pro ACAP 3.3″) would be identified as acap version 3.3. However, there are a couple of thousand fingerprints on the Null probe, and I want to randomly use them all. Therefore, I wrote a tool to reverse the regexps and put them into a python declaration.

Regexp reverse engineering

I was surprised to find that I could not locate any existing library to reverse regular expressions in the way I wanted : from a regexp, create ONE simple string that matches it. The closest I could find was a perl implementation which found as many matches it could given a regexp and a fixed length. Therefore, I wrote my own tool.

Maybe it is pretty simple to reverse a regexp, if you are a great programmer who knows what you are doing. It took me a while, though – and still I used the parts building the object-tree represention of the regexp from the built-in python regexp implementation. Basically, it does this :
* Remove all optional groups (foo)* => gone
* Insert 1 for digits, insert a for alpha, etc…
* Insert the first char in any group-of-characters [abcd] => a
* Insert the first in any OR (abc|def) => abc
… the list goes on a bit. It gets tricky when it comes to group-references and stuff. The source is available at http://martin.swende.se/hgwebdir.cgi/pxeger/file/tip/Revexpy.py

Putting it together

So, did it work? Well, yes :

#: nmap -p 82 66.249.7.26 -sV
 
PORT   STATE SERVICE VERSION
82/tcp open  ftp     Xitami ftpd
 
...again...
 
PORT   STATE SERVICE VERSION
82/tcp open  smtp    (Wanadoo blocks smtp - NOT A REAL smtpd!)
 
...again...
 
PORT   STATE SERVICE VERSION
82/tcp open  nngs    No Name Go Server

Alas, also no. After a while, the python process died – probably because it was running on a machine with 128 MB of memory, which it had to share with Apache and Mysql…

Disclaimer

While this was a really fun project which touched on a lot of interesting fields, I did not do it as a security-tool to actually use in a real scenario.

All the codez are available at:
http://martin.swende.se/hgwebdir.cgi/fuzzywall/
http://martin.swende.se/hgwebdir.cgi/pxeger/

Jag lyssnade på honom och Robert E Lee på SEC-T i Stockholm i höstas. Ett geni som avlidit alldeles för tidigt och kommer att saknas av många.

Mer info:

• Memorial

• Facebook memorial
• Robert E Lee’s blog
• Mer om Sockstress från Security Now!

En liknande föreställning är att så länge man inte “öppnar” en fil så gör det inget om den är skadlig – den kan inte göra något. Didier Stevens har postat en video ( http://blog.didierstevens.com/2009/03/04/quickpost-jbig2decode-trigger-trio/ )på tre scenarior där malware lyckas exekvera utan att filen “öppnas” genom att lägga kod i metadata. Denna exekveras sedan av Explorer när dokumentet blir valt, eller när på dokumentets thumbnail visas i fönstret, eller när användaren hovrar med musen över det. Ta en titt på videon, det är intressanta grejer.

Jinx is a javascript-based tester for cross site scripting. Currently based on GreaseMonkey, but can be easily decoupled to be used without GM support. It is a ‘quick-n-dirty’ way of testing a page you are on, not an exhaustive way of testing an entire domain.

What it does is that it gathers all the links from the page you are at (including document.location), and the ones that contain parameters are then permutated so a payload is injected into the parameters, one at a time. If you select “Quick check”, then the payload tries to inject the characters ‘ ” and < and get them reflected into the page. Any of these three characters often signal lack of filtering and probable XSS exploitation.

For example "test?a=b&c=d" generates two requests : "test?a=b &c=d” and “test?a=b&c=d “. A prompt tells you how many requests it wants to make, you can look at them or run them.

It is a pretty nice way to do some basic testing and often finds interesting stuff – I test all kinds of pages every now and then, it is staggering how many flaws are out there.

Fatherland av Robert Harris är en klassisk spänningsroman av gammalt stuk; Polis utreder mord, kommer för nära makten och hamnar risigt till. Samt blir ihop med The Woman på vägen. En twist är att storyn är förlagd i 60-talets tyskland efter att tyskland vunnit andra världskriget. Okej bok. Men läs hans bok Pompeii istället.

Bedroom Secrets of the Master Chefs, Irvine Welsh. Efter att Welsh tappade formen lite med boken “Filth” så hittade han den igen snabbt. Högklassig Welsh.

The Black Swan, av Nassim Nicholas Taleb, handlar om hur de svarta svanarna formar världen mycket mer än vad de flesta vill erkänna. Taleb ger kängor åt alla möjliga håll, inte minst åt ekonomi-”vetenskapen”. Denna bok är en av de mest intressanta jag läst under 2009.

Lords of the Bow samt Bones of the Hill, av Conn Iggulden, del två och tre i trilogin om Gengis Kahn. Iggulden är nog den bästa författaren av “historisk fiction”, och trilogin om Gengis är minst lika bra som den tidigare serien om Romarriket.

Crooked little vein, Warren Ellis – Warren Ellis har mest gjort sig känd som författare till grafiska noveller, detta är hans debutroman. Det märks att detta är en författare som är dels väldigt van att skriva, men inte så väldigt van att skriva böcker. Det är annorlunda, helt enkelt. Det är det roligaste jag läst på länge, även om man också märker att han hittat på lite “as he goes along” – när han skrev början hade han ingen aning om hur det skulle fortsätta, vilket kan vara lite vanskligt när man skriver boken i första person singular imperfekt. Galen, svart och rolig.

För ung för att dö, Christer Isaksson – Christer Isaksson gör kulturgärning genom att berätta om en av Sveriges sista avrättningar : avrättningen av Theodor Sallrot. Detta är berättelsen om honom och människorna i hans närhet, samt i viss mån samhället de levde i och samhället vi lever i. Fantastisk bok, rekommenderas starkt.

Netherland, av Joseph O’Neill – Netherland är en ganska hårt hypad roman som utkom strax före sommaren – författaren är bosatt i New York och har skrivit om det post-9/11-traumatiserade New York. Som vanligt när en bra bok kommer ut i USA ställer sig folk frågan :”Den är bra, men är den The Great American Novel?”, och som vanligt besvaras det med “Nej”. Det verkar finnas nån förväntning på att nu, snart, nån gång kommer the novel to end all novels (och den kommer vara amerikansk). Ganska lustigt det där.

Iallafall, det är en ruskigt fingerspitzgefûhl som genomsyrar denna bok. Författaren skiljdrar ytterst finkänsligt vad som pågår i huvudet på stackars NN , som snubblar sig fram i livet genom äktenskaps- och livskris. Det är inte årets största sensation men en väldigt bra roman. Litteratur av hög kvalitet.

Allah is not obliged, Ahmadou Korouma. Meh. Fick tipset från DN, men denna bok var inget vidare. Beskriver en barnsoldats liv – men ska man gestalta en liten obildad gosse som inte gått i skolan så får man hålla sig från långrandiga politiska utläggningar. Är man en sjuttioårig, högutbildad och extremt insatt i politik kanske man ska hålla sig till sitt eget perspektiv.

En annan tid, ett annat liv, Leif GW Persson. Jorå, den var bra.

I just read (though not the entire thing, just the first 40 pages) Securing WebGoat using ModSecurity. Basically, Stephen Craig Evans configured ModSecurity to patch most of the vulnerabilities in a very vulnerable web application – WebGoat. Using both traditional regexp-based ModSecurity rules and combining this with Lua-scripting he has shown how even complexities such as business logic flaws can be mitigated by a WAF. That is, basically a filtering layer (proxy) between the end user and the (insecure) web application.

It is not a silver bullet and yada yada, but WAFs seem to me to be a great tool to use for certain purposes: it gives you the chance to not touch a line of code on the application. For the purpose of patching painfully old systems, that should really have been taken out and shot but are kept running for ‘business-reasons’, I’d rather learn ModSecurity + e.g Lua properly than having to learn every thinkable and unthinkable language and platform ever used for throwing together web content.

Read it, if you are into security, which you must be if you made it past the first sentence of this entry

-Project wiki: http://www.owasp.org/index.php/OWASP_Securing_WebGoat_using_ModSecurity_Project
-Doc : http://www.owasp.org/index.php/Image:OWASP_ModSecurity_Securing_WebGoat_wiki_28Nov2008.zip