Troubleshooting 101: The Power of netstat

I was reading up on ETW, and discovered the TCPView tool.  Having recently worked with the joys of ip_conntrack on Linux hosts with iptables and bad PHP frameworks leaving TCP sessions in TIME_WAIT, this was a nice find for Windows.

But in reviewing this article, its curious that Microsoft doesn't demonstrate the use of native OS tools to triage a simple port binding issue.  So here's a quick example to help you find what's listening on TCP 80.

C:\> netstat -p TCP -on | find ":80 "
  TCP    172.16.37.100:80    157.56.98.83:443       ESTABLISHED     320

This shows us a list of TCP ports with the numeric port value and the PID.

C:\> tasklist /FI "PID eq 320"
Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
w3wp.exe                       320 Services                   0     99,344 K

Pass the PID in a predicate filter to tasklist, and we find the process name.  Since it's an IIS worker process, we must go a step further to identify the Application Pool:

C:\> %windir%\system32\inetsrv\appcmd.exe list wp
WP "320" (applicationPool:WsusPool)

Another great feature of netstat is it's statistics feature, which a peer pointed out to me:

C:> netstat -s

It's left as an exercise to the reader to peruse the value that single command can have in troubleshooting down the stack before you worry about data link and the physical layer.  You've already swapped out that crossover cable for a regular patch, right?

Now on IPv6!

On a whim, I configured CloudFlare today.  Then happily discovered, it helps my site go into the future, and not just faster, but also with IPv6.  On a side note, since they'll be handling my name servers for the purposes as a CDN, I can eliminate my name server hosting costs, saving $12/year, unless I put those $12 towards premium features as the site grows.

But wait, there's more!

Looking at their apps, I found Blitz.io, which is a sweet simple load-testing service.  After testing it out a bit, I misspelled Brazil, and found this fun fact:



"s" isn't just British anymore

Hm, those regional mappings seem very familiar.  Quick to the bat-lookup (or bat-dig, your preference):

C:\>nslookup www.blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
Name:    elb014717-1106725811.us-east-1.elb.amazonaws.com
Addresses:  54.243.112.187
          54.243.77.105
          54.243.229.217
Aliases:  www.blitz.io
          mie-8036.herokussl.com

Ha, I thought so.  It looks like they're using Heroku, who's a well know user of AWS.

That's not all, Bob.  Tell 'em what they've won!

When I first ran the query, I mistakenly specified the apex zone:

C:\>nslookup blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
Name:    blitz.io
Address:  50.31.209.229

They aren't using the ELB in their apex zone.  I happen to know that feature is an AWS specific implementation of Route 53.  So let's check name servers:

C:\>nslookup -q=ns blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
blitz.io        nameserver = ns4.dnsimple.com
blitz.io        nameserver = ns2.dnsimple.com
blitz.io        nameserver = ns3.dnsimple.com
blitz.io        nameserver = ns1.dnsimple.com

Yup, they're using name servers elsewhere.  But why not use CloudFlare for the optimizations?  Perhaps its something that only their engineers can answer.

From Whence?

Ever wonder where a website's hosted?  Or who's that strange IP in your logs came from?  DNS can be more helpful than you think, if it was configured right.  So, let's have some fun, shall we?

Who's there?

In lieu of know how a site is configured, or finding a reputable site with enough foresight to configure their site in a fashion to be interesting to look at.  So who might we inspect?  How about one of the popular open-source relational database systems, Postgresql?

Staring with the basics, let's do a quick A record lookup and check the name servers for the zone:

C:\>nslookup www.postgresql.org
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
Name:    www.mirrors.postgresql.orgAddresses:  2001:4800:7903:4::126
          2a02:c0:301:0:ffff::32
          2a02:16a8:dc51::50
          217.196.149.50
          87.238.57.232
          98.129.198.126
Aliases:  www.postgresql.org

Ooooh, not only do we have six endpoints, it's a CNAME record with IPV6 addresses.  It's left as an exercise to the reader to play with IPv6 and discuss the merits of CNAMES, so let's look at name servers.

C:\>nslookup -query=ns postgresql.org
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
postgresql.org  nameserver = ns4.postgresql.org
postgresql.org  nameserver = ns3.postgresql.org
postgresql.org  nameserver = ns2.postgresql.org
postgresql.org  nameserver = ns1.postgresql.org

Looks like these guys are smart enough to host and manage their own name servers.  Not something I'd personally want to spend my time on (thanks to Dyn and now Route 53).

Where are they coming from?

So obviously they're behind three endpoints, so a quick WHOIS lookup should give us an idea of where they're at, right?

217.196.149.50 :  RIPE Network Coordination Centre
87.238.57.232 : RIPE Network Coordination Centre
98.129.198.126 : Rackspace Hosting RSCP-NET-4

We all know who Rackspace is and that's about all I can get there short of inside knowledge of Rackspace's netblock allocation, but who on earth is RIPE Network?  And why is this interesting?  So how can I know where those two IPs are hosted?  tracert is always a nice idea, but sometimes it yields the same results.

Six hops out of the interesting bits of my own network and we find:

  7    13 ms    13 ms    12 ms  208.178.58.89
  8   193 ms   191 ms   193 ms  64.209.102.34
  9   202 ms   201 ms   208 ms  asr01-0-0-0.dc1.conova.com [195.70.99.142]
 10   204 ms   198 ms   200 ms  217.196.158.12
 11   198 ms   197 ms   197 ms  217.196.158.38
 12   195 ms   198 ms   204 ms  zalkon.postgresql.org [217.196.149.50]

And also:

 11    80 ms    81 ms    88 ms  xe-4-3-0-0.nyk2nqp1.us.ip.tdc.net
 12   187 ms   188 ms   186 ms  static-213.50.153.37.addr.tdcsong.se
 13   191 ms   184 ms   192 ms  ge-1-0-0.cr2-ksd1.n.bitbit.net [213.50.153.38]
 14   187 ms   185 ms   192 ms  vlan-1.cs1-ksd1.n.bitbit.net [87.238.62.145]
 15   191 ms   186 ms   197 ms  bond0.fw2-ksd1.n.bitbit.net [87.238.62.151]
 16   189 ms   185 ms   187 ms  zetar.postgresql.org [87.238.57.232]

Certainly interesting and partially information.  I'm guessing bitbit.net might be one hosting provider or ISP.

How do we find out more?

Let's have even more fun, courtesy of DNS.  Reverse lookups were popularized a few weeks back by the tracert replay of the intro to Star Wars: A New Hope, but let's use this fact to our advantage.

Since we know that a reverse DNS lookup will give us the name of an IP and given that in most cases ISPs or service providers allocate chunks of RIR networks in contiguous blocks, let's try looking up a CIDR block of the respective ARPA addresses of those IPs:

C:\>nslookup -query=ptr 149.196.217.in-addr.arpa
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
149.196.217.in-addr.arpa
        primary name server = adns001.dc1.conova.com
        responsible mail addr = domainadmin.conova.com
        serial  = 2012092001
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 1814400 (21 days)
        default TTL = 86400 (1 day)

And then:

C:\>nslookup -query=ptr 57.238.87.in-addr.arpa
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
57.238.87.in-addr.arpa
        primary name server = ns-foo.linpro.net
        responsible mail addr = hostmaster.linpro.no
        serial  = 2013032700
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 2419200 (28 days)
        default TTL = 86400 (1 day)

And that's a wrap.  There's a good chance that the other two IPs are hosted by Conova and LinPro, who's TXT record has this little nugget:

C:\>nslookup -query=txt linpro.no
Server:  google-public-dns-b.google.com
Address:  8.8.4.4
Non-authoritative answer:
linpro.no       text =
        "never-gonna-give-you-up"
linpro.no       text =
        "google-site-verification=ChNtbQK1HLTM6oP5E3yV5SLo8NOQJmwJibARqdN0Ypo"