Friday, July 19, 2013

Life with DRM

So my Sony Blu-Ray player can't play Netflix today.  Wonder why?  Well it's not Netflix, that works great on my PC.  And my iPhone and my Wii.  tcpdump to the rescue!

That's all fine an dandy if my DD-WRT device was my router instead of just an AP, I'd have to mount a CIFS share and download a package in order to do a tcpdump.

Instead, I've got a hand-me-down Juniper with ScreenOS.  Not being intimately familiar, I did recall it had some equivalent facilities if using the CLI, so without further ado, here's a cheat sheet for working with snoop:

ssg-> clear dbuf
ssg-> snoop info
Snoop: OFF
Filters Defined: 1, Active Filters 1
Detail: ON, Detail Display length: 96
Snoop tunnel traffic: ON
Snoop filter based on:
id 1(on): IP src-ip 172.16.37.101 dir(B)
ssg-> snoop filter ip src-ip 172.16.37.100 direction both
snoop filter added
ssg-> snoop
Start Snoop, type ESC or 'snoop off' to stop, continue? [y]/n y


clear dbuf clears the in memory buffer, then snoop info shows the current snoop state.  Then I use a filter to just watch one device, so I can determine what destination IPs are involved.  Later, we can re-run with a different filter to get a better sampling. 

I did a quick telnet test to www.google.com then stopped the snoop trace.

ssg-> snoop off
Snoop off

To view the trace, we send the dbuf to stream output and page through results.  If you're using a more detailed filter it will decode unencrypted HTTP requests a bit more.

ssg-> get dbuf stream91638.0: bgroup0(i) len=74:0019d28739e8->001db5ba738b/0800
              172.16.37.100 -> 8.8.8.8/17
              vhl=45, tos=00, id=2844, frag=0000, ttl=128 tlen=60
              udp:ports 56461->53, len=40
              00 1d b5 ba 73 8b 00 19 d2 87 39 e8 08 00 45 00     ....s.....9...E.
              00 3c 0b 1c 00 00 80 11 4e 11 ac 10 25 64 08 08     .<......N...%d..
              08 08 dc 8d 00 35 00 28 51 91 61 1e 01 00 00 01     .....5.(Q.a.....
              00 00 00 00 00 00 03 77 77 77 06 67 6f 6f 67 6c     .......www.googl
              65 03 63 6f 6d 00 00 01 00 01                       e.com.....   

91638.0: bgroup0(i) len=66:0019d28739e8->001db5ba738b/0800
              172.16.37.100 -> 173.194.33.17/6
              vhl=45, tos=00, id=2845, frag=4000, ttl=128 tlen=52
              tcp:ports 49295->80, seq=787237732, ack=0, flag=8002/SYN
              00 1d b5 ba 73 8b 00 19 d2 87 39 e8 08 00 45 00     ....s.....9...E.
              00 34 0b 1d 40 00 80 06 4f 5f ac 10 25 64 ad c2    
.4..@...O_..%d..
              21 11 c0 8f 00 50 2e ec 4b 64 00 00 00 00 80 02     !....P..Kd......
              20 00 73 98 00 00 02 04 05 b4 01 03 03 08 01 01     ..s.............
              04 02                                               ..           

91638.0: bgroup0(i) len=60:0019d28739e8->001db5ba738b/0800
              172.16.37.100 -> 173.194.33.17/6
              vhl=45, tos=00, id=2846, frag=4000, ttl=128 tlen=40
              tcp:ports 49295->80, seq=787237733, ack=1688811284, flag=5010/ACK
              00 1d b5 ba 73 8b 00 19 d2 87 39 e8 08 00 45 00     ....s.....9...E.
              00 28 0b 1e 40 00 80 06 4f 6a ac 10 25 64 ad c2     .(
..@...Oj..%d..
              21 11 c0 8f 00 50 2e ec 4b 65 64 a9 37 14 50 10     !....P..Ked.7.P.
              00 43 38 5b 00 00 00 00 b7 3d 3e 89                 .C8[.....=>. 

91641.0: bgroup0(i) len=60:0019d28739e8->001db5ba738b/0800
              172.16.37.100 -> 173.194.33.17/6
ssg-> snoop filter delete
All filters removed
ssg-> clear dbuf
ssg->


Lastly, I drop my filter and clear the in memory buffer to clean up. 

Later I facilitated this with a filter between my device and the device provisioning service over at Sony to determine why my Blu-Ray player couldn't play Netflix.  Sure enough, the device makes a connection to ssm1.internet.sony.tv and does a GET for /BDP/stv/STVgetTime/ and gets an HTTP 404 response.  Using IE10's F12 Developer Tools, I captured the response header and confirmed it's an Apache host.  Beyond that reverse lookup tricks and tracert didn't tell me much.

For fun, I also tested the Slacker radio plugin, and it fails hitting a similar host, ssm.internet.sony.tv and ssm2.internet.sony.tv, but there were also some exciting UDP packets going by on 1900.

So after contacting Sony support, I was directed to call their support team as my issue couldn't be solved on chat.  I feel sorry for the CS agent I spoke with because their first reaction after demonstrating my case was to use an HDMI cable instead.  So much for the OSI model.  This really points at how DRM can affect customer experience, especially if it's something the hardware vendor tacked on to their device on top of a 3rd party service that doesn't require it for other devices.  Even though I was (and still am) a PlayStation 3 fanboy, I'm having second thoughts about Sony devices. 

I watched Netflix on Wii today.   Because it just worked.

Wednesday, May 22, 2013

Troubleshooting 101: The Power of netstat

I was reading up on ETW, and discovered the TCPView tool.  Having recently worked with the joys of ip_conntrack on Linux hosts with iptables and bad PHP frameworks leaving TCP sessions in TIME_WAIT, this was a nice find for Windows.

But in reviewing this article, its curious that Microsoft doesn't demonstrate the use of native OS tools to triage a simple port binding issue.  So here's a quick example to help you find what's listening on TCP 80.

C:\> netstat -p TCP -on | find ":80 "
  TCP    172.16.37.100:80    157.56.98.83:443       ESTABLISHED     320

This shows us a list of TCP ports with the numeric port value and the PID.

C:\> tasklist /FI "PID eq 320"
Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
w3wp.exe                       320 Services                   0     99,344 K


Pass the PID in a predicate filter to tasklist, and we find the process name.  Since it's an IIS worker process, we must go a step further to identify the Application Pool:

C:\> %windir%\system32\inetsrv\appcmd.exe list wp
WP "320" (applicationPool:WsusPool)


Another great feature of netstat is it's statistics feature, which a peer pointed out to me:

C:> netstat -s

It's left as an exercise to the reader to peruse the value that single command can have in troubleshooting down the stack before you worry about data link and the physical layer.  You've already swapped out that crossover cable for a regular patch, right?

Sunday, April 7, 2013

Now on IPv6!

On a whim, I configured CloudFlare today.  Then happily discovered, it helps my site go into the future, and not just faster, but also with IPv6.  On a side note, since they'll be handling my name servers for the purposes as a CDN, I can eliminate my name server hosting costs, saving $12/year, unless I put those $12 towards premium features as the site grows.

But wait, there's more!


Looking at their apps, I found Blitz.io, which is a sweet simple load-testing service.  After testing it out a bit, I misspelled Brazil, and found this fun fact:


"s" isn't just British anymore
Hm, those regional mappings seem very familiar.  Quick to the bat-lookup (or bat-dig, your preference):

C:\>nslookup www.blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
Name:    elb014717-1106725811.us-east-1.elb.amazonaws.com
Addresses:  54.243.112.187
          54.243.77.105
          54.243.229.217
Aliases:  www.blitz.io
          mie-8036.herokussl.com


Ha, I thought so.  It looks like they're using Heroku, who's a well know user of AWS.

That's not all, Bob.  Tell 'em what they've won!


When I first ran the query, I mistakenly specified the apex zone:

C:\>nslookup blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
Name:    blitz.io
Address:  50.31.209.229


They aren't using the ELB in their apex zone.  I happen to know that feature is an AWS specific implementation of Route 53.  So let's check name servers:

C:\>nslookup -q=ns blitz.io
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
blitz.io        nameserver = ns4.dnsimple.com
blitz.io        nameserver = ns2.dnsimple.com
blitz.io        nameserver = ns3.dnsimple.com
blitz.io        nameserver = ns1.dnsimple.com


Yup, they're using name servers elsewhere.  But why not use CloudFlare for the optimizations?  Perhaps its something that only their engineers can answer.

Tuesday, April 2, 2013

From Whence?

Ever wonder where a website's hosted?  Or who's that strange IP in your logs came from?  DNS can be more helpful than you think, if it was configured right.  So, let's have some fun, shall we?

Who's there?


In lieu of know how a site is configured, or finding a reputable site with enough foresight to configure their site in a fashion to be interesting to look at.  So who might we inspect?  How about one of the popular open-source relational database systems, Postgresql?

Staring with the basics, let's do a quick A record lookup and check the name servers for the zone:

C:\>nslookup www.postgresql.org
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
Name:   
www.mirrors.postgresql.org
Addresses:  2001:4800:7903:4::126
          2a02:c0:301:0:ffff::32
          2a02:16a8:dc51::50
          217.196.149.50
          87.238.57.232
          98.129.198.126
Aliases: 
www.postgresql.org


Ooooh, not only do we have six endpoints, it's a CNAME record with IPV6 addresses.  It's left as an exercise to the reader to play with IPv6 and discuss the merits of CNAMES, so let's look at name servers.

C:\>nslookup -query=ns postgresql.org
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
postgresql.org  nameserver = ns4.postgresql.org
postgresql.org  nameserver = ns3.postgresql.org
postgresql.org  nameserver = ns2.postgresql.org
postgresql.org  nameserver = ns1.postgresql.org


Looks like these guys are smart enough to host and manage their own name servers.  Not something I'd personally want to spend my time on (thanks to Dyn and now Route 53).

Where are they coming from?


So obviously they're behind three endpoints, so a quick WHOIS lookup should give us an idea of where they're at, right?

217.196.149.50 :  RIPE Network Coordination Centre
87.238.57.232 : RIPE Network Coordination Centre
98.129.198.126 : Rackspace Hosting RSCP-NET-4

We all know who Rackspace is and that's about all I can get there short of inside knowledge of Rackspace's netblock allocation, but who on earth is RIPE Network?  And why is this interesting?  So how can I know where those two IPs are hosted?  tracert is always a nice idea, but sometimes it yields the same results.

Six hops out of the interesting bits of my own network and we find:

  7    13 ms    13 ms    12 ms  208.178.58.89
  8   193 ms   191 ms   193 ms  64.209.102.34
  9   202 ms   201 ms   208 ms  asr01-0-0-0.dc1.conova.com [195.70.99.142]
 10   204 ms   198 ms   200 ms  217.196.158.12
 11   198 ms   197 ms   197 ms  217.196.158.38
 12   195 ms   198 ms   204 ms  zalkon.postgresql.org [217.196.149.50]


And also:

 11    80 ms    81 ms    88 ms  xe-4-3-0-0.nyk2nqp1.us.ip.tdc.net
 12   187 ms   188 ms   186 ms  static-213.50.153.37.addr.tdcsong.se
 13   191 ms   184 ms   192 ms  ge-1-0-0.cr2-ksd1.n.bitbit.net [213.50.153.38]
 14   187 ms   185 ms   192 ms  vlan-1.cs1-ksd1.n.bitbit.net [87.238.62.145]
 15   191 ms   186 ms   197 ms  bond0.fw2-ksd1.n.bitbit.net [87.238.62.151]
 16   189 ms   185 ms   187 ms  zetar.postgresql.org [87.238.57.232]


Certainly interesting and partially information.  I'm guessing bitbit.net might be one hosting provider or ISP.

How do we find out more?


Let's have even more fun, courtesy of DNS.  Reverse lookups were popularized a few weeks back by the tracert replay of the intro to Star Wars: A New Hope, but let's use this fact to our advantage.

Since we know that a reverse DNS lookup will give us the name of an IP and given that in most cases ISPs or service providers allocate chunks of RIR networks in contiguous blocks, let's try looking up a CIDR block of the respective ARPA addresses of those IPs:

C:\>nslookup -query=ptr 149.196.217.in-addr.arpa
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

149.196.217.in-addr.arpa
        primary name server = adns001.dc1.conova.com
        responsible mail addr = domainadmin.conova.com
        serial  = 2012092001
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 1814400 (21 days)
        default TTL = 86400 (1 day)


And then:

C:\>nslookup -query=ptr 57.238.87.in-addr.arpa
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

57.238.87.in-addr.arpa
        primary name server = ns-foo.linpro.net
        responsible mail addr = hostmaster.linpro.no
        serial  = 2013032700
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 2419200 (28 days)
        default TTL = 86400 (1 day)


And that's a wrap.  There's a good chance that the other two IPs are hosted by Conova and LinPro, who's TXT record has this little nugget:

C:\>nslookup -query=txt linpro.no
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
linpro.no       text =

        "never-gonna-give-you-up"
linpro.no       text =

        "google-site-verification=ChNtbQK1HLTM6oP5E3yV5SLo8NOQJmwJibARqdN0Ypo"

Monday, March 18, 2013

Why I'm a PC User

I've been a prosumer of Apple products since my college graduation and have just sold my last Mac, an early model Intel based iMac.  I had saved up for the purchase for the purpose of getting the 256MB ATI Radeon X1600 and a 20" display.  A huge upgrade from my vastly underpowered 1.4Ghz AMD Athlon with a nVidia GeForce 4600 8x AGP.

It was a the Golden Age of World of Warcraft, and with a peer running Ventrilo server, I was enjoying maxed out graphics (AAx4 aside) at 1680x1050.  Next to my old 12" PowerBook G4 it was greased lightning.

I didn't leave Apple, Apple left me


When I started using Macs, the G5 had just come out and after taking my Java and Linux courses in college, I was apt to get my hands on something with UNIX frameworks under the hood.  Everything was Fedora vs Mandriva back then, and SuSE was coming in strong with the YaST experience, but I choose Gentoo.  There was a fork of the portage tool that ran under Mac OS X and it was a dream for playing with open source software.

Indie developers like Rogue Amoeba, Flight Meat, and Delicious Monster were making waves, years ahead of the App Store market.  I remember going to the MacWorld Expo just to have the chance to meet the guys behind the software.  Never mind that I never wrote a lick of code myself.

Wither OS X


Fast forward a few years.  Introduce the iPhone.  iTunes is still making a killing.  And now merge the two platforms together.  And remove mouse buttons from track pads.  Flip scrolling directions.

Don't get me wrong, some of the apps being developed for the mobile platform are amazing.  And conversely, some of my favorite apps are still awesome.

And never mind that I've been supporting Microsoft infrastructure applications for the past eight years.  I've done my fair share of USB-serial cables passed-thru to VMware Fusion to reprogram APC UPS equipment in HyperTerminal in third-world countries, but it's time to move on.

Going Blue, in the Cloud


The best Microsoft experience is on the Microsoft platform.  Now that I'm a quintuplet MCITP and dual MCSA, I've clearly chosen my platform of choice.  But given that most of that isn't on bare metal anymore, I do just about everything in The Cloud.

The only problem keeping me from switching to something like a Surface RT for the majority of my home life is storage (where do you put 40GB of music and 20GB of photos?) and x86 support.  But I'm sure that'll come along shortly.

To be honest


Once the warchest is built up, I'll still probably get a replacement Mac at some point.  But it'll have to be the right price and the right hardware.  I've got my budget to stick to after all, and I can't stand Intel HD Graphics chips.  I've experienced too many strange glitches and performance issues.

Saturday, January 12, 2013

Troubleshooting 101: Resolution (Part 1 of 5)

Resolution is the first stop when it comes to troubleshooting.  If a user, application, or protocol can't determine the address of it's destination by name, then we'll never take the first step on a journey of hopefully less than 64 hops.

This troubleshooting step typically revolves around DNS, but could also refer to NetBIOS, or even at the application layer.  Since you're already on the Internet, let's discuss DNS issues in particular.

Tools of the Trade

The most common tools used for testing DNS are dig and nslookup.  Hailing from a Microsoft background, I usually find myself using nslookup:
C:\>nslookup
Default Server:  google-public-dns-b.google.com
Address:  8.8.4.4
> www.eveningread.com
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
Name:    ghs.l.google.com
Addresses:  2607:f8b0:400e:c01::79
          173.194.79.121
Aliases: 
www.eveningread.com
          ghs.google.com
From this short example, you can see several things.  First, my laptop is configured to query Google's Public DNS servers.  I selected their secondary server as my primary name server based on performance testing with namebench. This tool identified the server as the fasted for my geographic location.

Secondly, you'll see I requested this site's fully qualified domain name for my query which defaults to searching for A records.  This returns ghs.l.google.com, which in a second, automated lookup, returns the actual public IP address for this site.

Over-riding this query is a farily simple task of appending a name server to a single line nslookup query:
C:\> nslookup www.eveningread.com 8.8.8.8
Or interactively setting it in the session:
C:\>nslookup
Default Server:  google-public-dns-b.google.com
Address:  8.8.4.4


> server 8.8.8.8
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8
 
Searching for different record types is similar:
C:\> nslookup -query-type=mx www.eveningread.com
Or interactively setting it in the session:
C:\>nslookup
Default Server: google-public-dns-b.google.com
Address: 8.8.4.4


> set q=mx
> eveningread.com
Server:  google-public-dns-b.google.com
Address:  8.8.4.4

Non-authoritative answer:
eveningread.com MX preference = 10, mail exchanger = 91b72d3b13b94f9d8f11a10f3e76c6.pamx1.hotmail.com
Here we see the results of an MX record lookup.  (Don't try to email me here, this is configured for demonstration purposes only.  Email is a whole different topic.)

To perform these queries to a name server, DNS clients typically initiate a connection to UDP port 53 of a name server, though some uses of TCP port 53 have been seen. 

It's left as an exercise to the reader to leverage packet capture utilities to watch DNS traffic on their client workstation or DNS server.

Example Use Case: Active Directory Specific DNS Records

One of the most common issues I run into is workstations failing to authenticate with Active Directory (unless they're unplugged from the network, in which case they use cached credentials), or computers that cannot outright join a domain.

Active Directory Domain Services uses some unique records to locate a Domain Controller via the Netlogon process.  Details on these records are outlined here, in this ancient Windows Server 2000 documentation.

When we can't query Active Directory, I typically drop to a command prompt and run the following query (for the corp.contoso.com Active Directory forest, in this example):
C:\>nslookup -query-type=srv _ldap._tcp.dc._msdcs.corp.contoso.com
Default Server: dc1.corp.contoso.com
Address: 10.0.0.14
_ldap._tcp.dc._msdcs.corp.contoso.com SRV service location:
 priority = 0
 weight = 0
 port = 389
 svr hostname = dc1.corp.contoso.com
_ldap._tcp.dc._msdcs.corp.contoso.com SRV service location:
 priority = 0
 weight = 0
 port = 389
 svr hostname = dc2.corp.contoso.com
dc1.corp.contoso.com internet address = 10.0.0.14
dc2.corp.contoso.com internet address = 10.0.0.15
 
 Anything other than the above (provided those are the correct IP addresses), and I've got a resolution problem that's keeping Netlogon from finding a Domain Controller.  Usually updating the DNS settings to query the appropriate name server does the trick at this point.

Caveat Emptor

Another issue I commonly see pertains to client side DNS caching.  Let's say you connect to a highly available database server, which fails over by updating a CNAME for it's endpoint.  One moment your clients are connecting to 10.0.0.20, the next database.corp.contoso.com points to 10.0.0.21.  But if your client application didn't query database.corp.contoso.com a second time, it'll be using the cached 10.0.0.20 IP until TTL expires, or worse never!

Wrap Up

So what's so important about resolution from a troubleshooting perspective?  Well, if it's not done right, you're talking to no one, a server that won't answer for your application, or stuck with all your queries running to a malicious website!  Even if you don't know when or why resolution is a part of your problem, knowing how to test it and remediate it is key to narrowing down your list of issues.