Hello, Kossacks!
Everyday Magic is a new diary series that seeks to explain many of the computing concepts we use on a regular basis in a way that non-techies can understand, and introduce them to the lingo we techies use in the process so that people know what tech types are talking about (and how technology changes in the news will actually affect you). I'm hoping to make it a regular series, both as an educational service to the Daily Kos community, and to improve my own knowledge of topics I'll be presenting on.
So before I dive right into our first topic, I'll answer a few questions I expect to get up front, then we'll get into our first Everyday Magic article.
Why is the new series called Everyday Magic when it's about computing and technology?
The series name (and my username) are derived from a quote from Arthur C. Clarke that is known as Clarke's Third Law: "Any sufficiently advanced technology is indistinguishable from magic." It's also based off of a lingo term in tech circles called automagically, which refers to an automated process that works, but nobody's dug deep enough to find out exactly why it works. I'm hoping to reduce the amount of computing topics that seem like magic to everyone!
What topics will you be covering?
My professional background is systems administration and the development of software that automates systems administration, so there are the topics I have content ready to go for. However, if you have a request, please drop it in the comments as a reply to the tip jar replacement, and I'll add it to the list of topics to get to.
Why are you doing this?
A few reasons. One, it's a way I can give back to the community and the 'Net as a whole. I've made a career off of computing, and I make a fine living off of it. The least I can do is give knowledge back to the culture that spawned me.
Secondly, I also believe that one's mastery of a topic is directly reflected by how wide of an audience they can explain the topic to, and that the inability to explain a concept to someone reflects on a lack of subject mastery on the part of the explainer. If I can't explain something, it lets me know where my own knowledge is lacking.
Finally, I'm on the autism spectrum, in the range formerly referred to as Asperger's Syndrome. I'm blessed with enough brain processing power to learn what comes naturally to many insofar as social interaction is concerned, but I need practice to continue to get better at social interaction and communication.
Who are you, and why/how are you qualified to teach to on this topic?
I'm a 32 year old tech worker that lives in the San Francisco Bay Area. I've been doing this stuff professionally for 15 years for companies you've heard of and companies you probably haven't, and I've been using computers since before I was potty trained. I'm also Red Hat Certified Engineer (for Red Hat Enterprise Linux 5).
However, like with any other information -- go verify it yourself! And if you find that I'm not correct about something, please correct me! I'm not a fan of being wrong. Do note that sometimes there will be a bit of hand-waving (jargon for skipping over details) going on to stop from making these any longer than they already are.
What's with the bolded words or phrases?
Bolded words or phrases indicate key terms and jargon that are used when describing topics. It's also stuff you may or may not hear on a daily basis, so now when you hear these terms, you know what the geek you're talking to means!
Well, with that out of the way, let's dive into our first topic: How your computer knows where DailyKos and other sites are on the Internet when you tell it to come here!
IP Addresses and the Domain Name System
www.dailykos.com, as far as a computer is concerned, means nothing on its own. www.dailykos.com is there for humans to remember, because it's easier to remember something that like than the strings of numbers that represent an IP address, which is the method computers use to find each other (at a high level, at least).
The Domain Name System (DNS) is what keeps track of the mappings of those human-readable names and the IP addresses the computer cares about. Nearly Internet service providers (ISPs) provide DNS servers for their customers to use as part of delivering Internet service. There are also public ones out there as well (Google, OpenDNS, etc), and paid services for consumers and businesses that provide additional services tacked on to standard DNS service.
So, how does a DNS server work?
A DNS server works by telling other DNS servers about the domains it's responsible for if they ask, and by asking the DNS servers for records about domains they control if a user tries to ask the DNS server about an address it doesn't know about yet.
The main DNS record that determines what IP address a domain like www.dailykos.com maps to is called an address record, commonly referred to as an A record. This type of record directly maps a name like www.dailykos.com to an IP address or set of IP addresses. I can use a program called dig to ask my DNS server what that mapping is:
$ dig a www.dailykos.com +short
173.231.134.22
173.231.134.18
There's also another type of record called a
canonical name record, commonly called a
CNAME record that points to another DNS address. It basically acts as a forwarder. If I'm the owner of the blah.com domain, and I want www.blah.com and website.blah.com to go to the same IP address, I can either make an A record for both of those addresses and point them at the same IP address, or I can make an A record for www.blah.com that points to the IP address, and then make a CNAME record for website.blah.com that tells a DNS server that website.blah.com points to www.blah.com. This is useful because if my IP address changes for some reason, I only need to update the A record for www.blah.com, and website.blah.com changes right along with it! Saves a lot of work.
There's a few other records in DNS, like those that say what server handles email for a domain, or ones that describe other properties of the domain, but for our purposes, these are the main ones.
Well, except for two more...
How does the DNS server I use know which DNS server to ask where www.dailykos.com is if it doesn't already know?
Each top level domain (or TLD), like .com, .net, .mil, .uk, etc., has a server (or set of servers) that is considered authoritative, or the source of truth, for that TLD. These servers are kept track of by what's known as the root DNS servers, which manage the entirety of the domain name service.
A DNS record called a start of authority record (SOA record) lets other DNS servers know which servers should be trusted to provide the correct information for a DNS zone, such as com, net, org, or a subset of those, like google.com, root-servers.net, and on up the portions of a DNS address. However, in most cases, the actual answer is provided by other servers identified by name server records (NS records) that handle the DNS requests for those domain.
So, if the DNS server that I'm using to resolve a DNS address, which is the term for figuring out what IP address a DNS address points to, like say, www.dailykos.com hasn't looked it up before, the process to find it out would go like this...if computers could talk, anyway:
Me: Take me to www.dailykos.com, dude. I've got diaries to comment on.
This is what you're doing when you type www.dailykos.com in your browser. Well, that does more than that, for today's purposes, that's all we care about.
My computer: I have no idea where that is. Hey, DNS server that this dude told me to use when he set me up to use the Internet, where's www.dailykos.com?
Unless you've filled out a hosts file on your computer that tells it explicitly what IP address it should go to when you give it www.dailykos.com, it needs to find out the IP address where those servers live. So it's going to contact the DNS server assigned to you by your ISP (or that you configured yourself) and ask for the IP address(es) where www.dailykos.com can be found.
DNS server: Hell if I know. I'm new around here. But, I know that I can find a root server at a.root-servers.net, which is what humans call those servers over at 198.41.0.4. Root server, who can I trust to give me the right info about .com domain names?
In order to reduce the amount of traffic other DNS servers have to handle, once they've looked a place up, they'll cache or store that record for as long as the time-to-live (or TTL) setting on that record is. The domain administrator sets that record, so other DNS servers know how often to check back in to see if the IP address changed.
But, we have a new DNS server here that has nothing cached yet. And to show how many lookups caching saves, we'll show a few of those interactions --- even here, I'm skipping steps in the interest of not wanting to make this a 3 hour long read.
When a DNS server is configured, the administrator lets it know which servers are the root servers. That's all it needs to start its search. So it's going to ask the root server for an SOA record for .com, so it can ask that which servers are authoritative for dailykos.com, so it can ask that server where www.dailykos.com is.
Root server: There's a server over at a.gtld-servers.net that can tell you. You can store that for 900 seconds before checking back in with me.
The root server here is responding with the SOA record for .com, and the TTL for that record.
DNS server: Yeah, about that. I'm kinda new here...
Doesn't help our DNS server that much, since it has no idea what IP address that maps to. So it's going to ask for the A record for that server so it knows where to go.
Root server: You are new, aren't you. Freshly hatched from the server farm, eh? I remember when I was your age... oh, right. Anyway, that dude's over at 192.5.6.30 for at least 74,440 more seconds. Ask him which of his buddies can tell you where dailykos.com is at. Talk to you later, dude.
The root server responds with the A record for the authoritative server, along with the TTL.
DNS server: Hey, server in charge of .com, which of your buddies can tell me where dailykos.com is at?
.com's authoritative server: Those dudes over there: A2, C2, D2, E2, F2, G2, H2, and L2.gtld-servers.net. My list tells me that A2 is up next to answer this sort of thing since he's first on this list, and he's right next door at 192.5.6.31.
Our brave DNS server sent off an NS record request to a.gtld-servers.net, which responded with that list of servers. When there's multiple servers like that in charge of providing records, the list will rotate (this is known as round-robin DNS load-balancing) to try to spread the load out. The first server provided in the list is where the next question goes.
DNS server: Thanks, boss. Hey A2.gtld-servers.net, can you tell me your counterpart over at dailykos.com?
A2@GTLD: Yeah. I've got that around here somewhere. It's ns1.p30.dynect.net. He's got 3 siblings named ns2, ns3, and ns4 if ns1 is busy.
This would be the response to an NS record request for dailykos.com.
DNS server: Where's that, and out of curiosity, who's the boss over there?
Here we're requesting the A record for ns1.p30.dynect.net, and for teaching purposes, the SOA record.
A2@GTLD: NS1@Dyn's over at 208.78.70.30 for at least 86400 more seconds. Oh, and the boss is the same guy. Pulling double duty. Just ask for the NS servers though, let that root server worry about the authoritative record from now on -- just ask for the NS records.
A2 saves us some time and helps us out by giving us the A record it has cached. The SOA record for dailykos.com is also ns1.p30.dynect.net, but really, to find things, all we care about is the NS record unless we have reason to believe a DNS hijacking is happening and someone's intentionally misdirecting us.
DNS server: Finally, almost done. Hey, NS1 over at Dynect, I got told you know where www.dailykos.com is. Mind helping a classy chassis like me out?
Our intrepid explorer is now asking for the A record for www.dailykos.com from one of the NS record-listed servers for the dailykos.com domain.
NS1@Dyn: Yeah, The Great Orange Satan's over at 173.231.134.18 and 172.231.134.22 for at least 1800 seconds. 172.231.134.18's ready for you.
This is the A record response for www.dailykos.com with TTL.
DNS Server: Thanks! Hey, Technomancer's computer, go talk to 172.231.134.18 so you can show that human www.dailykos.com.
My computer: Thanks! Hey Technomancer, here's orange!
Me: Yay Great Orange Satan!
There are, of course, steps the browser needs to take and servers the browser need to talk to for this site to actually display, but that's another topic for another Everyday Magic. But that, my friends, is a high-level overview of how DNS resolution can be done recursively, or by starting from the root server on up!
Thanks for reading! Feedback is always appreciated.
-- The Technomancer
Want to request a topic for a future Everyday Magic? Drop your requests as responses to the first comment. Feel free to ask questions and otherwise use this diary as a computer/internet/technology related open thread as well!