Saturday
All week the Internet has been crisp uploading my ftp fast; but today it is down to a crawl. Just recording it.
===
It's even worse at 10:00 pm. Most web sites take too long to connect. Something is going on.
===
It is now impossible to do serious research on the Web. Many sites time out while trying to connect. It's pretty annoying.
===
It's worse. It is now nearly impossible even to get to Google.
===Sunday
There is definitely something wrong with the Internet. It is bursty: for a few seconds things happen fast, then it all slows to a crawl again. It can take a long time to open Google or other pages that are usually nearly instant. It has been this way all weekend.
===
Jerry Pournelle, this weekend.
Jerry has been complaining about the internet for quite a while now, and he's not alone.
The reason I quote Mr. Pournelle isn't to make you think I'm important; I don't think we've never met. It's just that he's writing publicly, and most of the information I have is semi-private. I don't figure they'd mind sharing the information, but I just don't know.
In "The Sting", Johnny Hooker is contemptuous of Chicago detective William Snyder. "Aw Christ, it doesn't make no difference now. If Snyder knows about it so does everybody else. He never gets anything first..." That's pretty much how the IT profession views Homeland Security.
It wasn't that they didn't give Homeland Security a chance, but when a government agency chews up and spits out two good men, one after another, ignoring their skill sets in favor of those whose skills are located below the neck, it's pretty obviously a case of clue deficiency syndrome. Having a pretty good time in the 100-year-dash doesn't mean you can run down cyber-criminals. All things being equal, I'd bet on the paraplegic, rather than the former jock.
But they're warning that the terrorists will be attacking the internet soon. Army Brigadier General John Custer, head of intelligence responsible for Iraq and Afghanistan, says Al Queda is using the internet to recruit members, learn where to attack, coordinate activities, and manage their affairs. It sure seems like a no-brainer, that by developing expertise in how the internet works, they are becoming familiar with the weak points.
Erik at ProductWiki.com observed this afternoon that "the Internet went down yesterday, and nobody noticed" He blames it on "GoDaddy" and makes several observations:
- a huge portion of the Internet was down yesterday because of a single point of failure
- a ton of people submitted the story to digg, and, surprisingly, it DIDN'T make it to the front page
- GoDaddy covered it up very well; they only once, on one page, mentioned they were "having technical difficulties" without any information as to why, or when they would be coming back up
- No mention of it on Bob Parson's blog
- This event made DST 2007 more significant than Y2K (which is kinda pathetic)
The problem is, it didn't happen that way.
We'd like to point out that Bob Parson's GoDaddy is a competitor. They're one of the biggest registrars out there, and one of the biggest web hosts. If the point were merely that dealing with us is better, because we encourage everybody to phone the guy in charge (me) any hour of the day, every day of the year, for almost any reason. (If you wake me to sell me replacement windows, or to leave a message for Marge, I'm likely to be a little testy.)
But it wasn't GoDaddy's fault. When you register an ORG domain with them, they give Afilias, the ORG registry, the information. For a COM domain, it's not Afilias, it's Network Solutions. Each extension has its own registry, and GoDaddy doesn't run ANY of them. GoDaddy could go down 5 minutes after you buy a domain name, and remain offline for six months, and your domain name would work perfectly fine. Their site is only necessary when you want to buy or renew a domain registration, or change your WHOIS information or name servers.
Now, they're also a big hosting company, but there aren't very many sites on any one server. The Howard Street Tunnel Fire in 2001 caused severe disruptions to big chunks of the internet for weeks of months.
It seems likely, Erik writes, "that this was connected to the new DST time shift that had happened the night before. GoDaddy said they were ready, but it seems not." Or perhaps they were. Their admission that they were "having technical difficulties" isn't a concession that they were the cause of the difficulties.
We've been having our own technical difficulties of late. Since May, I've been keeping track of brute force attacks on our servers. There've been 435,257 of them.
Not everybody who tries to log in, and fails, is attacking the server. Sometimes, I wash my fingers in the morning, and can't do a thing with them all day. And sometimes, I wash my head, too. One of our customers called this afternoon; he was sure he remembered his password (and he was right, although I'm not sure he was spelling it right.) However, his domain name is a long one (think "FrenchFumbleFingers.com") and his username is a short one (think "french", or "fumble" or "fingers".)
He had tried to log in about twenty times over a two-hour period. Brute force attacks typically jam in hundreds of attempts in a period of seconds. The username will also be something no legitimate user would ever try, such as "root", or "admin". They're pretty easy to identify as break-in attempts.
The chart above, however, will tell you that they're increasing in number. For months, they rarely reached 2000 attacks in a day. Recently, there was one day with 14,661 attacks.
The amount of spam increased dramatically in November and December, and it really jammed up a lot of server owners. Microsoft was really ailing, as was Yahoo, and even though Google rarely has trouble, I heard complaints about their Gmail. We didn't have much fun, either.
Mail hosting is "expensive" in terms of server resources. If a user comes to us for a web page, he tells us what his IP is, and stays connected while we pipeline multiple files to him. It's like dealing cards off the top of the deak. Outgoing mail, though, requires that the server do a DNS lookup, to find out the IP of the other domain's mail server. Once the mail is gone, we have to delete it as well. And that's for ONE piece of outgoing mail, not a whole bunch of them.
Incoming mail is much worse. When the server gets contacted, the distant server will say, "We have mail for support@domania.org". The server (mabel.observerco.com, in this case), has to check to see if it can accept mail for domania.org. When the answer is yes, it accepts the mail, then has to figure out what to do with it. First, it is scanned for viruses. Then the server looks to see what happens to mail addressed to "support". There are a number of rules to be applied, such as "if it's from X, Y, or Z, delete it at once" or "if the subject of the mail mentions Cialis, delete it". If it is deliverable, of course, it has to go to support - except support isn't a mailbo. It's an alias. It goes in another mailbox.
And when it goes into the mailbox, the server has to find a space on the hard drive to store the email, then mark the space used in the disk directory, then put an entry into the file system directory with the name of the file, and finally write the file to the disk.
If that sounds involved to you, well, that's because it is. If someone were trying to slip something past the defenses of a server, it'd be smart to do it while the server is deluged with spam. I haven't heard anyone else say that, but that thought surely hasn't occurred only to me.
The weekend is over, but the internet hasn't gotten healthy again. North America shows a 67ms ping - fairly fast - but 16% packet loss. South America has only a 7% packet loss, but their pings are running 168ms. Australia is even more pronounced, with a 201 ms ping and -zero- packet loss, but Asia is running a 380ms ping and 19% packet loss. Overall, the global index is only a 78 - fair, not good.
You don't need to be me to see those numbers, or even a Jerry Pournelle. They're from the Internet Traffic Report site, and while the numbers may seem confusing at first, you'll soon figure out their maps and charts, and tables, and figure out whether it's broad sections of the internet, or just your local ISP, that's having the problem.
It's been a long day and I'm exhausted. I'd like to go to bed, but that's just asking for trouble....