Jim Goings

I host two sites on Media Temple; this one and my Dungeons and Dragons community site, Dragon Avenue. I have been using them for a few months now and I can say without hesitation two things:

1. Media Temple has some of the slickest features out there.
2. My sites either go down or take over ten seconds to load frequently. By frequently, I mean daily.

I have had a couple of phone calls with their tech support folks, including one manager, who seemed sympathetic but unable to offer any advice on how to address the issue. At first, (mt) seemed to sweep the issue under the rug. Communication was infrequent and it took days (no kidding) to even get a response from tech support (as reported by not just me, but my friends who use them as well).

Now they seem to have taken to apologizing for the outages each time they occur.I recently commented on one of their outage reports, but I don’t expect I’ll hear anything back… at least anything that is tangible. I wonder if they’ll even approve my comments to be revealed to the public. Given that none of their recent blogs have comments, I suspect not.

They still aren’t talking about the slowness at all though. As long as their servers respond to web requests within a minute or so, then all is right with the world in their eyes. Yes, that was a bit of hyperbole, but my points are valid.

I recently opened a ticket with them asking about the outages and even asked if switching to a DV plan would eliminate them:

Support response: 2008/04/01 16:37
This depends on the cause of the issue. I have found a good number of causes for timeouts having to do with external resources being pulled. Now if it is a routing issue, moving to a (dv) would help . One thing that would help us in troubleshooting your issue would be to provide us with trace route results to your server.

Just let us know which path you will want to go.

I replied:

Site is timing out currently at 11:10PM on 4/8/2008. Here is a tracert as requested. As you can see, the issue isn’t network related (at least on the Internet side of things). I should note that this isn’t rare. The site times out often, sometimes many times per day.

I suspect that your servers are over loaded, but that’s just my speculation. Either way, I need to have my site reliably available. Please help.

tracert www.dragonavenue.com

Tracing route to www.dragonavenue.com [64.13.232.65]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms FIOS_Router.home [192.168.1.1]
2 8 ms 7 ms 6 ms L100.VFTTP-02.PTLDOR.verizon-gni.net [xxx.xxx.xxx.xxx]
3 8 ms 6 ms 7 ms P2-2.LCR-02.PTLDOR.verizon-gni.net [xxx.xxx.xxx.xxx]
4 26 ms 24 ms 24 ms so-6-0-0-0.PEER-RTR1.SJC80.verizon-gni.net [130.81.17.133]
5 24 ms 24 ms 24 ms 130.81.15.26
6 81 ms 32 ms 31 ms avanz.ge3-1.2.cr01.lax03.pccwbtn.net [63.218.91.30]
7 32 ms 32 ms 32 ms cr01-2-2.lax4.net2ez.com [64.93.64.74]
8 33 ms 32 ms 32 ms cust-77.lax4.net2ez.com [64.93.75.14]
9 33 ms 32 ms 31 ms 72.10.63.210
10 33 ms 32 ms 31 ms acmkoiegkq.gs02.gridserver.com [64.13.232.65]

Trace complete.

To ensure that they had as much information as possible, I even provided them with 3rd party verification:

Further verification from monitus that the issue isn’t related to my home network:

** PROBLEM alert - www.dragonavenue.com HTTP PROBLEM **
PROBLEM Alert
Service: HTTP
URL: http://www.dragonavenue.com
Tag: Default
State: CRITICAL
Date/Time: 2008-04-09 06:14:47 GMT

Additional Information: Connection refused.

Get your own monitoring at http://monitis.com
Best regards, monitis team.

Here is their response.

The brief outage you have experienced tonight was due to an outstanding issue we have with our (gs) hardware which you can read more about here:

http://weblog.mediatemple.net/weblog/category/system-incidents/gs-grid-service-intermittent-service-availability/

The trace route that you posted is exceptional, your problem is not network related. Contrary to what the last technician said, moving to a (dv) hosting plan you would not have these problems anymore. The issues are isolated to our (gs) hosting plan only. If you wish to try out our (dv) package, I can set one up for you with 1 free month so you can check it out. Let me know if you would like to do that and I’ll set that up for you.

Lame. Just lame. No acknowledgment of the fact that the outages occur daily for minutes at a time or the general slowness that is epidemically affecting their two “grids”.

Here is a screenshot of just one page full of alerts from Monitus, a third party monitoring service. Check the dates on the right column. Sad.

Switching hosts is a PITA. In fact, I tried to switch today and ran into some issues around the Apache mod-rewrite rules that make Dragon Avenue work correctly. As such, I bailed out until I can get some help figuring out how to change the rules appropriately. In addition, there is a rumored compatibility issue with Apache 2.x and Expression Engine. I couldn’t find the info on that, but the host I attempted to move to indicated that there was a problem and the fix was a code change.

As I said, switching hosts is a PITA. I can only hope that Media Temple will really fix their issues or that my patience will end and I will be forced to tackle the burden of switching. Sigh.

UPDATE 6/24/2008 - Media Temple claims to have fixed the major issue causing downtime. And it seems they may be right. My site has been up for 3 weeks now without fail. That may not seem like much, but it’s a huge win over the daily outages I was experiencing. I’ll update this again in a few weeks.

UPDATE 7/13/2008 - The uptime is holding and performance has been good.  There was only one outage since the last update and it was very brief.

However, I am still very disappointed that Media Temple’s customer service during the poor service.  Their response to my issues were slow and pointless.  I expect accountability and help for my money.  I will evaluate their customer service again when the need to contact them arises.

30 years after making mixtapes for your friends was all the rage, muxtape enters the scene with the new simple, digitial way of doing the same thing.

My muxtape has some tunes that I think are interesting and worth a listen.

I’ve been an avid reader of the Sam Lawrence vs Lawrence Liu discussions on social software. Sam is the Chief Marketing Officer at my present employer, Jive Software. Lawrence is the Senior Technical Product Manager for SharePoint technologies at Microsoft. I thought I would take the time to post my thoughts after using SharePoint for several years to aid in the debate.

Getting users to understand and adopt social software can be hard in general, but SharePoint has so many options and options within options that users are literally scared of using it. I saw the glazed-over stare of users time and time again when they attempted to use SharePoint. Generally, users would have a specific idea in mind and would attempt to click around hoping the answer would appear. This resulted in one of two things: they would give up or the very persistent would ask IT to walk them through it.

For example, one of the managers at my previous employer wanted to “have a web page where sales folks could find and share information about partners”. He tried to implement some things on his own and by the time he called me for help, he had repeat a cycle of “I don’t get it” to “a ha!” to “wait, I don’t get it” several times. Should he use a blog, wiki, document library, web page widget, or perhaps a custom data sheet?

The good news is that he already understood how blogs, wikis and document libraries worked. The custom data sheet blew his head clean off with complexity however. He looked into the other options while I pointed out certain aspects of each to him. He was indecisive… which one of these best fit what he was trying to do? The answer was not clear has each option had merit, but all weren’t quite what he wanted and didn’t seem collaborative enough.

He was dead right. He had 50 choices about how to store the information, but they were all very similar to each other and none of them offered real collaboration. Sure, he could enable version control and edit partner information with sales that way, but what he really wanted was to collaborate about the partners and take the collected information and share it with everyone. The bottom line is that SharePoint just can’t do this in a seamless way.

All of the parts of SharePoint exist within their own bubbles so-to-speak. You can have a discussion, but it has nothing to do with your wiki. You can edit a document with 5 people, but you can’t really collaborate that way - who made the last edit? More importantly, why did they make the last edit?

The confusion around how to fit SharePoint into the many business needs became an epidemic. SharePoint literally became the highest source of helpdesk requests in the company. The majority of these requests where of the “how do I” nature. People weren’t getting it.

When we migrated from SharePoint 2003 to 2007, the primary reason was in the hopes that usability improvements would help users adopt it’s use. To ensure maximum success, we created many documents, send emails with tips, and moved all IT and HR information to SharePoint to “force” users to use it regularly, if only for that information. The hope was that if they could get used to using it for everyday stuff like benefits information, w-2’s, IT FAQs, etc., they would start to feel comfortable enough to use it for their own needs as well. I even started a regular blog with my IT peers that was regularly read by most of the company.

It didn’t work. People were reading stuff on SharePoint, but they weren’t collaborating at all. I resorted to the ultimate in trickery… pizza. I sent out a lunch meeting invite to the entire company. “Come and eat pizza and I will show you the magic of SharePoint!” was how the subject line read.

85 people showed up to watch my demonstration while they ate delicious Pizza Schmizza. I specifically presented real world uses of SharePoint that applied to the company. I had a very polite, attentive, and engaged audience. They asked many questions and most stayed an extra 30 minutes so they could get their questions answered. It was clear that they really wanted to use it! They were hungry to collaborate! My presentation ended in heartfelt thanks and applause.

I had a gathering of about 15 people (mostly department managers) that hung around to chat with me about their specific needs. I answered as many questions as I could and offered to personally give everyone 2 hours of my time to personally help them setup SharePoint to meet their needs. In other words, all they had to do was share their vision with me and I would do all of the setup for them!

The next month was very busy for me, needless to say. Things were looking up as I was setting up dozens of spaces and helping people leverage the system.

You know what we had in the end? A freakin’ glorified file server.

We had thousands of word and excel documents in many different SharePoint sites. We also had a few folks who converted their shared excel docs into custom data sets within SharePoint. Excel in a web browser. Cool, but not exactly the social storm I was hoping for.

• Discussions? Zero.
• Team blogs? One (just IT).
• Personal blogs? Zero.
• Personal sites setup? About a 12 out of 325+ employees, mostly IT.

After about a year after the migration to 2007, all we had was files stored inconveniently in a SQL database. Sure, they were easier to see, sort, and use in many ways. But damnit, people were supposed to be collaborating. Where were the team interactions? Where were the people working together within SharePoint?

I didn’t ever see collaborate happen with SharePoint and helpdesk requests kept coming… people just couldn’t get passed some sort of barrier with it.

When I came to Jive, my eyes were opened on what collaboration is. Employees were really working together and producing results all within Clearspace. Before the day was out, I had blogged, discussed several topics with many different departments, and created several documents… all before I had spent 8 hours at my new work-home.

How the hell was this possible? What’s the major driver of success here? It was easy to use. Unlike SharePoint, clicking around and experimenting for results yields success.

SharePoint is a Mack truck. Clearspace is a jet-powered party bus with leather seats. Sure, SharePoint will get your stuff there. Clearspace will get *you* there.

Sometime in the near future, I’ll post about the IT administration side of SharePoint in more detail. In fact, I’ve got a title already: “SharePoint: Not the IT Answer”

Blogger treated me well for many years, but I finally made the call to switch to something a little more powerful. In order to use the latest features of Blogger, you have to host your site on their blogspot service which I don’t want to do. I prefer the freedom of using my own hosted site for whatever I please.

Wordpress imported all 4 years of my blog posts in about 60 seconds. How awesome is that? While I was at it, I updated the site theme. I greatly prefer this theme to my old one. What do you think?

I finally switched from Comcast to Verizon. I did a lot of research on which service was better and why and found the details in most reviews to be lacking. As such, I thought I would take the time to post what I’ve learned along with some real world comparisons and metrics.

Installation
Installation took about 7 hours total. It would have taken less time, but I apparently had a couple of bad barrel connectors that wouldn’t pass some of the technicians tests. The tech was very thorough, did a very neat and tidy install. he also picked up his trash which was very courteous.

Here’s some pics of the ONT box on the side of the house.

The ONT box gets power from a Verizon UPS located inside the closet in my office where all of my networking terminates.

The coax on the left side of the above pic goes straight into the FIOS router. The tech said that it’s possible to run Ethernet straight from the ONT, but he’s had mixed luck with it working that way. He said about half of the time it just doesn’t cooperate, but that the coax (moca) always works. Here’s a pic of the router.

FIOS vs Comcast Internet
The router is made by ActionTec and has 4 10/100 Mbps ports. I was disappointed that it didn’t support gigabit on the switch side, but oh well. The configuration of the router is done through a nice web GUI. I was using DD-WRT on a Linksys and was concerned this would be a downgrade. However, the processor is significantly faster than the linksys.

Note the center column that shows all connected devices on the network. This is where things get interesting. The coax leading to the Router is not just the Internet feed, but it also connects your TV STBs (Set Top Boxes) to your network. This is how the STBs get program guides and other data services. Pretty neat. Note that I it shows that I have 3. The first two are actually ONE HD-DVR, but apparently each tuner makes it’s own network connection. The other STB is my kids TV.

There are many tools out there to test your speed capabilities. I prefer speedtest.net as it’s free, pretty, and easy to use. Check out the Comcast and Verizon speed test results:

The above tests were done to Olympia, WA which always showed the highest throughput for Comcast from my home in the Portland area. However, FIOS tests much higher to the Portland location (which is what you would expect). I can’t explain the difference other than to assume that Comcast has a peering location in Olympia and FIOS has a closer one in Portland somewhere. Just a guess though.

It is important to note that Comcast cheats like crazy on the speedtest reports, regardless of which one you use. They even have a marketing term for their cheat… SpeedBoost. Basically, Comcast will give you excellent bandwidth (in my case, 20Mbps) for the first 10 seconds or so of any download. After that 10 seconds of bliss, however, you are knocked back down to a more modest 6-8Mbps. This makes Comcast look great on speed and bandwidth reports, but has little bearing on what you can get sustained. Sure, this will make web pages load quicker, but won’t help you with any data transfer.

To illustrate my point, I downloaded Elephant’s Dream (an open source movie) via NNTP using Comcast and took a snapshot of the network throttling behavior.

As the breaks in between each set of data is when I manually stopped the transfer. As each transfer began, the throughput spiked to about 20Mbps or so, but quickly settled on a much lower sustained rate. I stopped and started the transfer a couple of times for illustration as the graph shows. Now check out the same exact download from the same NNTP servers (Giganews) on FIOS.

As you can see, I was able to get 1830KBps (roughly 14Mbps) SUSTAINED. No spikes here folks, just real throughput without any tricks.

Some things you should know. All computers connected to the FIOS network require some tweaking to get full speed. Thankfully, Verizon has a FIOS speed optimizer tool you can use to obtain the best performance that tweaks your OS for you. It doesn’t appear to leave any malware or adware behind… just modifies the registry to optimize the TCP settings. Vista users will need to grab the KB939006 patch from Microsoft first.

FIOS vs Comcast TV
My initial impressions are very positive.

The HD-DVR has revealed little to no menu lag like the Comcast DVR does. Further, the menus are much more elegant than the Comcast ones. The STBs for Comcast and FIOS are made by Motorola, so I can only assume that the firmware developers on the FIOS side are much better at their jobs. The guide is easy to use. You can scroll rapidly through channels by holding the up or down arrow. With Comcast, I had to hit page down or page up. FIOS scrolling is fast, fast, fast. The channel logos appear next to each channel for easy visual recognition as well.

I haven’t used the DVR to it’s fullest extent, so I will refrain from commenting on it. Nothing negative in my initial impressions came up however.

Compression artifacts are apparently no where to be seen on the regular channel line-up. With Comcast, the cartoon network showed clear compression artifacts, even on the kids 36″ standard def TV. The HD channels are also very clean, crisp, and free of artifacts.

The Video On Demand was easy to use, but has a real ugly side… it’s compressed like crazy and the artifacts were very big and very ugly on my 57″ HDTV. Click on the pic below to see the full size image. The artifacts are readily apparent and very disappointing. Also, I wasn’t able to find any HD VOD. Perhaps it’s there somewhere, but I didn’t see it.

As far as phone service goes… it works. There isn’t much to review other than I am able to call people and they can call me. Someday I plan to ditch my phone service entirely and just use our cell phones so I don’t care for much more than a dial tone and clear voice, which FIOS has.

Summary
FIOS is about $30 per month cheaper than Comcast in my neck of the woods for the “Freedom” package which includes TV, Phone, and Internet service.

The Internet downstream clearly faster than Comcast by more than double when comparing sustained speeds. FIOS upstream is about 7 times faster than Comcast. In general latency also appears to be somewhat lower than Comcast. Winner: FIOS.

The high definition TV service from FIOS is similar to Comcast, although Comcast had visible compression artifacts on many channels that FIOS does not. However, the FIOS VOD service is horrible compressed and it shows. Winner: FIOS, but if you’re a heavy VOD user, you may want to keep Comcast.

Phone service seems virtually identical.

I hope this review was helpful to you. Enjoy FIOS!