Archive for the ‘technology’ Category

By the numbers

June 20, 2008 Comments off

There’s nothing like a little bit of shameless self-promotion every once in a while.  :-)  We just sent out the following email internally to remind everyone of what IT “brings to the table”.  Fun times.

There are some things normal people can do. For everything else, there’s IT.

Number of employees and contractors that keep clients happy, loyal and paying: 600+

Number of distribution lists delivering vital targeted information to interested people: 711

Number of emails a month that attempt to infiltrate unsuspecting inboxes under false pretenses: 1,255,000

Amount of MBs it takes to backup all Civica files so that there’s no bloodshed when a server crashes:13,000,000

The 11 IT staffers who keep it all running without concern for a life, limb or sanity: Priceless

Without our IT department, most of what we do every day wouldn’t be remotely possible. So if you cross paths with any IT staffers in the near future, tell them thanks. Don’t send them emails though… they’re kind of busy.

If you’re interested, take a look below for all of the gritty details. Sure we don’t necessarily all know what some of these things actually mean, but IT says they’re important, so we should probably take their word for it. Either way, these are some pretty impressive numbers.

·         570+ employees

·         40+ contractors

·         9 offices in 7 cities on 2 continents, plus a number of “remote” folks in additional cities and countries

·         Servers

o   90+ physical servers in production and hosting of dev VM’s

o   220 IT/AMG hosted dev virtual machines and 22 production VM’s (142/12 in Bellevue alone)

o   8.75 TB of SAN storage capacity in the Civica datacenter (6 TB of capacity allocated)

o   6.8 TB of file server storage used across the organization

·         Email

o   875 Exchange mailboxes (users, testing, project use, etc.)

o   38 Conference Room scheduling mailboxes

o   711 Distribution Lists on Exchange

o   1.54:1 ratio of DL’s to employees (!!!)

o   520 GB of mailbox and public folder storage

o   Monthly inbound email traffic from Internet (through MXLogic filters)

§  1.4M messages sent to company-owned accounts (multiple domains)

§  84% denied, 5% quarantined as spam or virus infected

§  145k messages delivered (11% of total received)

·         Backups

o   13+ TB of data written to tape each week

o   22 tapes used per week (600GB/tape)

o   50 hours to do a full back up the Civica file server (includes data replicated from other offices)

·         Network

o   28 data circuits and 35 voice circuits (230 lines) across all offices

o   57 Mbps of Internet connectivity across all offices

o   144 Mbps of inter-office backbone connectivity

o   93 networking devices in use

§  15 firewalls

§  11 routers

§  47 switches

§  20 wireless access points

·         Civica Datacenter

o   15,000 Watts / 52 Amps of load the UPS (40kW/139A capacity)

o   72,000 BTU’s of heat generated in the room (average gas grill puts out 35-45,000 BTU’s)

o   6 tons of cooling used to keep equipment cool

Categories: technology Tags:

We got BLAST-ed!

May 9, 2008 1 comment

I just called Comcast support to get them to undo a filter that was placed on our home Internet connection a couple years ago.  We used to use a program that sent a challenge email to each spam message we got to give the sender a chance to prove they’re human.  🙂  We’re not using that anymore, rather our home email is filtered by Postini before it even gets to our server mailbox.

I got a tech on the phone after a couple minutes of waiting – not too bad really.  I explained to Neil that we were being blocked from sending email, and how I know WHY we were getting blocked and that we’ve stopped using that tool.  I also mentioned that having SMTP (email sending) traffic blocked made difficult to my job at times, since it’s hard to test email if I can’t send an email.  🙂

He said he’d take the filter off, but it would take a reboot of our cable modem, which runs our Comcast phone line, so he’d call us back in a few minutes.  He did, and then he gave me a present.  He said that he increased my upload bandwidth 4X to over 2Mbps!  WOOHOO!

This is Comcast’s new “Blast” service, and Neil told me they just started the beta of the program here in the Seattle market a few days ago.  Blast is an enhancement to the top-tier of cable modem service that gives you constant faster bandwidth, not just short bursts.  They’re slowly rolling it out around the country, and I had heard a while back from my good friend that it was coming here this summer.

So much for the wait!  🙂  2.7Mb uploads instead of the previous 500-700Kb.  OH YEAH!


Thanks Neil!!!

Categories: technology

How to apply WSUS updates to a Windows Server 2008 Core machine

April 21, 2008 24 comments

Windows Server 2008 Core is an old new concept in the Windows world.  We’ve spent the last couple decades trying to put a fancy user interface onto our computers to make tasks easier for the user.  The server world has followed suit as well with the same GUI advancements.  And I’ll be the first to admit that I like working in a GUI for most small or one-off tasks on a server.

The problem is that with all that code to make a pretty interface you get a much larger attack surface for your environment.  All of a sudden you need to apply patches Internet Explorer and media player to your critical SQL server!

So along comes Windows Server 2008 and the option of a “Core” installation.  Server Core removes pretty much all of the user interface components leaving a greatly reduced attack (and patch) surface that you have to manage.  I recently heard the comment that if Windows Server 2000 had a Core install mode only 40% of patches for the platform would apply (60% for Win2003).  Check out more about Server Core here.

So great – I’ve got a lot fewer patches to deploy … but I DO need to still deploy patches.  I work in a smaller-scale environment; roughly 50 servers in my core datacenter with another 40+ scattered in local offices around the country and the UK, and about 650 users.  We’re not big enough to have the right folks on staff to run Microsoft’s System Center Configuration Manager (formally SMS) to push out applications and patches to computers.

Instead we rely on Windows Server Update Service (WSUS) to manage the Windows Update agent already on each machine (the local agent actually does the install).  Our user and dev machines automatically download and install patches, but our servers just download the patches (pre-staged for install) so we can manually manage when each server installs and reboots.  This way we make sure servers are rebooted in the right order and are around to ensure everything comes back up.  🙂

The problem comes in with Server Core … there’s no way for you to see the “you have updates to install” notice in the system tray when you log on to the computer because, well, there’s no system tray!  I looked and looked and couldn’t figure out how to still leverage WSUS to push patches out and kick off an install.

I recently attended an event on the Microsoft campus in nearby Redmond where I had access to some of the key players in the Windows 2008 product world.  The event is under NDA so I can’t give a lot of details, but let’s just say I’M TOTALLY SOLD!

I reached out to a couple of Microsoft contacts from that event to get some help and was rewarded with a link to an MSDN article the solves my issue!  Effectively you get a command-line representation of the Windows Update user interface by calling a VBscript!

Copy the code to a Notepad window and save the file as a .vbs file accessible on the Server Core machine you want to patch (I copied it to a directory on the local disk).  From the Server Core command line execute that script via ‘cscript nameofscripthere.vbs’ and let it work it’s magic.  You’ll see the patches it scans and finds you need (according to your WSUS group policies … guessing this would work against the public Microsoft Update servers too), will download them (or leverage the pre-staged bits already downloaded), and then ask if you if you want to proceed to install.  Give it a “Y” and you’re on your way!

Special thanks to Brendan and Paul for finding this solution!


As an aside, you can also manually download each patch’s .MSU installer to the Server Core box (using a file copy from another machine) and then call each patch individually from the command line using ‘wusa.exe patchnamehere.msu’.  You’ll actually still get the standard Windows Update user interface dialog where you can prompt it not to reboot.  You can even use a couple command line options to make wusa.exe run in a quite mode and not reboot, thus allowing you to apply multiple patches via a batch script.  Still kind of clunky though.

The problem is that the WSUS/WUA tools don’t download the .MSU files (or if they do they store them in an “exploded” form) so you can’t use that combination to push out the patches to your boxes for deployment – thus the need for the VBS script above and my 4 day search for a good solution.

IT Gone Wild

February 17, 2008 Comments off

A la “Office Space”…

This is what happens when you get tired of servers crashing on you.  You make a preemptive strike.  This one won’t be bothering us anymore (okay it’s been off for over a year, but it still felt good after everything we’ve been through).

In addition to a couple screw drivers and a hammer impaled into the box, I discovered you can make a great musical instrument (a shaker) out of a hard drive.  If you hit it hard enough the platters inside shatter into thousands of little pieces that then float around inside the case.

IMG_0137 IMG_0144

Don’t piss off IT guys.  🙂

Categories: humor, technology


February 16, 2008 Comments off

What’s a FLD?  That’s a F#%$ Load of Disk.  🙂


We just put hooked up a Dell PowerVault MD1000 RAID array to a file server in our Spokane office.  The chassis has 15 750GB drives in it, arranged in a RAID 50 array (2 sets of 7 disks in RAID 5 with a RAID 0 stripe across both RAID 5’s, 1 global hot spare).  Over 8TB of online storage … that’s 8,000GB!

Categories: technology

PC vs Mac

January 22, 2008 Comments off

I’ve been working (my butt off) to integrate a new company into our email system over the past week and weekend.  It’s an ad agency so there are a lot of Macs around … which definitely complicates things and created about an extra day of work for me.  I’m not a Mac guy, so a lot of my work has been trial and error with a healthy bit of intuition and a pinch of luck.  That being said, I just had this conversation…

Lead Mac Guy: I just want to say we’ve had more Mac tech support around here in the last 3 days than we have in 9 years.

Me: Wow – thanks, but that’s sad because I don’t have a clue.

[banter about PC’s vs Macs…]

LMG: You’re just jealous because you use a second grade system.

Me: Better second grade than kindergarten!  [walks off to a room of raucous laughter knowing he got in the winning shot]

To quote Toyota … oh what a feeling!

Categories: humor, technology

Tech meets Toys

January 12, 2008 1 comment

I thought this was a great comparison of the current market forces behind competitive high definition DVD standards Blu-Ray and HD-DVD.  KC is a HUGE Thomas fan.  🙂

HD DVD vs blu-ray

Categories: Kaitlyn, technology