There are a lot of wires…
9 racks of equipment, an old Nortel PBX the render farm, SAN, Blades.
Before I talk about the wires and equipment though, a little background. On March 31, 2015 I received an email announcing the exciting news about our office. “The lease for our current building is up in December so we are moving to the Bindery on Blake.” The email had a nice picture of a room.
On April 10th I went for a tour. The Bindery is an old building that was built in 1927 and was home to Magnus Metals, a foundry. In 1976 Eastwood Printing started operations in the building and it was a book bindery for the next 30 years. In 2014 Sonia Danielsen, owner of the property, decided to convert it to a multi-tenant mixed use building and we were moving there! Sprung Construction was busy with the shell getting it ready for new interior construction.
The interior space was empty on the tour but the Facebook page for the Bindery showed the drop ceilings, silver paint, and messy warehouse which Sprung had already been busy fixing up.
The plan initially was to lease area A, D, E & F. We called them bays. After negotiations it was decided that we would only lease Area A and D. These are 220′ x 45′ roughly 20,000 sq ft of space. Substantially smaller than the space we were in. Over the next few months there were meetings and talk about the design for the new space. We had surveys and inquiries into what we liked and disliked with the current space.
In June there was finally a space plan presented to us. The IT area was, lacking. How do you fit 9 racks of equipment, storage and work space and 4 desks into the little orange space.
The saving grace is that the construction was a long way from being done. I took this picture on June 10th 2015 of the Bindery parking lot.
Just a month later the building was looking even more demolished and I figured there was no way the place would be ready to move by Thanksgiving.
In order to concede space the plan is to move our server equipment to the Fortrust data center and connect the new facility using a 1 Gb E-Line.
The new plans in August reflect a much improved IT area.
The “Server” area is really network switch and Audio / Video equipment. The cozy work room would have five desks and a work space. Then everything went quiet. No more meetings. No more discussion. When I walked by the construction site stuff was going on outside but it didn’t seem like much was happening inside.
About the same time in August I was given approval to ordered a new SAN. I convinced management that it was not wise to expect us to move hard drives that have been on and running for 8 years without having catastrophic failure. The existing HP EVA 4400 had to be upgraded and we would move the new SAN.
So begins what this blog post said it was all about, how to move a server room.
Step 1. Identify what you have and decide what to do with it.
- HP EVA 4400 SAN – upgrade it to a new SAN
- HP ML370 Server and direct attached external storage – migrate to a VM and move the storage into the SAN
- Stand alone server – migrate to a virtual machine
- Data Domain DD530 – ship to the Alexandria office and replace with a new Exagrid
- HP C3000 Blade enclosure with a mix of Gen 2 and Gen 7 blades – Upgrade to a C7000 with Gen 9 blades.
This process continued until we had identified everything in all 4 server racks and 5 two post telco racks. The inventory was compared to the asset database to make sure nothing was missed. Lots of Visio drawings like the one above were made, marked up, revised, and printed to paper.
This process required that every cable was labeled. The Switch port was identified. The UPS was identified. The photographs were labeled and printed to go into the paper documentation.
Step 3. IP Documentation
The next step was to gather all the intellectual property, passwords, network addresses, configuration files anything I could think of that may be needed during the move. I printed everything that I could and added it to my paper documentation. I wrote shutdown procedures and startup procedures for every server, service, and piece of equipment.
Why paper documentation? I figured if things went south I might need to refer to something and it needed to be in paper so there was no need for any compute resource or network connectivity. Just to be certain I had the information.
Step 4. Publish the book.
Step 5. Open the book and do a dry run.
I scheduled maintenance windows and tested the shutdown and start up procedures. Verified the IP address and switch port configurations for every server. Check and double check.
Now that the plan was ready we needed to order the SAN. We looked at several SAN vendors. It came down to a bake off between Nimble and HP. Both had their advantages and I was really leaning toward Nimble but HP made us a heck of a deal so we went with the HP 3PAR 8200. We ordered the 3PAR on October 6th. It is brand new technology and I am fairly certain we were one of the first customers to install one.
While I waited for the SAN to arrive I started working on two other big initiatives. The first was to upgrade VMWare vSphere from 5.1 to 6.0. That meant migrating the hosts from the old Gen 7 blades in the C3000 enclosure to new Gen 9 blades in a new C7000 enclosure that we had purchased earlier in the year. The second was to migrate our Citrix XenServer to the new workstation blades in the C7000. That involved upgrading to XenServer 6.2 and XenDesktop 7.2. This was important as we decided that we would use the Project Cloud (XenDesktop) machines for our staff so they would be functional while their regular desktop PC’s were moved to the new office. Both of these projects will be future posts.
But there was still no way we were moving over Thanksgiving weekend. The new move date was scheduled to December 22nd.
The E-Line was not installed and Level(3) was giving us a date of mid February. The construction was still going on and was far from complete. The Building’s MDF had no lock on the door and no rack for equipment to be mounted in. Level(3) kept failing the inspection.
On November 20th the MDF received doors that locked. Level(3) pulled the fiber and then sent the process to the next group to schedule and come out to rack equipment and turn up service. We still had to get a rack installed into the room.
The implementation engineers from HP came out on November 16th to complete the 3PAR install. After a bunch of turmoil and hoops we completed the installation by Friday morning and I started migrating data from the EVA 4400 Friday evening. We ran a handful of servers from the 3PAR for a week and then I migrated the bulk of the 30 TB over the Thanksgiving weekend. The new SAN was alive.
From Thanksgiving weekend through mid January I worked 90 hour weeks. The whole thing was a blur. I had engineers from Lewan & Associates out to certify the VMWare upgrade, configure Cisco Nexus switches for the data center, We set the Nexus up on a baker’s rack at the office and got it all working and tested prior to racking at Fortrust. We also configured Cisco Catalyst POE Switches for the Bindery office.
A week later David and I hauled over the Render Farm and racked it up. Thankfully the team was finished with a render project and since the holidays were getting ready to start they hadn’t scheduled any new projects. Woo Hoo I got to move something early!
Brad and I configured the new Cisco Unified Communications environment for our new phone system. We had lots of fun with the SIP line. We had hauled over the router and done a test call back when the circuit was installed. Now that we were trying to use the line we found that Level(3) had turned off the inbound SIP test & turnup phone number.
PANIC! – NOT ME!
The move hadn’t even started and I had over half of a rack loaded with equipment.
December 18th brought the installation of the E-Line into the Bindery. Level(3) just needed to extend the Fortrust side into the rack.
All the documentation was ready and the move was a go for December 22nd. Time to take a weekend off. On Monday the 21st late in the day we received temporary occupancy to the new space. I ran over to look at the space.That was it. I wrote a final email to the office and went home for the night.
The VMWare environment was back up and running with email flowing in and out by 10 PM on the 22nd. On the 23rd I moved the Exagrid and BackupExec Server and tested replication back to the DR location. Made sure the VPN connectivity was working between offices and remote users. Tested the Citrix Environment.
No need to PANIC!
That is pretty much how to move a server room.
The E-Line was still not up and running, but the electrician didn’t have the power to the desks either. The race was on to see who could have working services first. (I won by an hour, but not until January 4th).