Matt Genelin

  • Daly City CA
Matt Genelin

Sys Admin / Security minded / DevOps

Work History

Work History
Apr 2014 - Mar 2015

Network Administrator

  • Upgraded five ESXi hosts (entire production infrastructure) in two Data centers (Minneapolis and Chicago).

    • Upgraded existing infrastructure from VMWare ESXi 5.1 to 5.5,

    • HP Proliant mixture to HP Proliant DL380 G8's.

    • Installed, configured and placed into production Simplified Disaster Recovery (SDR) with Dell Equal logic SAN.

    • Installed and began collecting performance data with vCenter Operations Management Server.

    • Installed, configured and placed in production F5's BIG-IP high-availability clustering across two data centers,

    • Ran a successful Disaster Recovery test of the above equipment, including failover in front of board-level management.

  • Performed desktop support when needed, but focused primarily on:

    • Public-facing web vulnerability mitigation.

    • Centralized log review.

    • Equipment Decommissioning.

    • Vendor management.

Apr 2004 - Dec 2014

Fast Computer Service Co.


I was a for-hire contract hourly System Administrator for the Small and Medium (SMB) marketplace in Minneapolis – Saint Paul. Focused primarily on Legal, Medical and Real estate markets, although I had several hundred clients.

  • Troubleshooting network, software and hardware problems including:

    • Malicious software detection and removal.

    • Data recovery.

    • Software installation and software vendor management.

    • Windows share setup and maintenance.

    • Active directory administration.

    • Hardware replacement.

    • Printer installation and laptop repair.

  • For one healthcare client in Mendota Heights, MN:

    • Performed SSO integration of eight different linux OS's against existing Active Directory domain in heterogeneous environment (linux, windows) using Samba / Winbind.

    • Implemented best practices and audit controls for SAS-70 audited clients.

    • Met with outside audit agency and corrected exceptions to dwindle down SAS-70 audit exception list for one client to one page in a non-best-practices environment. (Windows 2000 in the year 2008 with all users having shared administrator credentials.)

Apr 2012 - Dec 2013

System Administrator, Patch Management

Thomson Reuters
  • Implemented Splunk as a centralized logging solution from ground up.

    • Achieved passing results on audit of systems two years running on SSAE-16, SOC-I, SOC-II and self-defined SOC-III audits for centralized logging.

  • Implemented patching program based on corporate standards, SOC-I, SOC-II and SOC-III and NIST SP800-40 Version 2.0.

  • Achieved monthly patching goal for 18 consecutive months.

  • Reduced Qualys Vulnerability score from 5.0 (bad) to 4.6 in first six months.

    • Implemented weekly scans (up from monthly.)

    • Implemented ticketing/Tracking project in JIRA to track progress on vulnerabilities that came out.

    • Implemented Spiceworks Inventory / Audit tool to track progress and generate reports for management.

    • Implemented Spacewalk version 1.7 for RPM Deploys and centralized /etc/sudoers management.

  • Found and corrected under-performing Java running on TOMCAT application for a major customer.

    • Used Splunk to correlate events.

    • Worked in the blind to troubleshot vendor-managed Checkpoint Firewall and F5 Big-IP Load Balancer, found overload issue on firewall.

    • Used Cacti graphs as a first-line pre-warning before customer web application slowness, and a one-day prediction of site down.

    • Correlated Apache Proxy Logs (load balancing three tomcat application instances) for instance failure to demonstrate to management exactly when performance issues arose.

    • Found old version of Java application running against production database by moving current application to new virtual server. Shutting down this old application led to solving the performance scaling issue; both applications were hitting the same production database instance, causing the current version of the application to perform badly when loaded.

  • Performed the day to day administration tasks that you would expect in a mixed Windows / EPEL 5 and 6 environment.
    • Wrote scripts with VMWare CLI to inventory machines, capacity planning and management, and removal of older snapshots from existing virtual machines.
    • Deployed WSUS (two data centers) and SCCM (LAN) updates during monthly maintenance outage window, automating reboots with group policy. Created custom package groups centric for remote office deployments (on a slow, constrained, network link) and os-specific deployments via SCCM.

    • Ran Mandiant Redline analysis tool on suspected infected virtual machine memory image while still running in production. Troubleshooted with security resource processes in use, basic use of tool before handing virtual machine image to corporate infosec.

    • Fixed up and updated legacy Python scripts for customer-facing reporting. Scheduled via cron.

    • Scripted, automated email alarms for system and application services that were not able to be easily monitored by existing Nagios monitoring system.

    • Answered and scheduled staff of on-call System Administartors.

    • Implemented database logging for SLA up time reporting via Nagios' ndo2db plugin for mysql. Maintained and updated as needed by business.

  • Performed software deploys of eGRC – Enterprise Governance Risk and Compliance – in-house software to public-facing websites running on Apache Tomcat and JBoss. Supported 57 minor versions and 7 major versions dating back to 2006 of this application. Tuned Apache 2.22, 2.24 and 2.26 front-ends for load balancing, reliability and performance.

  • Designed and implemented high-volume Postfix mail relay (12 messages/sec, average) for outgoing mail, including two Postfix servers that were automatically redundant using UCARP, an IP-failover software for Linux. These replaced one failing mail server that ran on windows and had no patch upgrade availability (no longer supported).

  • Implemented HP Service Manager 7 within business unit for ITIL-like Change Management program (4 month project.) Maintained status as HPSM Librarian within company to keep inventory in compliance with corporate 98% objectives.

  • I decommissioned old virtual machines and old hardware using newly established Change Management guidelines.

Jan 2010 - Aug 2011

Support Genius

Code 42 Software
  • Installed and configured enterprise backup servers (CrashPlan PRO) in data centers for Fortune 100 companies.

    • Performed on-site, customer-facing engagements where I custom configured CrashPlan PRO backup clients

    • Trained customer's IT staff on how CrashPlan PRO worked.

    • Performed push installs in heterogeneous environments (MSI's / exe for Windows 32-bit and 64-bit, DMG packages for Mac OS X, custom scripting of tarball installer for any Linux with kernel 2.6 and higher) in front of enterprise clients.

      Tested and pushed with Altiris, Apple Remote Desktop Software Distribution, Perl / Expect.

    • Integrated backup appliance with enterprise customer's existing LDAP or Active Directory environment to use existing LDAP structure for backup client authentication. (Wrote and tested regular expressions to provide appropriate security controls for authentication.)

    • Provided basic “defense in depth” network design analysis and consulting for client's placement of CrashPlan PRO Backup Server.

    • Performed firewall / network troubleshooting and resolution to insure remote monitoring enabled correctly for CrashPlan PRO Backup Server.

  • Fielded Zendesk support tickets for 50k+ user base that grew to 150k in an public-internet facing role.

    • Became escalation point for 10 other support technicians.

  • Managed Professional Services engagements for remote installations between my on-site engagements.

Apr 2008 - Jul 2008

Ms Lan Engineer

  • Acted as a contract System Administrator for an up-and-coming ISP / Managed Services Provider

    • Provided on-site break/fix computer repair for existing client base.

    • Led strategy for repair and transition between customer Windows SBS 2003 virtual servers.

    • Transitioned new clients from physical server to a VMWare infrastructure (P2V).

    • Performed weekly maintenance on physical and virtual Windows Server on and off client site.

    • Fielded help desk calls on overflow basis.

    • Planned outage of NetApp SAN.

    • Upgraded VMWare infrastructure (ESXi hosts, virtual machines.)

Aug 1999 - Dec 2002

Online Editor / Webmaster

The Minnesota Daily
  • As a student-worker, worked 20-40 hours per week while holding down a full-time course load and:

    • Supervising two student workers.

    • Produced and double-checked daily production of online newspaper for website.

    • Planned, directed and implemented new website design (front end and back end, Dreamweaver, Perl) of website on Jan 1, 2000.

    • Guided strategy in implementation of domain purchase and transition.

Jan 2001 - May 2002


U of MN Computer Science Department
  • As a student-worker, provided help desk support for U of MN Computer Science department students and faculty.

    • Network troubleshooting.

    • Physical network provisioning in a port-security enabled environment.

    • Password resets on NFS /YP and Active Directory.

    • Backup supervision and lost / deleted file restores on Amanda backup and Veritas Backup Exec

    • Fire and security system monitoring.

    • Provisioned door / card access to building.

    • Provided emergency (overnight) support for failing data center air conditioner.


Aug 1999 - May 2005

Bachelors of Computer Engineering

University of Minnesota

Full Electrical Engineering curriculum with half of the Computer Science curriculum in one degree; Computer Engineering.



Hetrogenius Environment Administration

Windows Domain Administration in heterogeneous environments. Installation of backup clients and custom installer config and push testing of applications in Mac OS X, Windows 2000 through Windows 7, and Linux 2.4 (and newer) environments. I once administrated eight different Linux OS's for a single client.

VMware Admin, install and patching

VMware Server 2 through 5.5 / ESXi patching and administration including VMWare Simplified Disaster Recovery (SDR).

Red Hat Linux EPEL 5 / 6

I have administrated and patched Red Hat Enterprise Linux 5 and 6.

Bash and Perl / Expect Scripting

Perl / Expect scripting - bash shell scripting – code deployments with svn, git.

  Hardware troubleshooting - LDAP / Active Directory integration in mixed environments – working knowledge of Windows, linux filesystems including ext2, ext3 and ext4 and experience upgrading these filesystems in place. I have experience with Windows update and application push-deploy tools, including SCCM and PDQ Deploy. I have three years experience with upgrading major versions of VMWare ESXi, and patching the minor patch releases of VMWare ESXi 3.5 through 5.5. I have experience with provisioning LUNs, fixing snap-mirror synchronization issues with a NetApp FAS3020.

Troubleshooting Apache, Tomcat and JBoss

Apache 2.x configuration and performance troubleshooting proxying for Tomcat and JBoss applications