Download PDF

Doug Needham

Author Data Structure Graphs, #DataOps Engineer, and Analytics Architect



Work experience


Oct 2013Present

Principal Consultant

Illumination Works Senior Consultant My current project is with Cengage Learning. We are using Amazon Redshift as a data repository for a clickstream analytics project. Our analytics tools are R, Tableau and Cognos. Our homegrown, Python based, data movement processes managed by Jenkins run a continuous pull cycle from our S3 integration point with our data provider. I also manage our Data Science lab. Cloudera Director built out our AWS Data Science lab, based on CDH 5.2. A short-term analytics project completed in December 2014 used Spark MLLib, and Spark GraphX to prove out different models for use in making recommendations as part of the Cloudera Data Science Challenge. Clear Measures Principal Consultant As a Data Architecture consultant, I provide expertise in Data Architecture to clients of Clear Measures. In addition to my project responsibilities, I am also the Big Data Competency lead for Clear Measures. This requires me to internal training on Hadoop and Cassandra, as well as review project candidates for these technologies. My final project was to make recommendations for the adoption of a Polyglot Analytics Framework for a large grocer based in Cincinnati My first project was to lead a data warehouse implementation using Data Vault Methodology for an origination analytic project. We were using Informatica as the ETL platform, Oracle as our database platform, and Cognos as the reporting platform. The entire data warehouse Architecture, designed in ERWin, pulls data from various sources and populate our data marts visualized via Cognos.


Dec 2011Oct 2013

Lead Data Engineer

Lead Data Engineer I engineered a cloud-based infrastructure for the management of health care data for 3M customers. Using RightScale to manage our clouds(Rackspace, Azure, AWS), and direct configuration tools for our hosted data centers(SunGard). Our data platforms are diverse: DataStax Enterprise Cassandra, SQL Server 2008, Cloudera Hadoop, and Percona MySQL using Gazzang as an encryption technology to ensure we comply with HIPAA in protecting our PHI data. kI perform Cassandra Administration, data server configuration, capacity planning, data modeling, Pentaho Deployment, and performance tuning and on-call support as the need arises. I also managed a team of DBA's supporting the operational systems that stored our customers PHI data. Our primary application supports the medical coding industry by using natural language processing on physician transcriptions. The enrichment and analysis of this medical data is the main goal of our organization.

Sunrise Senior Living

Oct 2008Jul 2011

Data Warehouse Developer

Data Warehouse Developer The financial data mart I worked on brought together actual and budget data from PeopleSoft. We developed the data models, created the PL/SQL code to populate the models and assisted the BI team with query optimization and tuning. Our PL/SQL code ran on a nightly basis for loading the data models. The prior system ran once a month and loaded the data into a macro controlled excel spreadsheet. I specifically worked on loading the data into fact tables, performance optimization of the queries generated by OBIEE, and custom calculations done at the data layer to improve performance of OBIEE.

Robert Half Technology

Apr 2008Oct 2008

Data Management Consultant

Data Management Consultant I worked as a data management consultant working with various customers in the Washington D.C. area. The systems that I have worked on include migration of Microsoft Access based systems to SQL Server 2005; conversion of applications from Microsoft Access based system to LAMP based PHP/MySQL back-end. Provide review and recommendations services for LAMP based web-site upgrades. I have also deployed the Zabbix network-monitoring tool to increase the reliability of one of the network.


Feb 2007Mar 2008

Director of Data Warehousing

Director of Data Warehousing I led our Enterprise Data Warehouse project. The EDW was an Oracle based data warehouse that brought in data from Oracle, SQL Server and MySQL based systems. We used Business Objects Data Integrator and Pentaho Data Integrator for our ETL tools. The data warehouse, designed as a Data Vault architecture, used for consolidation of multiple system and key integration of our disparate systems. From the Data Vault, we began feeding traditional star schema based data marts for reporting via Business Objects. In addition to my duties managing the Data Warehouse infrastructure, I also participated in the review, and maintenance of our PL/SQL code to find opportunities for improvement to our Java based storefront.


Mar 2006Feb 2007

Senior Dba

SAIC Senior DBA There are a number of Biometric Intelligence Resource(BIR) applications based on SQL Server 2000 for the identification of non-allied personnel in combat areas deployed throughout the world. The project I worked on was an effort to consolidate all of the data from each of the BIR applications. The goal was for a consolidated identification of non-allied personnel in an Oracle 10g environment. Our ETL infrastructure was SAS Data Integrator that brought data from SQL Server to flat files. These files where then placed on the Oracle server as external tables. PL/SQL code then merged this data into the staging area for change data capture. I was able to contribute to this project by turning around the data infrastructure. By educating the project management team on industry best practices for data warehousing solutions I was able to show exactly where we needed to focus the project going forward. This re-alignment of priorities directly contributed to the completion of the project on time.


Jul 2005Mar 2006

Senior Dba

Senior DBA As a Senior Database Administrator in a SQL Server 2000, I worked to define an Enterprise Data Architecture that will support our Data Warehouse, Business Intelligence, and Enterprise Application Architectures

Chase Student Loans

Nov 2002Jul 2005

Manager of Database

Manager of Database Operations As a Database administrator manager in a SQL Server 2000 and Oracle environment, I managed a group of eight people supporting 16 production/development database servers as well as a number of MSDN licensed SQL Server instances installed on local machines of developers and DBA's. These databases range from Clustered SAN based systems to stand-alone servers. My groups' responsibilities include: Production Operational Data Store Our legacy application was built on the Cache' database and all of our reporting was done in MS SQL Server. As we were beginning to exceed the capabilities of our legacy telemarketing systems, we deployed a new internally developed.Net application. This new application interfaced directly to multiple AVAYA dialers based on location. The Operational Data Store was the centralized repository for the consolidation of data from our diverse systems. We loaded the campaign data from the marketing department, and processed it such that it could be loaded onto both the dialer as well as the.Net application. At the end of the day extracts were done from the dialer, the Cache' system, and the.Net application for the production of consolidated reports. BI and ETL infrastructure As new users were migrated into the Business Intelligence environment, we certified them in training, and then added them in Business Objects security. We also maintained backups of the Universe. We did installation and configuration of our Informatica services, as well as maintenance, upgrades, and troubleshooting of any problems the ETL developers ran into. Database Server infrastructure We supported 16 production and development database servers as well as a number of MSDN licensed SQL Server instances installed on local machines of developers and DBA's. These databases range from Clustered SAN based systems to stand-alone servers. We ensured that scheduled DTS packages, jobs and processes complete in a timely manner.

Lockheed Martin


Software Developer

Software Developer GTN Project(08/01 to 02/02): I worked as a Database administrator for a Sybase and Oracle based project. This was a web application that has to use Sybase's DCO product to integrate data from the Oracle database with the Sybase database maintained by our application. The front end is a Java Server Pages based application using BEA's WebLogic web server.


Nov 1999Feb 2000

Software Developer

MITRE: I worked on a team to convert an embedded application from VxWorks to Yellow Dog Linux running on an embedded Radstone PPC1A board. As part of this conversion I Worked on the port of DCE/RPC to the PowerPC platform, as well as the port of the application itself. This was a short-term proof of concept project. MITRE needed to prove that an existing VxWorks application could be ported over to LINUX in a timely fashion and we proved the concept. EXCEL(11/98 to 11/99): Responsibilities include participating in a project to port part of the dispatch application from Win32 to HP-UX11. This consists of porting Windows DLL's written with MFC to Unix Shared Libraries. I also wrote a quarterly Payphone reporting system in PERL. I built and maintained a LINUX test environment where I have been testing the feasibility of a port of our legacy systems to LINUX. Management Controls Software Developer 1998 My responsibilities included developing and maintaining a Visual C++ application that used embedded SQL with ODBC to communicate to SQL Server 6.5, installation and configuration of the Oracle Test Database, and porting the code to use Oracle as a database in addition to SQL Server. Writing and distributing SQL scripts for maintenance of the database. Maintenance to the database and stored procedures, enhancements to our existing products, interfacing to new products or systems, and traveling to customer sites to troubleshoot application, database and network configuration problems.



Software Developer Consultant

Software Developer Consultant QWEST(2/00 to 06/00): I worked on a team to update a sales commissioning system. This is a C++ application running on a HP-UX 11 with Oracle 8i as the backend. The application creates commission data based on customer orders for sales agents. I also co-maintain our local Linux server.

El Paso Energy


Software Developer

Software Developer Led a team that maintained and supported a pseudo real-time DOS based C application deployed at 20 different field locations. This application communicated with specialized hardware to enable Fuel trucks to load petrochemical products onto the truck. I was in charge of redesigning this application into a more Real-time application.