What does it mean to build data infrastructure at a 100 year old company? We're finding out at Condé Nast.

Condé Nast has an incredible amount of data from decades of magazines, a score of websites, and thousands of partnerships, surveys, and special projects. Unfortunately, (and inevitably) it is scattered through the same volume of archives, databases, and long-untouched cubbyholes; accessible in militia fashion by the silo'ed interests that have historically "owned" it, and un-productizable.

Late in 2015 we decided to change that. By bringing on an SVP of Data Strategy (Karthic Bala), CN made the statement that data will play a major role in the future of our company. The first step to making this powerful asset actionable, was to build modern systems to access and maintain it.

So, how do we approach this huge task? We started last year by pulling apart and auditing our myriad of sources. From our library management databases, to our click and impression-streams, we looked at the storage mechanisms, data shape, and integrity of dozens of datasets.

Once we'd done that, we put into place a simple plan.

Large-scale data stores like digital event-streams, content change-logs, and server logs will be stored in an immutable "big data" file-based warehouse on S3 and accessed via Hadoop clusters running Presto; while more traditional stores will be standardized and made accessible via Presto on top of Redshift warehouses.

This combination means that there is no superficial difference between querying a 100Tb clickstream, and querying a 100-record brand revenue table, and we can optimize storage mechanisms around the size and shape data that they represent, without redesigning the query interfaces above.

During Q1 of 2016 our data engineering, warehousing, and operations teams worked closely together to take this plan and turn it into reality. We established an ingestion architecture and applied it to our biggest streaming datasets; we added Luigi to standardize our more common jobs; and we built several internal tools to allow for easier management, and wider access to our amazing data.

In a following post, I'll go into more detail about our ETL process, realtime streaming sources, and other technologies preparing for 2016.