Data Science at Zefr

We have always known at Zefr that whenever a video ad is placed around relevant content the viewer will have a more engaging experience. We have largely delivered on this promise by offering our customers content “packages” which provide an assortment of well-defined online video content targeting solutions around specific themes, trends, or categories beside which our customers can choose to place their digital ads.  

Some clients, however, are less interested in “packages” and simply want to run their campaigns against all YouTube content that is suitable to their brand (or campaign).  This requires a higher level of customization wherein we must identify subtle, yet important, nuances among brand preferences at the video level which means we end up making a unique list of videos for each customer.  The challenge for us then is to make these brand-customized content targeting solutions scalable across campaigns. This is all made possible by data science.

Data Science

We have always relied heavily on data science to generate business growth in a variety of ways. However, as we begin shifting our approach to curation more and more towards these highly customized solutions for every campaign, data science has innervated more of our processes.  Our customers lean on us to provide detailed insights around every online video on YouTube; we use data science to create unique video lists that would be most appropriate for each customer.

To do this, three components must work seamlessly together.  First, we must create campaign-level labeled data to identify videos either as “relevant” or “irrelevant” for each campaign.  Second, we must extract key bits of knowledge about videos to allow machines to “learn” and “understand” what a specific video is all about. This allows us to create context logs.  Finally, we then must leverage machine learning to detect patterns in all this data, helping to more accurately distinguish relevant from irrelevant videos for each customer and every campaign.

Human Review

The most critical step in this process is human review.  Humans must watch every video to determine which videos are “good” or “bad” for any given campaign.  Video-level review is the fuel for the data science engine. At Zefr, our video reviewers are the account executives, account managers, and ad operations experts working on every campaign. They know our customer’s preferences inside and out.  Here’s a great example. Our team has the knowledge to help a major makeup brand ensure their ads are only seen adjacent to videos about “daily makeup application” (and not videos about “Halloween makeup”). This is because, in data science, when humans make decisions holistically, the discoverable patterns in the data are much richer. Combining this with our deep understand of YouTube allows us to make customer- and campaign-level decisions that take all elements of a video into account appropriately.

Information Extraction

The data science team springs into action once the human review of videos is complete. This begins with featurization: the process of taking a complex piece of data, in our case a YouTube video, and breaking it down into its component parts.

No machine learning algorithm that we know of currently exists that can natively “understand” a YouTube video.  Each video consists of numerous pieces of information (i.e. thumbnails, title, description, number of views, publish date, etc.).  During featurization our data scientists extract this information from a video and organize it to be ingested by pattern recognition algorithms.  This is known as multi-modal learning because we are consuming many different kinds of data –  image, audio, text, and numerical. Multi-modal learning allows us to find patterns in the different components of a video. This helps give Zefr a competitive advantage over other solutions that analyze only a single component of the data (i.e.text).

Machine Learning

Machine learning boils down to this: algorithms that find patterns in data.  In Zefr’s case, our objective is to find patterns that distinguish “relevant” videos from “irrelevant” videos for every campaign we run. Human review and featurization are the first steps in this process; machine learning comes next.  

Our data scientists use state-of-the-art algorithms – including random forests, gradient boosting machines, and deep neural networks – to find distinct patterns in our data.  We also tap into some of the best machine learning platforms like Vowpal Wabbit, H2O, MxNet, and scikit-learn to ensure we find every pattern present in our data. Using a variety of machine learning platforms allows us to select the best algorithm to suit both our data and the desired customer outcome.

There are billions videos in the YouTube ecosystem – and we know a lot about them. Because of this, we need both a rapid experimental environment to test new ideas as well as a robust production environment to deploy successful experiments.  We are able to accomplish both using high-quality open source tools as part of our process. In fact, we’ve also contributed to the ecosystem with a paper about our production environment, Aloha.


Our customers get the best outcomes when their ads are aligned with relevant content. Improving our machine learning capabilities is one of the ways we achieve this at the campaign-level.  We start with human review, watching videos on YouTube with our customers’ brand guidelines in mind to create a new data source that is wholly unique to and deeply aligned with our customers’ needs.  Data science allows us to derive actionable insight using the learnings from our own human review of the videos with our customer’s needs in mind. We can then predict, for every video on YouTube, whether or not it will be suitable for every customer.