DMEXCO 2018 Zefr
EventsTrail Blazers

Zefr addresses the technology of storytelling

This year’s Digital Marketing and Expo Conference (DMEXCO) brought together key players in business and online marketing to discuss the needs and opportunities of today and tomorrow and the ways in which technology is the driving force behind it all.

Our CTO, Oded Noy, was invited to speak on a panel specifically focusing on how technology is affecting storytelling. “The Technology of Storytelling” panel gave Noy an opportunity to discuss ad tech and how Zefr is approaching widespread challenges like brand safety, artificial intelligence and GDPR compliance.

Here’s what Noy had to say –

Question: How has Zefr evolved over the past year? Have you seen more advertisers take direct control? 

Noy: Over the past year, Zefr has evolved in the industry by understanding the value of listening to brands and helping them target their video ads based on what they uniquely consider brand safety.  For example, for clients like P&G who have been voluntarily absent from YouTube, Zefr has solely focused on providing them with control and the resulting precision needed.

To what extent are artificial intelligence, etc., already starting to transform the ad industry? Are they leading to more relevant personalized content being delivered through the right channels?

Noy: Artificial intelligence, machine learning and blockchain have transformed the online ad industry, but these aren’t terms to be feared; these are all fancy names for math!

At Zefr we see it as less about the evolution through technology and more about the harnessing of human-nuanced comprehension of video with the scale only a machine can provide. We call this human cognition.

For some brands personalization is key, for others it’s about the implicit endorsement that an ad provides to the content it runs next to. For the latter, we provide a way to bridge that gap.

While it’s still early, have you seen the impact of GDPR on brand and publisher behavior? Will it lead to significant change, for example, in the use of third-party data? 

Noy: There has definitely been a significant shift in the market conversation since GDPR, particularly around targeting advertising. With GDPR, and the rise of individuals concerns around privacy , we have seen a renewed interest in the type of targeting we provide. This targeting is free of personal identifiable information. Instead of thinking about consumers PII, we think about consumers behaviors and how viewing creates mindsets in order to help brands target. Our goal is to get the right ads in front of the right people at the right time.he need and desire for this type of targeting has been the most significant change.

What does the future hold?

Noy: In 5 simple words, “Brands in the driver’s seat.”

Product & Tech

Inside the Minds of Zefr Technology

On this month’s episode of Technically Speaking, I got to sit down with Principal Data Scientist, Ryan Deak and Product Manager, Rob Knopfler. We discussed how Zefr is using both human review and machine learning to scale YouTube more efficiently and effectively, ultimately leading to the highly relevant targeting capabilities we are known for. 

John Merrifield: Tell us a little bit about the role you each play in building Zefr’s technology.

Ryan Deak: I work on the data science team, where our primary focus is understanding video content and relating that to the needs of our advertising partners. My job is to create statistical models to help curate our advertisers’ campaigns by finding the right video content to pair with their brand and help them steer clear of content that doesn’t fit with their desired message.

Rob Knopfler: I wear a lot different hats as a product manager, but what I really do is connect the dots for customers, commercial teams, and engineering in order to solve problems, and build great products. In this context, the problem we are solving is ensuring that our partners only run their campaigns in front of highly safe and highly relevant video content. This problem is especially challenging because YouTube is a rapidly growing ocean of video content that changes every single second.

JM: How do machine learning and human review work together to review all the videos on YouTube at scale?

RK: So, human review and machine learning are kind of like peanut butter and jelly. Machine learning is an algorithmic approach to scanning a ton of video and finding patterns between these videos. However, you cannot build these models without human review. Our team of experts reviews this content, finds details in it and makes nuanced decisions that models aren’t capable of making on their own. They are both really important, but you can’t have one without the other.

RD: Ya, I think Rob’s right—we really do need both. We use the human review to supply the behavior that we want the models to have for the specific videos that we review. But for a model to be useful, it has to be able to generalize to videos it hasn’t seen before. The interesting thing is that we get a confidence associated with each prediction it makes. So, if a model isn’t confident about a particular video, we can tell the review team to go back and review that video and thereby help that model to learn that type of content.

JM: Ok, Rob—question for you. YouTube is an ever-changing platform, so how do you not only keep up with it but take things to the next level?

RK: The beauty and the challenge of working on YouTube is Zefr’s commitment to building great products and move quickly with it. Today, we evaluate a ton of different data within a video: titles, descriptions, duration, imagery, amongst other things. We use all of these data points to evaluate videos holistically which allows our models to become even stronger and learn much faster.

JM: What’s the key for Zefr in bringing this all together to best help brands navigate YouTube?

RD: I think efficiency is key. Because YouTube is already so big, with more content being added all the time, it is crucial that we are smart about how we review videos. We need to spend our monetary, time, and computational budgets efficiently. By continuing to search out the best videos to make our models even better, we can scale this to brands and their specific campaigns needs.

technically speaking: machine learning episode
Product & Tech

Inside the Minds of Zefr Technology

The key to success in video advertising today? Relevance. It sounds simple enough, but once you layer on challenges such as 3rd party data accuracy, reporting reliability, scalability and cost, it’s no wonder brand advertisers often feel like this –

overwhelmed face emoji

Like with traditional TV advertising, the best results in digital video advertising are not going to be achieved through audience targeting at the channel level, but rather, content targeting at the video level. But with the vast amount of content on YouTube, that is impossible to do with human review alone. This is where machine learning comes in. Simply put, machine learning is a process that leverages knowledge (in the form of patterns) extracted from human review and feeds it to a machine that applies those patterns to algorithms that allow human-guided review at a massive scale.

In our latest episode of Technically Speaking, I sat down with the Head of Data Science at Zefr, Jon Morra, to discuss how Zefr is using machine learning to scale ad relevance through content targeting at the video level across all of YouTube.

How does it work? Morra explains,

What machine learning is about is extracting patterns. We have a team here at Zefr, who watches a sampling of videos on behalf of brands to identify them as either “relevant” or “irrelevant” for each brand or specific campaign. We look for patterns that identify videos that are good for one brand and bad for another. Once data science has extracted those patterns, the machine can then “watch” every video on YouTube, using the knowledge of the human-identified patterns to deliver results at scale.

Morra acknowledges that understanding videos is at the core of data science at Zefr,but understanding brands is the real differentiator. When you know your clients and you can speak their language, you can deliver the results they are looking for. Morra adds that,

We take our understanding of what is important to brands and we align it with the right content on YouTube at a scale that no one else can do in a brand-specific way.

Watch the full episode to hear more about machine learning and a sneak peek at what’s next for the data scientists at Zefr

Want to hear more about how our content targeting solutions can help you reach your audience?

Product & Tech

We have always known at Zefr that whenever a video ad is placed around relevant content the viewer will have a more engaging experience. We have largely delivered on this promise by offering our customers content “packages” which provide an assortment of well-defined online video content targeting solutions around specific themes, trends, or categories beside which our customers can choose to place their digital ads.  

Some clients, however, are less interested in “packages” and simply want to run their campaigns against all YouTube content that is suitable to their brand (or campaign).  This requires a higher level of customization wherein we must identify subtle, yet important, nuances among brand preferences at the video level which means we end up making a unique list of videos for each customer.  The challenge for us then is to make these brand-customized content targeting solutions scalable across campaigns. This is all made possible by data science.

Data Science

We have always relied heavily on data science to generate business growth in a variety of ways. However, as we begin shifting our approach to curation more and more towards these highly customized solutions for every campaign, data science has innervated more of our processes.  Our customers lean on us to provide detailed insights around every online video on YouTube; we use data science to create unique video lists that would be most appropriate for each customer.

To do this, three components must work seamlessly together.  First, we must create campaign-level labeled data to identify videos either as “relevant” or “irrelevant” for each campaign.  Second, we must extract key bits of knowledge about videos to allow machines to “learn” and “understand” what a specific video is all about. This allows us to create context logs.  Finally, we then must leverage machine learning to detect patterns in all this data, helping to more accurately distinguish relevant from irrelevant videos for each customer and every campaign.

Human Review

The most critical step in this process is human review.  Humans must watch every video to determine which videos are “good” or “bad” for any given campaign.  Video-level review is the fuel for the data science engine. At Zefr, our video reviewers are the account executives, account managers, and ad operations experts working on every campaign. They know our customer’s preferences inside and out.  Here’s a great example. Our team has the knowledge to help a major makeup brand ensure their ads are only seen adjacent to videos about “daily makeup application” (and not videos about “Halloween makeup”). This is because, in data science, when humans make decisions holistically, the discoverable patterns in the data are much richer. Combining this with our deep understand of YouTube allows us to make customer- and campaign-level decisions that take all elements of a video into account appropriately.

Information Extraction

The data science team springs into action once the human review of videos is complete. This begins with featurization: the process of taking a complex piece of data, in our case a YouTube video, and breaking it down into its component parts.

No machine learning algorithm that we know of currently exists that can natively “understand” a YouTube video.  Each video consists of numerous pieces of information (i.e. thumbnails, title, description, number of views, publish date, etc.).  During featurization our data scientists extract this information from a video and organize it to be ingested by pattern recognition algorithms.  This is known as multi-modal learning because we are consuming many different kinds of data –  image, audio, text, and numerical. Multi-modal learning allows us to find patterns in the different components of a video. This helps give Zefr a competitive advantage over other solutions that analyze only a single component of the data (i.e.text).

Machine Learning

Machine learning boils down to this: algorithms that find patterns in data.  In Zefr’s case, our objective is to find patterns that distinguish “relevant” videos from “irrelevant” videos for every campaign we run. Human review and featurization are the first steps in this process; machine learning comes next.  

Our data scientists use state-of-the-art algorithms – including random forests, gradient boosting machines, and deep neural networks – to find distinct patterns in our data.  We also tap into some of the best machine learning platforms like Vowpal Wabbit, H2O, MxNet, and scikit-learn to ensure we find every pattern present in our data. Using a variety of machine learning platforms allows us to select the best algorithm to suit both our data and the desired customer outcome.

There are billions videos in the YouTube ecosystem – and we know a lot about them. Because of this, we need both a rapid experimental environment to test new ideas as well as a robust production environment to deploy successful experiments.  We are able to accomplish both using high-quality open source tools as part of our process. In fact, we’ve also contributed to the ecosystem with a paper about our production environment, Aloha.


Our customers get the best outcomes when their ads are aligned with relevant content. Improving our machine learning capabilities is one of the ways we achieve this at the campaign-level.  We start with human review, watching videos on YouTube with our customers’ brand guidelines in mind to create a new data source that is wholly unique to and deeply aligned with our customers’ needs.  Data science allows us to derive actionable insight using the learnings from our own human review of the videos with our customer’s needs in mind. We can then predict, for every video on YouTube, whether or not it will be suitable for every customer.

Product & Tech

The popularity of YouTube, the world’s largest video platform, continues to soar and with that comes an increased appetite for online video content. This comes at a time when (US) online video viewers are expected to hit 236 million by 2020, up from 213.2 million in 2016. For advertisers, the situation is pretty straightforward: spend more advertising dollars where there are more eyeballs. However, achieving precise content targeting at scale remains a big hurdle.

Zefr harnesses the power of machine learning – with a touch of human curation – to wade through YouTube content and classify it at the individual video level. Why is this important? By doing so, we’re able to provide brands and advertisers with a content targeting solution that is relevant (at the campaign level), brand safe, and highly scalable. Only machine learning can do this – effectively – at the truly massive scale of YouTube. (We’re talking billions of videos!)

Machine Learning

Let’s start with the basics: what is machine learning? Machine learning is a computer’s ability to learn and improve from experience without being explicitly programmed to do so. It is another way of talking about artificial intelligence, which was specifically designed, in its early days, to improve the overall speed and efficiency of data-heavy processes. Today, this allows us to tackle otherwise costly and time-consuming tasks at a manageable scale.

As you would expect, machine learning isn’t a one-size-fits-all concept. There are actually three primary types of machine learning: 1) unsupervised, 2) supervised, and 3) semi-supervised. All three types require data input – either “labeled” (manually done by humans) or “unlabeled.”

  • Unsupervised Machine Learning
    How an “intelligent” system learns through unlabeled data inputs. Because this isn’t human-curated learning, there is no easy way to determine learning accuracy through the output (as it’s hard to identify the relationship between input and output). If the output of unsupervised machine learning eventually delivers against a pre-stated hypothesis, you know that “learning” is taking place and delivering the desired end result(s).
  • Supervised Machine Learning  
    How an “intelligent” system learns through a combination of labeled input and output examples. Because the inputs and outputs have been guided by humans upfront, this is the easiest way to ensure learning objectives and, once achieved, begin applying that learning to new functions.
  • Semi-supervised Machine Learning  
    As you may have guessed, this is a hybrid of the two learning models above. This process is guided primarily by unlabeled data, but influenced by a small amount of labeled data to guide and expedite the learning process to achieve the desired outcomes. Combining the two processes have not only been found to be more effective overall, but more cost-effective as well because it requires less time and attention from a human to curate that learning process.

How it Works

The team at Zefr primarily relies on supervised and semi-supervised machine learning models to make the magic happen. According to Zefr Principal Data Scientist, Ryan Deak:

“We have millions of videos to categorize in some way. If we had an infinite amount of time, we (humans) could go through them one-by-one and say, for example, ‘this video has X property, so we want to group it with other videos that also have X property.’”

Unfortunately, we don’t have an infinite amount of time nor would anyone in their right mind want to dedicate that much budget to tackle a process like this manually. Doing so is simply not scalable in any way. That’s precisely where the supervised machine learning comes into play.

“We can go through the ‘cherry-picked’ process for a select number of cases to label videos based on the properties we find. At that point, however, we pass those labeled videos over to the machine learning algorithm, and it analyzes that input to predict which unlabeled videos should be associated with a particular category. The ultimate goal of machine learning, therefore, is to be able to generalize what it’s learned, based on the labeled data provided, and accurately apply that learning to data it has never analyzed before. This is how Zefr scales a massive amount of online video content to deliver a precise content targeting solution to brands and advertisers.

Precise content targeting for online video content at scale is the challenge that Zefr's technology solves for.

One of our key differentiators are the proprietary tools we use to streamline this machine learning process. We use a framework called Aloha, which makes it easy to extract data (from content) and integrate it into our learning and prediction pipelines. This is huge! All too often, as much as 75 percent of a data engineer’s time is spent on manual tasks – like extracting and cleaning data – instead of focusing on more complex, problem-solving-oriented tasks.

“In the past 5 or 10 years, there has been an explosion of higher quality open source machine learning libraries, like Spark, Vowpal Wabbit,, and MXNet to support our deep learning needs. The fact that we operate on a single framework (Aloha) to integrate all these libraries means both our data scientists and data engineers can do their jobs much more efficiently without being bogged down by the typical troubleshooting that comes with model deployment.”

*More information about these libraries is outlined in “Hidden Technical Debt in Machine Learning Systems” and in our own SysML 2018 conference paper, focused specifically on Aloha.”

Human Review

The massive volume of videos on YouTube means that machine learning is table stakes when it comes to developing and deploying a precise content targeting solution. It is the only efficient way to navigate – and make sense of – the totality of the YouTube online video ecosystem. As Deak points out:

“It’s not like TV where you have maybe 1,000 TV shows at any given time in a year vs. a billion videos on YouTube. It’s both a problem and an opportunity for us as well as for advertisers. The only hitch: it’s an opportunity that’s about a million times the size of TV.”

The truth is, without some human review mixed into this process, it’s difficult to accurately assess the relevant alignment of a particular piece of video content to a particular brand message. That’s all about brand safety and, in spite of how advanced machine learning has become, we’re not ready to let artificially-intelligent algorithms take complete control of the driver’s seat here. There’s a certain ethos involved with human review and curation that rounds out this process – and all with the goal of driving the most value to brands and advertisers at the end of the day. Eliminating human review from the machine learning process raises the chances of “unsafe” content slipping through the cracks or, more generally speaking, content misalignment with brands. At this point in time, that’s not a gamble we’re willing to take.

Our machine learning process starts and ends with human review. We feed our algorithms useful data and then assess and evaluate the output from those algorithms. And it’s thanks in big part to this human-machine collaboration that we’ve been able to create highly accurate models that can blacklist unsafe content, whitelist safe content and deliver more precise content targeting.

The Zefr Touch

The team at Zefr is evolving alongside machine learning. We’re creating new ways of making machine learning more accurate, effective, and efficient vis-à-vis YouTube while simultaneously taking those learnings to make our (human) team more effective as well. The foundation of our system is based on modeling and scoring. This means that our algorithms are built specifically to provide high quality machine learning models that improve as they learn over time. Adding the human layer to this dynamic is our “secret sauce.” We know the ins and outs of the YouTube ecosystem better than anyone else (except maybe for YouTube). For this reason, Zefr is uniquely positioned to provide brands and advertisers with a precisely targeted opportunity to align their promotional messages with contextually relevant online video content – all at scale!