FUNCORP’s services and advertising infrastructure have recently undergone significant changes. In addition to the Prebid Mobile, we now also support and develop the Prebid Server to work with our apps.
We chose Prebid because we’ve been using its Software Development Kit for a long time and have enough experience and competence in working with it. This expertise allows us to improve and develop the service without compromising its stability. As a result, Prebid has become one of the critical services within our infrastructure.
Before telling you about the decision, the implementation, and future plans, let’s review the Prebid basics.
This article covers not only the technical side of the Prebid Server but also product metrics.
To begin with, let me remind you that FUNCORP develops entertainment services, and the primary monetization method is displaying ads.
To display ads, you need someone to publish those ads in your apps in the first place. There exist multiple advertising networks with whom you can sign a contract and set up an integration to get the ads for your users.
Some of them you can see in the picture below:
Our app supports ads of different formats and sizes. It can look like this:
When you open the app, you get ads creatives — ads shown to users through some digital platform (in this case, our mobile apps). To deliver creatives to the user, we send requests to different ad networks, aggregate the received ads on our site and display them.
Previously, the advertising market used the waterfall advertising mediation process (the so-called waterfall auction). Each publisher sets up requests to ad networks with a minimum ad price. These ad networks go through a prioritized list of bids from top to bottom and may return the first bid that satisfies the condition without reaching the most favorable recommendation on the list. The whole process was prolonged because bids were checked one by one (like water flow in a waterfall — hence this comparison).
Copyright — https://prebid.org/why-prebid/
An alternative to the “waterfall” is the header bidding auction (HB). Unlike the previous method, requests are sent out to all bidders, and the best bid wins. This approach worked faster and made it possible to find the best possible price.
Prebid, an open-source unified worldwide auction solution, was launched in 2015. Prebid is the product of the joint effort of three major advertising companies — AppNexus, Rubicon Project, and PubMatic. Prebid made it possible to set a standard for working with HB that simply didn’t exist before and create a large community around the product.
Prebid is developing three products — a web product and products for server and mobile platforms:
Prebid Mobile is a mobile SDK (Software Development Kit) that works on the header bidding principle just like the other solutions. It can be used to receive ads directly from or in conjunction with the Prebid Server. This SDK is implemented in the most popular mobile operating systems: iOS and Android.
We will focus on the Prebid Server. With this solution, you send a request to the server, and the server, in turn, requests creatives from all selected partners.
Why do we need our own Prebid server?
Let’s see what options are available for working with Prebid Server:
Renting the Prebid Server There is a comprehensive list of providers on the Prebid website, each offering different terms of a partnership. Until recently, we used the services of one of these companies.
Creating your own Prebid Server. Advertising is the primary source of our income, so we want to have maximum control over the server, the process of receiving ads, and security.
You can use the Prebid Server to flexibly configure your work with partners, and you don’t need to modify the client-side code. If you have the necessary expertise and resources, hosting your own server is more cost-efficient in the long run despite the possible maintenance challenges.
Previously, we were limited to improving and optimizing the Prebid experience on the client-side only, but with our own Prebid Server, we can make various improvements to both parts of the system, increasing their effectiveness many times over.
An important aspect here is that now we provide the service infrastructure by ourselves, which means we can ensure the same high standards of support and stability that we apply to all of our other services.
Now, let’s talk about the technical aspects of this solution.
The main components of the Prebid server-side solution are Prebid Server, Prebid Cache Server, and the NoSQL database for data storage.
The tasks of each component are as follows:
Processes requests from the client, send them to selected partner ad networks, and save bids via the Prebid Cache Server to the NoSQL DB (Ads Creative DB).
Prebid Cache Server
Saves results received from the Prebid Server to the NoSQL DB and outputs them on user request.
Ads Creative DB
Stores results based on the configured Time to live (TTL) parameter.
The deployment and testing of our server were split into several stages:
Let’s go through all of them.
We looked at the available Prebid solutions. In short, there are two server-side solutions: Go- and Java-based. We chose the Java implementation because the technology stack used in the Prebid Java Server and Cache was more familiar to us.
Next, we deployed a cluster of multiple servers and caches with a load balancer. We analyzed how a failure of a particular node would affect application performance, checking each node (balancer, server, cache, database, bidders) and conducted a series of load tests for a single machine and cluster. To test the server, we set up services on an independent device to simulate the response of real bidders. It was necessary not to launch a DDOS attack on an advertising network.
We tested the following parameters:
The only element of the system we cannot control is external bidders. We had to ensure maximum reliability and uptime for the rest of the features because a complete failure of any of the nodes (i.e., if an entire cluster fails) would make it impossible to get creatives.
We implemented a mechanism for fast switching the users from the partner server to our own Prebid server to conduct client testing. During this period, some users were switched to our servers and received ads through them for a few days. We, in turn, collected real-time advertising statistics and data on the performance of both the partner server and our own Prebid server. We analyzed statistics from both servers, and the results were the same, which confirmed the correctness of our configuration.
Then, we had to move the servers from AWS to the data center and run another series of tests with a higher percentage of users. The data acquired from the tests allowed us to forecast the RPC, CPU, etc., load quite well. As a result, the maximum percentage of users in the trial was 35% on Android devices and 45% on iOS devices. The server load reached 35k RPS (Requests Per Second).
To manage the configuration of requests to partner ad networks, we created our own service with a frontend to edit, save and delete stored requests.
A few words about what a stored request is and why you need it:
Technically, we could hardcode all the necessary parameters for receiving ads from partners in the client — banner sizes, ad types and networks themselves. But this would increase the complexity of development and make our work with ad networks less flexible.
To avoid this, the Prebid Server has a stored request functionality. It allows you to save chunks of configurations in the database and use only the identifier of this configuration on the client.
So, when the Prebid Server receives a request with a stored request ID, it requests it via HTTP from the administration service and caches it for the selected time.
To reduce hardware costs, we did component profiling and chose several areas where we could try to improve performance:
After testing all the hypotheses, we improved performance by ~10%.
Our system consists of separate Prebid Server and Prebid Server Cache clusters, an administration system, NoSQL storage for creatives, and long-term storage for winning bids.
Long-term storage is necessary if any complaints regarding creatives arise, and further investigation is required. The storage time is configurable.
We constantly collect and analyze various technical metrics from system components to ensure stable operation and timely response to problems. We run health checks and monitor resource consumption, response delays, and the number of different types of errors.
All this data is displayed in real-time in monitoring systems, and automatic alerts are set up for the most critical metrics.
Very few technical improvements are left — we are now completing an audit of system security and finalizing the administration tools to make handling stored requests easier.
To improve the efficiency of our advertising campaigns, it is essential to monitor the Prebid server operation in real-time. Obviously, technical monitoring is crucial in the deployment of such high-load services. In addition to technical monitoring, we also consider it essential to monitor product metrics that indicate the quality of the execution of business processes.
So we’ve defined a set of metrics to monitor the operational quality of components of our own deployed infrastructure and our partners’ infrastructure. We also configured alerts for critical events to rapidly respond to emerging issues.
The selected metrics are classified based on the following attributes:
Degree of integration. Metrics indicate the operational quality:
Source of the events:
A bidder is a provider of advertising. With Prebid, the advertisement is returned in the form of a bid. The bids of different bidders participate in the auction on the server. The winning bid is returned to the publisher. The publisher loads the ad into one of its placements based on the bid. This means that we can indirectly influence an individual bidder’s quality. One way or another, we think it is crucial to integrate monitoring here as well. Firstly, to let a partner know that there is a problem if they don’t already know about it. Secondly, to reconfigure the list of enabled bidders or their settings if issues with a particular bidder affect the performance or other characteristics already implemented in our infrastructure.
Types of available monitoring metrics and what they indicate:
Number of bid requests to the bidder
The number of bids returned from bidders
Bidder fill rate
Bidder request errors, the average time to get bids from the bidder
The efficiency of requests to the bidder (auction win-rate)
Showing ads in the dimension by the bidder on the client
Bidder ad creatives errors, the average time to get bids from the bidder
We faced some limitations when implementing alerting in our internal infrastructure. It’s impossible, for example, to get reliable information on admin panel problems via product metrics. You can figure out indirectly that the connection with the admin panel is lost when the configuration is invalidated. In this case, requests to bidders will just not be sent. This is why we rely primarily on technical alerts, but product metrics can also be useful when conducting investigations.
If the metrics were collected from the server in the previous cases, the main source of events is the client analytics.
Rate is the number of requests to the service per unit of time. The fill rate is the percentage of requests served by the advertising partner (relative to their total number).
The server’s total number of bid requests does not change much if seasonality is taken into account and DAU is maintained. The same goes for the fill rate unless individual bidders are added or disabled. In this case, the fill rate can be calculated by the placement in the application, not by the bidder. Metrics indicate connectivity to the server and the correctness of integration with the clients.
These metrics are calculated similarly to the previous section but for the server rather than for individual bidders.
Metrics on creatives indicate cache availability and ad traffic quality because issues related to receiving the creatives are often localized within individual companies.
The advertising infrastructure is critical to the operation of FUNCORP services. We’ve always paid a lot of attention to Prebid, one of the main header bidding auction mechanisms for our apps. The decision to support the client and the server-side of Prebid increases the responsibility manifold.
Therefore, in addition to the effort we’ve already made, we are going to develop our own support for Prebid infrastructure in the following areas:
We’ll keep you informed about all the changes and improvements in these areas, as well as about the development of our Prebid service. Follow us on Medium for updates!