Richard Kahn
Bot detection is the process of finding and identifying bots or non-human, automated programs designed to carry out tasks. Detection in theory is simple, look for data points from your website or on the visitor’s side that indicate if a visitor is human or not. At its core that’s all bot detection is; does this visitor display signals that indicate it’s not a real human. However, the execution of this becomes extremely difficult because bot detection is like shooting a moving target while standing on a moving platform. How is it a moving target? Bot infiltration methods and automations are always changing. As soon as one method of tricking a system is identified, fraudsters and programs evolve in a new way and give off different data points. How is it like standing on a moving platform? The internet itself is constantly changing. Browsers and operating systems, the things that give off all of the data, are constantly changing as well, making it harder to home in on which data points are most important in identification.
Bots fall under two basic categories: GIVT (General Invalid Traffic) and SIVT (Sophisticated Invalid Traffic).
The difference between GIVT & SIVT comes down to the difficulty to detect. Think of GIVT as “easy to identify” bots. These can be both good and bad bots. An example of a good bot would be an SEO crawler, which is a bot google uses to look through your web pages and determine things like schema, load speeds, and other indicators that show how accessible and easy your site is to navigate. These good bots usually identify themselves in the data they give when landing on a page or site. However, there are some good bots that don’t identify themselves specifically but still are good. Microsoft Azure bots don’t announce themselves, but they are still easily detected because the information they pass along are consistent data points. GIVT can also be malicious, but still easy to detect.
SIVT is where bot detection becomes much harder. Sophisticated Invalid Traffic is harder to detect because the signals on both ends of the spectrum, the visitor and the behavior, can be gamed or be randomized in a way where the patterns are harder to see. This is complicated further by the advent of AI, which has automated many processes that used to be manual. This has made it easier to bypass existing systems.
The old process for a fraudster writing bot scripts would look like this:
Manually Write Code → Test → Fail
Manually Write Code → Test → Fail
Manually Write Code → Test → Pass
Manually Write Deployment → Exploit
Now the process looks like this:
Write Script → Run 100’s of Tests with variation → Fail → Run 100’s of Tests with variation → Fail → Run Successful Test → Pass → Exploit
AI allows fraudsters and hackers to validate and test at vastly accelerated rates that before would have been impossible without huge quantities of developers. Now one person with one computer can exploit processes in real-time, 24/7.
What does SIVT Look Like?
Cookie Injection
Cookie injection is the process of artificially adding a cookie, or a way to track where a visitors source, artificially to falsify where a transaction or submission came from. A widely publicized example of this is the Phia debacle with Phoebe Gates. Phia was a startup that tracked affiliate links to assign credit and payouts to publishers that sent traffic and drove sales to websites. Internally, Phia was fraudulently adding cookies to purchases where visitors came from other affiliates (non-Phia affiliates) or organically found the purchase on their own, to take credit for purchases and receive payouts without real attribution. For advertisers and retailers, cookie injection provides fake data that obscures what channels are really driving your conversions, leading to decisions that waste time and money on channels that don’t actually convert, effectively wasting your spend.
Residential Proxies
Residential Proxies are proxy servers that route internet traffic through legitimate IP addresses, usually through malware if happening for malicious purpose. There was a real world example of this on Samsung TV’s. Samsung had an app on their marketplace that was loaded with malicious malware. When a Smart TV owner downloaded the app, they unknowingly downloaded malware that allowed fraudsters and bots to use the owners IP address and devices as their environmental data (see below), making it look like they were originating from legitimate IP’s instead of data centers or bot farms. Residential proxies can make bot identification much harder, which is why you need a wide array of data points for accurate bot detection.
AI Enabled Bots
AI enabled bots are scripts written and edited by AI to perform tasks that require human interaction. Scripts used to be written manually, which made their patterns and behavior easier to detect. With the advent of AI, bots are being written and are adapting in real time to bot detection strategy. These bots are the hardest to identify, especially when relying on behavioral analysis or captcha challenges alone. (Anura does not use behavioral analysis because of its weaknesses).
How Bot Detection Works: Catching the Scripts
Identifying bots accurately can be done in lots of different ways, ranked here in order of effectiveness: environmental analysis, fingerprinting, challenges, behavioral analysis, and honeypots IP analysis.
Environmental Analysis
Environmental analysis is the process of collecting data from the visitor’s environmental setup. This can be as simple as a geolocation, or as complicated as seeing if the user has dev tools on while browsing a website (which can be an indicator of malicious activity, but not always). Think of environmental analysis as the visitor’s information being collected. More examples of environmental analysis could be device characteristics, operating systems, browser information, IP address and browser setup (automated or headless browsers could give off unique data points). Environmental analyses is important for bot detection because it takes multiple types of data into account when considering whether or not a visitor is a bot.
Fingerprinting
Fingerprinting is simply taking a lot of information and converting it to a string of data that is unique to that visitor. Think of it like a digital name tag. An example would be a visitor hits your site, the visitor’s user environment tells you 4 data points: their location, IP address, browser, and browser version. Those 4 data points would be converted from a long line of code into a unique sequence of numbers and letters that is now assigned to that visitor. This “fingerprint” is then used to track a visitor while keeping tabs on their information. The weakness of fingerprinting alone is that hackers and bots can “spoof” pieces of their information to fake or change their fingerprint. If you change one data point, like your IP address in the example, you will generate an entirely new fingerprint.
Verification Challenges
Verification challenges are “tests” that challenge digital visitors to see how they react. CAPTCHA are the well-known versions of this, which challenge a visitor to solve a non-structured puzzle like re-typing squiggly text or identifying all the pictures with part of a fire hydrant in them. Challenges can also be code on a page that sends a request to a visitor and analyzes the response to determine if the visitor is a bot. Verification challenges alone are not great at detecting bots but are an important piece of the greater process of bot identification.
Behavioral Analysis
This is the process of collecting data on a user's behavior on your website. An example of this would be a visitor visits a website and clicks through a massive number of pages and links at inhuman speed, which could be an indicator that they are a bot. Behavioral analysis is not always a great indicator for bot detection and identification because it requires a ton of data and patterns to be accurate, and with the proliferation of AI enabled fraud, bots are increasingly better at mimicking human behavior which trick normal behavioral analysis (Again, Anura does not use behavioral analysis).
HoneyPots
Honeypots are fields that trick bots into interacting with them that are usually hidden from real users. A good example of a honeypot would be putting a hidden checkbox on your form that doesn’t appear on the page, so real visitors would be able to interact with it, but bots would see it in the code and check it. If the checkbox is ticked, that form submission in theory was a bot. While this sounds fool proof, there are ways that bots can detect honeypots and get around them. While honeypots will catch basic bots, more sophisticated automations can find ways around them.
IP Analysis
IP analysis is the act of analyzing the IP and network information of a user to determine where they are coming from, whether that be a residence, business, data center or from a IP with a poor reputation. IP analysis is the most basic detection and can be used to detect bots when it comes to known IP reputations. If traffic comes from a known fraudulent IP, then you can safely assume, for a certain period, that the traffic is illegitimate. IP’s rotate, so illegitimate IP’s can be reassigned to legitimate users, and some IP addresses are shared (think about Starbucks, a lot of new people can come from the same IP with varying intention) making the practice of IP blocking alone a poor indicator for advanced bot identification.
The Goal of Bot Detection
Again, the goal here is to determine whether each visitor is Human or not in the most accurate way possible, and the emphasis for good bot detection software is to allow real humans through and stop 100% of the bots. This binary determination is what matters most, especially for digital advertisers who rely on conversions from real people to get paid. Knowing which bots are on your website and stopping the fraudulent ones is obviously important, but the real value is in making sure real people aren’t blocked. Additionally, just because a bot is on your site does not mean they need to be blocked, so having a solution that understands what type of bot you’re dealing with is crucial. To determine your best bot detection use case, check to see if the vendor you’re looking at offer a no obligation audit that allows you to understand the amount of bots and the types of bots that you are dealing with.
