Prevent and Block Bots | Detect Non-Human Traffic
Real-Time Bot Detection
Machine Learning Bot Detection Adapts to Your Traffic to Prevent Non-Human Behavior
Cybercriminals, fraudsters and even competitors use bots to do their evil bidding. Bots can carry out specific actions such as generating duplicate accounts, engaging in user generated content SPAM, creating fraudulent transactions, or breaking into existing user accounts. Bots and non-human traffic are even responsible for billions of dollars per year in advertisement & click fraud. Bot Detection sounds complicated, but IPQS has made it just another challenge easily solved by our anti-fraud tools.
Filtering bad actions from your site in real-time without penalizing your legitimate users can be a difficult task. Captchas are frustrating for even the most internet savvy humans and can easily frustrate valid users. However, proactively filtering bots doesn’t need to be difficult. IPQualityScore detects bots and non-human traffic from botnets & malware, automated scripts, fake click programs, compromised devices, invalid browsers, and other poor quality traffic sources. Let us show you how simple bot detection can be. Deploy our tools to any site or app in just a few minutes to block bot traffic in real-time and instantly improve the quality of your traffic, users, and transactions.
Bot Detection API & Bot Detection Tools
Industry Leading Bot Detection Tools To Detect Bot Traffic
Easily deploy the IPQS bot detection API to quickly mitigate bot traffic with precision accuracy. Identify even the most sophisticated bots which can bypass Google recaptcha and similar captcha services. IPQS can prevent bots before unwanted traffic reaches your website at any point during a user path, such as initiating the bot check before checkout, login, or registration. Quickly deploy our bot detection tools on your website with our easy-to-use bot detection API in JSON or XML format with low-latency response times.
Mitigate & detect even the most sophisticated bots that mimic human behavior. Prevent scraping, account login abuse, fraudulent accounts & purchases, and other types of automated fraudulent behavior.
Precise Bot Detection
How Do We Detect Bots?
IPQS scores hundreds of millions of transactions and user events per day. We leverage this cyber threat network to identify devices that have recently engaged in malicious behavior and detect devices that exhibit patterns of known fraud. Our bot detection tools feature over 10 years of technology in bot protection solutions. A mix of device fingerprinting, browser verification, forensic analysis, behavior monitoring, machine learning, and IP reputation is performed on the user to determine if they fit the profile of a fraudulent user or non-human bot. IPQS detection methods are always evolving to mitigate any threats from new bots and emerging fraud rings. Completely stop bots using our suite of bot detection tools.
Additionally, create custom rules to force scoring to be more aggressive or more lenient for specific tests that our anti-fraud tools perform. Adaptive filters learn from your audience and user base to avoid flagging legitimate traffic and easily identifying irregular behavior generated by bots. When a bot is detected, your site or app has the flexibility to perform any custom behavior, such as blocking the user or transaction, setting its status for further review, or automatically limiting the user all in real-time. Our flexible APIs provide over 20 data points to define a risk profile, so your platform can make the best decisions.
Expert Bot Mitigation
Mitigate bot traffic without hurting the user experience for legitimate users. IPQS solutions are perfect for bot protection with seamless integration in the background. Our tools do not display any loading messages or delay the page’s display in any way. Frictionless filtering provides a smooth user experience for your audience while removing malicious bots and clicks from suspicious web traffic. Keep your business protected from the web’s worst bad actors by adding just a few lines of code to your site.
Quick Integration for Bot Detection Tools
Fraudsters tend to pray on easy victims. Once you begin to mitigate bots, cybercriminals typically move on to easier targets. Implementing IPQS’s bot detection service only takes a few minutes and will instantly block bots. Automate quality control and regulate fraudulent accounts & transactions in real-time. Increase the integrity and security of your website by minimizing fraudulent behavior, fake traffic, and non-human visitors.
View available IPQS fraud detection plugins that make it easy to enable bot management and mitigate bot traffic for over 2, 000 apps.
What is bot traffic? How to stop unwanted bot traffic to your website.
Bot traffic is internet traffic coming from automated software that is designed to perform repetitive, mostly simple tasks. These bots, the automated software, can perform these tasks around the clock, and often much quicker than any human ever could.
Around half of all internet traffic comes from web bots. While there are good bots that can be beneficial for your website, approximately 30 percent of all traffic comes from bad bots. These bots are designed to perform all sorts of malicious tasks, from scraping and stealing web content to usurping user accounts and scalping inventory.
Even when bot attacks are unsuccessful in executing their malicious objectives, they can still strain your web servers and hurt your website’s performance, in some cases to the point of making the website unavailable for human visitors. Effective management of bot traffic is therefore very important for any business with an online presence—but as we will see, this is not an easy task.
To really understand what bot traffic is and how to effectively manage it, let’s first explore these two different types of bots available.
The different types of web bot traffic
Web bot traffic can be divided into three broad categories:
In managing bot traffic, recognizing that there are good bots that are only trying to help is very important. Good bots are, in fact, primordial to the success and performance of your site.
Here are two typical examples of good bots:
Search engine bots
The most important type of good bots is the crawler bots owned and operated by Google, Bing, Baidu, Yandex, and other search engines. Their task is fairly obvious: they constantly crawl the internet to find the content that they will show to people in search of information. Search engine bots help you get your website in front of potential buyers, and you definitely want their traffic.
These bots are sent by various third-party service providers you use. For example, if you use SEO tools like Ahrefs or SEMRush, they will use their bots to crawl your site to check your SEO performance (link profile, traffic volume, etc. ). Performance measurement tools such as Pingdom also fall in this category. Like search engine bots, partner bots render useful services. However, on certain occasions (such as a major sales event with a significant traffic spike), you may want to limit the number of requests they are allowed to make to your website in order to optimize the performance for human visitors.
The DataDome bot management solution classifies “commercial bots” in a separate category. These bots are operated by legitimate companies, typically for collecting and exploiting online content. They are mostly honest about their identity, but they may or may not be beneficial to your business. Commercial bot traffic will also drain your server resources and impact your website performance.
Here are some examples:
These bots crawl websites to find attractive and relevant content to be featured on aggregator sites and platforms. They can help promote your content and amplify its reach, but most website owners prefer to control which aggregator bots can access their content and at what rates.
Price comparison bots
Price comparison bots are similar to aggregator bots, but instead of online content they are looking for prices. For example, a flight comparison website may use scraper bots to scan the websites of different airlines and compile the prices in a comparison tool. Price comparison bots can help get your offers in front of more buyers, but most website owners prefer to work with approved comparison partners who get their data from a pricing feed.
These bots crawl the internet to search for copyrighted images, videos, and other content to ensure nobody is illegally using this content without permission.
Both good bots and commercial bots generally meet the following three main criteria:
They come from well-known, legitimate sources (Google, Bing, etc. ), and they are transparent about the owner/operator of the bot
They perform mostly beneficial tasks
They will follow the rules and policies in your file.
Unlike good bots, bad (malicious) bots won’t follow your rules. They also tend to hide their identity and source, and often try to pose themselves as legitimate human users.
The main thing differentiating these bad bots from good bots, however, is the type of tasks they perform: bad bots are programmed with malicious intent, to perform disruptive and even destructive tasks. Bad bots can cause a lot of (permanent) damage when left unchecked.
Here are some common examples:
Web scraping bots
These bots steal content and information on your site and then publish or sell in on other sites. This can create content duplication issues, among other problems. The bot, for example, can steal private price information regarding your products and release them to your competitors, so you lose your competitive advantage. This is pretty common in websites/businesses that display prices and where the price is a very important purchase decision factor, such as ticketing websites, travel agents, etc.
Credential stuffing bots
These bots use stolen credentials (typically sourced from data breaches) to “stuff” known usernames and passwords into the login pages on other sites. The purpose is to gain access to (and abuse) user accounts. Because people tend to use the same username-password combination for all their accounts, they often have a quite high success rate.
Read more: Behind the scenes of a massively distributed credential stuffing attack
These bots post spam content or send spam emails in bulk, often including links to fraudulent websites. We commonly see these bots leaving comments on blogs, social media posts, and forums, among other mediums.
Ad fraud bots
These bots click on PPC ads to generate extra revenue or skew the cost of the ad. As a result, the advertiser is charged with high advertising fees for a campaign that is not effective.
Denial of Service (DoS) bots
In Layer 7 DDoS attacks, bots are making repeated requests to resource-hungry elements of a web application, such as large file downloads or form submissions. This causes slowdowns in performance, or even complete downtime.
Credit card fraud bots
These bots are used to test combinations of credit card information in order to identify missing data like CVV code, expiry date, etc. by trying small transactions repeatedly. Their activity might cause chargebacks for the e-commerce site, and might damage the fraud score of the business.
Gift card fraud bots
These bad bots steal money from gift card accounts, which can lead to damages in reputation and loss of future revenue.
To summarize, bad bot activities may cause:
Sudden spikes and abnormal increases of page views
Higher bandwidth usage
Skewed Google Analytics reports and other KPIs, which may lead to business decisions based on inaccurate data
Lower conversion rates
Poor website performance
Increased strain on data centers and higher server costs
How to identify bot traffic
To manage bot traffic, it must first be correctly identified. Here are a few things to look out for in your traffic and business metrics.
Increase in traffic and bounce rate
If you notice a sudden rise in both traffic volume and bounce rate at the same time, it is a strong indication of bad bot traffic. An abnormal increase in traffic usually means a high number of bots coming to your site, or a single bot repeatedly coming to your site again and again. The increase in bounce rate indicates that the bot just leaves without exploring more pages after it has fulfilled its task.
Page load speed
A dramatic fall in page-load speed—especially if you haven’t made any significant changes to your website— is a telltale sign of bad bot traffic. Although bot traffic is not the only possible reason for slower site performance, it’s an indication that you should take a closer look at your other KPIs.
While one single bot is unlikely to make a significant impact on your site’s overall speed, many malicious activities involve a lot of bots entering a website at the same time, like in the event of Layer 7 DDoS attacks.
An abnormal decrease in bounce rate
Another important factor to check is when your bounce rate dips to a suspiciously low level. This is a strong indicator of web scraping bots stealing your content, scanning a very large number of pages.
This one might be more difficult to measure right away, but when web scraping bots steal your content and publish it on other sites, it might impact your site’s SERP ranking in the long run.
Not only there’s a chance that your site might be outranked by the site dubiously publishing your content, but your site might also get penalized by Google for duplication issues. Make sure to always set up canonical tags on every blog post so your article is always considered canonical even when your content is stolen.
Customer complaints about unavailable goods
If your customers repeatedly complain that they’re unable to purchase the products they want from your site, you may be the victim of scalper bots. These bots are designed for ultra-fast online purchasing, and can be a cause of great frustration for real customers who are unable to beat them to the checkout page.
How to stop bot traffic
So you’ve checked your stats and come up with the verdict: you definitely have a bot traffic problem. What now?
Although our main focus will be to stop bad bot traffic, we also need to manage traffic from good and commercial bots. Not all good and commercial bots might be useful for your site, and while they won’t deliberately hurt your site, they might strain your site’s performance with unnecessary traffic. Also, properly managing these good bots will also help us in differentiating them from bad bots.
Managing the good bots
Thankfully, since good bots are open about their identity and are mostly willing to be managed, managing their traffic should be fairly easy. There are two main approaches we can use:
The main approach in managing good bots is to set up rules and policies in your file. The basic principle is to allow the good bots that are going to benefit your site and block those who might not help your site at all. For example, if you don’t serve the Chinese market and there’s no Chinese language version of your site, there’s no need for Baidu’s bots to crawl your site and you may want to block them.
You can follow this guide on how to manage your
If you have a bot management solution, the other approach is to set up a blacklist and/or whitelist (allowlist). We can, for example, set up a white list of what kinds of good bots are allowed to roam our site, if we are 100% sure that these are the only bots that are going to be beneficial for our site. A good bot management solution should also let you manage good bot traffic with features such as rate limiting or timeboxing, so you can allow access on your own terms.
Managing the bad bots
In managing and stopping bad bot traffic, here are several different approaches we can try:
Investing in a good bod management solution
With how bad bots are becoming more advanced and adept at imitating human behaviors, an advanced bot management solution is required. Bots may even use AI and machine learning technologies to achieve their tasks and to mask their identity, and so an AI-based bot management solution like DataDome is now a necessity.
DataDome performs real-time, behavioral-based bot detection to effectively identify even the most sophisticated bots, which can forge their user agent (UA) and rotate between hundreds if not thousands of perfectly clean IP addresses.
A lot of these bot management solutions are now fairly affordable and easy to use, so if you are serious about your cybersecurity, investing in a proper bot detection and mitigation solution is a must.
A basic approach to stop bot traffic is to use Captchas. Services like Google’s reCaptcha are easy to implement, but we shouldn’t think of Captchas as a one-size-fits-all answer to bot management. There are two reasons for this:
Using too many Captchas on your site can ruin the user experience and increase your site’s bounce rate
Not only are today’s bots getting better, but there are also various Captcha farm services available for hackers where humans will solve the Captcha before passing it to the bot.
So, think of Captchas as prerequisite protection, and not the final answer to your bot management strategy.
Using a WAF (Web Application Firewall)
Another common solution for stopping bot traffic is using a WAF. A WAF, in a nutshell, is a firewall (shield) placed between a web application (or web page) and the client. Traffic and resources first go to the WAF before they are sent to the client. We can think of a WAF as a reverse proxy server.
A WAF can be useful for protecting applications against the most common types of attacks, and may block a part of your unwanted bot traffic. However, WAFs are designed for application protection—not bot detection—and are powerless against sophisticated bots that actively try to circumvent your security solutions.
Although today’s bots typically use vast numbers of different IP addresses, making IP-based protection rather ineffective, we can still implement the practice of blocking IP addresses that are obvious sources of malicious bots. Be very careful, however, in the case of blocking public IPs, since you could be blocking legitimate users as well.
Stricter access controls
On sensitive areas of your website (i. e. areas intended for admins) where users can access your database), you can implement stricter access controls like requiring multi-factor authentication (MFA). This can be effective in stopping bot traffic that is performing credential stuffing attacks and other malicious activities.
The idea is that, in the event that a bot has successfully cracked your site with brute force and/or credential stuffing attacks, it won’t be able to access your network fully, minimizing potential damage.
Unmanaged bot traffic can be very costly for any business with a website/online presence. Effectively identifying and stopping abusive bot traffic is therefore extremely important.
While there are various approaches we can use to mitigate bad bot traffic, investing in a specialized bot management solution remains the most effective one.
Since today’s most advanced bots are extensively using machine learning technologies, an AI-based bot management solution is preferred. The best bot management solutions leverage machine learning to analyze visitor behavior and stop malicious bots before they even reach your network.
How to Filter Bot Traffic Activity in Google Analytics – Codehouse
Have you recently observed suspicious increases or fluctuations in your analytics data or an increase in traffic from unknown sources?
Possible spam or bot traffic (bad bot) activities can cause this issue and it can impact your website data accuracy and Google Analytics (GA) results.
What is bot traffic?
Bot traffic is the name given to non-human traffic and traffic created by various spiders and programs.
This traffic can cause misleading results in your web page data and Google Analytics reports. It could also have an impact on overall business performance.
More importantly, it’s vital for your business to be proactive in detecting bot traffic, and then to filter out bot activity. This will help you make effective data driven decisions.
Detecting suspicious activity in GA
Do you review your web traffic and analytics results on a daily or weekly basis? Examining your analytics results regularly and knowing your web traffic and traffic sources can help you identify bot and spam traffic. You should also set up alerts that notify you automatically when a threshold is reached.
The screenshot below shows an increase in traffic for March. There are also two other spikes that aren’t in the normal flow of traffic distribution.
Was there a special campaign or an activity that would lead to an increase in web traffic in March? If not, and the traffic is not internally sourced, it can be concluded that this traffic may be generated by bots.
Before starting this research, it is very important to check whether the standard Google Analytics bot traffic and spider filter is selected in the Analytics settings or not. This option filters many bots that Google can identify as a bot or spider. To view this option, go to Admin and then View Settings in Google Analytics. Make sure that the Bot Filtering box is ticked in this section.
Three steps to identify bot traffic in GA
Step 1: In the Google Analytics Master View section, select Acquisition from the left side column and select All Traffic and then Channels. Click Referral in the Default Channel Grouping list. In this section, you can see various referral sourced visits and referral information.
Do you know all the listed sources? Examine them in detail by clicking the Show Rows option at the bottom of the page. Then browse the traffic distribution by choosing a wide date range.
In the referral information on this screenshot, we can see the visits from various sources. The traffic from a source called ” is an unknown source and looks suspicious.
Step 2: Examine the traffic sources in the source list you think is suspicious. Use different metrics such as Bounce Rate, Average Visit Duration, Sessions and Users.
If a 100% bounce rate has a visit duration close to 00:00:00, this traffic may be bot traffic.
Step 3: If there’s referrer spam that disguises itself, use the Secondary Dimension option to search for different options. For example, Hostname, Network Domain, City, Campaign.
Any website without your website’s hostname can be defined as bot traffic.
An example of some bot traffic sources is:
/Ashburn City bot traffic/
Enjoying this article? Sign up to our newsletter
Four options to filter bot traffic from GA
To filter bot traffic from your GA, consider four options:
Use the Referral Exclusion list
Use GA filters
Use cloud service offerings
1. Use the Referral Exclusion list
A popular way to exclude bot traffic referral sources from analytics results is the Referral Exclusion List. This is in the Admin section of GA.
You can filter any suspicious traffic or traffic coming from a bot traffic source from our results by using this feature.
However, this filtering doesn’t impact the previous results and website data. This feature filters referral from your results, starting from the date of application.
It’s also important to test the Referral Exclusion List under Testing View, or a view created for the purpose of preserving your current collection methods.
2. Use GA filters
Filter bot traffic by using Google Analytics’ existing filtering feature. This provides more advanced options and filtering capabilities compared to the Referral Exclusion List as you can create more detailed filters.
You can also create filters based on City, Network Domain, Hostname or Campaign Source by using the Filters section in the Admin section. Since some bot traffic doesn’t have referral information, you can use this section to create detailed filters.
The screenshot below is a bot filtering example. It filters “traffic” and “bot” keywords sources from analytics results. This was tested under Test View before applying it to the Master View.
3. Create segments
Bot or spam traffic in your GA results may cause problems when creating reports or making decisions about your business.
Get accurate results by creating Google Analytics Segments. This allows you to separate bot traffic from your previous data (data before the filter is applied).
In the example below, we created a segment to filter bot traffic originating from Ashburn from the web page results.
You too can create a segment like this one and use it when reporting your data retrospectively.
4. Use cloud service offerings
As another level of protection, you can use a bot protection service by different cloud services providers such as Azure, AWS Cloudflare or Barracuda.
For instance, Microsoft’s Azure Web Application Firewall can help you to filter potential bot traffic and spiders. Bots and spiders can be blocked by the firewall and you can also specify and block the specific bot traffic sources. Thus, they can’t use up the resources and services of your website and impact your Analytics results. Also, some services such as Amazon Bot Management feature can provide you Bot Analytics and give insights related to your website’s bot traffic activity.
Our Digital Experience team works with many customers to get the best from Google Analytics by creating insightful reports and dashboards that help influence decisions.
Learn more about analytics and data visualisation:
Cohort analysis and how to use it in Google Analytics
Sitecore SEO and how to optimise for Google, Yandex and Bing
What is Google Core Web Vitals?
How to set up Goal Funnel Visualisation
To find out more about how we can enable your business to get the very best from your website data, get in touch. We’re here to help your current digital marketing challenges.
Frequently Asked Questions about bot traffic detected
What does bot detected mean?
When a bot is detected, your site or app has the flexibility to perform any custom behavior, such as blocking the user or transaction, setting its status for further review, or automatically limiting the user all in real-time.
Why is my site getting bot traffic?
There are two reasons for this: Using too many Captchas on your site can ruin the user experience and increase your site’s bounce rate. Not only are today’s bots getting better, but there are also various Captcha farm services available for hackers where humans will solve the Captcha before passing it to the bot.
What is bot traffic on Google Analytics?
Bot traffic is the name given to non-human traffic and traffic created by various spiders and programs. This traffic can cause misleading results in your web page data and Google Analytics reports. It could also have an impact on overall business performance.Apr 27, 2021