Are you a good bot or a bad bot? We’re talking to you: the majority of internet traffic now originating from bots.
You can’t respond? Aren’t you a bot?
The latest report from web security and research firm Imperva suggests that – once again – over half the internet traffic today comes from bots. More formally and completely known as web robots the ubiquitous creatures are software which run automated tasks over the internet. Some are very useful and increased productivity because they can perform tasks much more quickly than can humans. Many bots are really destructive and dangerous.
The Imperva report for 2016 finds that humans produced 48.2 percent of all internet traffic while bots – good and bad – account for 51.8 percent of all internet traffic. Of that majority bot traffic, says the Imperva report, 28.9 percent is generated by bad bots while 22.9 percent is generated by good bots.
The good news is good bots traffic is up. The bad news is that generally speaking and according to the Imperva report every third visitor to any website is an attack bot.
None of this is new phenomena. But it does signal a resurgence in bot traffic.
“In 2015 we documented a downward shift in bot activity on our network, resulting in a drop below the 50 percent line for the first time in years,” explains Imperva. “In 2016 we witnessed a correction of that trend, with bot traffic scaling back to 51.8 percent—only slightly higher than what it was in 2012.”
The good news is the increase is bot traffic comes from an increase in good bot activity, according to Imperva.
“In 2016 we tracked 504 unique good bots—278 of which were active enough to generate at least 1,000 daily visits to our network,” explains the report. “Of these, 57.2 percent displayed an increase in activity, while only 29.4 percent saw their activity decrease year-over-year.”
The bad bot traffic, says Imperva, has remained constant around the 30 percent mark over the past five years.
And the bad bots tend to go after websites will very little overall activity.
“For instance, on the least trafficked domains—those frequented by ten human visitors a day or less—bad bots accounted for 47.7 percent of visits while total bot traffic amounted to 93.3 percent,” says the Imperva report. “Simply put, good bots will crawl your website and bad bots will try to hack it regardless of how popular it is with the human folk. They will even keep visiting a domain in absence of all human traffic.”
Okay, so what’s a good bot and what’s a bad bot?
The most common form of a bad bot is what’s called an impersonator bot.
“Impersonators are attack bots masking themselves as legitimate visitors so as to circumvent security solutions,” Imperva explains. “Clearly such bypass capabilities complement all malicious activities and that makes impersonators the ‘weapon of choice’ for the majority of automated attacks. More so, because a rudimentary level of obfuscation is relatively easy to achieve.
“For the fifth year in a row, impersonator bots continue to be the most active offenders. In 2016 they were responsible for 24.3 percent of all traffic on our network and 84 percent of all bad bot attacks against Incapsula-protected domains.”
Impersonator are most frequently used in Denial of Service Attacks – or DDOS. These are the type of attack one hears about most frequently in the news. DDOS attacks use the bot to send traffic “hits” to websites at a rate of thousands per second, overwhelming the site’s server and crashing the site.
Good bots, on the other hand, help make the internet work more productively. Google Search Bots are examples of good bots. Imperva puts good bots into four overall categories:
- Feed fetcher – Bots that ferry website content to mobile and web applications, which they then display to users.
- Search engine bots – Bots that collect information for search engine algorithms, which is then used to make ranking decisions.
- Commercial crawlers – Spiders used for authorized data extractions, usually on behalf of digital marketing tools.
- Monitoring bots – Bots that monitor website availability and the proper functioning of various online features.
But the work of the evil bots never ends.
“The most alarming statistic in this report is also the most persistent trend it observes: for the past five years every third visitor website visitor was an attack bot,” Imperva says.
“The implications of this trend are felt by many digital business owners, the majority of whom are facing non-human attackers on a regular basis. Specifically, out of 100,000 domains in this survey, 94.2 percent experienced at least one bot attack over the 90 day period.”