Do IPs have personality?

Looking for evidence of what type of attacker do we have

AI in the news... but attacks are nothing new!

There's been quite a lot on the news recently about AI agents (or robots) attacking websites. Some years ago our office received about 1.5 million attacks over a weekend. For context, we were a very small NGO and we repelled the attackers. A banking services company across the road wasn't so lucky and it cost them tens of thousands of Euros. Another occasion some years back one of our servers was attacked, enough that it started to overheat!

We don't enjoy attacks or indeed over zealous crawlers, so we developed a system called Pynblok

Repeated bahaviour

As we are collecting data and at the same time blocking attackers, we are noticing dimensions of repeated behaviour:

Not simple but multidimensional

The first thing we realised was that the characteristics were multidimensional, which also meant it was all too easy to get caught in the trap of complexity.  Let's give you an example. Over a 10 second burst one attacker identified by it's IP made 221 attacks; 22 attacks per second. Here's a few of them:

Now this was the same IP, so it's the same attacker most likely, and in this case all of the attacks were looking for password credentials which many people store in what are called 'environmental variables'. One would be very unwise to do so but many people apparently do! The interesting thing about the list to is the 'User-Agent'. Notice how each different attack is using a different apparent 'User-Agent'. Of all the 221 attacks in that burst almost all of them are different. Many pretending to be legitimate bots like Google or Bytespider or Perplexity. 

So for that IP we might say

So we might have called that the The Opportunistic Reconnaissance Bot. Another IP might emerge as The Persistent Specialist or the The Noisy Brute. But what we wanted with Pynblok was not twenty five attributes which would have been difficult to remember or understand but to collapse the presentation onto two axes: Horizontally we made that narrow to broad and vertically low activity to high activity. 

Two dimensions on an axis

Looking at the horizontal axis we go from focus on the left to opportunism on the right. Focused IPs repeatedly attack using the same technology, consistent target using specialised exploits or probes. Opportunistic IPs try everything, look for passwords, look for Wordpress exploits, PHP exploits etc etc then follows whatever looks potentially exploitable.

Looking at the vertical axis we go from noisy to stealthy. Sometimes the stealthy is not a single IP but a cluster of IPs working in concert to try to get through what otherwise might be blocked. Noisy IPs run huge bursts with lots of requests, are obviously scanning and have short attack campaigns. When they try against version 2 of Pynblok we try to cut them off within 3 seconds.

Stealthy attacks work on low frequency often with long intervals between attacks, which can be minutes or hours between and have carefully distributed requests which are harder to notice.

Hence the example that I cited earlier would be categorised as a Scanner: High-volume, semi broad-spectrum reconnaissance. Like human personality tests the four quadrants aren't a measure of badness. Unlike human tests, in our measure they are all bad. They are all attackers! What they help us do is describe how the IP behaves. 

It also helps us evaluate how we respond to that IP personality. That part I won't describe as it would facilitate attackers to optimise their attacks against a Pynblok protected server. However, the Pynblok system doesn't use external AI nor is it externally accessible from the web.

Crawlers

Crawlers (bots from AI engines or search engines) are different. One wants search engines to crawl one's site so that people can find it. AI is potentially different, some people want to block them, some people encourage them. However, very aggressive crawlers can put unreasonable load on your server. There are also loads of fake crawlers pretending to be Google or Bing or whatever. In Pynblok we added processing to identify real vs fake crawlers.

So in the two axes for crawlers we had similarities with attackers: horizontally we had focused vs exploratory. The focused crawler concentrates on a small set of known URL types or pages, whereas the exploratory, discovers broadly and follows lots of different paths.

The vertical axis is gentle vs aggressive. Gentle crawlers have a low request rate and respect normal site behaviour. Aggressive crawlers have a high request rate with lots of parallel requests and little regard for load.

This table gives you a clue as to how variable the bots are at crawling sites. 

Obviously OpenAI's GPTBot is by a long way the most aggressive followed by Meta's Facebook AI crawler and Anthropic's ClaudeBot. Identifying and verifying bots is almost more art form than science!

Different sites will have very different patterns, this is purely an example from one server, not something to use as common to all sites on the Internet. Those figures are just for one server over a couple of weeks. 

Names for the crawler personalities

So, in consequence we called those four personalities The Harvester, The Scraper, The Librarian and The Explorer.

The aim was to answer two questions:

What type of attacker are you?

What type of crawler are you?

Every IP has a personality. It's not just an address, it's behaviour. Finding out their behaviour helps us to know how to react to them.

And that is what Pynblok is about... blocking the pain of attackers and unfriendly crawlers!