As we step around the smoldering husks of systems from the holiday shopping season and those that were affected by Spectre and Meltdown madness, we can’t help but to look at the road that took us to this point. Of course I’m being somewhat tongue and cheek.  

Years ago I spent some time working in a record shop. It was a part time job while I took courses in university and worked in a local radio station. I was an aspiring bass player for a band, a radio DJ and I was working in a record shop. I’m not sure if I could have been more of a cliché had I tried.  

But back then I learned the ins and outs of retail business. I learned the types of customers who would come into the store to browse. The people that came in with a purpose to buy and then get out and, invariably, the kid that wanted to stuff a CD down the front of his pants and attempt to walk out in an inconspicuous manner.  

It got to a point where it was a trivial exercise to be able to recognize someone who was intent on purchasing as soon as they came through the door. This, by no means, meant we treated them any differently. More to the point it was just a skill that one developed over time as we analyzed the customer traffic coming into the store.  

After a couple of years, I was interviewed for a job with a record company. I was bursting at the seams at the prospect of realizing my dream. As we went through the interview questions I was handling them well until we got to the question: “Where do you see music retail going in 5 years?” Easy answer I thought. “Online sales will dominate future growth,” I said with some modicum of confidence.  

My heart sank. My interviewer blew up laughing. “That’s silly, people would have to buy a device that they could play the music on.” I felt like I had completely ruined the interview. Thankfully, this was not the case, but I did learn how the music retail space was diametrically opposed to the perceived shift to online markets at that point in time.  

This interview took place back in 1994. I wish I had realized how right I was about the shift to online retail. In June of 1999 the peer-to-peer file sharing program Napster was released for the first time. The rest quickly became historical lessons. The shift had occurred.  

Nowadays, it is commonplace to shop online for most things. I was able to do all of my Christmas shopping in my pajamas without ever leaving the sofa. Now, however, the landscape has changed but, a lot of what I learned from my time in retail remains. Customers behave much in the same manner as they do for the traditional bricks and mortar operations with the added variable of the robots, or rather, bots. 

These automated programs scan websites looking for competitive information, better deals (offering better prices on their site), or in some cases they’re just written so badly that they can potentially negatively impact your website. How do you manage that sort of traffic? Each bot has a purpose that may or may not be in line with your company goals. They might eat away at expensive resources causing you a financial impact. This doesn’t even take into account the negative actors on the Internet.  

So this begs the question, what are you doing for managing your bot traffic? You really need to be able to identify, categorize, and apply different behaviors to different bots based on your requirements. If you are running an airline you want to avoid bad booking attempts, whereas a hotel operator would love nothing more than to move all of their surplus inventory. 

You can lower your operating costs if you manage these bots so that a badly configured bot does not inadvertently cost you needless overhead. Bots can account, in some cases, for half of all the web traffic on your site. If you can reduce the strain and latency on your website, you can improve the experience for the actual humans who are trying to utilize your web presence. 

You need to have a clear and concise idea as to what bots are on your site so that you can marshal your resources accordingly. Availability is as much a part of the security posture as anything else. When I worked in retail, we had to understand what the customer’s requirements were as they came through the door. In that same vein there is a real need to be aware of what requirements your customers, and our digital scavengers, have from your websites. In this day and age, you should not have to worry about some misconfigured bot or scraper trying to smuggle a copy of Ugly Kid Joe’s, “America's Least Wanted” down the front of its pants.