for players, by players

Guidelines For Manufacturers

Loadscout utilizes crawling technologies in order to access you website and fetch required data for the platform. Our team creates specialized crawler bots for every manufacturer's website according to its structure. Loadscout bot is planned to make a regular visit to the manufacturer website (once or twice per week) in order to get the latest products. On every site our bot is seeking for the product title, code, image, manual, brochure, exploded views and product properties, data collected are stored in our server in order to avoid excessive traffic load on manufacturer's website.

Building crawler bot for hundreds of manufacturer's website we have gain a lot of experience from studying website structure and creating appropriate algorithms for parsing data, therefore we would like to make a list about things that you should not do on your website, most of the following guidelines are similar to Google. Keep in mind that we put this guidelines list in order to help make your website better and not to criticize your hard effort.

  1. When you change your urls use 301 redirect. When you want to make a new design to you website and change the url from domain/category/product to domain/product for example keep in mind that the best practice is to make a 301 redirect on you .htaccess file and point the old url to the new one, don't just vanish the old url. The reason is simple, if your website with the url domain/category/product is on the internt for let's say 2 years, google will have index it, some people may have share it on a social network, forum etc or may bookmark it, imagine now that when you change it to domain/product the old url will be pointless for all these peaople including google. The solution is to simple tell to your .htaccess that the old url points to the new, so every time that some of those with the old url clicks on it, it will work. These procedure can be painless and will probably require to put someone to make it, but it's necessary.
  2. Don't use flash. By saying don't use flash for loadscout bot means don't use flash for the entire site or the main image. Our bot cannot read and intepret the flash content.
  3. Create a table with product attributes / specifications. It's much easier for the visitor but also for our bot if the product attributes like material, energy, velocity, capacity etc are on a html structure rather than merged in the description separated by p html tag,in the same line. Although we are creating specialized bot for every manufacturer, it is difficult and requires a lot of effort (which at this point we are not capable of provide) in order to create an algorithm that can extract this information from a whole text.
  4. Put property values for your products. The more detail presentation you make the more attractive it will be, and the more information our bot will extract and the better presentation we can provide for your product in So in your product webpage create a good looking table and put some product attributes / specifications etc. We acknowledge that this is really hard work and some times difficult if you have hundreds of products, but we just proposing the optimum.

In any case our team will be happy to provide any technical advice or help about your website structure and loadscout bot.