Running an eCommerce store requires you to double down the efforts that you would otherwise put into managing your website. Not only does your website bring sales for you, but it also acts as your storefront. So, the experience that you deliver to your customers has to rival and even exceed the one that they would have in a traditional brick-and-mortar store.
If you have already set your foot in the eCommerce industry — Kudos! You have already taken a step in the right direction leading to success. But the success of eCommerce stores relies heavily on a lot of factors. You will need to have a strong SEO strategy to establish dominance over the SERPs, and a sound content strategy to keep your customers engaged.
Of course, this is only the starting point. You will need to use myriad strategies and tools to get it right and, more importantly, keep it on the right track over time. Log files are one of those tools that you can leverage to monitor and improve your website performance consistently.
With that said, let us dive right into what log files are, and all that they have to offer.
If you are one of those eCommerce entrepreneurs who has been shrugging off log files as just another part of technical SEO, think again. You might be missing out on a plethora of insights and information that you might otherwise want to be equipped with.
Every action taken by a user or a bot gets registered in a file that is stored on the server end, which is known as a log file. The actions, in this case, are initiated every time a user or bot sends a request for any resource on your website — so for each ‘hit’. This might all seem like menial data unless you decide to dig deeper and want details related to each of these actions.
At this point, we should mention that these log files contain quite a few valuable pieces of information. This information is stored anonymously. Typically, the time of each action, the status code returned, and the corresponding IP address are some of the details that you can fetch from the log file.
Needless to say, for a growing eCommerce website, a log file can easily be analysed to create useful insights. Sure, it can and is already used by technical site personnel, but when used right, it can have several advantages for SEO auditing as well. Due to this reason, developers, as well as SEO experts, analyse log files to identify any malfunctions and errors.
You can always opt for an SEO agency to help you with seeking these technical insights. But it is never a bad idea to learn how to do it yourself, is it?
Curious to check out what these highly notorious log files look like? Take a look at this example:
Now that you have a bit of knowledge to give you a better understanding of log files, let us explore their various types.
The most general form of logs that are maintained by web servers is access logs. They store information on each activity, such as their HTTPS status code. These logs are generally stored in a text format, such as Combined Log Format or Common Log Format.
Remember that these logs are stored and can be retrieved directly from the web server that received the access requests. You may need to gather your log files from multiple sources and reformat them so that you can easily access and interpret them.
If you have a large website that witnesses a decent amount of traffic, you probably have had to implement a Content Delivery Network (CDN) already. CDNs play a vital role in setting up a network of servers in different geographical locations so that your users can easily access your content regardless of their location.
But keep in mind that these networks are massive in scale, meaning you can not always have smooth access to your logs.
The other type of log file that you will encounter on your web server is error logs. A web server may not always be able to process your requests. At times, in the process of fulfilling the request, it might encounter errors in fetching the required information or resources. Every time such an event occurs, the related information is stored in an error log.
Say a user initiated a download of a PDF that is on your website. The PDF did exist on your website earlier but does not anymore, so the webserver runs into an error because the requested file is missing. This error is logged into the error log file.
Here is an example of what it may look like:
Additionally, some of the common formats that log files can be found in, include:
- Apache: Used by Nginx and Apache servers
- ELB: Used by Amazon Elastic Load Balancers
- W3C: Used by Microsoft IIS servers
- Other Custom Formats: Supported by servers that use a custom log format for outputs
Where to Find Your Log Files?
All said and done — it is not understanding log files that is tricky. It is the process of locating and accessing them that non-technical executives often have trouble with.
The first step to finding your log files is understanding more about your hosting setup. As discussed before, the more complicated your hosting setup, the longer it takes to find your log files. The other thing you need to know is that log files work differently for web servers and load balancers.
The process of finding log files can be overwhelming due to the sheer volume of files and information that you will end up uncovering. It is always a good idea to have a categorisation in place to keep the process clutter-free. Needless to say, you can always seek help of a digital agency in this process.
So, if you are trying to locate log files for conducting a technical SEO analysis, it is better to differentiate the relevant files from the rest of the records before going further.
Log files do not just offer you data that user actions have triggered, but also indicate updates triggered by bots. However, when we talk about log files in the context of technical SEO, there are quite a few insights that can be extracted from log files.
Here are a few types of important data points that you can derive from log files on the webserver:
Crawl Budget Loss
SEOs need to analyse the ratio between the number of pages on your website and the number of pages that Google has visited on your website. The crawl budget is the total number of pages that Google will visit on your website on any given day. In some cases, your most crucial web pages could take the brunt of it.
Log files can provide insights about the various kinds of bots that have crawled your website, tools that have accessed your website, and additional information about referring pages and organic traffic.
We just talked about a scenario wherein your most crucial web pages might not get crawled due to the predefined crawl budget. However, one way around it is to set the crawl priority in your XML sitemap. You have to simply prioritise your most crucial web pages before the others.
Analysing your log files will allow you to get a better understanding of your internal mesh, wherein you can also check the pages that Google prioritises the most.
Crawl Date and Frequency
Considering the importance that the process of crawling holds in the larger scheme of things, companies should undoubtedly pay attention to it. Another metric that you should monitor to assess the effectiveness of crawling is the crawl frequency. It is basically the frequency at which your web pages are crawled by search engines.
Interesting pages often get multiple visits in a day from Google. It is thus important to keep a check on the pages that are being crawled often, and the frequency at which it happens.
In this regard, log files will give you a complete picture of the crawl data as well as the frequency at which the bots crawled your website so far.
Running a functional website does not just end at taking a website live — it has much more to do with maintaining it over time. It entails ensuring that whether it is users or bots that land on your website; the experience remains seamless.
One way to ensure that this happens is with the help of status codes. Whether it is a 404 Error, or any other that can derail your SEO efforts — paying attention to it is imperative. To actively avoid such status codes, all the pages that present 4XX or 5XX errors should be tracked and removed.
Doing so can demonstrate in advance whether your website can handle a certain level of traffic and how it is likely to perform if that were to increase. Conducting a server log file analysis can help you with this and much more.
Your website should and could go into bouts of frequent updates or redesigning, and most times, it would even be necessary. In such cases, companies often prefer 302 redirects which indicate to search engines that the page has been temporarily moved to a new site or page. This is an alternative to a 301 redirect, which indicates that the page has permanently moved.
However, 302 redirects are not considered to be that great an option from the SEO perspective in most cases. It prevents optimal referencing, and the link juice from any external links on the original page will not be passed to the temporary address. So, what is the point?
Such pages should be removed in order to portray more stability on your website. Log files can help you identify such redirects more easily and allow you to make changes accordingly.
Duplicate URLs can be quite a complicated problem to handle for eCommerce websites. You will probably have similar content in multiple URLs. However, different URLs with the same content in the absence of canonical tags can lead to heavy penalties for your website.
What’s more, Googlebot can also slow down and prove less efficient in such cases. Why? Because it will end up spending a large amount of your crawl budget on duplicate URLs, instead of the unique content that you have on your website.
Identifying such URLs with the help of log files and eliminating them from your website will help elevate your website performance from an SEO standpoint.
Apart from these areas, SEOs will also be able to identify and optimise the Crawl Window and eliminate any blocked resources.
As we discussed before, log files have a common purpose to fulfil and a good one at that. But they exist in multiple formats based on the web server that you might be using. With that in mind, let us look at how you can go about retrieving your log files from these different kinds of servers.
How to Get Server Log Files from Apache?
If you have run a website long enough, you probably have an entanglement with Apache every single day. Apache HTTP Server is an open-source, and free software with cross-platform capabilities maintained by a community of developers. Apache is responsible for dispensing web content across the internet, with the help of HTTP requests by users.
Since it is the most popular web server used in the world, you need to understand how to fetch log files from Apache, better than any other alternative. Let us now look at two methods using which you can accomplish this:
Method #1: Using cPanel
cPanel is a server and site management platform that offers quite a few intuitive and user-friendly features. The platform allows you to manage all things server-related through a graphical interface, making it easier for you to carry out tasks like accessing logs.
If you are using cPanel, you can easily use Apache access logs through the dashboard. It is a simple 3-step process that is unlikely to take much time.
- Start by accessing the section ‘Metrics’ on your cPanel dashboard. Once you locate it, click on the ‘Raw Access’ option.
- This will lead you to a page that will display a list of access log files. If you have enabled archiving, you should be able to see a list of monthly access logs presented in the form of hyperlinks.
- Click on the log file that you want to access, which should prompt you to either open or save the file. These files will be in gzip format, which can be accessed directly if you are using Linux. But for any other OS, you may need to use a decompressing tool.
Once decompressed, the contents of the file can be accessed using any text editor.
Method #2: Using Terminal Commands
While you can easily use cPanel to access logs using a graphical interface, you can also use terminal commands to take those logs. So if you have access to the computer that is hosting Apache or even have remote access to it, you can display and even filter the contents of the logs.
You can use any of these default commands to locate the access logs:
You can use the cd command to find the exact location where the access logs might be stored. Additionally, you can use different kinds of terminal commands to filter through the information stored in the logs.
For instance, if you are looking to find specific terms from your access logs, you can use the grep command to filter information with the help of keywords. So, if you are only looking for entries with the GET request, you will essentially need to use the following command:
sudo grep GET /var/log/apache2/access.log
You can replace GET with any other specific type of information that you are looking for, in this command and filter the log files as per your need. You can also access the error logs using the terminal command method.
How to Get Server Log Files from NGINX?
NGINX is another type of web server that plays multiple roles, such as that of a load balancer and reverse proxy. Like Apache, accessing and configuring logs is much easier in NGINX if you are on Linux. It is also a free and open-source software similar to Apache.
NGINX stores two types of log files — access logs and error logs. The location of these log files depends on the operating system that you are using.
Again, you can view and access the log files using terminal commands. The most common command to get to the access logs is:
sudo tail -f /var/log/nginx/access.log
Using this command, you can fetch the contents of your log files and display them in real-time in the form of output on the terminal.
You can use a variety of different commands to filter the contents of the log files and for displaying them as per your need.
With the help of the data that you will fetch from your access logs, you will be able to get a detailed overview of the resources that are most frequently requested by users. On NGINX, you configure log files, get access to multiple ones, and create custom log formats easily with the help of the right terminal commands.
How to Get Server Log Files from IIS?
While Apache and NGINX have been designed to run smoothly on Linux, IIS (Internet Information Services) by Microsoft does the same for Windows environments. IIS is known to be one of the more efficient and logging-friendly architectures. To enable logging in to IIS, you need to have a Windows web hosting set up integrated.
Once you do that, follow these steps to find your log files easily:
- Go to Start > Control Panel > Administrative Tools
- On this screen, all you need to do is run Internet Information Services (IIS).
- Doing so will help you find your website on the left-side menu. Take note of your site ID from here.
- Now, click on your site and then select the ‘Logging’ icon. You will now be able to view your Logging settings screen.
- If you scroll down on this screen, you will be able to find the path to your log files.
And so, with these easy steps, you will easily be able to locate and access your log files. So, now that we have got that out of the way, let us look at how you can read the contents of your log files. You will need Log Parser Studio for this purpose.
Here is a step-by-step process that you can follow:
- Log in to your Log Parser Studio account. Go to File > New Query.
- Here, click on ‘Open’ and select the log files that you want to access.
- Now, select the log type as IISW3CLOG.
Now, you can create queries to filter and read your log files as you please. You can also export your log files in CSV format and run them on MS Excel or similar tools using Log Parser Studio.
Now, we have discussed extensively on accessing and reading log files for various web servers, and operating systems. We have finally reached the point where we can start delving deeper into the analysis part of it all.
After all, wasn’t that the main purpose all along?
But before we launch into the hows of it, let us first understand what log file analysis entails.
Log file analysis is the process of dissecting and evaluating the information in your log files so that you can identify pertinent information about your audience and any crawling and indexing issues. The main purpose of conducting an SEO log file analysis is to equip you with enough user insights to inform your SEO and website decisions.
Log files are more informative than any other tool that you can find when it comes to understanding how Google is crawling your website. You can get a complete picture of when, and how your website was crawled by Googlebot and whether any issues were encountered.
You will be able to resolve any major issues in your website right away and prevent it from experiencing the brunt of ineffective web pages. Depending on your website type, the analysis of your log files can be more impactful for you than others.
If you have a dynamic website that entails frequent updates to web pages and elements, it would bode well for you to leverage log file analysis to ensure sustainable website performance.
The eCommerce industry is bigger and more popular than ever before — and there is no plateau in sight. Within this big industry and the tough competition, companies need to be on top of their games to survive in the race and continue growing.
What we mean is that a missed opportunity or an issue can be worth much more for an eCommerce website than any other. With that in mind, technical SEOs working for eCommerce companies need to monitor how the website is performing consistently to deliver a uniform and seamless user experience.
What’s more, eCommerce websites need to perform well at all times, including fast loading times, and smoother transactions. Did you know that more than 46% of visitors hardly or never revisit a website that portrays a poor performance?
An SEO Log file analysis can upgrade your technical SEO performance significantly by putting your website on the fast track to the high ranks on Google SERPs. eCommerce websites can elicit a lot of insights from the following types of log files:
- Access Logs
- Error Logs
- Cookie Logs
All these log files can offer up a ton of information about how the pages on the website are performing, and the user reactions. Regardless of the log analysis tool that you use to access the files, you will end up with quite a bit of information about the website's performance.
Some of the common parameters that you would measure through the log file analysis include:
- Page Views
- Website Visits
- Average time on site
We shall discuss these parameters in more detail later in this guide.
But when you have a rich repository of such valuable information, you can do wonders as an eCommerce store. You can optimise the pages that are not performing as well as you want them to, figure out the elements that are working the most for you, and make tweaks to the website to level up the user experience.
Apart from helping you understand how your website is being crawled and indexed, and identifying pages prioritised by search engines, we have a few more points to talk about. Here are two other reasons why you should be investing time in conducting a log file analysis for your eCommerce website.
Identify Bugs and Check Overall Site Health
Even if you are a small eCommerce brand, there is no doubt that your website would be more voluminous than most. Needless to say, multiple changes might take place on your website simultaneously and get quite difficult to track.
Unfortunately, when you make a lot of changes to your website, you may often run into errors which can get quite tricky to track.
You can, of course, take the long route and run regular site audits and checks using tools like Google Search Console (GSC). But that would mean that you will need to spend a lot of time and resources and still run into roadblocks.
Instead, log file analysis will help you identify instances where search engine bots run into errors while crawling your website. So, using it in tandem with GSC might be much more fruitful in proactively identifying and resolving website errors.
Measure the Freshness of Content on Your Site
Fresh content is quite important when you are trying to strengthen your content marketing strategy and measure the quality of content on your website.
But the process of fresh content does not just entail posting new content frequently. It will also include posting it at regular intervals, and tracking how frequently Google bots crawl your website. You can post all the content you want, but that would be of little use if search engines do not discover it.
Log files will be able to provide insights about how frequently your website has been crawled, and which pages have been prioritised. You can also measure the average time a page is crawled in a day and subsequently find out the actual impact of how your fresh content is performing
Verify Alignment With Your Business Goals
Your SEO strategy is no doubt a product of a lot of hard work and time. As an eCommerce store, you need to take extra steps to create a winning strategy, and take a lot of factors into consideration.
So when you actually put your content to work, it is crucial to determine if the pages that are spearheading your strategy are actually being discovered. Similarly, you should also determine if unimportant pages are getting more attention than they warrant.
When you monitor these pages, you should also decide if your business goals are being met with the help of your website and SEO strategy. If not, it is a good time to take a step back and reevaluate your efforts.
Keeps Everything in Check During Migration
Content migration can be a long and arduous process if you do not have everything organised and sorted. Even after you do that, there is always a risk of diminishing the ranking and performance that you have worked so hard to build.
More importantly, when you are migrating your website, the crawling behaviour of Google can get quite erratic and unpredictable. When you have an eCommerce website, it is especially important to monitor how bots are crawling your website.
You will need to have a redirect plan, which defines the new URLs that your existing ones will move to. Even after all this, you can use your SEO log file analysis to identify if there are any crawling issues and fix them immediately so that your newly migrated website’s performance is maintained.
Even though log file analysis can add a lot of value to ensuring eCommerce performance, not all eCommerce CMSs allow users to access and analyse logs. In some of these cases, users can opt for a third-party tool to find a way around it. Even then, you will find that CMSs such as Wix, BigCommerce, Webflow and SquareSpace do not allow the analysis of log files.
Still, if you are plagued with questions about how analysing log files can help you out exactly, we understand. But if you want to understand how it can aid you in your strategic endeavours, here are a few questions that it can help you seek answers for:
1. What pages are search engines prioritising?
One of the most useful takeaways that you will get from your log files is a detailed overview of the pages that Googlebot has crawled. You will be able to get details such as the crawl time, frequency and any potential errors that it might have encountered. You can check whether the pages that you are banking on are getting enough priority or not.
2. What is Google ignoring that directly affects your revenue?
If Google or other search engines are deprioritising some of your most high-valued and crucial pages, you might be losing out on revenue. As an eCommerce website, you should be monitoring this and eliminate such instances the right way by adjusting the crawl priority.
3. How many crawled pages are actually indexed?
Since log file analysis can show how pages are being crawled, you will be able to identify any major issues that may be encountered in the process. For eCommerce websites, log file analysis is almost inevitable when it comes to understanding whether crucial parameters are being skipped or unnecessary ones are getting crawled. Even the smallest of errors can put a major dent in your crawl budget.
4. Are your revenue-generating landing pages visible on your eCommerce site?
Sitemaps, internal links, and site structure — are just some of the primary signals that familiarise search engines with your priorities in terms of which pages should be crawled and indexed. Even then, if any of your important pages are being deprioritised, you will now have a good starting point to understand why.
So, be on the lookout for these answers so that you can leverage any technical SEO optimisation opportunities.
Now that you know the plethora of benefits that conducting an analysis of your logs can offer for boosting your SEO strategy and efforts, you must be eager to get started.
One final thing before we dive into how log file analysis can be performed — it is important to be proactive in conducting log file analysis so that any issues can be identified and resolved in advance.
Once you get access to log files, there are two methods that you can use to perform log file analysis. The first option is to carry out the analysis manually. But warning where it is due — carrying out the log file analysis manually requires you to have an advanced working knowledge of Excel.
But fret not, there are now many tools that you can instead use to perform a detailed log file analysis. These log file analysers can save you a lot of time, are easy to learn and will ensure that you get accurate results without putting in too much effort.
So, without further ado, let us dive right into understanding the best log file analysis tools.
Screaming Frog SEO Spider has been synonymous with any kind of SEO analysis that you can think of — especially technical. So, it is no surprise that it features on the list of the best log file analysers.
The SEO Log File Analyser is a tool that allows you to upload your log files, and then get the insights that you are seeking from these files. It is a light software that runs with ease and is highly user-friendly.
More importantly, the Screaming Frog Log File Analyser can help you to do the following:
- Identifying the crawled URLs
- Finding broken URLs
- Discovering uncrawled and orphan pages
- Identifying fast and slow pages
- Audit Redirects
- Improving the crawl budget
- Identifying the crawl frequency
So, if you are a small eCommerce store, and looking for free log file analysis tools, Screaming Frog can be a great option.
Price: Free Version (Log Event Limit), Paid Version - £99/year
If you have dabbled in SEO at all, you need no introduction to SEMrush. It is known to be one of the top SaaS platforms to manage everything in content marketing and SEO. With a comprehensive suite of tools to accomplish any task that you may take up in the world of SEO, SEMrush is easily one of those tools that can help you take your SEO strategy to the next level.
SEMrush’s Log File Analyser is no exception.
You can generate two types of reports using the tool:
- Googlebot’s activity
- Pages’ Hits
It is a browser-based tool, so you do not have to download it, and use it seamlessly with the online version. The tool offers up insights about:
- Crawled pages and frequency
- Efficiency of crawl budget
- Crawl errors
- Referral URLs
Price: Free, Three pricing plans - Pro ($119.95/month), Guru ($229.95/month), Business ($449.95/month)
JetOctopus is a comprehensive tool that focuses specifically on crawling and log analysing. The Log analyser by JetOctopus is one of the more affordable options available out there and is just as easy to use.
With a simple two-click connection integration method, you can get started with analysing your files right away and expect the same quality of detailed analysis as SEMrush or Screaming Frog.
This powerful log analyser tool allows you to evaluate:
- Crawl budget waste
- Pages visited by bots
- Fake bots that hurt your website
- Pages not visited
So, if you are a budding or medium-sized eCommerce company that is seeking an affordable tool that can
Price: 7-day free trial, €120/month (billed annually)
Like JetOctopus, OnCrawl primarily evaluates the crawl activity on your website by different kinds of bots, and how efficiently your crawl budget is being spent. One of the best factors that works for On Crawl is its visually stunning interface and detailed insights.
A significant factor that you should know about OnCrawl is that it is designed specifically for large and enterprise websites that have thousands of web pages. So, if you are a small eCommerce business, skip this tool for now.
OnCrawl also allows you to carry out dynamic segmentation, which helps you divide URLs and links based on specific patterns and criteria. The tool is highly secure and GDPR-compliant, in addition to having the ability to easily adapt to evolving processing and storage requirements.
With OnCrawl, you can use your log file analysis to:
- Monitor and prevent crawl budget waste
- Monitor Googlebot’s activity
- Understand how your pages are crawled and indexed
- Audit your technical SEO performance
Price: 14-day free trial, Explorer (€49/month), Business (€199/month), Infinity & Beyond (custom quotes)
Choosing the right tool to carry out your log file analysis can get quite confusing. There are several features and factors that you should consider. Most importantly, you should ensure that you select a tool that meets your business requirements and has features that you can fully leverage.
You can also use Data Studio to make the analysis a lot easier and interactive. This Data Studio template for Log file analysis using Logflare might come in handy for that.
By now, you know what the hype about log files and their analysis is all about. You even know the tools that you should use for carrying out a comprehensive evaluation of your log files, depending on the scale of your eCommerce business.
It is important to know that most log file analysers follow a similar process for carrying out the analysis. Follow this step-by-step process for conducting an effective log file analysis using a log analyser:
Step 1: Download a Copy of Your Log File
Earlier in this guide, we detailed the process that you can follow to find and download your log files. Depending on the web server that you are using, go ahead with seeking and securing a copy of the log files that you want to analyse.
If you have a webmaster or IT staff who takes care of the technical things, acquire a copy of your log files from them. Remember to obtain log files of a wider timeframe so that you can effectively analyse insights for a definitive period.
Step 2: Verify that the Log File is Formatted Correctly
When you are conducting a manual log file analysis, you can download the log files in the .csv format. This can be a daunting process if you are not very familiar with technical SEO. However, you can always familiarise yourself with what the various terms and columns mean.
Instead, you can opt for the more preferable method of using a log analyser tool for conducting the analysis. Depending on the tool that you have chosen for your log file analysis, ensure that all your log files are downloaded in an acceptable format.
Step 3: Upload Your Log File on a Log File Analyzer Tool
Most log file analysers require log files to be in a tabular format or a spreadsheet format. Prepare your log files based on what is required by the tool that you are using.
Once you do that, you can upload the log files to the analyser tool. Ensure to look for the specific directions for uploading, as each log file analyser takes a different approach to it.
Step 4: Initiate the Log File Analysis Process
Now that you are fully prepared, it is time to get to work!
Initiate your log file analysis process in the tool, after you have uploaded the log files and are prompted to start the analysis. All you need to do now is sit back and let the tool carry out a thorough analysis of your log files.
Depending on the volume of data that you have uploaded to the tool, this process can take a few seconds or even a few hours.
Step 5: Analyse Results
Once you are done with analysing log files, the tool will present you with a detailed report of the findings. Equipped with this report, you can go ahead and conduct a comprehensive analysis of the findings and figure out the insights that you can derive from them.
The report will be able to provide you with a lot of insights into the way Googlebot and other crawlers access content on your website. This is the point at which you will need to seek answers to pertinent questions, such as:
- Are all the important web pages being crawled?
- Which pages are getting deprioritised and why?
- How frequently are the web pages crawled?
- Are any faulty status codes disrupting the user experience?
Remember, these questions are just suggestions and you can tailor them as per your goals.
Equipped with all this knowledge, let us now circle back to why it is so important for eCommerce websites to invest efforts into analysing logs. Of course, we are talking about the many SEO benefits that you can unlock with the help of analytical insights from the log files.
Even though we have already discussed some of these benefits in this guide, it would bode well to take a look at the many more advantages that you stand to gain.
Let’s get right into it.
Discovers If and Where the Crawl Budget is Being Wasted
Google defines the crawl budget as the number of URLs that Googlebot can and wants to crawl depending on factors such as the crawl rate and crawl demand. Other factors influencing your crawl budget include the link equity flowing through your website and your domain authority.
It is important to monitor where your crawl budget is being spent otherwise there is a risk of wasting it on insignificant pages, while your fresh content stays ignored.
Here are some factors that can lead to wastage of your crawl budget:
Log analysis will identify how exactly your website is being crawled and its frequency. It will thus help you crack down on, and minimise potential crawl budget wastage. Talking about the crawl budget, a log file analysis can also help you ensure that the important pages on your eCommerce website are being crawled.
Checks if Your Store has Switched to Google’s Mobile-first Index
As Google’s efforts to make the web more mobile-friendly continue to gradually increase, mobile-first indexing is one of the first things you should know about. As an eCommerce store, you should anyway be targeting a seamless user experience on mobile devices, but it will go a long way in appeasing the Google gods as well.
After all, the m-commerce revolution has already started. Nearly 43% of eCommerce value today stems from mobile commerce.
So, how do you know if you have made that big switch to the Mobile-first index?
Well, when you do make that leap, you should see a steady increase in traffic from Googlebot Smartphone. Generally, your website will get 80% of crawling traffic from a desktop, and 20% from mobile. However, when you switch to Google’s mobile-first index, these numbers will essentially reverse. Most of your crawling traffic will now come from mobile.
You will be able to locate if this has happened, using any of the log analysers that we have talked about in this guide.
Verifies if All Targeted Search Engine Bots Can Access Your Pages
Although we talk about Googlebot many times in this guide, targeting other search engine bots is equally important. Sure, Google dominates the game, but the other search engines are just as crucial.
With that in mind, your eCommerce store should monitor whether all the search engines that you are targeting can discover you. Assign importance to getting crawled by Bingbot, Baiduspider, and Yandexbot just as much, especially if you are targeting a global audience.
With the help of log file analysers, you can ensure that you are getting crawl traffic from the search engines that you are targeting. At the same time, you will also be able to monitor if you are getting unwanted traffic from a specific search engine.
For instance, if your target audience is not in China, excessive traffic from BaiduSpider would be completely unnecessary. If you experience a lot of unusual activity from these crawlers, it may be a good idea to block them.
Identifies Any Incorrect Status Codes
If you use Google Search Console to monitor your website performance, you may be able to acquire insights about 404s and 302s. But log files go a step further and provide you with a detailed overview of the status codes pertaining to each page.
You can use log file analysers to understand which specific response code a search engine might have experienced in the end. You will also identify the specific URLs that you should fix based on these status codes and understand the improvements that are required.
Highlights Inconsistent Response Codes
Along with getting an overview of the status codes, you should also pay attention to inconsistent response codes. If you look at the last few response codes of your search engines, you will be able to conclude your technical checks as long as there are no spikes in 4XX or 5XX errors. It is possible to fetch these insights using a log file analysis tool.
When there are mix-ups in the issues related to servers and broken links, can often lead to inconsistent response codes. But with such insights from your log file analysis, you will be able to proactively work on resolving these issues.
Locates Slow or Large Pages
It is not just crawl-related issues and errors that log file analysis can point out. It is important to pay attention to the Core Web Vitals (CWV) as they influence the way your website performs. Metrics such as Time to First Byte (TTFB), and Time to Last Byte (TTLB) play a crucial role in getting your website crawled and indexed.
When you have an eCommerce website, fast loading times are not just preferable, they are also mandatory. Every second of delay in loading a website is likely to reduce customer satisfaction by 16%.
The size and speed of your website will significantly impact not just your customers, but also your SEO performance. You will be able to figure out the Average Response Time of your website through your analysis. You can then use those insights to improve your website performance.
Finds Orphaned Pages
Orphaned pages can hurt the ranking of your website. Search engines will crawl such pages, but since they are not a part of your website’s internal linking structure, they exist as standalone pages.
Orphaned pages exist as a result of:
- Content updates
- Incorrect internal or external linking
- Changes to site structure
- Old and redirected URLs
Taking a timely call on such orphaned pages is crucial for the success of any eCommerce website. If not, it can have a detrimental effect on your SEO performance.
Discovers Indexable Pages that Google isn’t Crawling
When you have an eCommerce website, there may be many pages on your website that are aimed at generating a lot of revenue. It is also important to ensure that your low-value pages are not eating away at your crawl budget. After all, there may be plenty of indexable pages that may get ignored in the process.
You can use the log file analysis to find whether these indexable pages are getting crawled. You can accordingly set priorities for the pages that you want Googlebot to crawl and index.
When you conduct a log file analysis, you will inevitably generate and process a large volume of information and insights. Some of this information may not make much sense to you, but some of it will give you a lot of insights into the performance of your eCommerce store.
With the help of your log file analysis, you will be able to measure what parts of your eCommerce store work for you, and which ones don’t.
Here are a few metrics that you can get from log files:
When you are trying to get the word out about your eCommerce website, every visitor that lands on it is important. Tracking the number of website visitors will tell you how popular your store is, and how much they are contributing to your business.
For this purpose, you need to consider SEO metrics such as page visits, and average time users spend on the website so that you can get a comprehensive view of your website performance. You should also track the time of their visits and their hits on your website, as these insights will help you identify patterns and trends in the industry.
More often than not, when your website traffic peaks, you can identify that you are getting your content strategy right. But it would be impractical to think that all of your content is performing equally well.
This is where pageviews can help.
Page Views can point out the main pages that drive the most traffic to your website. Whether it is an informative blog or a highly effective landing page — every page matters. Measuring your page views will also help you understand which pages are engaging with your users the most. You can then design your content strategy to include more of such content and successfully amplify your website traffic.
The downloads on your website will tell you how well your users are interacting and engaging with your content. After all, why would they waste time downloading a resource they do not assign importance to?
But eCommerce websites need to ensure that users experience a seamless process even while downloading resources. Log files give you the opportunity to assess how much time servers are taking to respond to requests for a resource on your website.
In other words, how long does your website make your user wait before delivering the resources they are requesting?
If your website is taking way too much time, it may be time to take a second look at your website performance and reduce this timespan.
The sources of your website traffic are crucial in determining how well your website is doing externally. It is also a great way to figure out if all those link building and social media campaigns are generating results for your website.
By analysing the referring sites, you will also be able to understand the persona of your visitors. With the help of your log file analysis, you will be able to figure out the sources and partnerships that generate the best results and continue focusing efforts on them.
Bounce Rate & Exit Pages
There is nothing more detrimental and disheartening for an eCommerce store than visitors landing on the website and abandoning it before the page is completely loaded. Your bounce rate will rise if this is happening too often on your website.
When you analyse log files and it suggests that your bounce rate is too high, you can then evaluate the reasons behind it. Similarly, you will also identify the pages that a visitor has landed on right before they leave your website. These are called exit pages.
This information is quite crucial for eCommerce websites. Being transaction-driven websites, eCommerce stores have to work harder to map the entire customer journey from start to end.
Lastly, you will also need to pay attention to the keywords that are definitive for your website performance.
You will need to ask yourself questions such as:
- Which keywords are working best for my website?
- Which keywords are satisfying the user's search intent?
- Which keywords are not at all generating any impact?
Log file analysis will not provide the exact answers to these questions, but will help you guide you in the right direction so that you can seek them. Remember, when you do your log file analysis right, you will be able to get expert insights into consumer behaviour, market demand and website traffic.
Log files can be rich sources of information, providing you with useful insights and data that can inform your website strategy. Analysing your log files can help you delve deeper into the factors that are making and breaking your website performance.
While having knowledge of technical SEO is useful in the process — even if you do not have it, you would be able to easily conduct server log file analysis. Your eCommerce store would be able to derive a lot of data from the log files that can then be used to level up the customer experience that you are offering. Use any of the log file analysers that we have listed in this guide to evaluate insights from your log files.
If you are still confused about how to go about the whole process, we understand. Get in touch with us, so that we can team up with you to level up your website performance.
Enter Your Website & get
an instant SEO Report for FREE