seo posts - idmontie's Portfolio

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    blockquote: "blockquote",
    img: "img",
    p: "p",
    ...props.components
  }, {Separator} = _components;
  if (!Separator) _missingMdxReference("Separator", true);
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "What happens to SEO when the crawler itself becomes one of your primary consumers?"
    }), "\n", _jsx(_components.p, {
      children: "We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website."
    }), "\n", _jsxs(_components.p, {
      children: ["We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With ", _jsx(_components.a, {
        href: "https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age",
        children: "53% percent of traffic"
      }), " is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing."]
    }), "\n", _jsx(Separator, {}), "\n", _jsxs(_components.p, {
      children: ["When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - ", _jsx(_components.a, {
        href: "https://claude.com/",
        children: "Claudebot"
      }), ", ", _jsx(_components.a, {
        href: "https://meta.com/",
        children: "meta"
      }), ", other nefarious bots that would not return correct User Agent strings."]
    }), "\n", _jsx(_components.p, {
      children: "Why were AI crawlers hitting the application so differently from Google bot?"
    }), "\n", _jsx(Separator, {}), "\n", _jsx(_components.p, {
      children: "To understand why this was so surprising, we have to understand who the site was originally designed for."
    }), "\n", _jsx(_components.p, {
      children: "When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress."
    }), "\n", _jsx(_components.p, {
      children: "With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?"
    }), "\n", _jsx(_components.p, {
      children: "During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required."
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.a, {
        href: "https://nextjs.org/learn/seo/webcrawlers",
        children: "NextJS"
      }), " and ", _jsx(_components.a, {
        href: "https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics",
        children: "Google Search Central"
      }), " have typically described the indexing process like this:"]
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png",
        alt: "Crawling Indexing Process"
      })
    }), "\n", _jsx(_components.p, {
      children: "Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step."
    }), "\n", _jsx(_components.p, {
      children: "But Googlebot wasn’t the problem."
    }), "\n", _jsx(Separator, {}), "\n", _jsx(_components.p, {
      children: "AI Crawlers hammer the application."
    }), "\n", _jsx(_components.p, {
      children: "Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see."
    }), "\n", _jsx(_components.p, {
      children: "We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses."
    }), "\n", _jsx(_components.p, {
      children: "Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers."
    }), "\n", _jsx(_components.p, {
      children: "The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them."
    }), "\n", _jsx(Separator, {}), "\n", _jsxs(_components.p, {
      children: ["Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, ", _jsx(_components.a, {
        href: "https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/",
        children: "The Web Is All Bots Now"
      }), ", which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots."]
    }), "\n", _jsxs(_components.p, {
      children: ["Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post ", _jsx(_components.a, {
        href: "https://people.kernel.org/monsieuricon/creepy-crawlies",
        children: "Creepy Crawlies"
      }), " that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html."
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt."
    }), "\n", _jsx(_components.p, {
      children: "And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now."
    }), "\n", _jsx(_components.p, {
      children: "The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};
function _missingMdxReference(id, component) {
  throw new Error("Expected " + (component ? "component" : "object") + " `" + id + "` to be defined: you likely forgot to import, pass, or provide it.");
}


<p>What happens to SEO when the crawler itself becomes one of your primary consumers?</p>
<p>We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website.</p>
<p>We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With <a href="https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age">53% percent of traffic</a> is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - <a href="https://claude.com/">Claudebot</a>, <a href="https://meta.com/">meta</a>, other nefarious bots that would not return correct User Agent strings.</p>
<p>Why were AI crawlers hitting the application so differently from Google bot?</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>To understand why this was so surprising, we have to understand who the site was originally designed for.</p>
<p>When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress.</p>
<p>With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?</p>
<p>During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required.</p>
<p><a href="https://nextjs.org/learn/seo/webcrawlers">NextJS</a> and <a href="https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics">Google Search Central</a> have typically described the indexing process like this:</p>
<p><img alt="Crawling Indexing Process" src="/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png" style="max-height:500px;margin:auto;text-align:center"/></p>
<p>Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step.</p>
<p>But Googlebot wasn’t the problem.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>AI Crawlers hammer the application.</p>
<p>Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see.</p>
<p>We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses.</p>
<p>Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers.</p>
<p>The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, <a href="https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/">The Web Is All Bots Now</a>, which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots.</p>
<p>Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post <a href="https://people.kernel.org/monsieuricon/creepy-crawlies">Creepy Crawlies</a> that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:</p>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.</p>
</blockquote>
<p>At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt.</p>
<p>And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now.</p>
<p>The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps.</p>


What happens to SEO when the crawler itself becomes one of your primary consumers?

We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website.

We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With [53% percent of traffic](https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age) is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing.

<Separator />

When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - [Claudebot](https://claude.com/), [meta](https://meta.com/), other nefarious bots that would not return correct User Agent strings.

Why were AI crawlers hitting the application so differently from Google bot?

<Separator />

To understand why this was so surprising, we have to understand who the site was originally designed for. 

When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress.

With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?

During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required. 

[NextJS](https://nextjs.org/learn/seo/webcrawlers) and [Google Search Central](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) have typically described the indexing process like this:

![Crawling Indexing Process](/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png)

Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step.

But Googlebot wasn’t the problem.

<Separator />

AI Crawlers hammer the application.

Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see.

We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses.

Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers.

The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them.

<Separator />

Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, [The Web Is All Bots Now](https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/), which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots.

Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post [Creepy Crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies) that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:

> TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
> 

At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt.

And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now.

The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps.


Are Crawlers Our Primary Consumers?

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    blockquote: "blockquote",
    code: "code",
    h2: "h2",
    h3: "h3",
    li: "li",
    p: "p",
    table: "table",
    tbody: "tbody",
    td: "td",
    th: "th",
    thead: "thead",
    tr: "tr",
    ul: "ul",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This post was originally on a blog I started for creating web applications. This post goes into SEO considerations for web applications."
    }), "\n", _jsx(_components.h2, {
      children: "Use Search Engine Friendly URLs"
    }), "\n", _jsxs(_components.p, {
      children: ["Use ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Semantic_URL",
        children: "semantic URLs"
      }), ".  Semantic URLs help improve usability and accessibility of a website.  The following table from the Wikipedia entry illistrates how to use semantic URLs:"]
    }), "\n", _jsxs(_components.table, {
      children: [_jsx(_components.thead, {
        children: _jsxs(_components.tr, {
          children: [_jsx(_components.th, {
            children: "Non-semantic URL"
          }), _jsx(_components.th, {
            children: "Semantic URL"
          })]
        })
      }), _jsxs(_components.tbody, {
        children: [_jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/index.php?page=name"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/name"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/index.php?page=consulting/marketing"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/consulting/marketing"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/products?category=2&pid=25"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/products/2/25"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/cgi-bin/feed.cgi?feed=news&frm=rss"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/news.rss"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/services/index.jsp?category=legal&id=patents"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/services/legal/patents"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/kb/index.php?cat=8&id=41"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/kb/8/41"
            })
          })]
        }), _jsxs(_components.tr, {
          children: [_jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/index.php?mod=profiles&id=193"
            })
          }), _jsx(_components.td, {
            children: _jsx(_components.code, {
              children: "http://example.com/profiles/193"
            })
          })]
        })]
      })]
    }), "\n", _jsx(_components.h2, {
      children: "Use Hashbangs for Dynamic Content When You Normally Use Hashes"
    }), "\n", _jsxs(_components.p, {
      children: ["If you normally use the hash ", _jsx(_components.code, {
        children: "#"
      }), " character for dynamic content, change it to the hashbang ", _jsx(_components.code, {
        children: "#!"
      }), ".  This allows googlebot to use ", _jsx(_components.code, {
        children: "#_REQUEST[\"_escaped_fragment_\"]"
      }), " instead of ", _jsx(_components.code, {
        children: "#!"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Use Push State If You Can"
    }), "\n", _jsxs(_components.p, {
      children: ["If the user is using Firefox or Chromium, you can use ", _jsx(_components.code, {
        children: "history.pushState( object, pageName, \"./?param=1\");"
      }), " so that the address bar changes, but the page does not reload.  This allows you to use ", _jsx(_components.code, {
        children: "?"
      }), " instead of ", _jsx(_components.code, {
        children: "#!"
      }), " to keep dynamic content."]
    }), "\n", _jsx(_components.h2, {
      children: "Use Descriptive Links"
    }), "\n", _jsx(_components.p, {
      children: "When creating links, refrain from using \"this\" or \"click here\" or similar phrases for your links.  Link descriptions should be descriptive."
    }), "\n", _jsx(_components.h2, {
      children: "Have an XML Sitemap"
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling."
      }), "\n"]
    }), "\n", _jsxs(_components.p, {
      children: ["Have a ", _jsx(_components.a, {
        href: "http://www.sitemaps.org/",
        children: "sitemap"
      }), " at ", _jsx(_components.code, {
        children: "/sitemap.xml"
      }), " for search engines to crawl."]
    }), "\n", _jsxs(_components.p, {
      children: ["Have a ", _jsx(_components.a, {
        href: "https://support.google.com/webmasters/answer/178636",
        children: "image sitemap"
      }), " for your images!"]
    }), "\n", _jsxs(_components.p, {
      children: ["Create a ", _jsx(_components.a, {
        href: "https://support.google.com/webmasters/topic/2370586?rd=1",
        children: "mobile sitemap"
      }), "!"]
    }), "\n", _jsx(_components.h2, {
      children: "Use Canonical URLs For Duplicate Content"
    }), "\n", _jsxs(_components.p, {
      children: ["Use ", _jsx(_components.code, {
        children: "<link rel=\"canonical\" ... />"
      }), " when you have duplicate content on multiple pages. Remember, multiple URLs that point to the same content coint as dusplicate content. Also remember that ", _jsx(_components.code, {
        children: "?page=1&subject=my-subject"
      }), " and ", _jsx(_components.code, {
        children: "?subject=my-subject&page=1"
      }), " are different pages to web crawlers!"]
    }), "\n", _jsx(_components.h2, {
      children: "Setup Webmaster Tools"
    }), "\n", _jsxs(_components.p, {
      children: ["Use ", _jsx(_components.a, {
        href: "http://www.google.com/webmasters/",
        children: "Google Webmaster Tools"
      }), " and ", _jsx(_components.a, {
        href: "http://www.bing.com/toolbox/webmaster",
        children: "Bing Webmaster Tools"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Setup Google Analytics"
    }), "\n", _jsxs(_components.p, {
      children: ["Have ", _jsx(_components.a, {
        href: "http://www.google.com/analytics/",
        children: "Google Analytics"
      }), " set up.  Or something like ", _jsx(_components.a, {
        href: "http://piwik.org/",
        children: "Piwik"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Have robots.txt Configured Properly"
    }), "\n", _jsxs(_components.p, {
      children: ["Set your ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Robots_exclusion_standard",
        children: "robots.txt"
      }), " file properly.  Each subdomain must have its own robots.txt file."]
    }), "\n", _jsx(_components.h2, {
      children: "301 Redirect for WWW Requests"
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.code, {
        children: "www.example.com"
      }), " or ", _jsx(_components.code, {
        children: "example.com"
      }), " should be your primary domain, not both. Have one 301 Redirect (Moved Permanently) to the other."]
    }), "\n", _jsx(_components.h2, {
      children: "Read the Search Engine Optimization Starter Guide"
    }), "\n", _jsxs(_components.p, {
      children: ["The ", _jsx(_components.a, {
        href: "http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/webmasters/docs/search-engine-optimization-starter-guide.pdf",
        children: "starter guide is viewable here"
      }), "."]
    }), "\n", _jsx(_components.h3, {
      children: "The Basics"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Create unique, accurate page titles. These titles are displayed in the search results. Titles reflect what is on each individual page."
      }), "\n", _jsx(_components.li, {
        children: "Use the description meta tag. Google will use them as snippets for your pages."
      }), "\n", _jsxs(_components.li, {
        children: ["Use simple to understand (ie \"friendly\") URLs.  URLs should contain relevant words. ", _jsx(_components.code, {
          children: "http://mysite.com/article/basket-ball-scores"
        }), " is a friendly URL."]
      }), "\n", _jsx(_components.li, {
        children: "Navigation is very important for search engines.  Think of you website as folders that should be easy to navigate.  Use breadcrumbs!"
      }), "\n", _jsxs(_components.li, {
        children: ["Use ", _jsx(_components.a, {
          href: "#have-an-xml-sitemap",
          children: "sitemaps"
        }), "!  Prepare one for users and one for search engines."]
      }), "\n", _jsx(_components.li, {
        children: "Use 404s when necessary."
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "Improving Site Structure"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Create a naturally flowing hierarchy."
      }), "\n", _jsx(_components.li, {
        children: "Avoid having all navigation based entirely on dropdown menus.  Your website should be accessible, pages should be conntected with normal text links."
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "Optimizing Content"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Create compelling and useful content."
      }), "\n", _jsx(_components.li, {
        children: "Offer unique and exclusive content."
      }), "\n", _jsxs(_components.li, {
        children: ["Checkout the ", _jsx(_components.a, {
          href: "https://adwords.google.com/KeywordPlanner",
          children: "Keyword Planner"
        }), " by Google AdWords."]
      }), "\n", _jsx(_components.li, {
        children: "Write easy-to-read text.  Don't embed text in images"
      }), "\n", _jsx(_components.li, {
        children: "Create fresh content."
      }), "\n", _jsx(_components.li, {
        children: "Create content for users, not for search engines."
      }), "\n", _jsxs(_components.li, {
        children: ["Write good anchor text: ", _jsx(_components.code, {
          children: "<a href=\"...\">Top Ten Basketball Shots</a>"
        }), " is a descriptive link."]
      }), "\n", _jsxs(_components.li, {
        children: ["Use ", _jsx(_components.code, {
          children: "alt"
        }), " tags on your images! This also helps with accessibility."]
      }), "\n", _jsx(_components.li, {
        children: "Use descriptive filenames for your images and assets!"
      }), "\n", _jsxs(_components.li, {
        children: ["Supply an ", _jsx(_components.a, {
          href: "https://support.google.com/webmasters/answer/178636",
          children: "image sitemap file"
        }), "!"]
      }), "\n", _jsxs(_components.li, {
        children: ["Use heading tags correctly!  ", _jsx(_components.code, {
          children: "<h1>"
        }), " is the most important. ", _jsx(_components.code, {
          children: "<h6>"
        }), " is the least important."]
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "Dealing with Crawlers"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Restrict crawling with robots.txt."
      }), "\n", _jsxs(_components.li, {
        children: ["Use ", _jsx(_components.code, {
          children: "rel=\"nofollow\""
        }), " when necessary for links. Normally, user generated links through comment systems should be tagged with nofollow."]
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "SEO for Mobile Phones"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Configure mobiles sites so that they can be indexed accurately."
      }), "\n", _jsxs(_components.li, {
        children: ["Create a ", _jsx(_components.a, {
          href: "https://support.google.com/webmasters/topic/2370586?rd=1",
          children: "mobile sitemap"
        }), "!"]
      }), "\n", _jsxs(_components.li, {
        children: ["If you use User-agents to generate your mobile site, make sure to allow ", _jsx(_components.code, {
          children: "Googlebot-Mobile"
        }), "!"]
      }), "\n", _jsx(_components.li, {
        children: "If you use a seperate domain for your mobile app, redirect them to the correct page!"
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "Promotions and Analysis"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Backlinks increase the value of your site ( backlinks are links that other people post to your site)."
      }), "\n", _jsx(_components.li, {
        children: "Know how to utilize your social media sites."
      }), "\n", _jsx(_components.li, {
        children: "Utilize the Google Webmaster Tools."
      }), "\n"]
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This post was originally on a blog I started for creating web applications. This post goes into SEO considerations for web applications.</p>
<h2>Use Search Engine Friendly URLs</h2>
<p>Use <a href="http://en.wikipedia.org/wiki/Semantic_URL">semantic URLs</a>.  Semantic URLs help improve usability and accessibility of a website.  The following table from the Wikipedia entry illistrates how to use semantic URLs:</p>
<table><thead><tr><th>Non-semantic URL</th><th>Semantic URL</th></tr></thead><tbody><tr><td><code>http://example.com/index.php?page=name</code></td><td><code>http://example.com/name</code></td></tr><tr><td><code>http://example.com/index.php?page=consulting/marketing</code></td><td><code>http://example.com/consulting/marketing</code></td></tr><tr><td><code>http://example.com/products?category=2&amp;pid=25</code></td><td><code>http://example.com/products/2/25</code></td></tr><tr><td><code>http://example.com/cgi-bin/feed.cgi?feed=news&amp;frm=rss</code></td><td><code>http://example.com/news.rss</code></td></tr><tr><td><code>http://example.com/services/index.jsp?category=legal&amp;id=patents</code></td><td><code>http://example.com/services/legal/patents</code></td></tr><tr><td><code>http://example.com/kb/index.php?cat=8&amp;id=41</code></td><td><code>http://example.com/kb/8/41</code></td></tr><tr><td><code>http://example.com/index.php?mod=profiles&amp;id=193</code></td><td><code>http://example.com/profiles/193</code></td></tr></tbody></table>
<h2>Use Hashbangs for Dynamic Content When You Normally Use Hashes</h2>
<p>If you normally use the hash <code>#</code> character for dynamic content, change it to the hashbang <code>#!</code>.  This allows googlebot to use <code>#_REQUEST[&quot;_escaped_fragment_&quot;]</code> instead of <code>#!</code>.</p>
<h2>Use Push State If You Can</h2>
<p>If the user is using Firefox or Chromium, you can use <code>history.pushState( object, pageName, &quot;./?param=1&quot;);</code> so that the address bar changes, but the page does not reload.  This allows you to use <code>?</code> instead of <code>#!</code> to keep dynamic content.</p>
<h2>Use Descriptive Links</h2>
<p>When creating links, refrain from using &quot;this&quot; or &quot;click here&quot; or similar phrases for your links.  Link descriptions should be descriptive.</p>
<h2>Have an XML Sitemap</h2>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.</p>
</blockquote>
<p>Have a <a href="http://www.sitemaps.org/">sitemap</a> at <code>/sitemap.xml</code> for search engines to crawl.</p>
<p>Have a <a href="https://support.google.com/webmasters/answer/178636">image sitemap</a> for your images!</p>
<p>Create a <a href="https://support.google.com/webmasters/topic/2370586?rd=1">mobile sitemap</a>!</p>
<h2>Use Canonical URLs For Duplicate Content</h2>
<p>Use <code>&lt;link rel=&quot;canonical&quot; ... /&gt;</code> when you have duplicate content on multiple pages. Remember, multiple URLs that point to the same content coint as dusplicate content. Also remember that <code>?page=1&amp;subject=my-subject</code> and <code>?subject=my-subject&amp;page=1</code> are different pages to web crawlers!</p>
<h2>Setup Webmaster Tools</h2>
<p>Use <a href="http://www.google.com/webmasters/">Google Webmaster Tools</a> and <a href="http://www.bing.com/toolbox/webmaster">Bing Webmaster Tools</a>.</p>
<h2>Setup Google Analytics</h2>
<p>Have <a href="http://www.google.com/analytics/">Google Analytics</a> set up.  Or something like <a href="http://piwik.org/">Piwik</a>.</p>
<h2>Have robots.txt Configured Properly</h2>
<p>Set your <a href="http://en.wikipedia.org/wiki/Robots_exclusion_standard">robots.txt</a> file properly.  Each subdomain must have its own robots.txt file.</p>
<h2>301 Redirect for WWW Requests</h2>
<p><code>www.example.com</code> or <code>example.com</code> should be your primary domain, not both. Have one 301 Redirect (Moved Permanently) to the other.</p>
<h2>Read the Search Engine Optimization Starter Guide</h2>
<p>The <a href="http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/webmasters/docs/search-engine-optimization-starter-guide.pdf">starter guide is viewable here</a>.</p>
<h3>The Basics</h3>
<ul>
<li>Create unique, accurate page titles. These titles are displayed in the search results. Titles reflect what is on each individual page.</li>
<li>Use the description meta tag. Google will use them as snippets for your pages.</li>
<li>Use simple to understand (ie &quot;friendly&quot;) URLs.  URLs should contain relevant words. <code>http://mysite.com/article/basket-ball-scores</code> is a friendly URL.</li>
<li>Navigation is very important for search engines.  Think of you website as folders that should be easy to navigate.  Use breadcrumbs!</li>
<li>Use <a href="#have-an-xml-sitemap">sitemaps</a>!  Prepare one for users and one for search engines.</li>
<li>Use 404s when necessary.</li>
</ul>
<h3>Improving Site Structure</h3>
<ul>
<li>Create a naturally flowing hierarchy.</li>
<li>Avoid having all navigation based entirely on dropdown menus.  Your website should be accessible, pages should be conntected with normal text links.</li>
</ul>
<h3>Optimizing Content</h3>
<ul>
<li>Create compelling and useful content.</li>
<li>Offer unique and exclusive content.</li>
<li>Checkout the <a href="https://adwords.google.com/KeywordPlanner">Keyword Planner</a> by Google AdWords.</li>
<li>Write easy-to-read text.  Don&#x27;t embed text in images</li>
<li>Create fresh content.</li>
<li>Create content for users, not for search engines.</li>
<li>Write good anchor text: <code>&lt;a href=&quot;...&quot;&gt;Top Ten Basketball Shots&lt;/a&gt;</code> is a descriptive link.</li>
<li>Use <code>alt</code> tags on your images! This also helps with accessibility.</li>
<li>Use descriptive filenames for your images and assets!</li>
<li>Supply an <a href="https://support.google.com/webmasters/answer/178636">image sitemap file</a>!</li>
<li>Use heading tags correctly!  <code>&lt;h1&gt;</code> is the most important. <code>&lt;h6&gt;</code> is the least important.</li>
</ul>
<h3>Dealing with Crawlers</h3>
<ul>
<li>Restrict crawling with robots.txt.</li>
<li>Use <code>rel=&quot;nofollow&quot;</code> when necessary for links. Normally, user generated links through comment systems should be tagged with nofollow.</li>
</ul>
<h3>SEO for Mobile Phones</h3>
<ul>
<li>Configure mobiles sites so that they can be indexed accurately.</li>
<li>Create a <a href="https://support.google.com/webmasters/topic/2370586?rd=1">mobile sitemap</a>!</li>
<li>If you use User-agents to generate your mobile site, make sure to allow <code>Googlebot-Mobile</code>!</li>
<li>If you use a seperate domain for your mobile app, redirect them to the correct page!</li>
</ul>
<h3>Promotions and Analysis</h3>
<ul>
<li>Backlinks increase the value of your site ( backlinks are links that other people post to your site).</li>
<li>Know how to utilize your social media sites.</li>
<li>Utilize the Google Webmaster Tools.</li>
</ul>


This post was originally on a blog I started for creating web applications. This post goes into SEO considerations for web applications.

## Use Search Engine Friendly URLs

Use [semantic URLs](http://en.wikipedia.org/wiki/Semantic_URL).  Semantic URLs help improve usability and accessibility of a website.  The following table from the Wikipedia entry illistrates how to use semantic URLs:

| Non-semantic URL                                                | Semantic URL                              |
|-----------------------------------------------------------------|-------------------------------------------|
| `http://example.com/index.php?page=name`                          | `http://example.com/name`                   |
| `http://example.com/index.php?page=consulting/marketing`          | `http://example.com/consulting/marketing`   |
| `http://example.com/products?category=2&pid=25`                   | `http://example.com/products/2/25`          |
| `http://example.com/cgi-bin/feed.cgi?feed=news&frm=rss`           | `http://example.com/news.rss`               |
| `http://example.com/services/index.jsp?category=legal&id=patents` | `http://example.com/services/legal/patents` |
| `http://example.com/kb/index.php?cat=8&id=41`                     | `http://example.com/kb/8/41`                |
| `http://example.com/index.php?mod=profiles&id=193`                | `http://example.com/profiles/193`           |

## Use Hashbangs for Dynamic Content When You Normally Use Hashes

If you normally use the hash `#` character for dynamic content, change it to the hashbang `#!`.  This allows googlebot to use `#_REQUEST["_escaped_fragment_"]` instead of `#!`.

## Use Push State If You Can

If the user is using Firefox or Chromium, you can use `history.pushState( object, pageName, "./?param=1");` so that the address bar changes, but the page does not reload.  This allows you to use `?` instead of `#!` to keep dynamic content.

## Use Descriptive Links

When creating links, refrain from using "this" or "click here" or similar phrases for your links.  Link descriptions should be descriptive.

## Have an XML Sitemap

> Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.

Have a [sitemap](http://www.sitemaps.org/) at `/sitemap.xml` for search engines to crawl.

Have a [image sitemap](https://support.google.com/webmasters/answer/178636) for your images!

Create a [mobile sitemap](https://support.google.com/webmasters/topic/2370586?rd=1)!

## Use Canonical URLs For Duplicate Content

Use `<link rel="canonical" ... />` when you have duplicate content on multiple pages. Remember, multiple URLs that point to the same content coint as dusplicate content. Also remember that `?page=1&subject=my-subject` and `?subject=my-subject&page=1` are different pages to web crawlers!

## Setup Webmaster Tools

Use [Google Webmaster Tools](http://www.google.com/webmasters/) and [Bing Webmaster Tools](http://www.bing.com/toolbox/webmaster).

## Setup Google Analytics

Have [Google Analytics](http://www.google.com/analytics/) set up.  Or something like [Piwik](http://piwik.org/).

## Have robots.txt Configured Properly

Set your [robots.txt](http://en.wikipedia.org/wiki/Robots_exclusion_standard) file properly.  Each subdomain must have its own robots.txt file.

## 301 Redirect for WWW Requests

`www.example.com` or `example.com` should be your primary domain, not both. Have one 301 Redirect (Moved Permanently) to the other.

## Read the Search Engine Optimization Starter Guide

The [starter guide is viewable here](http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/webmasters/docs/search-engine-optimization-starter-guide.pdf).

### The Basics

- Create unique, accurate page titles. These titles are displayed in the search results. Titles reflect what is on each individual page.
- Use the description meta tag. Google will use them as snippets for your pages.
- Use simple to understand (ie "friendly") URLs.  URLs should contain relevant words. `http://mysite.com/article/basket-ball-scores` is a friendly URL.
- Navigation is very important for search engines.  Think of you website as folders that should be easy to navigate.  Use breadcrumbs!
- Use [sitemaps](#have-an-xml-sitemap)!  Prepare one for users and one for search engines.
- Use 404s when necessary.

### Improving Site Structure

- Create a naturally flowing hierarchy.
- Avoid having all navigation based entirely on dropdown menus.  Your website should be accessible, pages should be conntected with normal text links.

### Optimizing Content

- Create compelling and useful content.
- Offer unique and exclusive content.
- Checkout the [Keyword Planner](https://adwords.google.com/KeywordPlanner) by Google AdWords.
- Write easy-to-read text.  Don't embed text in images
- Create fresh content.
- Create content for users, not for search engines.
- Write good anchor text: `<a href="...">Top Ten Basketball Shots</a>` is a descriptive link.
- Use `alt` tags on your images! This also helps with accessibility.
- Use descriptive filenames for your images and assets!
- Supply an [image sitemap file](https://support.google.com/webmasters/answer/178636)!
- Use heading tags correctly!  `<h1>` is the most important. `<h6>` is the least important.

### Dealing with Crawlers

- Restrict crawling with robots.txt.
- Use `rel="nofollow"` when necessary for links. Normally, user generated links through comment systems should be tagged with nofollow.

### SEO for Mobile Phones

- Configure mobiles sites so that they can be indexed accurately.
- Create a [mobile sitemap](https://support.google.com/webmasters/topic/2370586?rd=1)!
- If you use User-agents to generate your mobile site, make sure to allow `Googlebot-Mobile`!
- If you use a seperate domain for your mobile app, redirect them to the correct page!

### Promotions and Analysis

- Backlinks increase the value of your site ( backlinks are links that other people post to your site).
- Know how to utilize your social media sites.
- Utilize the Google Webmaster Tools.
