8 articles tagged “web” — software, engineering, and related writing by idmontie.

Web blog posts - idmontie's Portfolio

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    blockquote: "blockquote",
    img: "img",
    p: "p",
    ...props.components
  }, {Separator} = _components;
  if (!Separator) _missingMdxReference("Separator", true);
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "What happens to SEO when the crawler itself becomes one of your primary consumers?"
    }), "\n", _jsx(_components.p, {
      children: "We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website."
    }), "\n", _jsxs(_components.p, {
      children: ["We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With ", _jsx(_components.a, {
        href: "https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age",
        children: "53% percent of traffic"
      }), " is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing."]
    }), "\n", _jsx(Separator, {}), "\n", _jsxs(_components.p, {
      children: ["When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - ", _jsx(_components.a, {
        href: "https://claude.com/",
        children: "Claudebot"
      }), ", ", _jsx(_components.a, {
        href: "https://meta.com/",
        children: "meta"
      }), ", other nefarious bots that would not return correct User Agent strings."]
    }), "\n", _jsx(_components.p, {
      children: "Why were AI crawlers hitting the application so differently from Google bot?"
    }), "\n", _jsx(Separator, {}), "\n", _jsx(_components.p, {
      children: "To understand why this was so surprising, we have to understand who the site was originally designed for."
    }), "\n", _jsx(_components.p, {
      children: "When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress."
    }), "\n", _jsx(_components.p, {
      children: "With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?"
    }), "\n", _jsx(_components.p, {
      children: "During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required."
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.a, {
        href: "https://nextjs.org/learn/seo/webcrawlers",
        children: "NextJS"
      }), " and ", _jsx(_components.a, {
        href: "https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics",
        children: "Google Search Central"
      }), " have typically described the indexing process like this:"]
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png",
        alt: "Crawling Indexing Process"
      })
    }), "\n", _jsx(_components.p, {
      children: "Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step."
    }), "\n", _jsx(_components.p, {
      children: "But Googlebot wasn’t the problem."
    }), "\n", _jsx(Separator, {}), "\n", _jsx(_components.p, {
      children: "AI Crawlers hammer the application."
    }), "\n", _jsx(_components.p, {
      children: "Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see."
    }), "\n", _jsx(_components.p, {
      children: "We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses."
    }), "\n", _jsx(_components.p, {
      children: "Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers."
    }), "\n", _jsx(_components.p, {
      children: "The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them."
    }), "\n", _jsx(Separator, {}), "\n", _jsxs(_components.p, {
      children: ["Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, ", _jsx(_components.a, {
        href: "https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/",
        children: "The Web Is All Bots Now"
      }), ", which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots."]
    }), "\n", _jsxs(_components.p, {
      children: ["Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post ", _jsx(_components.a, {
        href: "https://people.kernel.org/monsieuricon/creepy-crawlies",
        children: "Creepy Crawlies"
      }), " that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html."
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt."
    }), "\n", _jsx(_components.p, {
      children: "And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now."
    }), "\n", _jsx(_components.p, {
      children: "The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};
function _missingMdxReference(id, component) {
  throw new Error("Expected " + (component ? "component" : "object") + " `" + id + "` to be defined: you likely forgot to import, pass, or provide it.");
}


<p>What happens to SEO when the crawler itself becomes one of your primary consumers?</p>
<p>We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website.</p>
<p>We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With <a href="https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age">53% percent of traffic</a> is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - <a href="https://claude.com/">Claudebot</a>, <a href="https://meta.com/">meta</a>, other nefarious bots that would not return correct User Agent strings.</p>
<p>Why were AI crawlers hitting the application so differently from Google bot?</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>To understand why this was so surprising, we have to understand who the site was originally designed for.</p>
<p>When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress.</p>
<p>With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?</p>
<p>During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required.</p>
<p><a href="https://nextjs.org/learn/seo/webcrawlers">NextJS</a> and <a href="https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics">Google Search Central</a> have typically described the indexing process like this:</p>
<p><img alt="Crawling Indexing Process" loading="lazy" width="1" height="1" decoding="async" data-nimg="1" class="mx-auto block !h-auto !w-auto max-h-[500px] max-w-full" style="color:transparent;width:auto;height:auto;max-width:100%;max-height:500px;margin:auto" src="/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png"/></p>
<p>Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step.</p>
<p>But Googlebot wasn’t the problem.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>AI Crawlers hammer the application.</p>
<p>Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see.</p>
<p>We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses.</p>
<p>Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers.</p>
<p>The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them.</p>
<div class="flex w-full items-center justify-center gap-2 py-4"><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div><div class="!m-0 h-1 w-1 rounded-full bg-gray-300"></div></div>
<p>Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, <a href="https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/">The Web Is All Bots Now</a>, which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots.</p>
<p>Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post <a href="https://people.kernel.org/monsieuricon/creepy-crawlies">Creepy Crawlies</a> that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:</p>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.</p>
</blockquote>
<p>At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt.</p>
<p>And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now.</p>
<p>The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps.</p>


What happens to SEO when the crawler itself becomes one of your primary consumers?

We all intuitively know how SEO is supposed to work. Google and other search engines have been working for decades to make sure that search returns high quality content, rather than spam that stuffs the page with keywords. We would write meaningful, quality content, and Google with crawl the page, index the data, and humans would search and visit the website.

We built the modern web around the assumption that crawlers are lightweight readers of web pages. Increasingly, they are behaving more like automated users. With [53% percent of traffic](https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age) is bot traffic on the internet, how we write web applications, serve them, and the end goal of SEO may be changing.

<Separator />

When we launched the new version of Couchsurfing, we expected all sorts of issues. We anticipated a large spike in traffic when we launched, even though we weren’t planning on any immediate marketing push. But what we didn’t expect was to continue to see large spikes in traffic after we launched. These spikes wouldn’t just get a web page via a cURL GET request, but they would request all the different locales, all the image assets on the page, and all the other assets on the page. Some of the traffic we looked at would even attempt to create URLs that didn’t exist in our web application, hammering us with requests that we would 404. These weren’t from Googlebot. These were tons of other crawlers that were scraping our site for information - [Claudebot](https://claude.com/), [meta](https://meta.com/), other nefarious bots that would not return correct User Agent strings.

Why were AI crawlers hitting the application so differently from Google bot?

<Separator />

To understand why this was so surprising, we have to understand who the site was originally designed for. 

When we started thinking about the next version of Couchsurfing, we decided that we would want to focus on “SSR” - Server Side Rendering” - as much of the page as possible. That is essentially how the existing application worked and we wanted to make sure the majority of the page wouldn’t have cascading loaders like Single Page Applications often have. We also knew wanted to move the application into React. The previous team had already tried implementing a few React components into the application, but it was slow progress.

With SSR and React as the baseline of what we wanted for the web application, we settled on NextJS as the framework. From there, every product kick-off would start with a few key questions: is this a public facing feature or page? And how much of it should be server-side rendered?

During the inception of this new project, there was quite a lot of discussion by Vercel, Next.js and web engineering at large about how much SSR was really required. 

[NextJS](https://nextjs.org/learn/seo/webcrawlers) and [Google Search Central](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) have typically described the indexing process like this:

![Crawling Indexing Process](/media/2026-09-08-are-crawlers-our-primary-consumers/crawlqueue.png)

Where the crawler will have items added to the render queue to actually execute the JS on the page. When we kicked off the project, we assumed that with the number of pages we were going to have to re-index with Google, that this step would take awhile to happen for our site. And when it did happen, we wouldn’t expect all our pages to be rendered in a reasonable timeframe. We estimated that our new multi-sitemap would have millions of pages, rather than the single 500,000 entries we had in the existing sitemap. We would have to worry about the crawl budget, so we settled on SSR’ing as much of the page as possible so that indexing could happen without the rendering step.

But Googlebot wasn’t the problem.

<Separator />

AI Crawlers hammer the application.

Crawling traffic. Thousands, hundreds of thousands of requests a day. We had anticipated Googlebot - the traditional bot that crawls pages to index them for Google search results. And that is the bot we had optimized SEO for. But we saw way more traffic than we expected from other crawl bots. We saw Meta’s bot crawling every single one of our user pages, location pages. We saw claude bot hit every single page in every single language we had localizes We saw weird scrapers that didn’t properly represent themselves try to hit thousand of user pages that required users to be logged in to see.

We were seeing lots of burst traffic. Claudebot for example would suddenly hit us with thousands of requests all at once. Our WAF had 429 rules to tell it to back-off, and we even had additional rate-limits on certain APIs that return 429 as well. This helped when the requests would come from the same IP address, but we would often see these large bursts come from clusters of IP addresses.

Pages would start responding slow due the insane influx of traffic, which wasn’t just to the page it was crawling, but to all the assets on the page, including images, JS files, and additional APIs that the page would normally request once hydrated on the client. This is the opposite of what we had expected - we expected crawling bots to simple do a GET to the page they were requesting and crawl the contents of that page. But crawling traffic for these LLM bots would actually evaluate the page like an end-user - some explicitly using tools like Puppeteer or Playwright according to the request headers.

The crawler was no longer simply requesting the server-side rendered version of the page. Instead the crawlers were acting like full users - hitting the page, requesting all the assets and image, and then following links on that page. Worse, some acted like malicious attacked - trying to enumerate pages that didn’t even exist to see if there was content on them.

<Separator />

Before AI, we would have considered these attacks. After AI, this is the new normal. As one Redditor describes, [The Web Is All Bots Now](https://www.reddit.com/r/webdev/comments/1vy80na/the_web_is_all_bots_now/), which means as engineers we have had to pivot from the old way of thinking about who our pages are serving to this new normal that most of the requests are going to bots.

Even Konstantin Ryabitsev, Director of the Linux Foundation, has come out with a blog post [Creepy Crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies) that details just how much CPU bots are using hitting the git.kernel.org site, using the HTML version of the site instead of more efficient mechanisms:

> TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
> 

At the end of the day, we have to decide from an engineering stand point what we do about this issue. We can rate-limit requests, we can block certain user-agents, and we can set-up robots.txt files with rules. But not all bots are playing by the rules. Some bots aren’t identifying themselves via unique user-agent strings. Some aren’t even looking at robots.txt.

And by blocking some agents, our sites may not be cited by agents, leading to less traffic – which is already decreasing since users are asking agents instead of doing text searches now.

The future is still foggy around SEO and AEO, but the current situation is clear: the majority of traffic on the internet is bots, and that is going to shape how we build and serve apps.


Are Crawlers Our Primary Consumers?

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    img: "img",
    li: "li",
    ol: "ol",
    p: "p",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "At the height of the GPU craze, around 2021, when everyone was holed up in their homes due to COVID, the 30 series of Nvidia GPUs was extremely difficult to purchase. The scarcity was due to a few factors: the COVID pandemic had disrupted supply chains across the world, Taiwan was experiencing a severe drought that was impacting chip production, and cryptocurrency, especially ETH, was gaining value, making mining it was the latest GPU series incredibly lucrative."
    }), "\n", _jsx(_components.p, {
      children: "Since the supply was low and demand was high, this created two types of markets: one was the second hand market for GPUs where an $800 GPU could go for $3500 in cash when sold to someone mining cryptocurrency. Because of the massive mark up, buying and selling software to automate the process of buying GPUs when they \"dropped\" on a site was also growing as a market. These were called \"bots\" in the community, where customers trying to get a single GPU for their gaming PC were competing against hundreds of not thousands of bots flooding websites when they dropped. Common sites like Nvidia, AMD, Best Buy, and Zotac would get flooded with millions of requests when they added GPU stock the site."
    }), "\n", _jsx(_components.p, {
      children: "To combat this, different market places tried different techniques to verify that humans were the ones trying to access the site. But due to time constraints, these never seemed to be properly implemented."
    }), "\n", _jsx(_components.p, {
      children: "Zotac added a Cloudflare verification to their site that would try to verify that a user was real. The setting they used though attempted to not disrupt real users, but only issue a challenge to visitors it deemed \"bots\". A puppeteer script could easily bypass this by first visiting other sites that used cloud flare (but didn't issue challenges) and then visiting the Zotac site. A fresh puppeteer instance would be issued a challenge, but one that visited other sites before would not."
    }), "\n", _jsx(_components.p, {
      children: "There were other tricks people used to help them attempt to secure GPUs. My favorite was adding a Zotac TShirt to your cart before a drop so that you could input all of your shipping and payment information. Then when a drop was detected, you'd simply issue the add to cart API call, and then the checkout API call. This would bypass the checkout walkthrough. When the website was flooded with millions of API calls, you only needed two API calls to succeed, rather than 10 or more."
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2025-06-16-gpus-best-buy-queues/do-something.jpg",
        alt: "Do something zotac"
      })
    }), "\n", _jsx(_components.p, {
      children: "Automating this way worked so well that a puppeteer script that I wrote was able to add 4 GPUs to one cart, send me a Pushbullet notification to wake me up in the middle of the night, and I just had to click one last step to verify the checkout manually since I didn't trust a script with actually making card payments on my behalf."
    }), "\n", _jsx(_components.p, {
      children: "Zotac did finally learn from this and finally upped the Cloudflare bot check to issue a challenge to all users during the rest of the 30 series release. They also released a new site that made it harder to track when GPU stock was released."
    }), "\n", _jsx(_components.p, {
      children: "Best Buy must have had many complaints about the terrible system they had. Any GPU stock they had would almost immediately be gone by the time s human would show up to try to purchase one. At the time, they had a graphql-like API that allowed anyone to request with any frequency the in-stock status of any number of skus on their system. If you used puppeteer to visit the site, grab the cookies, and then issue an API request to this API, you could consistently check the status of all GPU skus every 5 or so seconds for the entire day. No rate limits, no other form of authentication other than public cookies."
    }), "\n", _jsx(_components.p, {
      children: "So that explained how bots were able to detect GPUs so quickly and with no form of bot checking or challenge system, once the GPU was in someone's cart, it was very easy to checkout since the Best Buy site would continue to work even when flooded with requests."
    }), "\n", _jsx(_components.p, {
      children: "To mitigate this, some poor group of developers was tasked with creating a queue system. I'm guessing the time crunch was real because when I went to investigate how this queue worked, it was entirely determined by the user cookie that was issued."
    }), "\n", _jsx(_components.p, {
      children: "Here's how the queue worked for the end user: the user would get a notification from their favorite discord server that there was a drop (since the graphql-like endpoint was still available). They would go to the page for the GPU they wanted to purchase and click \"add to cart\". The button would then change to a \"waiting in queue\" message and be disabled. There was no indication how long that queue would be. Then after some time, the button would change back to \"add to cart\". The user clicks it, and if there was still stock, the GPU would be added to cart. Then the user needed to quickly checkout before the stock ran out."
    }), "\n", _jsx(_components.p, {
      children: "I was really curious on the part where there was no indication of how long the queue was. When testing using a 5950 AMD CPU that still had stock, but had the queue system enabled, I determined that the wait time was effectively random. It didn't matter if you were the first one to click that button, it would just assign a random queue time to you before you could cart the item."
    }), "\n", _jsx(_components.p, {
      children: "I ran more tests. I recorded a small sample of wait times and they would vary from a minute to over an hour."
    }), "\n", _jsx(_components.p, {
      children: "I decided to investigate what code determined when the button changed back to \"add to cart\". No API calls were made while waiting in queue. Huh okay so the browser code already knows how long the queue time is."
    }), "\n", _jsx(_components.p, {
      children: "The UI code on the Best Buy site is not just minified, but also mangled. This process will replace function names, introduce additional function calls to obfuscate what functions are doing, and generally makes it very difficult to debug. After tracing back what changed the button and the call stack, I was able to finally piece together how the queue time was determined."
    }), "\n", _jsx(_components.p, {
      children: "When a user visited the best buy site, they were issued a public cookie. It looked like a special UUID. The middle section of the UUID must have some special properties to meet some hash check value from what I could tell. The point is that I must have had some guaranteed randomness to it that gave the developers the idea to reuse this section to deterministically evaluate a queue time on both the browser and the server."
    }), "\n", _jsx(_components.p, {
      children: "I created a browser extension to test the hypothesis:"
    }), "\n", _jsxs(_components.ol, {
      children: ["\n", _jsx(_components.li, {
        children: "Show the queue time for the current cookie on the page by injecting some HTML."
      }), "\n", _jsx(_components.li, {
        children: "Add a button to \"roll for a better queue time\". This would issue a request to get 20 new cookies, figure out which queue time was best, set the browser cookie to that cookies and display the new queue time."
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "I tested with the 5950 CPU and saw my queue time go from 30 minutes to 1 minute. Success!"
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2025-06-16-gpus-best-buy-queues/queue-times.png",
        alt: "Queue times"
      })
    }), "\n", _jsx(_components.p, {
      children: "I had a close group of friends that I rolled this out to. The next Best Buy drop happened and we were all able to get the founders edition cards. Finally!"
    }), "\n", _jsx(_components.p, {
      children: "There was a popular bot net that effectively did the same thing. I don't know if they knew how to calculate the queue times, but they would spin up 20+ browsers with the best buy site and wait for the first one to have the add to cart button, and then give the purchaser of the bot net subscription the cookies for that session."
    }), "\n", _jsx(_components.p, {
      children: "Times were dire and I thought this approach from best buy was garbage. It effectively made individuals - who might have stood a chance before - now have to wait some random amount of time against those with those who had hundreds of bots trying to get a good queue time."
    }), "\n", _jsx(_components.p, {
      children: "Conveniently someone reached out to me to commission a chrome extension related to getting GPUs. They wanted to know what was possible and I told them I figured out the queue system times, and how an average person could get a better queue time. We released a branded version of my extension and some 10,000 people downloaded it."
    }), "\n", _jsx(_components.p, {
      children: "And then I waited for the next Best Buy drop."
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2025-06-16-gpus-best-buy-queues/a-rare-occurence.png",
        alt: "A rare occurrence"
      })
    }), "\n", _jsx(_components.p, {
      children: "The alert went out that there was stock."
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.img, {
        src: "/media/2025-06-16-gpus-best-buy-queues/extension.png",
        alt: "extension"
      })
    }), "\n", _jsx(_components.p, {
      children: "And the website crashed 20 seconds into the drop."
    }), "\n", _jsx(_components.p, {
      children: "I was actually surprised since their site had done so well before. But I guess people rolling for new cookies 20 times in a second was enough to bring the site down momentarily."
    }), "\n", _jsx(_components.p, {
      children: "After that drop, Best Buy held off on GPU drops for over a month. When the next drop finally happened, they had changed the queue system slightly. The cookie was no longer the source for the queue time, and rolling for a new cookie did not give you a new queue time. Finally."
    }), "\n", _jsx(_components.p, {
      children: "Your queue position was still effectively random, but at least it could no longer be gamed by requesting new sessions. Nor could you know your queue time even if you did ask for new sessions."
    }), "\n", _jsx(_components.p, {
      children: "Props to Best Buy for continuing to experience with this too. Pokemon cards are the new drops and they force people to use the mobile app now to request a time in queue."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>At the height of the GPU craze, around 2021, when everyone was holed up in their homes due to COVID, the 30 series of Nvidia GPUs was extremely difficult to purchase. The scarcity was due to a few factors: the COVID pandemic had disrupted supply chains across the world, Taiwan was experiencing a severe drought that was impacting chip production, and cryptocurrency, especially ETH, was gaining value, making mining it was the latest GPU series incredibly lucrative.</p>
<p>Since the supply was low and demand was high, this created two types of markets: one was the second hand market for GPUs where an $800 GPU could go for $3500 in cash when sold to someone mining cryptocurrency. Because of the massive mark up, buying and selling software to automate the process of buying GPUs when they &quot;dropped&quot; on a site was also growing as a market. These were called &quot;bots&quot; in the community, where customers trying to get a single GPU for their gaming PC were competing against hundreds of not thousands of bots flooding websites when they dropped. Common sites like Nvidia, AMD, Best Buy, and Zotac would get flooded with millions of requests when they added GPU stock the site.</p>
<p>To combat this, different market places tried different techniques to verify that humans were the ones trying to access the site. But due to time constraints, these never seemed to be properly implemented.</p>
<p>Zotac added a Cloudflare verification to their site that would try to verify that a user was real. The setting they used though attempted to not disrupt real users, but only issue a challenge to visitors it deemed &quot;bots&quot;. A puppeteer script could easily bypass this by first visiting other sites that used cloud flare (but didn&#x27;t issue challenges) and then visiting the Zotac site. A fresh puppeteer instance would be issued a challenge, but one that visited other sites before would not.</p>
<p>There were other tricks people used to help them attempt to secure GPUs. My favorite was adding a Zotac TShirt to your cart before a drop so that you could input all of your shipping and payment information. Then when a drop was detected, you&#x27;d simply issue the add to cart API call, and then the checkout API call. This would bypass the checkout walkthrough. When the website was flooded with millions of API calls, you only needed two API calls to succeed, rather than 10 or more.</p>
<p><img alt="Do something zotac" loading="lazy" width="1" height="1" decoding="async" data-nimg="1" class="mx-auto block !h-auto !w-auto max-h-[500px] max-w-full" style="color:transparent;width:auto;height:auto;max-width:100%;max-height:500px;margin:auto" src="/media/2025-06-16-gpus-best-buy-queues/do-something.jpg"/></p>
<p>Automating this way worked so well that a puppeteer script that I wrote was able to add 4 GPUs to one cart, send me a Pushbullet notification to wake me up in the middle of the night, and I just had to click one last step to verify the checkout manually since I didn&#x27;t trust a script with actually making card payments on my behalf.</p>
<p>Zotac did finally learn from this and finally upped the Cloudflare bot check to issue a challenge to all users during the rest of the 30 series release. They also released a new site that made it harder to track when GPU stock was released.</p>
<p>Best Buy must have had many complaints about the terrible system they had. Any GPU stock they had would almost immediately be gone by the time s human would show up to try to purchase one. At the time, they had a graphql-like API that allowed anyone to request with any frequency the in-stock status of any number of skus on their system. If you used puppeteer to visit the site, grab the cookies, and then issue an API request to this API, you could consistently check the status of all GPU skus every 5 or so seconds for the entire day. No rate limits, no other form of authentication other than public cookies.</p>
<p>So that explained how bots were able to detect GPUs so quickly and with no form of bot checking or challenge system, once the GPU was in someone&#x27;s cart, it was very easy to checkout since the Best Buy site would continue to work even when flooded with requests.</p>
<p>To mitigate this, some poor group of developers was tasked with creating a queue system. I&#x27;m guessing the time crunch was real because when I went to investigate how this queue worked, it was entirely determined by the user cookie that was issued.</p>
<p>Here&#x27;s how the queue worked for the end user: the user would get a notification from their favorite discord server that there was a drop (since the graphql-like endpoint was still available). They would go to the page for the GPU they wanted to purchase and click &quot;add to cart&quot;. The button would then change to a &quot;waiting in queue&quot; message and be disabled. There was no indication how long that queue would be. Then after some time, the button would change back to &quot;add to cart&quot;. The user clicks it, and if there was still stock, the GPU would be added to cart. Then the user needed to quickly checkout before the stock ran out.</p>
<p>I was really curious on the part where there was no indication of how long the queue was. When testing using a 5950 AMD CPU that still had stock, but had the queue system enabled, I determined that the wait time was effectively random. It didn&#x27;t matter if you were the first one to click that button, it would just assign a random queue time to you before you could cart the item.</p>
<p>I ran more tests. I recorded a small sample of wait times and they would vary from a minute to over an hour.</p>
<p>I decided to investigate what code determined when the button changed back to &quot;add to cart&quot;. No API calls were made while waiting in queue. Huh okay so the browser code already knows how long the queue time is.</p>
<p>The UI code on the Best Buy site is not just minified, but also mangled. This process will replace function names, introduce additional function calls to obfuscate what functions are doing, and generally makes it very difficult to debug. After tracing back what changed the button and the call stack, I was able to finally piece together how the queue time was determined.</p>
<p>When a user visited the best buy site, they were issued a public cookie. It looked like a special UUID. The middle section of the UUID must have some special properties to meet some hash check value from what I could tell. The point is that I must have had some guaranteed randomness to it that gave the developers the idea to reuse this section to deterministically evaluate a queue time on both the browser and the server.</p>
<p>I created a browser extension to test the hypothesis:</p>
<ol>
<li>Show the queue time for the current cookie on the page by injecting some HTML.</li>
<li>Add a button to &quot;roll for a better queue time&quot;. This would issue a request to get 20 new cookies, figure out which queue time was best, set the browser cookie to that cookies and display the new queue time.</li>
</ol>
<p>I tested with the 5950 CPU and saw my queue time go from 30 minutes to 1 minute. Success!</p>
<p><img alt="Queue times" loading="lazy" width="1" height="1" decoding="async" data-nimg="1" class="mx-auto block !h-auto !w-auto max-h-[500px] max-w-full" style="color:transparent;width:auto;height:auto;max-width:100%;max-height:500px;margin:auto" src="/media/2025-06-16-gpus-best-buy-queues/queue-times.png"/></p>
<p>I had a close group of friends that I rolled this out to. The next Best Buy drop happened and we were all able to get the founders edition cards. Finally!</p>
<p>There was a popular bot net that effectively did the same thing. I don&#x27;t know if they knew how to calculate the queue times, but they would spin up 20+ browsers with the best buy site and wait for the first one to have the add to cart button, and then give the purchaser of the bot net subscription the cookies for that session.</p>
<p>Times were dire and I thought this approach from best buy was garbage. It effectively made individuals - who might have stood a chance before - now have to wait some random amount of time against those with those who had hundreds of bots trying to get a good queue time.</p>
<p>Conveniently someone reached out to me to commission a chrome extension related to getting GPUs. They wanted to know what was possible and I told them I figured out the queue system times, and how an average person could get a better queue time. We released a branded version of my extension and some 10,000 people downloaded it.</p>
<p>And then I waited for the next Best Buy drop.</p>
<p><img alt="A rare occurrence" loading="lazy" width="1" height="1" decoding="async" data-nimg="1" class="mx-auto block !h-auto !w-auto max-h-[500px] max-w-full" style="color:transparent;width:auto;height:auto;max-width:100%;max-height:500px;margin:auto" src="/media/2025-06-16-gpus-best-buy-queues/a-rare-occurence.png"/></p>
<p>The alert went out that there was stock.</p>
<p><img alt="extension" loading="lazy" width="1" height="1" decoding="async" data-nimg="1" class="mx-auto block !h-auto !w-auto max-h-[500px] max-w-full" style="color:transparent;width:auto;height:auto;max-width:100%;max-height:500px;margin:auto" src="/media/2025-06-16-gpus-best-buy-queues/extension.png"/></p>
<p>And the website crashed 20 seconds into the drop.</p>
<p>I was actually surprised since their site had done so well before. But I guess people rolling for new cookies 20 times in a second was enough to bring the site down momentarily.</p>
<p>After that drop, Best Buy held off on GPU drops for over a month. When the next drop finally happened, they had changed the queue system slightly. The cookie was no longer the source for the queue time, and rolling for a new cookie did not give you a new queue time. Finally.</p>
<p>Your queue position was still effectively random, but at least it could no longer be gamed by requesting new sessions. Nor could you know your queue time even if you did ask for new sessions.</p>
<p>Props to Best Buy for continuing to experience with this too. Pokemon cards are the new drops and they force people to use the mobile app now to request a time in queue.</p>

At the height of the GPU craze, around 2021, when everyone was holed up in their homes due to COVID, the 30 series of Nvidia GPUs was extremely difficult to purchase. The scarcity was due to a few factors: the COVID pandemic had disrupted supply chains across the world, Taiwan was experiencing a severe drought that was impacting chip production, and cryptocurrency, especially ETH, was gaining value, making mining it was the latest GPU series incredibly lucrative.



Since the supply was low and demand was high, this created two types of markets: one was the second hand market for GPUs where an \$800 GPU could go for \$3500 in cash when sold to someone mining cryptocurrency. Because of the massive mark up, buying and selling software to automate the process of buying GPUs when they "dropped" on a site was also growing as a market. These were called "bots" in the community, where customers trying to get a single GPU for their gaming PC were competing against hundreds of not thousands of bots flooding websites when they dropped. Common sites like Nvidia, AMD, Best Buy, and Zotac would get flooded with millions of requests when they added GPU stock the site.

To combat this, different market places tried different techniques to verify that humans were the ones trying to access the site. But due to time constraints, these never seemed to be properly implemented.

Zotac added a Cloudflare verification to their site that would try to verify that a user was real. The setting they used though attempted to not disrupt real users, but only issue a challenge to visitors it deemed "bots". A puppeteer script could easily bypass this by first visiting other sites that used cloud flare (but didn't issue challenges) and then visiting the Zotac site. A fresh puppeteer instance would be issued a challenge, but one that visited other sites before would not.

There were other tricks people used to help them attempt to secure GPUs. My favorite was adding a Zotac TShirt to your cart before a drop so that you could input all of your shipping and payment information. Then when a drop was detected, you'd simply issue the add to cart API call, and then the checkout API call. This would bypass the checkout walkthrough. When the website was flooded with millions of API calls, you only needed two API calls to succeed, rather than 10 or more.

![Do something zotac](/media/2025-06-16-gpus-best-buy-queues/do-something.jpg)

Automating this way worked so well that a puppeteer script that I wrote was able to add 4 GPUs to one cart, send me a Pushbullet notification to wake me up in the middle of the night, and I just had to click one last step to verify the checkout manually since I didn't trust a script with actually making card payments on my behalf.

Zotac did finally learn from this and finally upped the Cloudflare bot check to issue a challenge to all users during the rest of the 30 series release. They also released a new site that made it harder to track when GPU stock was released.

Best Buy must have had many complaints about the terrible system they had. Any GPU stock they had would almost immediately be gone by the time s human would show up to try to purchase one. At the time, they had a graphql-like API that allowed anyone to request with any frequency the in-stock status of any number of skus on their system. If you used puppeteer to visit the site, grab the cookies, and then issue an API request to this API, you could consistently check the status of all GPU skus every 5 or so seconds for the entire day. No rate limits, no other form of authentication other than public cookies.

So that explained how bots were able to detect GPUs so quickly and with no form of bot checking or challenge system, once the GPU was in someone's cart, it was very easy to checkout since the Best Buy site would continue to work even when flooded with requests.

To mitigate this, some poor group of developers was tasked with creating a queue system. I'm guessing the time crunch was real because when I went to investigate how this queue worked, it was entirely determined by the user cookie that was issued.

Here's how the queue worked for the end user: the user would get a notification from their favorite discord server that there was a drop (since the graphql-like endpoint was still available). They would go to the page for the GPU they wanted to purchase and click "add to cart". The button would then change to a "waiting in queue" message and be disabled. There was no indication how long that queue would be. Then after some time, the button would change back to "add to cart". The user clicks it, and if there was still stock, the GPU would be added to cart. Then the user needed to quickly checkout before the stock ran out.

I was really curious on the part where there was no indication of how long the queue was. When testing using a 5950 AMD CPU that still had stock, but had the queue system enabled, I determined that the wait time was effectively random. It didn't matter if you were the first one to click that button, it would just assign a random queue time to you before you could cart the item.

I ran more tests. I recorded a small sample of wait times and they would vary from a minute to over an hour.

I decided to investigate what code determined when the button changed back to "add to cart". No API calls were made while waiting in queue. Huh okay so the browser code already knows how long the queue time is.

The UI code on the Best Buy site is not just minified, but also mangled. This process will replace function names, introduce additional function calls to obfuscate what functions are doing, and generally makes it very difficult to debug. After tracing back what changed the button and the call stack, I was able to finally piece together how the queue time was determined.

When a user visited the best buy site, they were issued a public cookie. It looked like a special UUID. The middle section of the UUID must have some special properties to meet some hash check value from what I could tell. The point is that I must have had some guaranteed randomness to it that gave the developers the idea to reuse this section to deterministically evaluate a queue time on both the browser and the server.

I created a browser extension to test the hypothesis:

1. Show the queue time for the current cookie on the page by injecting some HTML.
2. Add a button to "roll for a better queue time". This would issue a request to get 20 new cookies, figure out which queue time was best, set the browser cookie to that cookies and display the new queue time.

I tested with the 5950 CPU and saw my queue time go from 30 minutes to 1 minute. Success!

![Queue times](/media/2025-06-16-gpus-best-buy-queues/queue-times.png)

I had a close group of friends that I rolled this out to. The next Best Buy drop happened and we were all able to get the founders edition cards. Finally!

There was a popular bot net that effectively did the same thing. I don't know if they knew how to calculate the queue times, but they would spin up 20+ browsers with the best buy site and wait for the first one to have the add to cart button, and then give the purchaser of the bot net subscription the cookies for that session.

Times were dire and I thought this approach from best buy was garbage. It effectively made individuals - who might have stood a chance before - now have to wait some random amount of time against those with those who had hundreds of bots trying to get a good queue time.

Conveniently someone reached out to me to commission a chrome extension related to getting GPUs. They wanted to know what was possible and I told them I figured out the queue system times, and how an average person could get a better queue time. We released a branded version of my extension and some 10,000 people downloaded it.

And then I waited for the next Best Buy drop.

![A rare occurrence](/media/2025-06-16-gpus-best-buy-queues/a-rare-occurence.png)

The alert went out that there was stock.

![extension](/media/2025-06-16-gpus-best-buy-queues/extension.png)

And the website crashed 20 seconds into the drop.

I was actually surprised since their site had done so well before. But I guess people rolling for new cookies 20 times in a second was enough to bring the site down momentarily.

After that drop, Best Buy held off on GPU drops for over a month. When the next drop finally happened, they had changed the queue system slightly. The cookie was no longer the source for the queue time, and rolling for a new cookie did not give you a new queue time. Finally.

Your queue position was still effectively random, but at least it could no longer be gamed by requesting new sessions. Nor could you know your queue time even if you did ask for new sessions.

Props to Best Buy for continuing to experience with this too. Pokemon cards are the new drops and they force people to use the mobile app now to request a time in queue.

GPUs and Best Buy Queues

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    li: "li",
    p: "p",
    ul: "ul",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "I was thinking about all the little projects I work on and how they grow over time. The applications end up hitting some milestones and end up needing similar functionality that compliments the core features. A small project ends up getting complex enough that it requires some unit tests. I’ll go to deploy the project and now I need some deployment scripts and analytics to ensure the application is running correctly. The technology may change between each project, but web applications always seem to have the same steps that need to be taken to strengthen the application as it is scaled up."
    }), "\n", _jsx(_components.p, {
      children: "At a large company, adding a new set of functionality always has a suite of concerns to think through before implementation: how will we deploy this feature, how do we validate that users are using the feature like we expected them to, how do we monitor for bugs and errors? This is on top of the basic functionality of actually writing and testing that new feature."
    }), "\n", _jsx(_components.p, {
      children: "What if we thought about this like Maslow’s Hierarchy of Needs, but in the context of a web application."
    }), "\n", _jsx(_components.p, {
      children: "Maslow's Hierarchy of Needs is a psychological model of human motivation proposed by Abraham Maslow in 1943. The model describes a hierarchy of human needs, beginning with basic physiological needs such as food and shelter and progressing upwards to higher-level needs such as self-actualization. Maslow argued that as humans satisfy their basic needs, they can move on to satisfy their higher-level needs."
    }), "\n", _jsx(_components.p, {
      children: "Applying Maslow's Hierarchy of Needs to web applications, we can identify the different levels of needs that need to be met in order to make a web application successful."
    }), "\n", _jsx(_components.p, {
      children: "Basic functionality:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "A single build/run script"
      }), "\n", _jsx(_components.li, {
        children: "Basic functionality (e.g. CRUD operations for a web app)"
      }), "\n", _jsx(_components.li, {
        children: "User interface (e.g. layout, navigation, responsive design)"
      }), "\n", _jsx(_components.li, {
        children: "Integration with external services (e.g. databases, APIs)"
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Safety and security:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Linting (e.g. ESLint, Prettier)"
      }), "\n", _jsx(_components.li, {
        children: "Unit tests (e.g. Jest, Mocha)"
      }), "\n", _jsx(_components.li, {
        children: "Basic error handling and reporting (e.g. logging, alerting)"
      }), "\n", _jsx(_components.li, {
        children: "Input validation (e.g. form validation)"
      }), "\n", _jsx(_components.li, {
        children: "Security best practices (e.g. encryption, password hashing, session management)"
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Love and belonging:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Basic analytics (e.g. page views, user engagement)"
      }), "\n", _jsx(_components.li, {
        children: "User authentication and authorization (e.g. sign-up, login, role-based access control)"
      }), "\n", _jsx(_components.li, {
        children: "User feedback (e.g. contact form, survey)"
      }), "\n", _jsx(_components.li, {
        children: "Social media integration (e.g. sharing, commenting)"
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Esteem:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Advanced analytics (e.g. user behavior tracking, A/B testing)"
      }), "\n", _jsx(_components.li, {
        children: "Performance monitoring (e.g. load testing, monitoring of server resources)"
      }), "\n", _jsx(_components.li, {
        children: "User experience optimization (e.g. user testing, usability analysis)"
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Self-actualization:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "End-to-end testing (e.g. Selenium, Cypress)"
      }), "\n", _jsx(_components.li, {
        children: "Accessibility and internationalization (e.g. support for screen readers, translation)"
      }), "\n", _jsx(_components.li, {
        children: "Scalability (e.g. load balancing, caching)"
      }), "\n", _jsx(_components.li, {
        children: "Continuous integration and delivery (e.g. Jenkins, Travis CI)"
      }), "\n", _jsx(_components.li, {
        children: "Deployment (e.g. Docker, Kubernetes)"
      }), "\n", _jsx(_components.li, {
        children: "Automated testing (e.g. unit test, integration test)"
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Of course this list isn’t exhaustive, but I’ve been thinking about it as more of a checklist to build upon when working on small side-projects that end up getting significant attention and development time."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>I was thinking about all the little projects I work on and how they grow over time. The applications end up hitting some milestones and end up needing similar functionality that compliments the core features. A small project ends up getting complex enough that it requires some unit tests. I’ll go to deploy the project and now I need some deployment scripts and analytics to ensure the application is running correctly. The technology may change between each project, but web applications always seem to have the same steps that need to be taken to strengthen the application as it is scaled up.</p>
<p>At a large company, adding a new set of functionality always has a suite of concerns to think through before implementation: how will we deploy this feature, how do we validate that users are using the feature like we expected them to, how do we monitor for bugs and errors? This is on top of the basic functionality of actually writing and testing that new feature.</p>
<p>What if we thought about this like Maslow’s Hierarchy of Needs, but in the context of a web application.</p>
<p>Maslow&#x27;s Hierarchy of Needs is a psychological model of human motivation proposed by Abraham Maslow in 1943. The model describes a hierarchy of human needs, beginning with basic physiological needs such as food and shelter and progressing upwards to higher-level needs such as self-actualization. Maslow argued that as humans satisfy their basic needs, they can move on to satisfy their higher-level needs.</p>
<p>Applying Maslow&#x27;s Hierarchy of Needs to web applications, we can identify the different levels of needs that need to be met in order to make a web application successful.</p>
<p>Basic functionality:</p>
<ul>
<li>A single build/run script</li>
<li>Basic functionality (e.g. CRUD operations for a web app)</li>
<li>User interface (e.g. layout, navigation, responsive design)</li>
<li>Integration with external services (e.g. databases, APIs)</li>
</ul>
<p>Safety and security:</p>
<ul>
<li>Linting (e.g. ESLint, Prettier)</li>
<li>Unit tests (e.g. Jest, Mocha)</li>
<li>Basic error handling and reporting (e.g. logging, alerting)</li>
<li>Input validation (e.g. form validation)</li>
<li>Security best practices (e.g. encryption, password hashing, session management)</li>
</ul>
<p>Love and belonging:</p>
<ul>
<li>Basic analytics (e.g. page views, user engagement)</li>
<li>User authentication and authorization (e.g. sign-up, login, role-based access control)</li>
<li>User feedback (e.g. contact form, survey)</li>
<li>Social media integration (e.g. sharing, commenting)</li>
</ul>
<p>Esteem:</p>
<ul>
<li>Advanced analytics (e.g. user behavior tracking, A/B testing)</li>
<li>Performance monitoring (e.g. load testing, monitoring of server resources)</li>
<li>User experience optimization (e.g. user testing, usability analysis)</li>
</ul>
<p>Self-actualization:</p>
<ul>
<li>End-to-end testing (e.g. Selenium, Cypress)</li>
<li>Accessibility and internationalization (e.g. support for screen readers, translation)</li>
<li>Scalability (e.g. load balancing, caching)</li>
<li>Continuous integration and delivery (e.g. Jenkins, Travis CI)</li>
<li>Deployment (e.g. Docker, Kubernetes)</li>
<li>Automated testing (e.g. unit test, integration test)</li>
</ul>
<p>Of course this list isn’t exhaustive, but I’ve been thinking about it as more of a checklist to build upon when working on small side-projects that end up getting significant attention and development time.</p>


I was thinking about all the little projects I work on and how they grow over time. The applications end up hitting some milestones and end up needing similar functionality that compliments the core features. A small project ends up getting complex enough that it requires some unit tests. I’ll go to deploy the project and now I need some deployment scripts and analytics to ensure the application is running correctly. The technology may change between each project, but web applications always seem to have the same steps that need to be taken to strengthen the application as it is scaled up.

At a large company, adding a new set of functionality always has a suite of concerns to think through before implementation: how will we deploy this feature, how do we validate that users are using the feature like we expected them to, how do we monitor for bugs and errors? This is on top of the basic functionality of actually writing and testing that new feature.

What if we thought about this like Maslow’s Hierarchy of Needs, but in the context of a web application.

Maslow's Hierarchy of Needs is a psychological model of human motivation proposed by Abraham Maslow in 1943. The model describes a hierarchy of human needs, beginning with basic physiological needs such as food and shelter and progressing upwards to higher-level needs such as self-actualization. Maslow argued that as humans satisfy their basic needs, they can move on to satisfy their higher-level needs.

Applying Maslow's Hierarchy of Needs to web applications, we can identify the different levels of needs that need to be met in order to make a web application successful.

Basic functionality:

- A single build/run script
- Basic functionality (e.g. CRUD operations for a web app)
- User interface (e.g. layout, navigation, responsive design)
- Integration with external services (e.g. databases, APIs)

Safety and security:

- Linting (e.g. ESLint, Prettier)
- Unit tests (e.g. Jest, Mocha)
- Basic error handling and reporting (e.g. logging, alerting)
- Input validation (e.g. form validation)
- Security best practices (e.g. encryption, password hashing, session management)

Love and belonging:

- Basic analytics (e.g. page views, user engagement)
- User authentication and authorization (e.g. sign-up, login, role-based access control)
- User feedback (e.g. contact form, survey)
- Social media integration (e.g. sharing, commenting)

Esteem:

- Advanced analytics (e.g. user behavior tracking, A/B testing)
- Performance monitoring (e.g. load testing, monitoring of server resources)
- User experience optimization (e.g. user testing, usability analysis)

Self-actualization:

- End-to-end testing (e.g. Selenium, Cypress)
- Accessibility and internationalization (e.g. support for screen readers, translation)
- Scalability (e.g. load balancing, caching)
- Continuous integration and delivery (e.g. Jenkins, Travis CI)
- Deployment (e.g. Docker, Kubernetes)
- Automated testing (e.g. unit test, integration test)

Of course this list isn’t exhaustive, but I’ve been thinking about it as more of a checklist to build upon when working on small side-projects that end up getting significant attention and development time.


The Hierarchy of Webapp Needs

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    h2: "h2",
    li: "li",
    p: "p",
    ul: "ul",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This short post was originally on a blog I started for creating web applications. This post goes into mobile considerations for web applications."
    }), "\n", _jsx(_components.h2, {
      children: "Tools to Sync Mobile and Web Development"
    }), "\n", _jsx(_components.p, {
      children: "These tools not only help with Mobile development, but they also help with cross-browser development as well (see Developer Tools):"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsxs(_components.li, {
        children: [_jsx(_components.a, {
          href: "http://vanamco.com/ghostlab/",
          children: "Ghostlab"
        }), " - Synchronized browser testing for web and mobile.  Test files and URLs against multiple devices by syncing clicks, scrolls, keystrokes across the devices. $49 for a license."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.a, {
          href: "https://creative.adobe.com/products/inspect",
          children: "Edge Inspect CC"
        }), " - Synchronized Browsing & Refreshing. Test files and URLs agaisnt multiple devices at once.  Part of the Adobe Creative Cloud."]
      }), "\n"]
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This short post was originally on a blog I started for creating web applications. This post goes into mobile considerations for web applications.</p>
<h2>Tools to Sync Mobile and Web Development</h2>
<p>These tools not only help with Mobile development, but they also help with cross-browser development as well (see Developer Tools):</p>
<ul>
<li><a href="http://vanamco.com/ghostlab/">Ghostlab</a> - Synchronized browser testing for web and mobile.  Test files and URLs against multiple devices by syncing clicks, scrolls, keystrokes across the devices. $49 for a license.</li>
<li><a href="https://creative.adobe.com/products/inspect">Edge Inspect CC</a> - Synchronized Browsing &amp; Refreshing. Test files and URLs agaisnt multiple devices at once.  Part of the Adobe Creative Cloud.</li>
</ul>


This short post was originally on a blog I started for creating web applications. This post goes into mobile considerations for web applications.

## Tools to Sync Mobile and Web Development

These tools not only help with Mobile development, but they also help with cross-browser development as well (see Developer Tools):

* [Ghostlab](http://vanamco.com/ghostlab/) - Synchronized browser testing for web and mobile.  Test files and URLs against multiple devices by syncing clicks, scrolls, keystrokes across the devices. $49 for a license.
* [Edge Inspect CC](https://creative.adobe.com/products/inspect) - Synchronized Browsing & Refreshing. Test files and URLs agaisnt multiple devices at once.  Part of the Adobe Creative Cloud.


Mobile

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    blockquote: "blockquote",
    code: "code",
    h2: "h2",
    p: "p",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This post was originally on a blog I started for creating web applications. This post goes into using web standards and best practices."
    }), "\n", _jsx(_components.h2, {
      children: "Check that GET and POST Requests are Used Properly"
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.code, {
        children: "GET"
      }), " request should be for requesting information."]
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.code, {
        children: "POST"
      }), " requests should be for sending information."]
    }), "\n", _jsxs(_components.p, {
      children: ["A ", _jsx(_components.code, {
        children: "GET"
      }), " request should not change the object that is being requests."]
    }), "\n", _jsx(_components.h2, {
      children: "Validate your HTML and CSS"
    }), "\n", _jsxs(_components.p, {
      children: ["Write your ", _jsx(_components.a, {
        href: "http://www.w3.org/TR/xhtml1/",
        children: "XHTML"
      }), " or ", _jsx(_components.a, {
        href: "http://www.w3.org/TR/REC-html40/",
        children: "HTML"
      }), " and ", _jsx(_components.a, {
        href: "http://www.w3.org/TR/CSS2/",
        children: "CSS"
      }), " according to ", _jsx(_components.a, {
        href: "http://www.w3.org/TR/",
        children: "the W3C specifications"
      }), " and make sure they ", _jsx(_components.a, {
        href: "http://validator.w3.org/",
        children: "validate"
      }), "."]
    }), "\n", _jsx(_components.p, {
      children: "Avoid browser quirks modes by doing this."
    }), "\n", _jsx(_components.h2, {
      children: "Use 301 and 302 Redirects Correctly"
    }), "\n", _jsxs(_components.p, {
      children: ["According to ", _jsx(_components.a, {
        href: "http://www.bigoakinc.com/blog/when-to-use-a-301-vs-302-redirect/",
        children: "Bigoak Inc"
      }), ":"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "All three major search engines handle 301 redirects the same, that is to say they ignore the original URL and instead index the destination URL."
      }), "\n"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "The three major engines handle 302 redirects very differently, and because of this 302s are typically not recommended."
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Best practice is to use 301 redirects (which are permanent and are usually cached by the browser).  For temporary moves, use 302 redirects."
    }), "\n", _jsx(_components.h2, {
      children: "Force IE to Use Unicode for Parameters"
    }), "\n", _jsxs(_components.p, {
      children: ["If you have ever seen ", _jsx(_components.code, {
        children: "utf8=✓"
      }), " in the parameter listing of a url, it's in order to ", _jsx(_components.a, {
        href: "http://stackoverflow.com/questions/3222013/what-is-the-snowman-param-in-ruby-on-rails-3-forms-for/3348524#3348524",
        children: "force Internet Explorer (5, 6, 7 and 8) to encode its parameters as unicode"
      }), "."]
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This post was originally on a blog I started for creating web applications. This post goes into using web standards and best practices.</p>
<h2>Check that GET and POST Requests are Used Properly</h2>
<p><code>GET</code> request should be for requesting information.</p>
<p><code>POST</code> requests should be for sending information.</p>
<p>A <code>GET</code> request should not change the object that is being requests.</p>
<h2>Validate your HTML and CSS</h2>
<p>Write your <a href="http://www.w3.org/TR/xhtml1/">XHTML</a> or <a href="http://www.w3.org/TR/REC-html40/">HTML</a> and <a href="http://www.w3.org/TR/CSS2/">CSS</a> according to <a href="http://www.w3.org/TR/">the W3C specifications</a> and make sure they <a href="http://validator.w3.org/">validate</a>.</p>
<p>Avoid browser quirks modes by doing this.</p>
<h2>Use 301 and 302 Redirects Correctly</h2>
<p>According to <a href="http://www.bigoakinc.com/blog/when-to-use-a-301-vs-302-redirect/">Bigoak Inc</a>:</p>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>All three major search engines handle 301 redirects the same, that is to say they ignore the original URL and instead index the destination URL.</p>
</blockquote>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>The three major engines handle 302 redirects very differently, and because of this 302s are typically not recommended.</p>
</blockquote>
<p>Best practice is to use 301 redirects (which are permanent and are usually cached by the browser).  For temporary moves, use 302 redirects.</p>
<h2>Force IE to Use Unicode for Parameters</h2>
<p>If you have ever seen <code>utf8=✓</code> in the parameter listing of a url, it&#x27;s in order to <a href="http://stackoverflow.com/questions/3222013/what-is-the-snowman-param-in-ruby-on-rails-3-forms-for/3348524#3348524">force Internet Explorer (5, 6, 7 and 8) to encode its parameters as unicode</a>.</p>


This post was originally on a blog I started for creating web applications. This post goes into using web standards and best practices.

## Check that GET and POST Requests are Used Properly

`GET` request should be for requesting information.

`POST` requests should be for sending information.

A `GET` request should not change the object that is being requests.

## Validate your HTML and CSS

Write your [XHTML](http://www.w3.org/TR/xhtml1/) or [HTML](http://www.w3.org/TR/REC-html40/) and [CSS](http://www.w3.org/TR/CSS2/) according to [the W3C specifications](http://www.w3.org/TR/) and make sure they [validate](http://validator.w3.org/).

Avoid browser quirks modes by doing this.

## Use 301 and 302 Redirects Correctly

According to [Bigoak Inc](http://www.bigoakinc.com/blog/when-to-use-a-301-vs-302-redirect/):

> All three major search engines handle 301 redirects the same, that is to say they ignore the original URL and instead index the destination URL.

> The three major engines handle 302 redirects very differently, and because of this 302s are typically not recommended.

Best practice is to use 301 redirects (which are permanent and are usually cached by the browser).  For temporary moves, use 302 redirects.

## Force IE to Use Unicode for Parameters

If you have ever seen `utf8=✓` in the parameter listing of a url, it's in order to [force Internet Explorer (5, 6, 7 and 8) to encode its parameters as unicode](http://stackoverflow.com/questions/3222013/what-is-the-snowman-param-in-ruby-on-rails-3-forms-for/3348524#3348524).  


Web Standards

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    blockquote: "blockquote",
    code: "code",
    h2: "h2",
    li: "li",
    p: "p",
    pre: "pre",
    strong: "strong",
    ul: "ul",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This post was originally on a blog I started for creating web applications. This post goes into security for web applications."
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Against SQL Injection"
    }), "\n", _jsxs(_components.p, {
      children: ["Read about ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/SQL_injection",
        children: "SQL injection"
      }), " and learn how to prevent it for the langauge and database you are using."]
    }), "\n", _jsx(_components.h2, {
      children: "All User Input is Filtered"
    }), "\n", _jsx(_components.p, {
      children: "Never trust user input.  This includes cookies and hidden form field values."
    }), "\n", _jsx(_components.h2, {
      children: "Read the Open Web Application Security Project"
    }), "\n", _jsxs(_components.p, {
      children: ["Read ", _jsx(_components.a, {
        href: "https://www.owasp.org/index.php/Category:OWASP_Guide_Project",
        children: "the Open Web Application Security Project"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "All Passwords are Hashed with a Salt"
    }), "\n", _jsxs(_components.p, {
      children: ["Hash all passwords using a ", _jsx(_components.a, {
        href: "http://security.stackexchange.com/q/21263/396",
        children: "salt"
      }), " and all salts are different to prevent against ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Rainbow_table",
        children: "rainbow attacks"
      }), "."]
    }), "\n", _jsxs(_components.p, {
      children: ["The hashing algorithm should be bcrypt or scrypt for storing passwords.  Look into ", _jsx(_components.a, {
        href: "http://codahale.com/how-to-safely-store-a-password/",
        children: "How To Safely Sroe a Password"
      }), "."]
    }), "\n", _jsxs(_components.p, {
      children: ["Do ", _jsx(_components.strong, {
        children: "not"
      }), " use MD5 or any SHA algorithm.  SHA algorithms are meant to be fast, which should not be used for trying to securely store passwords."]
    }), "\n", _jsx(_components.h2, {
      children: "Use a Standard Authentication System"
    }), "\n", _jsx(_components.p, {
      children: "Do not role your own authentication system."
    }), "\n", _jsx(_components.h2, {
      children: "Process Credit Card Information Correctly"
    }), "\n", _jsxs(_components.p, {
      children: ["Look into the ", _jsx(_components.a, {
        href: "https://www.pcisecuritystandards.org/",
        children: "PCI Security Standards"
      }), " for financial security information."]
    }), "\n", _jsx(_components.h2, {
      children: "Use HTTPS for Sensitive Data"
    }), "\n", _jsxs(_components.p, {
      children: ["Set up your ", _jsx(_components.a, {
        href: "http://www.mozilla.org/projects/security/pki/nss/ssl/draft302.txt",
        children: "SSL"
      }), "/", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Https",
        children: "HTTPS"
      }), " certs correctly and check it against ", _jsx(_components.a, {
        href: "https://www.ssllabs.com/ssltest/",
        children: "SSL Labs Server Test"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Session Hijacking"
    }), "\n", _jsx(_components.p, {
      children: "Built in session systems are usually premade to handle these types of attacks. It is not recommended that you roll your own, but it should have the following features:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Encrypted data traffic.  The session key in particular should be encrypted, but ideally all traffic should be encrypted by using SSL/TLS."
      }), "\n", _jsx(_components.li, {
        children: "Long random session key.  Attackers should not be able to guess valid session keys."
      }), "\n", _jsxs(_components.li, {
        children: ["Regenerating the session id after a successful login.  Prevents [session fixation](", _jsx(_components.a, {
          href: "http://en.wikipedia.org/wiki/Session_fixation",
          children: "http://en.wikipedia.org/wiki/Session_fixation"
        }), "]."]
      }), "\n", _jsx(_components.li, {
        children: "Change the cookie value with each and every request.  This reduces the window for an attacker to hijack the session."
      }), "\n"]
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Cross Site Scripting (XSS)"
    }), "\n", _jsxs(_components.p, {
      children: ["Unfiltered query parameters that are added to the DOM are an easy way for attackers to use cross site scripting to pull data from your users.  Filter all query parameters; see ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Cross-site_scripting",
        children: "Cross-site Scripting"
      }), " for more information on common attacks and prevention methods."]
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Page Embedding"
    }), "\n", _jsx(_components.p, {
      children: "You can prevent page embedding with the following header:"
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "X-Frame-Options: DENY\n"
      })
    }), "\n", _jsx(_components.p, {
      children: "This will cause browsers to refuse requests for framing in that page."
    }), "\n", _jsxs(_components.p, {
      children: ["You can also prevent page embedding with JavaScript to prevent users from accessing your website in an iframe.  Look into this ", _jsx(_components.a, {
        href: "http://stackoverflow.com/questions/7776281/javascript-jquery-how-to-detect-if-a-page-is-embedded-by-others",
        children: "StackOverflow solution"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Cross Site Request Forgeries"
    }), "\n", _jsx(_components.p, {
      children: "Cross site request forgeries are attacks by a malicious user whereby they craft a link to a site which a user is know to have been authenticated.  For example, say your site is a bank website; the malicious user may craft a URL for you site that when a user clicks on it, deposits money into the attacker's account.  The attacker then sends the link to users who may have been logged into your bank website."
    }), "\n", _jsxs(_components.p, {
      children: ["According to ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Cross-site_request_forgery",
        children: "Wikipedia"
      }), ":"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "At risk are web applications that perform actions based on input from trusted and authenticated users without requiring the user to authorize the specific action. A user who is authenticated by a cookie saved in the user's web browser could unknowingly send an HTTP request to a site that trusts the user and thereby causes an unwanted action."
      }), "\n"]
    }), "\n", _jsx(_components.h2, {
      children: "Prevent Click Jacking"
    }), "\n", _jsxs(_components.p, {
      children: ["Similar to ", _jsx(_components.a, {
        href: "#prevent-page-embedding",
        children: "Page Embedding"
      }), ", click jacking works by:"]
    }), "\n", _jsxs(_components.blockquote, {
      children: ["\n", _jsx(_components.p, {
        children: "A user might receive an email with a link to a video about a news item, but another valid page, say a product page on Amazon.com, can be \"hidden\" on top or underneath the \"PLAY\" button of the news video. The user tries to \"play\" the video but actually \"buys\" the product from Amazon."
      }), "\n"]
    }), "\n", _jsx(_components.p, {
      children: "Other known exploits include:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Tricking users into enabling their webcam and microphone through Flash"
      }), "\n", _jsx(_components.li, {
        children: "Tricking users into making their social networking profile information public"
      }), "\n", _jsx(_components.li, {
        children: "Making users follow someone on Twitter[8]"
      }), "\n", _jsx(_components.li, {
        children: "Sharing links on Facebook"
      }), "\n"]
    }), "\n", _jsxs(_components.p, {
      children: ["See ", _jsx(_components.a, {
        href: "http://en.wikipedia.org/wiki/Clickjacking",
        children: "Wikipedia"
      }), " for more information."]
    }), "\n", _jsx(_components.h2, {
      children: "Verify That Server Software is Up to Date"
    }), "\n", _jsx(_components.p, {
      children: "Make sure that the software on your system is up to date with the latest patches."
    }), "\n", _jsx(_components.h2, {
      children: "Verify That Packages, Modules, Plugins, etc are Up to Date"
    }), "\n", _jsx(_components.p, {
      children: "Make sure that the latest software used by your projest is up to date with the latest patches."
    }), "\n", _jsx(_components.h2, {
      children: "Verify That Your Database is Secured"
    }), "\n", _jsx(_components.p, {
      children: "Make sure that connections to the database are at least password protected.  Verify that only the least amount of permissions are being used."
    }), "\n", _jsx(_components.p, {
      children: "Make sure that confidential information is encrypted or hashed."
    }), "\n", _jsx(_components.h2, {
      children: "Attempt to Make Your App Run As Non-Root"
    }), "\n", _jsxs(_components.p, {
      children: ["Apply ", _jsx(_components.a, {
        href: "https://en.wikipedia.org/wiki/Principle_of_least_privilege",
        children: "the principle of least privilege"
      }), " and try to have your app run as non-root as often as possible."]
    }), "\n", _jsx(_components.h2, {
      children: "Use the Content-Security-Policy Header"
    }), "\n", _jsx(_components.p, {
      children: "Specify the following header to make the browser only run certain content depending on its origin:"
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "example"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "Content-Security-Policy: script-src 'self' https://apis.google.com\n"
      })
    }), "\n", _jsx(_components.p, {
      children: "You can also specify the following type of content:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "script-src (JavaScript)"
      }), "\n", _jsx(_components.li, {
        children: "connect-src (XMLHttpRequest, WebSockets, and EventSource)"
      }), "\n", _jsx(_components.li, {
        children: "font-src (Fonts)"
      }), "\n", _jsx(_components.li, {
        children: "frame-src (Frame Urls)"
      }), "\n", _jsx(_components.li, {
        children: "img-src (Images)"
      }), "\n", _jsx(_components.li, {
        children: "media-src (Audio and Videos)"
      }), "\n", _jsx(_components.li, {
        children: "object-src (Flash)"
      }), "\n", _jsx(_components.li, {
        children: "style-src (CSS)"
      }), "\n"]
    }), "\n", _jsx(_components.h2, {
      children: "Do Not Allow Mime Sniffing with User Uploaded Content"
    }), "\n", _jsx(_components.p, {
      children: "You can tell browsers to not try to guess Mime Types of files with the following header:"
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "X-CONTENT-TYPE-OPTIONS: nosniff\n"
      })
    }), "\n", _jsx(_components.p, {
      children: "For example, say a user uploads an HTML file. You intend to serve it to users as a text file, but the browser will attempt to serve it as an HTML page.  You can tell the browser that you know what you are doing and that the Mime Type you are providing is indeed correct by using the above header."
    }), "\n", _jsx(_components.h2, {
      children: "Force HTTPS Traffic"
    }), "\n", _jsx(_components.p, {
      children: "You can force HTTPS Traffic with the following header:"
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "Scrict-Transport-Security: max-age=31536000; includeSubDomains\n"
      })
    }), "\n", _jsxs(_components.p, {
      children: ["This will force ", _jsx(_components.code, {
        children: "https://website.com"
      }), " even when typing in ", _jsx(_components.code, {
        children: "http://website.com"
      })]
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This post was originally on a blog I started for creating web applications. This post goes into security for web applications.</p>
<h2>Prevent Against SQL Injection</h2>
<p>Read about <a href="http://en.wikipedia.org/wiki/SQL_injection">SQL injection</a> and learn how to prevent it for the langauge and database you are using.</p>
<h2>All User Input is Filtered</h2>
<p>Never trust user input.  This includes cookies and hidden form field values.</p>
<h2>Read the Open Web Application Security Project</h2>
<p>Read <a href="https://www.owasp.org/index.php/Category:OWASP_Guide_Project">the Open Web Application Security Project</a>.</p>
<h2>All Passwords are Hashed with a Salt</h2>
<p>Hash all passwords using a <a href="http://security.stackexchange.com/q/21263/396">salt</a> and all salts are different to prevent against <a href="http://en.wikipedia.org/wiki/Rainbow_table">rainbow attacks</a>.</p>
<p>The hashing algorithm should be bcrypt or scrypt for storing passwords.  Look into <a href="http://codahale.com/how-to-safely-store-a-password/">How To Safely Sroe a Password</a>.</p>
<p>Do <strong>not</strong> use MD5 or any SHA algorithm.  SHA algorithms are meant to be fast, which should not be used for trying to securely store passwords.</p>
<h2>Use a Standard Authentication System</h2>
<p>Do not role your own authentication system.</p>
<h2>Process Credit Card Information Correctly</h2>
<p>Look into the <a href="https://www.pcisecuritystandards.org/">PCI Security Standards</a> for financial security information.</p>
<h2>Use HTTPS for Sensitive Data</h2>
<p>Set up your <a href="http://www.mozilla.org/projects/security/pki/nss/ssl/draft302.txt">SSL</a>/<a href="http://en.wikipedia.org/wiki/Https">HTTPS</a> certs correctly and check it against <a href="https://www.ssllabs.com/ssltest/">SSL Labs Server Test</a>.</p>
<h2>Prevent Session Hijacking</h2>
<p>Built in session systems are usually premade to handle these types of attacks. It is not recommended that you roll your own, but it should have the following features:</p>
<ul>
<li>Encrypted data traffic.  The session key in particular should be encrypted, but ideally all traffic should be encrypted by using SSL/TLS.</li>
<li>Long random session key.  Attackers should not be able to guess valid session keys.</li>
<li>Regenerating the session id after a successful login.  Prevents [session fixation](<a href="http://en.wikipedia.org/wiki/Session_fixation">http://en.wikipedia.org/wiki/Session_fixation</a>].</li>
<li>Change the cookie value with each and every request.  This reduces the window for an attacker to hijack the session.</li>
</ul>
<h2>Prevent Cross Site Scripting (XSS)</h2>
<p>Unfiltered query parameters that are added to the DOM are an easy way for attackers to use cross site scripting to pull data from your users.  Filter all query parameters; see <a href="http://en.wikipedia.org/wiki/Cross-site_scripting">Cross-site Scripting</a> for more information on common attacks and prevention methods.</p>
<h2>Prevent Page Embedding</h2>
<p>You can prevent page embedding with the following header:</p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>X-Frame-Options: DENY
</code></pre></div>
<p>This will cause browsers to refuse requests for framing in that page.</p>
<p>You can also prevent page embedding with JavaScript to prevent users from accessing your website in an iframe.  Look into this <a href="http://stackoverflow.com/questions/7776281/javascript-jquery-how-to-detect-if-a-page-is-embedded-by-others">StackOverflow solution</a>.</p>
<h2>Prevent Cross Site Request Forgeries</h2>
<p>Cross site request forgeries are attacks by a malicious user whereby they craft a link to a site which a user is know to have been authenticated.  For example, say your site is a bank website; the malicious user may craft a URL for you site that when a user clicks on it, deposits money into the attacker&#x27;s account.  The attacker then sends the link to users who may have been logged into your bank website.</p>
<p>According to <a href="http://en.wikipedia.org/wiki/Cross-site_request_forgery">Wikipedia</a>:</p>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>At risk are web applications that perform actions based on input from trusted and authenticated users without requiring the user to authorize the specific action. A user who is authenticated by a cookie saved in the user&#x27;s web browser could unknowingly send an HTTP request to a site that trusts the user and thereby causes an unwanted action.</p>
</blockquote>
<h2>Prevent Click Jacking</h2>
<p>Similar to <a href="#prevent-page-embedding">Page Embedding</a>, click jacking works by:</p>
<blockquote class="border-l-4 border-gray-300 pl-4">
<p>A user might receive an email with a link to a video about a news item, but another valid page, say a product page on Amazon.com, can be &quot;hidden&quot; on top or underneath the &quot;PLAY&quot; button of the news video. The user tries to &quot;play&quot; the video but actually &quot;buys&quot; the product from Amazon.</p>
</blockquote>
<p>Other known exploits include:</p>
<ul>
<li>Tricking users into enabling their webcam and microphone through Flash</li>
<li>Tricking users into making their social networking profile information public</li>
<li>Making users follow someone on Twitter[8]</li>
<li>Sharing links on Facebook</li>
</ul>
<p>See <a href="http://en.wikipedia.org/wiki/Clickjacking">Wikipedia</a> for more information.</p>
<h2>Verify That Server Software is Up to Date</h2>
<p>Make sure that the software on your system is up to date with the latest patches.</p>
<h2>Verify That Packages, Modules, Plugins, etc are Up to Date</h2>
<p>Make sure that the latest software used by your projest is up to date with the latest patches.</p>
<h2>Verify That Your Database is Secured</h2>
<p>Make sure that connections to the database are at least password protected.  Verify that only the least amount of permissions are being used.</p>
<p>Make sure that confidential information is encrypted or hashed.</p>
<h2>Attempt to Make Your App Run As Non-Root</h2>
<p>Apply <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">the principle of least privilege</a> and try to have your app run as non-root as often as possible.</p>
<h2>Use the Content-Security-Policy Header</h2>
<p>Specify the following header to make the browser only run certain content depending on its origin:</p>
<p><strong>example</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>Content-Security-Policy: script-src &#x27;self&#x27; https://apis.google.com
</code></pre></div>
<p>You can also specify the following type of content:</p>
<ul>
<li>script-src (JavaScript)</li>
<li>connect-src (XMLHttpRequest, WebSockets, and EventSource)</li>
<li>font-src (Fonts)</li>
<li>frame-src (Frame Urls)</li>
<li>img-src (Images)</li>
<li>media-src (Audio and Videos)</li>
<li>object-src (Flash)</li>
<li>style-src (CSS)</li>
</ul>
<h2>Do Not Allow Mime Sniffing with User Uploaded Content</h2>
<p>You can tell browsers to not try to guess Mime Types of files with the following header:</p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>X-CONTENT-TYPE-OPTIONS: nosniff
</code></pre></div>
<p>For example, say a user uploads an HTML file. You intend to serve it to users as a text file, but the browser will attempt to serve it as an HTML page.  You can tell the browser that you know what you are doing and that the Mime Type you are providing is indeed correct by using the above header.</p>
<h2>Force HTTPS Traffic</h2>
<p>You can force HTTPS Traffic with the following header:</p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>Scrict-Transport-Security: max-age=31536000; includeSubDomains
</code></pre></div>
<p>This will force <code>https://website.com</code> even when typing in <code>http://website.com</code></p>


This post was originally on a blog I started for creating web applications. This post goes into security for web applications.

## Prevent Against SQL Injection

Read about [SQL injection](http://en.wikipedia.org/wiki/SQL_injection) and learn how to prevent it for the langauge and database you are using.

## All User Input is Filtered

Never trust user input.  This includes cookies and hidden form field values.

## Read the Open Web Application Security Project

Read [the Open Web Application Security Project](https://www.owasp.org/index.php/Category:OWASP_Guide_Project).

## All Passwords are Hashed with a Salt

Hash all passwords using a [salt](http://security.stackexchange.com/q/21263/396) and all salts are different to prevent against [rainbow attacks](http://en.wikipedia.org/wiki/Rainbow_table).

The hashing algorithm should be bcrypt or scrypt for storing passwords.  Look into [How To Safely Sroe a Password](http://codahale.com/how-to-safely-store-a-password/).

Do **not** use MD5 or any SHA algorithm.  SHA algorithms are meant to be fast, which should not be used for trying to securely store passwords.

## Use a Standard Authentication System

Do not role your own authentication system.

## Process Credit Card Information Correctly

Look into the [PCI Security Standards](https://www.pcisecuritystandards.org/) for financial security information.

## Use HTTPS for Sensitive Data

Set up your [SSL](http://www.mozilla.org/projects/security/pki/nss/ssl/draft302.txt)/[HTTPS](http://en.wikipedia.org/wiki/Https) certs correctly and check it against [SSL Labs Server Test](https://www.ssllabs.com/ssltest/).

## Prevent Session Hijacking

Built in session systems are usually premade to handle these types of attacks. It is not recommended that you roll your own, but it should have the following features:

- Encrypted data traffic.  The session key in particular should be encrypted, but ideally all traffic should be encrypted by using SSL/TLS.
- Long random session key.  Attackers should not be able to guess valid session keys.
- Regenerating the session id after a successful login.  Prevents [session fixation](http://en.wikipedia.org/wiki/Session_fixation].
- Change the cookie value with each and every request.  This reduces the window for an attacker to hijack the session.

## Prevent Cross Site Scripting (XSS)

Unfiltered query parameters that are added to the DOM are an easy way for attackers to use cross site scripting to pull data from your users.  Filter all query parameters; see [Cross-site Scripting](http://en.wikipedia.org/wiki/Cross-site_scripting) for more information on common attacks and prevention methods.

## Prevent Page Embedding

You can prevent page embedding with the following header:

```
X-Frame-Options: DENY
```

This will cause browsers to refuse requests for framing in that page.

You can also prevent page embedding with JavaScript to prevent users from accessing your website in an iframe.  Look into this [StackOverflow solution](http://stackoverflow.com/questions/7776281/javascript-jquery-how-to-detect-if-a-page-is-embedded-by-others).

## Prevent Cross Site Request Forgeries

Cross site request forgeries are attacks by a malicious user whereby they craft a link to a site which a user is know to have been authenticated.  For example, say your site is a bank website; the malicious user may craft a URL for you site that when a user clicks on it, deposits money into the attacker's account.  The attacker then sends the link to users who may have been logged into your bank website.

According to [Wikipedia](http://en.wikipedia.org/wiki/Cross-site_request_forgery):

> At risk are web applications that perform actions based on input from trusted and authenticated users without requiring the user to authorize the specific action. A user who is authenticated by a cookie saved in the user's web browser could unknowingly send an HTTP request to a site that trusts the user and thereby causes an unwanted action.

## Prevent Click Jacking

Similar to [Page Embedding](#prevent-page-embedding), click jacking works by:

> A user might receive an email with a link to a video about a news item, but another valid page, say a product page on Amazon.com, can be "hidden" on top or underneath the "PLAY" button of the news video. The user tries to "play" the video but actually "buys" the product from Amazon.

Other known exploits include:

- Tricking users into enabling their webcam and microphone through Flash
- Tricking users into making their social networking profile information public
- Making users follow someone on Twitter[8]
- Sharing links on Facebook

See [Wikipedia](http://en.wikipedia.org/wiki/Clickjacking) for more information.

## Verify That Server Software is Up to Date

Make sure that the software on your system is up to date with the latest patches.

## Verify That Packages, Modules, Plugins, etc are Up to Date

Make sure that the latest software used by your projest is up to date with the latest patches.

## Verify That Your Database is Secured

Make sure that connections to the database are at least password protected.  Verify that only the least amount of permissions are being used.

Make sure that confidential information is encrypted or hashed.

## Attempt to Make Your App Run As Non-Root

Apply [the principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) and try to have your app run as non-root as often as possible.

## Use the Content-Security-Policy Header

Specify the following header to make the browser only run certain content depending on its origin:

**example**

```
Content-Security-Policy: script-src 'self' https://apis.google.com
```

You can also specify the following type of content:

- script-src (JavaScript)
- connect-src (XMLHttpRequest, WebSockets, and EventSource)
- font-src (Fonts)
- frame-src (Frame Urls)
- img-src (Images)
- media-src (Audio and Videos)
- object-src (Flash)
- style-src (CSS)

## Do Not Allow Mime Sniffing with User Uploaded Content

You can tell browsers to not try to guess Mime Types of files with the following header:

```
X-CONTENT-TYPE-OPTIONS: nosniff
```

For example, say a user uploads an HTML file. You intend to serve it to users as a text file, but the browser will attempt to serve it as an HTML page.  You can tell the browser that you know what you are doing and that the Mime Type you are providing is indeed correct by using the above header.

## Force HTTPS Traffic

You can force HTTPS Traffic with the following header:

```
Scrict-Transport-Security: max-age=31536000; includeSubDomains
```

This will force `https://website.com` even when typing in `http://website.com`


Security

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    h2: "h2",
    p: "p",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This post was originally on a blog I started for creating web applications. This post goes into maintenance of web applications."
    }), "\n", _jsx(_components.h2, {
      children: "Setup an Error/Bug Reporting Solution"
    }), "\n", _jsx(_components.p, {
      children: "Use Github's issue tracker, or JIRA, or anything.  Keeping track of issues, enhancements, and bugs in progress helps tremendously down the line."
    }), "\n", _jsx(_components.h2, {
      children: "Setup a Contact Form"
    }), "\n", _jsx(_components.p, {
      children: "Simple email forms or using other services like Reddit or even forums can help users give you suggestions and criticism."
    }), "\n", _jsx(_components.h2, {
      children: "Documentation"
    }), "\n", _jsx(_components.p, {
      children: "Document how the application works, the deployment progress, and anything else you can!  Those who do future maintenance will thank you."
    }), "\n", _jsx(_components.h2, {
      children: "Backup/Restore Strategy"
    }), "\n", _jsx(_components.p, {
      children: "Have a backup strategy and a restore strategy."
    }), "\n", _jsx(_components.h2, {
      children: "Version Control Your Project"
    }), "\n", _jsx(_components.p, {
      children: "Use Subversion, Mecurial, Git to version control your files.  Preferably Git ;) ."
    }), "\n", _jsx(_components.h2, {
      children: "Setup Unit Tests/Acceptance Tests/Regression Test etc"
    }), "\n", _jsx(_components.p, {
      children: "Getting any type of tests is very useful, especially acceptance testing and integration testing."
    }), "\n", _jsx(_components.h2, {
      children: "Setup Logging"
    }), "\n", _jsx(_components.p, {
      children: "Log any application errors, user errors, exceptions, etc."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This post was originally on a blog I started for creating web applications. This post goes into maintenance of web applications.</p>
<h2>Setup an Error/Bug Reporting Solution</h2>
<p>Use Github&#x27;s issue tracker, or JIRA, or anything.  Keeping track of issues, enhancements, and bugs in progress helps tremendously down the line.</p>
<h2>Setup a Contact Form</h2>
<p>Simple email forms or using other services like Reddit or even forums can help users give you suggestions and criticism.</p>
<h2>Documentation</h2>
<p>Document how the application works, the deployment progress, and anything else you can!  Those who do future maintenance will thank you.</p>
<h2>Backup/Restore Strategy</h2>
<p>Have a backup strategy and a restore strategy.</p>
<h2>Version Control Your Project</h2>
<p>Use Subversion, Mecurial, Git to version control your files.  Preferably Git ;) .</p>
<h2>Setup Unit Tests/Acceptance Tests/Regression Test etc</h2>
<p>Getting any type of tests is very useful, especially acceptance testing and integration testing.</p>
<h2>Setup Logging</h2>
<p>Log any application errors, user errors, exceptions, etc.</p>


This post was originally on a blog I started for creating web applications. This post goes into maintenance of web applications.

## Setup an Error/Bug Reporting Solution

Use Github's issue tracker, or JIRA, or anything.  Keeping track of issues, enhancements, and bugs in progress helps tremendously down the line.

## Setup a Contact Form

Simple email forms or using other services like Reddit or even forums can help users give you suggestions and criticism.

## Documentation

Document how the application works, the deployment progress, and anything else you can!  Those who do future maintenance will thank you.

## Backup/Restore Strategy

Have a backup strategy and a restore strategy.

## Version Control Your Project

Use Subversion, Mecurial, Git to version control your files.  Preferably Git ;) .

## Setup Unit Tests/Acceptance Tests/Regression Test etc

Getting any type of tests is very useful, especially acceptance testing and integration testing.

## Setup Logging

Log any application errors, user errors, exceptions, etc.


Maintenance

"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    code: "code",
    em: "em",
    h2: "h2",
    h3: "h3",
    li: "li",
    p: "p",
    pre: "pre",
    strong: "strong",
    ul: "ul",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsx(_components.p, {
      children: "This post was originally on a blog I started for creating web applications. This post goes into RESTful interfaces and how to design them."
    }), "\n", _jsxs(_components.p, {
      children: ["RESTful APIs are a common way for applications to be able to communicate with each other.  An important part about RESTful APIs is that they are ", _jsx(_components.em, {
        children: "consistent"
      }), ".  Naming, verbage, actions, and errors should be consistent within your API and with how other APIs behave."]
    }), "\n", _jsx(_components.p, {
      children: "Here are some tips and questions that you should ask yourself when developing an API."
    }), "\n", _jsx(_components.h2, {
      children: "Are you using PUT and POST correctly?"
    }), "\n", _jsx(_components.p, {
      children: "Both PUT and POST can be used for creating. You do not need to support both PUT and POST. Consider the following design decisions:"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: "Do you name your URL objects you create explicitly, or let the server decide? If you name them then use PUT. If you let the server decide then use POST."
      }), "\n", _jsx(_components.li, {
        children: "PUT is idempotent, so if you PUT an object twice, it has no effect. This is a nice property, so I would use PUT when possible."
      }), "\n", _jsx(_components.li, {
        children: "You can update or create a resource with PUT with the same object URL"
      }), "\n", _jsx(_components.li, {
        children: "With POST you can have 2 requests coming in at the same time making modifications to a URL, and they may update different parts of the object."
      }), "\n"]
    }), "\n", _jsxs(_components.p, {
      children: ["See ", _jsx(_components.a, {
        href: "http://stackoverflow.com/questions/630453/put-vs-post-in-rest",
        children: "this StackOverflow question for more information"
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Are you using appropriate verbs?"
    }), "\n", _jsx(_components.p, {
      children: "When making a REST API, your verbs are your request types, not actual verbs in your URL!  Your URLs should use nouns, not verbs!  Remember to use the plural form for multiple results."
    }), "\n", _jsx(_components.p, {
      children: "Use your URLs to specify your objects.  Here are some examples of RESTful style requests:"
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Obtain 1 Question"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "GET /questions/<question>\n"
      })
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Obtain All Questions"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "GET /questions\n"
      })
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Append/Update Question"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "POST /questions/<existing_question>\n"
      })
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Create a New Question"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "PUT /questions/<new_question>\n"
      })
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Overwrite a Question"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "PUT /questions/<existing_question>\n"
      })
    }), "\n", _jsx(_components.p, {
      children: _jsx(_components.strong, {
        children: "Delete Question"
      })
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "DELETE /questions/<existing_question>\n"
      })
    }), "\n", _jsx(_components.h2, {
      children: "Nested Resources should be exenstions, not entirely different queries"
    }), "\n", _jsx(_components.p, {
      children: "If you want to get answers to a question, using the above examples as a starting point, the URL should be:"
    }), "\n", _jsx(_components.pre, {
      children: _jsx(_components.code, {
        children: "GET /questions/1/answers\n"
      })
    }), "\n", _jsx(_components.p, {
      children: "This is nested routing and it helps for a cleaner design."
    }), "\n", _jsx(_components.h2, {
      children: "Do you have a plan to version your API?"
    }), "\n", _jsx(_components.p, {
      children: "There are different ways to version your API, but it's nice to plan ahead in order for consumers of your API to be prepared."
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.a, {
        href: "https://developer.github.com/v3/media/#request-specific-version",
        children: "Github uses"
      }), " the ", _jsx(_components.code, {
        children: "Accept"
      }), " HTTP header to pass the desired version to the API."]
    }), "\n", _jsxs(_components.p, {
      children: ["Another less common method is to version the application in the URL: ", _jsx(_components.code, {
        children: "/api/v1/..."
      }), "."]
    }), "\n", _jsx(_components.h2, {
      children: "Paging"
    }), "\n", _jsxs(_components.p, {
      children: ["Large result sets should be paginated, you can use ", _jsx(_components.a, {
        href: "https://developer.github.com/guides/traversing-with-pagination/",
        children: "the Github method"
      }), " for indicated next and previous pages of results using the ", _jsx(_components.code, {
        children: "Link"
      }), " HTTP header."]
    }), "\n", _jsx(_components.h2, {
      children: "HTTP Status Codes"
    }), "\n", _jsxs(_components.p, {
      children: [_jsx(_components.a, {
        href: "https://bourgeois.me/rest/",
        children: "This list of HTTP Status codes"
      }), " and their meaning is very helpful for knowing what HTTP Status your API should be returning.  It has been duplicated here as a backup:"]
    }), "\n", _jsx(_components.h3, {
      children: "Success codes"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "201 Created"
        }), " should be used when creating content (INSERT)"]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "202 Accepted"
        }), " should be used when a request is queued for background processing (async tasks),"]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "204 No Content"
        }), " should be used when the request was properly executed but no content was returned (a good example would be when you delete something)."]
      }), "\n"]
    }), "\n", _jsx(_components.h3, {
      children: "Client error codes"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "400 Bad Request"
        }), " should be used when there was an error while processing the request payload (malformed JSON, for instance)."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "401 Unauthorized"
        }), " should be used when a request is not authenticiated (wrong access token, or username or password)."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "403 Forbidden"
        }), " should be used when the request is successfully authenticiated (see 401), but the action was forbidden."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "406 Not Acceptable"
        }), " should be used when the requested format is not available (for instance, when requesting an XML resource from a JSON only server)."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "410 Gone Should"
        }), " be returned when the requested resource is permenantely deleted and will never be available again."]
      }), "\n", _jsxs(_components.li, {
        children: [_jsx(_components.code, {
          children: "422 Unprocesable entity"
        }), " Could be used when there was a validation error while creating an object.\nA more complete list of status codes can be found in ", _jsx(_components.a, {
          href: "http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html",
          children: "RFC2616"
        }), "."]
      }), "\n"]
    }), "\n", _jsx(_components.h2, {
      children: "Further Reading"
    }), "\n", _jsxs(_components.ul, {
      children: ["\n", _jsx(_components.li, {
        children: _jsx(_components.a, {
          href: "https://pages.apigee.com/rs/apigee/images/api-design-ebook-2012-03.pdf",
          children: "API Design Ebook"
        })
      }), "\n", _jsx(_components.li, {
        children: _jsx(_components.a, {
          href: "http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api",
          children: "Best Practices for a Pragmatic RESTful API"
        })
      }), "\n"]
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


<p>This post was originally on a blog I started for creating web applications. This post goes into RESTful interfaces and how to design them.</p>
<p>RESTful APIs are a common way for applications to be able to communicate with each other.  An important part about RESTful APIs is that they are <em>consistent</em>.  Naming, verbage, actions, and errors should be consistent within your API and with how other APIs behave.</p>
<p>Here are some tips and questions that you should ask yourself when developing an API.</p>
<h2>Are you using PUT and POST correctly?</h2>
<p>Both PUT and POST can be used for creating. You do not need to support both PUT and POST. Consider the following design decisions:</p>
<ul>
<li>Do you name your URL objects you create explicitly, or let the server decide? If you name them then use PUT. If you let the server decide then use POST.</li>
<li>PUT is idempotent, so if you PUT an object twice, it has no effect. This is a nice property, so I would use PUT when possible.</li>
<li>You can update or create a resource with PUT with the same object URL</li>
<li>With POST you can have 2 requests coming in at the same time making modifications to a URL, and they may update different parts of the object.</li>
</ul>
<p>See <a href="http://stackoverflow.com/questions/630453/put-vs-post-in-rest">this StackOverflow question for more information</a>.</p>
<h2>Are you using appropriate verbs?</h2>
<p>When making a REST API, your verbs are your request types, not actual verbs in your URL!  Your URLs should use nouns, not verbs!  Remember to use the plural form for multiple results.</p>
<p>Use your URLs to specify your objects.  Here are some examples of RESTful style requests:</p>
<p><strong>Obtain 1 Question</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>GET /questions/&lt;question&gt;
</code></pre></div>
<p><strong>Obtain All Questions</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>GET /questions
</code></pre></div>
<p><strong>Append/Update Question</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>POST /questions/&lt;existing_question&gt;
</code></pre></div>
<p><strong>Create a New Question</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>PUT /questions/&lt;new_question&gt;
</code></pre></div>
<p><strong>Overwrite a Question</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>PUT /questions/&lt;existing_question&gt;
</code></pre></div>
<p><strong>Delete Question</strong></p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>DELETE /questions/&lt;existing_question&gt;
</code></pre></div>
<h2>Nested Resources should be exenstions, not entirely different queries</h2>
<p>If you want to get answers to a question, using the above examples as a starting point, the URL should be:</p>
<div class="overflow-auto rounded-2xl bg-[#0e005d6d] p-4 font-mono text-sm dark:bg-[#0e005d6d] dark:text-gray-100"><pre><code>GET /questions/1/answers
</code></pre></div>
<p>This is nested routing and it helps for a cleaner design.</p>
<h2>Do you have a plan to version your API?</h2>
<p>There are different ways to version your API, but it&#x27;s nice to plan ahead in order for consumers of your API to be prepared.</p>
<p><a href="https://developer.github.com/v3/media/#request-specific-version">Github uses</a> the <code>Accept</code> HTTP header to pass the desired version to the API.</p>
<p>Another less common method is to version the application in the URL: <code>/api/v1/...</code>.</p>
<h2>Paging</h2>
<p>Large result sets should be paginated, you can use <a href="https://developer.github.com/guides/traversing-with-pagination/">the Github method</a> for indicated next and previous pages of results using the <code>Link</code> HTTP header.</p>
<h2>HTTP Status Codes</h2>
<p><a href="https://bourgeois.me/rest/">This list of HTTP Status codes</a> and their meaning is very helpful for knowing what HTTP Status your API should be returning.  It has been duplicated here as a backup:</p>
<h3>Success codes</h3>
<ul>
<li><code>201 Created</code> should be used when creating content (INSERT)</li>
<li><code>202 Accepted</code> should be used when a request is queued for background processing (async tasks),</li>
<li><code>204 No Content</code> should be used when the request was properly executed but no content was returned (a good example would be when you delete something).</li>
</ul>
<h3>Client error codes</h3>
<ul>
<li><code>400 Bad Request</code> should be used when there was an error while processing the request payload (malformed JSON, for instance).</li>
<li><code>401 Unauthorized</code> should be used when a request is not authenticiated (wrong access token, or username or password).</li>
<li><code>403 Forbidden</code> should be used when the request is successfully authenticiated (see 401), but the action was forbidden.</li>
<li><code>406 Not Acceptable</code> should be used when the requested format is not available (for instance, when requesting an XML resource from a JSON only server).</li>
<li><code>410 Gone Should</code> be returned when the requested resource is permenantely deleted and will never be available again.</li>
<li><code>422 Unprocesable entity</code> Could be used when there was a validation error while creating an object.
A more complete list of status codes can be found in <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">RFC2616</a>.</li>
</ul>
<h2>Further Reading</h2>
<ul>
<li><a href="https://pages.apigee.com/rs/apigee/images/api-design-ebook-2012-03.pdf">API Design Ebook</a></li>
<li><a href="http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api">Best Practices for a Pragmatic RESTful API</a></li>
</ul>


This post was originally on a blog I started for creating web applications. This post goes into RESTful interfaces and how to design them.

RESTful APIs are a common way for applications to be able to communicate with each other.  An important part about RESTful APIs is that they are *consistent*.  Naming, verbage, actions, and errors should be consistent within your API and with how other APIs behave.

Here are some tips and questions that you should ask yourself when developing an API.

## Are you using PUT and POST correctly?

Both PUT and POST can be used for creating. You do not need to support both PUT and POST. Consider the following design decisions:

- Do you name your URL objects you create explicitly, or let the server decide? If you name them then use PUT. If you let the server decide then use POST.
- PUT is idempotent, so if you PUT an object twice, it has no effect. This is a nice property, so I would use PUT when possible.
- You can update or create a resource with PUT with the same object URL
- With POST you can have 2 requests coming in at the same time making modifications to a URL, and they may update different parts of the object.

See [this StackOverflow question for more information](http://stackoverflow.com/questions/630453/put-vs-post-in-rest).

## Are you using appropriate verbs?

When making a REST API, your verbs are your request types, not actual verbs in your URL!  Your URLs should use nouns, not verbs!  Remember to use the plural form for multiple results.

Use your URLs to specify your objects.  Here are some examples of RESTful style requests:

**Obtain 1 Question**

```
GET /questions/<question>
```

**Obtain All Questions**

```
GET /questions
```

**Append/Update Question**

```
POST /questions/<existing_question>
```

**Create a New Question**

```
PUT /questions/<new_question>
```

**Overwrite a Question**

```
PUT /questions/<existing_question>
```

**Delete Question**

```
DELETE /questions/<existing_question>
```

## Nested Resources should be exenstions, not entirely different queries

If you want to get answers to a question, using the above examples as a starting point, the URL should be:

```
GET /questions/1/answers
```

This is nested routing and it helps for a cleaner design.

## Do you have a plan to version your API?

There are different ways to version your API, but it's nice to plan ahead in order for consumers of your API to be prepared.

[Github uses](https://developer.github.com/v3/media/#request-specific-version) the `Accept` HTTP header to pass the desired version to the API.

Another less common method is to version the application in the URL: `/api/v1/...`.

## Paging

Large result sets should be paginated, you can use [the Github method](https://developer.github.com/guides/traversing-with-pagination/) for indicated next and previous pages of results using the `Link` HTTP header.

## HTTP Status Codes

[This list of HTTP Status codes](https://bourgeois.me/rest/) and their meaning is very helpful for knowing what HTTP Status your API should be returning.  It has been duplicated here as a backup:

### Success codes

- `201 Created` should be used when creating content (INSERT)
- `202 Accepted` should be used when a request is queued for background processing (async tasks),
- `204 No Content` should be used when the request was properly executed but no content was returned (a good example would be when you delete something).

### Client error codes

- `400 Bad Request` should be used when there was an error while processing the request payload (malformed JSON, for instance).
- `401 Unauthorized` should be used when a request is not authenticiated (wrong access token, or username or password).
- `403 Forbidden` should be used when the request is successfully authenticiated (see 401), but the action was forbidden.
- `406 Not Acceptable` should be used when the requested format is not available (for instance, when requesting an XML resource from a JSON only server).
- `410 Gone Should` be returned when the requested resource is permenantely deleted and will never be available again.
- `422 Unprocesable entity` Could be used when there was a validation error while creating an object.
A more complete list of status codes can be found in [RFC2616](http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html).

## Further Reading

- [API Design Ebook](https://pages.apigee.com/rs/apigee/images/api-design-ebook-2012-03.pdf)
- [Best Practices for a Pragmatic RESTful API](http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api)


web posts