Crawling as a discovery process in which search engines send out a team of robots, known either as crawlers or spiders, to find new and updated content. Content can vary and it can be a website page, an image, a video, a PDF, but regardless of the format, the content is discovered and the links are sent back to Google because they were fetched by these crawl bots. And Google will index it with the URL for the various websites. So let's look at Google Search Console now, and I'll show you what, what we're looking for in Google Search Console to see if our website's been crawled recently and what kind of information it collected. So, here is our website page. Again, this is our whirlpool site. And if we go here and we look at it's, the page is a redirect because it's HTTP and we're redirecting it to HTTPS. So in this case, I don't, I want it crawled, but as far as what we're doing, we're actually redirecting it to the secure site. So I really don't need it to index, but it is being crawled, normally as it should be. So, let's look at some of the issues we encounter when we're having crawling or indexing issues. Let's go over here and, of course, you can always request a index here in Google. When you request indexing, you're only going to have about 20 a day that Google Search Console will allow you to do. Once you're over that, you're gonna have to wait to the next day. And then after you've done that consecutive days of 20, say you have a large website, once you've done that, it will actually lock you out for seven days and you'll have to wait. So you know, about 20 a day. If you get 60 URLs in there, request for index, that's a lot and that's substantial and you should actually be considering what web pages are the most important, especially when you build a site. If it's well over 60 pages, what pages are the most important to have Google index first and then request those pages. And you can go right here in Google Search Console and look at, is the site available for indexing, and you can look at the results and look and see when was it last crawled, is indexing allowed? Right here with indexing, if it ever says indexing is not allowed or NA, or is it allowed to be crawled, that's going to be in your bot doc file. So here in Google Search Console, we can actually use what's called a bot.TXT tester, and it will allow us to pull up the bot doc and see if there's any issues here within the document that would limit what Google crawl bots are allowed to see and what they're allowed to index. It's important to note that we do have a crawl budget with our website and Google bots will only come and nibble at our website at any given time. So as you can see here, the last time the site was crawled was the 21st. It's not being crawled every day, it's not being crawled every week. It's one of those things that Google will come visit it and then maybe come back in a few weeks. So it's something to consider, especially when we're building sites and SEO in sites. And then sometimes it takes a little while for where the indexing to catch up to where we want to be. And part of that is we need to go into the backend of the website and dump the cache and do those kinds of things and basically tell Google, Hey, we've refreshed our site. You can do that here in Google Search Console, by request indexing or view crawled page, and even go a live test and then view the tested page there. And that will basically give you a manual crawl. So, even though it doesn't say crawl my website, if you do this live test, Google is basically sending bots to go look at your site on a particular URL. So that's a good way to get around and basically encourage Google to look at it. So as you can see right here, Google has crawled smartphone and it was crawled today. So, this will update that it was crawled here shortly and we won't have to worry about this website so much. So let's get into, we've talked about the bot doc. We talked about how to find the bot doc and how to edit it. So, here in Google Search Console, I can actually go into this file and actually block URLs here. I can put 'em in. I can put 'em in here to disallow a particular URL. In this case, we've disallowed the admin portals for our website. Let's go and look at how to request URLs be blocked in Google Search Console. So we can do a 30-day or temporary block in Google Search Console, and basically tell it to not cover certain pages or that we're, we're removing certain pages here in Search Console. So here's the removals in Search Console. And this is what we will use to go through and do a temporary removal. Let's say we're updating a page or we want to not have it indexed until it's ready. Let's say we're building a shop. But we're, what'll happen sometimes, and it's kind of embarrassing when it happens, is that sometimes people will build like a men's special discount page and they will semi launch that page unintentionally. So it's just better if you have kind of, either put it in a bot doc as temporarily blocked or put it in right here as your temporary removals and then go through and remove it out later when you're ready to launch that page. That happens sometimes, Google will actually crawl a page that will actually show up as a 404 for let's say a Father's Day special discount page. And it will actually index in Google before we're actually ready to launch that page. Sometimes that happens and when it happens, it's kind of rare now, but it used to happen way more often. And it's one of those things that we can go through right here and request it. We can also submit certain pages that have outdated content and basically say, Hey, we've got this outdated content still in Google. Can you crawl this page? So this is the way to do it. And this is a way to do temporary, to go through and add certain pages here. Okay, so in this video we've covered crawling and some of the tools for website crawling and how to submit a bot doc. So we can go to Google and open bot doc tester, and submit a bot doc for a given property, add a property, submit a property. So this is how we go through and actually add bot docs to properties to go to Google, and then you go to Search Console. All right, in a future video we'll cover keywords and keyword planning, as well as some more things about website design. All right, we'll see you in the next video.
