I read that these posts are all indexed by search engines in the Guidelines. I’m curious if Discourse purposely appeals to certain search engines, or if the search engines just find these because it’s a public website. Anyone know how search engines decide what/how to index forums?
1 Like
I’m sure it gives the spiders a better organized set of text to help with spidering. I’m not sure the specifics of how they do it though.
2 Likes
Spiders look up the robots.txt file at the root of the domain. For Discourse it looks like this:
https://forum.warp.world/robots.txt
It’s possible to follow links from the home page and reach every category and post; spiders find new content that way. Using semantic URLs also give spiders even more information on the pages.
I’m not seeing a <link rel="canonical"> in the page, but there is a <link rel="search"> as well as a <meta name="fragment">. Overall this is a pretty decent SEO setup.
2 Likes
DID SOMEBODY SAY SPIDER?
