Very true... But someone is wrong. 🫣😄
I just want to speak up for all those with complex websites worrying about AI crawl ability - and back up my recommendation with data. 😄
Oh, and happy Easter John. 🐣🐰
Perfect. Thanks @polemicdigital.com . I sent him an email Tuesday (I think it was), so will just let him respond after easter. 😊
Does any of my SEO connections know how to get an article published on Search Engine Land?
My interaction with @johnmu.com (quoted) has been cited in an article (searchengineland.com/technical-se...) in a context that I strongly disagree with, so would love to publish a counter point article.
Perfect. Thanks for a clear answer. 😊
Sorry for being late to the party, but I just spotted this ai.google.dev/api/llms.txt.
@johnmu.com Is this an endorsement from Google?
If not, would it be fair to assume that if the Gemini team tests this there are serious considerations of supporting it?
I know this is really geeky but would love to see examples from Google on what a Gbot server load/priority could look like based on request, response header, speed and content fetched.
Not giving away specifics but more so we gain a better understanding of what weighs when "budgeting" for Gbot.
Love the answer and completely agree!
Thanks.
My curiosity just drove me to looking into whether Google had made public declarations as to what constitutes a crawl/hit in terms of their own processes.
If it is the total load on Gbot infrastructure then why not just have article examplifying this?
Thanks. Yeah, I am trying to get to the bottom of it. It seems they are a part of some scripts - which seem to generate the URLs with unique ID's on each request.
Next up: reaching out to the developers 😅
Am I correct: The help docs has no definition of what constitutes a crawl in terms of budget?
Well, my plan is to tell the client to block those URLs with robots.txt.
I have no idea why it is crawlable 🙈
But it just made me think about the crawl budget - and made me wonder if returning no content is still classified as a crawl or not.
Hey @johnmu.com
I just discovered roughly 200K URLs w status 204 (no content) not blocked in robots.txt.
Would a status 204 waste crawl budget or not?
I guess the question is: What IS a crawl (in terms of crawl budget)?
Is it both request, response header and content OR just request + response?
Also my hunch. Would be quite impressive with timetravelling Gbot though!
Is the crawl time reported using PDT timezone (so when I see 21:28 in GSC and I am located in the UK I need to add 8 hours) or using the users timezone?
If PDT, where do I suggest that this is made much clearer in GSC? 😊
Is Googlebot capable of precognition?
Gbot reports a bug that was introduced after Gbot crawl.
Does anyone (perhaps @johnmu.com) know why GSC lists a page as last crawled May 21 21:28 - w user declared canonical showing a bug, that was introduced in a midnight release between the 21. and the 22?
Awesome. Thanks for that. I tried looking up the answer on your long page w. general feature info but this info wasn't listed. Might be worth adding in terms making sure your computer clock is set to the correct time zone 🤦♂️😅
@screamingfrog.co.uk Quick question regarding scheduled crawls: Does SF use the computer clock for activating the scheduled crawls at the right time or does SF use some other global clock and an internal sync'ed clock?
How did you identify the need for the content brief in the first place?
Often I would use a combo of 1 and 2 to identify main topics that do not overlap.
Then I would use 2 and create a page strategy matching intent with topic. From that I would build the brief.
Haha, they could just as well have said:
As famous captain Picard of Star Wars once said: "So long and thanks for all the fish"
But yes, love seeing how IT issues are combined with climate challenges... I guess cloud computing and AI enforces that.
Chart - Showing sharp rise in "Organic traffic". X/Horizontal Axis : Time (measured in 4-day blocks, early February to early March). Y/Vertical Axis : Traffic quantity (measured in 400K bars). Chart shows a flat line until near the end of February, then massive increases over a matter of days (from 0 to over 1.5 Million, in approx. a week).
.
:: Is Google showing Favouritism? ::
After years (and years) of complaints,
of people showing G examples of weak, bad, spammy, unhelpful, unsatisfactory content,
ranking on "brand sites" (particularly Large/Enterprise Publishers) ...
... G made the #SRA.
>>>
X: x.com/darth_na/sta...
Thanks! Will definitely also be my recommendation.
I also just learned that the server sometimes serves a different variant of the robots.txt file.
Sometimes it includes the line "Disallow: /oplevelser/*$" and sometimes it doesn't.
🤯
Hi @johnmu.com
Do you know how Gbot would interpret this in robots.txt:
/oplevelser/*$
GSC says it's crawlable when inspecting: www.dailys.dk/oplevelser/m...
technicalseo.com/tools/robots... + screaming Frog says its not due to robots.txt.
Is *$ an invalid combo making Gbot ignore that line?
😂
I am seeing more big websites being hit by "bad URLs" in Google Search Console which indicates that the problem could be on a rise - perhaps leveraged by AI mass analyzing weak spots on big reputable websites. 😱💩
I wrote an article showing you how to fix it: www.linkedin.com/pulse/scamme...
Awesome! I guess I could have told myself that. 🤦♂️😅
Will check it out.
Sounds really interesting!
Is there any way to watch your TikTok if I do not have TikTok installed (it's a tinfoil hat-thing)?
Interesting!
1) What do you monitor? Is it a certain prompt?
2) GPTs return different answers for each request, so how do evaluate output that can be both plain text, bullets and tables?
Thanks. Could be I should look into Make at some point to see if it makes sense to use for some projects.
What does the use of Make add to the process instead of just connecting chatGPT directly to the Google Sheet using an extension?
Just joined a week ago and already loving it!
So much more focused than Elons nightmare. It reminds me of old Twitter.
Zoomet lidt ud.
@mortendd.bsky.social Fruen og jeg har lige været ved Børnnerup Havn i håb om at se hærfuglen (desværre var den væk).
MEN vi så dyrespor i sandet ved havnen og lugtede kraftigt dyretis 50 meter derfra.
Aftrykkene var på størrelse med 2/5-krone ca.
Kan du se, om sporene kan passe med Mårhund?