#Bytespider

7小时前

主流AI爬虫User-Agent大盘点！朋友说OpenAI的 GPTBot抓的太凶。为了GEO效果，又不得不开放各种AI的爬虫。当然，应该也有伪装这些爬虫UA采集页面。常见的 AI Bot如下 OpenAI GPTBot 用于模型训练 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; + OpenAI OAI-SearchBot 用于OpenAI的搜索功能 OAI-SearchBot/1.0; + OpenAI ChatGPT-User 用户与ChatGPT聊天涉及网页访问时触发 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; + PerplexityBot/Perplexity-User 训练场景：Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +) 用户场景：Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +) Claude-User、Claude-SearchBot ClaudeBot/Claude-User/Claude-SearchBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) DeepSeekBot Mozilla/5.0 (compatible; DeepseekBot/1.0; +)。抖音/字节/豆包AI Bytespider Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.1511.1269 Mobile Safari/537.36; Bytespider

OpenAI GPT-5发布引发用户不满，阿尔特曼回应质疑· 72 条信息

#AI爬虫 #User-Agent #GPTBot #OpenAI #Bytespider