{"id":248,"date":"2026-10-31T15:00:00","date_gmt":"2026-10-31T09:30:00","guid":{"rendered":"https:\/\/akshaymaurya.in\/blog\/?p=248"},"modified":"2026-10-31T08:27:36","modified_gmt":"2026-10-31T02:57:36","slug":"robots-txt-ai-training-opt-out-openai-google-hindi","status":"publish","type":"post","link":"https:\/\/akshaymaurya.in\/blog\/robots-txt-ai-training-opt-out-openai-google-hindi\/","title":{"rendered":"\u090f\u0915 \u0932\u093e\u0907\u0928 \u0915\u093e robots.txt \u0905\u092c OpenAI, Google, Anthropic \u0915\u094b AI \u091f\u094d\u0930\u0947\u0928\u093f\u0902\u0917 \u0938\u0947 \u0930\u094b\u0915\u0947\u0917\u093e: \u0928\u092f\u093e \u0911\u092a\u094d\u091f-\u0906\u0909\u091f \u0938\u094d\u091f\u0948\u0902\u0921\u0930\u094d\u0921 \u0915\u094d\u092f\u093e \u0939\u0948"},"content":{"rendered":"\n

Website publishers \u0915\u0947 \u0932\u093f\u090f 30 October 2026 \u090f\u0915 \u092c\u0921\u093c\u0940 \u0924\u093e\u0930\u0940\u0916 \u092c\u0928 \u0917\u0908 \u0939\u0948\u0964 OpenAI, Google, Microsoft \u0914\u0930 Anthropic \u0928\u0947 \u092e\u093f\u0932\u0915\u0930 announce \u0915\u093f\u092f\u093e \u0939\u0948 \u0915\u093f \u0935\u0947 robots.txt \u092e\u0947\u0902 \u090f\u0915 \u0928\u090f single token \u0915\u094b honor \u0915\u0930\u0947\u0902\u0917\u0947, \u091c\u093f\u0938\u0915\u093e \u0928\u093e\u092e \u0939\u0948 __usage_ai-training\u0964 Website owner \u0915\u094b \u0938\u093f\u0930\u094d\u092b \u0926\u094b line \u0932\u093f\u0916\u0928\u0940 \u0939\u094b\u0902\u0917\u0940, User-Agent: __usage_ai-training \u0914\u0930 Disallow: \/, \u0914\u0930 \u091a\u093e\u0930\u094b\u0902 companies \u0909\u0938 site \u0915\u0947 content \u0915\u094b \u0905\u092a\u0928\u0947 AI models \u0915\u0940 training \u092e\u0947\u0902 use \u0928\u0939\u0940\u0902 \u0915\u0930\u0947\u0902\u0917\u0940\u0964<\/p>\n\n\n\n

\u091a\u093e\u0930\u094b\u0902 labs \u090f\u0915 \u0938\u093e\u0925, \u090f\u0915 \u0939\u0940 token \u092a\u0930<\/h2>\n\n\n\n

\u0907\u0938 announcement \u0915\u0940 \u0938\u092c\u0938\u0947 \u092c\u0921\u093c\u0940 \u092c\u093e\u0924 companies \u0915\u0940 list \u0939\u0948\u0964 OpenAI, Google, Microsoft \u0914\u0930 Anthropic, \u092f\u0939 \u091a\u093e\u0930\u094b\u0902 \u0906\u091c \u0915\u0940 AI race \u0915\u0947 \u0938\u092c\u0938\u0947 \u092c\u0921\u093c\u0947 \u0928\u093e\u092e \u0939\u0948\u0902\u0964 \u0907\u0928 \u091a\u093e\u0930\u094b\u0902 \u0928\u0947 coordinated statements \u092e\u0947\u0902 \u0915\u0939\u093e \u0915\u093f \u0935\u0947 __usage_ai-training token \u0915\u094b \u092a\u0939\u091a\u093e\u0928\u0947\u0902\u0917\u0947 \u0914\u0930 respect \u0915\u0930\u0947\u0902\u0917\u0947\u0964 \u092a\u0939\u0932\u0947 \u0939\u0930 company \u0915\u093e \u0905\u092a\u0928\u093e bot name \u0914\u0930 opt-out \u0924\u0930\u0940\u0915\u093e \u0925\u093e, publisher \u0915\u094b \u0939\u0930 lab \u0915\u0947 \u0932\u093f\u090f \u0905\u0932\u0917 directive \u0932\u093f\u0916\u0928\u0940 \u092a\u0921\u093c\u0924\u0940 \u0925\u0940\u0964 \u0905\u092c \u090f\u0915 token \u091a\u093e\u0930\u094b\u0902 \u092a\u0930 \u0932\u093e\u0917\u0942 \u0939\u094b\u0917\u093e\u0964 \u092e\u0947\u0930\u0947 \u0939\u093f\u0938\u093e\u092c \u0938\u0947 \u092f\u0939 coordination \u0916\u0941\u0926 \u092c\u0921\u093c\u0940 news \u0939\u0948\u0964<\/p>\n\n\n\n

\u0924\u0915\u0928\u0940\u0915\u0940 \u0930\u0942\u092a \u0938\u0947 \u092f\u0939 \u0915\u0948\u0938\u0947 \u0915\u093e\u092e \u0915\u0930\u0947\u0917\u093e<\/h2>\n\n\n\n

\u0924\u0930\u0940\u0915\u093e \u092c\u0939\u0941\u0924 simple \u0939\u0948\u0964 robots.txt \u0935\u0939 \u091b\u094b\u091f\u0940 text file \u0939\u0948 \u091c\u094b \u0939\u0930 website \u0915\u0947 root \u092a\u0930 \u0939\u094b\u0924\u0940 \u0939\u0948\u0964 \u0928\u090f system \u092e\u0947\u0902 publisher \u0938\u093f\u0930\u094d\u092b \u092f\u0939 \u0932\u093f\u0916\u0947\u0917\u093e, User-Agent: __usage_ai-training \u0914\u0930 Disallow: \/\u0964 \u092a\u0939\u0932\u0940 line \u0915\u0939\u0924\u0940 \u0939\u0948 \u0915\u093f \u092f\u0939 \u0928\u093f\u092f\u092e AI training \u0915\u0947 \u092e\u0915\u0938\u0926 \u0938\u0947 content use \u092a\u0930 \u0932\u093e\u0917\u0942 \u0939\u0948\u0964 \u0926\u0942\u0938\u0930\u0940 line \u0915\u0939\u0924\u0940 \u0939\u0948 \u0915\u093f training \u0915\u0947 \u0932\u093f\u090f \u092a\u0942\u0930\u0940 website off limits \u0939\u0948\u0964 \u091a\u093e\u0939\u0947\u0902 \u0924\u094b publisher \u0938\u093f\u0930\u094d\u092b \u0915\u0941\u091b sections block \u0915\u0930 \u0938\u0915\u0924\u093e \u0939\u0948, \u091c\u0948\u0938\u0947 Disallow: \/premium\/ \u0932\u093f\u0916\u0915\u0930 paid articles \u0915\u094b \u092c\u093e\u0939\u0930 \u0930\u0916 \u0938\u0915\u0924\u093e \u0939\u0948\u0964 \u092f\u0939 design robots.txt \u0915\u0947 \u092a\u0941\u0930\u093e\u0928\u0947 format \u092a\u0930 \u0939\u0940 \u092c\u0928\u093e \u0939\u0948, \u0907\u0938\u0932\u093f\u090f \u0915\u0941\u091b install \u0928\u0939\u0940\u0902 \u0915\u0930\u0928\u093e \u0939\u0948\u0964 \u091b\u094b\u091f\u093e blogger \u092d\u0940 \u0907\u0938\u0947 \u0926\u094b minute \u092e\u0947\u0902 \u0932\u0917\u093e \u0938\u0915\u0924\u093e \u0939\u0948\u0964<\/p>\n\n\n\n

\u092f\u0939 standard \u0915\u0939\u093e\u0902 \u0938\u0947 \u0906\u092f\u093e: AIPREF \u0914\u0930 IETF draft<\/h2>\n\n\n\n

\u092f\u0939 token \u0939\u0935\u093e \u092e\u0947\u0902 \u0928\u0939\u0940\u0902 \u0906\u092f\u093e \u0939\u0948\u0964 \u0907\u0938\u0915\u0947 \u092a\u0940\u091b\u0947 formal standards process \u0939\u0948\u0964 Anthropic \u0915\u0947 standards lead Kevin Kelley \u0928\u0947 IETF Internet-Draft \u0932\u093f\u0916\u093e \u0939\u0948, \u091c\u094b 30 October 2026 \u0915\u094b publish \u0939\u0941\u0906 \u0914\u0930 3 May 2027 \u0915\u094b expire \u0939\u094b\u0917\u093e\u0964 \u092f\u0939 draft IETF \u0915\u0947 AIPREF working group \u092e\u0947\u0902 discuss \u0939\u094b \u0930\u0939\u093e \u0939\u0948, \u091c\u094b decide \u0915\u0930\u0924\u093e \u0939\u0948 \u0915\u093f AI systems \u0915\u0947 \u0932\u093f\u090f content permission \u0915\u0948\u0938\u0947 express \u0915\u0940 \u091c\u093e\u090f\u0964 \u0907\u0938\u092e\u0947\u0902 Meta, Cloudflare, News Corp, Financial Times, BBC, EFF \u0914\u0930 Creative Commons \u091c\u0948\u0938\u0947 participants \u0939\u0948\u0902\u0964 \u092f\u093e\u0928\u0940 publishers, tech companies \u0914\u0930 digital rights groups, \u0924\u0940\u0928\u094b\u0902 \u090f\u0915 \u0939\u0940 table \u092a\u0930 \u0939\u0948\u0902\u0964<\/p>\n\n\n\n

\u0915\u094c\u0928 \u0915\u092c \u0932\u093e\u0917\u0942 \u0915\u0930\u0947\u0917\u093e: timeline \u0938\u092c\u0915\u0940 \u0905\u0932\u0917<\/h2>\n\n\n\n

Announcement joint \u0939\u0948, \u0932\u0947\u0915\u093f\u0928 implementation \u0915\u0940 speed \u0905\u0932\u0917 \u0905\u0932\u0917 \u0939\u0948\u0964 Anthropic already implement \u0915\u0930 \u0930\u0939\u093e \u0939\u0948, \u092f\u093e\u0928\u0940 support \u0938\u092c\u0938\u0947 \u092a\u0939\u0932\u0947 \u0909\u0938\u0940 \u0915\u0947 systems \u092e\u0947\u0902 \u0906\u090f\u0917\u093e\u0964 OpenAI \u0928\u0947 \u0915\u0939\u093e \u0939\u0948 \u0915\u093f support \u0906\u0928\u0947 \u0935\u093e\u0932\u0947 months \u092e\u0947\u0902 \u0906\u090f\u0917\u093e\u0964 Microsoft \u0928\u0947 concrete date \u0926\u0940 \u0939\u0948, 1 February 2027 \u0924\u0915 honor \u0915\u0930\u0928\u0947 \u0915\u093e target \u0939\u0948\u0964 Google \u0928\u0947 \u0938\u093f\u0930\u094d\u092b 2027 \u0915\u0939\u093e \u0939\u0948, \u0915\u094b\u0908 \u092e\u0939\u0940\u0928\u093e \u0928\u0939\u0940\u0902 \u092c\u0924\u093e\u092f\u093e\u0964 Publisher \u0915\u0947 \u0932\u093f\u090f practical \u092e\u0924\u0932\u092c \u092f\u0939 \u0939\u0948 \u0915\u093f token \u0906\u091c \u0932\u0917\u093e\u0928\u0947 \u0938\u0947 \u091a\u093e\u0930\u094b\u0902 labs \u090f\u0915 \u0938\u093e\u0925 \u0928\u0939\u0940\u0902 \u0930\u0941\u0915\u0947\u0902\u0917\u0947, \u0939\u0930 company \u0905\u092a\u0928\u0940 timeline \u092a\u0930 \u0930\u0941\u0915\u0947\u0917\u0940\u0964<\/p>\n\n\n\n

\u092a\u0941\u0930\u093e\u0928\u0947 opt-out \u0924\u0930\u0940\u0915\u094b\u0902 \u0938\u0947 \u0924\u0941\u0932\u0928\u093e<\/h2>\n\n\n\n

\u092a\u0939\u0932\u0947 publisher \u0915\u0947 \u092a\u093e\u0938 \u092c\u093f\u0916\u0930\u0947 options \u0925\u0947\u0964 OpenAI \u0915\u093e GPTBot block \u0915\u0930\u0928\u0947 \u0915\u0947 \u0932\u093f\u090f \u0905\u0932\u0917 directive \u0932\u093f\u0916\u0928\u0940 \u092a\u0921\u093c\u0924\u0940 \u0925\u0940\u0964 Google \u0928\u0947 Google-Extended token \u0926\u093f\u092f\u093e \u0925\u093e\u0964 Common Crawl \u0938\u0947 opt-out \u0915\u093e \u0924\u0930\u0940\u0915\u093e \u0905\u0932\u0917 \u0925\u093e\u0964 \u0939\u0930 lab bot name \u092c\u0926\u0932\u0924\u093e \u0930\u0939\u0924\u093e \u0925\u093e, publisher \u0915\u094b robots.txt \u092c\u093e\u0930 \u092c\u093e\u0930 update \u0915\u0930\u0928\u0940 \u092a\u0921\u093c\u0924\u0940 \u0925\u0940\u0964 \u0928\u092f\u093e __usage_ai-training token \u0907\u0938\u0940 fragmentation \u0915\u094b \u0916\u0924\u094d\u092e \u0915\u0930\u0928\u0947 \u0915\u0940 \u0915\u094b\u0936\u093f\u0936 \u0939\u0948\u0964 \u090f\u0915 token, \u091a\u093e\u0930 labs, \u090f\u0915 \u091c\u0948\u0938\u093e \u092e\u0924\u0932\u092c\u0964<\/p>\n\n\n\n

AI agents \u0915\u093e \u0926\u094c\u0930 \u0914\u0930 training rights \u0915\u093e \u092b\u0930\u094d\u0915<\/h2>\n\n\n\n

\u092f\u0939\u093e\u0902 \u090f\u0915 \u091c\u0930\u0942\u0930\u0940 distinction \u0939\u0948, crawling \u0914\u0930 training \u090f\u0915 \u091a\u0940\u091c \u0928\u0939\u0940\u0902 \u0939\u0948\u0902\u0964 AI agents \u0906\u091c websites \u0915\u094b real time access \u0915\u0930\u0924\u0947 \u0939\u0948\u0902, \u091c\u0948\u0938\u0947 assistant \u0906\u092a\u0915\u0947 \u0932\u093f\u090f flight search \u0915\u0930\u0947\u0964 \u0910\u0938\u093e \u0939\u0940 \u090f\u0915 system \u0939\u0948 Tasklet MCP<\/a>, \u091c\u094b cloud agents \u0915\u094b websites \u0938\u0947 connect \u0915\u0930\u0924\u093e \u0939\u0948\u0964 Agent \u0915\u093e site \u0916\u094b\u0932\u0928\u093e \u0914\u0930 model training \u0915\u0947 \u0932\u093f\u090f content scrape \u0915\u0930\u0928\u093e, \u0926\u094b \u0905\u0932\u0917 use \u0939\u0948\u0902\u0964 \u0928\u092f\u093e token \u0938\u093f\u0930\u094d\u092b training permission control \u0915\u0930\u0924\u093e \u0939\u0948, crawling \u0915\u094b \u0928\u0939\u0940\u0902\u0964 Publisher agents \u0915\u094b site access \u0926\u0947 \u0938\u0915\u0924\u093e \u0939\u0948, \u0932\u0947\u0915\u093f\u0928 \u0909\u0938\u0940 content \u0915\u094b future training \u0938\u0947 \u092c\u093e\u0939\u0930 \u0930\u0916 \u0938\u0915\u0924\u093e \u0939\u0948\u0964 \u092e\u0947\u0930\u0947 \u0939\u093f\u0938\u093e\u092c \u0938\u0947 \u092f\u0939\u0940 nuance \u0907\u0938 standard \u0915\u094b practical \u092c\u0928\u093e\u0924\u0940 \u0939\u0948\u0964<\/p>\n\n\n\n

\u0907\u0938 token \u0915\u0940 \u091a\u093e\u0930 \u092c\u0921\u093c\u0940 \u0938\u0940\u092e\u093e\u090f\u0902<\/h2>\n\n\n\n

\u092a\u0939\u0932\u0940, \u092f\u0939 \u0938\u093f\u0930\u094d\u092b \u0906\u0917\u0947 \u0915\u0947 \u0932\u093f\u090f \u0915\u093e\u092e \u0915\u0930\u0947\u0917\u093e\u0964 \u091c\u094b data \u092a\u0939\u0932\u0947 collect \u0939\u094b \u091a\u0941\u0915\u093e \u0939\u0948 \u0914\u0930 \u091c\u093f\u0928 models \u0915\u094b train \u0915\u093f\u092f\u093e \u091c\u093e \u091a\u0941\u0915\u093e \u0939\u0948, \u0909\u0928 \u092a\u0930 \u0915\u094b\u0908 \u0905\u0938\u0930 \u0928\u0939\u0940\u0902 \u0939\u094b\u0917\u093e\u0964 \u0926\u0942\u0938\u0930\u0940, \u092f\u0939 permission control \u0915\u0930\u0924\u093e \u0939\u0948, crawling \u0928\u0939\u0940\u0902\u0964 \u0915\u094b\u0908 lab site crawl \u091c\u093e\u0930\u0940 \u0930\u0916 \u0938\u0915\u0924\u093e \u0939\u0948, \u092c\u0938 training \u092e\u0947\u0902 use \u0928 \u0915\u0930\u0928\u0947 \u0915\u093e \u0935\u093e\u0926\u093e \u0915\u0930\u0924\u093e \u0939\u0948\u0964 \u0924\u0940\u0938\u0930\u0940, \u0907\u0938\u092e\u0947\u0902 enforcement power \u0928\u0939\u0940\u0902 \u0939\u0948\u0964 robots.txt \u0939\u092e\u0947\u0936\u093e voluntary convention \u0930\u0939\u0940 \u0939\u0948, \u0915\u094b\u0908 \u0915\u093e\u0928\u0942\u0928 \u0928\u0939\u0940\u0902\u0964 \u0915\u094b\u0908 company ignore \u0915\u0930\u0947 \u0924\u094b publisher \u0915\u0947 \u092a\u093e\u0938 penalty mechanism \u0928\u0939\u0940\u0902 \u0939\u0948\u0964 \u091a\u094c\u0925\u0940, \u092f\u0939 individual draft \u0939\u0948, formal IETF standard \u0928\u0939\u0940\u0902\u0964 \u0907\u0928 \u0938\u0940\u092e\u093e\u0913\u0902 \u0915\u094b \u091c\u093e\u0928\u0928\u093e \u091c\u0930\u0942\u0930\u0940 \u0939\u0948, \u0924\u093e\u0915\u093f publishers \u0907\u0938\u0947 \u091c\u093e\u0926\u0942 \u0915\u0940 \u091b\u0921\u093c\u0940 \u0928 \u0938\u092e\u091d\u0947\u0902\u0964<\/p>\n\n\n\n

\u092e\u0947\u0930\u0947 \u0939\u093f\u0938\u093e\u092c \u0938\u0947 \u092f\u0939 \u0915\u094d\u092f\u094b\u0902 matter \u0915\u0930\u0924\u093e \u0939\u0948<\/h2>\n\n\n\n

\u092e\u0947\u0930\u0947 \u0939\u093f\u0938\u093e\u092c \u0938\u0947 \u0907\u0938 announcement \u0915\u093e \u0905\u0938\u0932\u0940 weight legal battles \u0915\u0947 context \u092e\u0947\u0902 \u0939\u0948\u0964 Publishers \u092a\u0939\u0932\u0947 \u0939\u0940 AI labs \u092a\u0930 copyright cases \u0915\u0930 \u091a\u0941\u0915\u0947 \u0939\u0948\u0902\u0964 Courts \u092e\u0947\u0902 \u0938\u0935\u093e\u0932 \u0909\u0920\u0924\u093e \u0930\u0939\u093e \u0939\u0948 \u0915\u093f training \u0915\u0947 \u0932\u093f\u090f web content use \u0915\u0930\u0928\u093e fair use \u0939\u0948 \u092f\u093e infringement\u0964 Industry-wide opt-out standard labs \u0915\u094b \u092f\u0939 \u0915\u0939\u0928\u0947 \u0915\u093e \u092e\u094c\u0915\u093e \u0926\u0947\u0924\u093e \u0939\u0948 \u0915\u093f \u0939\u092e\u0928\u0947 choice \u0926\u0940 \u0925\u0940\u0964 Bigger picture \u092f\u0939 \u0939\u0948 \u0915\u093f AI industry \u0909\u0938 phase \u092e\u0947\u0902 \u091c\u093e \u0930\u0939\u0940 \u0939\u0948 \u091c\u0939\u093e\u0902 data \u0915\u0947 \u0932\u093f\u090f permission \u0915\u0940 system \u092c\u0928\u0947\u0917\u0940\u0964 \u092f\u0939 standard \u0909\u0938 system \u0915\u0940 \u092a\u0939\u0932\u0940 \u0908\u0902\u091f \u0939\u094b \u0938\u0915\u0924\u093e \u0939\u0948, \u092d\u0932\u0947 \u0939\u0940 \u0906\u091c voluntary \u0914\u0930 incomplete \u0939\u094b\u0964<\/p>\n\n\n\n

\u0906\u092e website owner \u0915\u0947 \u0932\u093f\u090f \u0907\u0938\u0915\u093e \u092e\u0924\u0932\u092c<\/h2>\n\n\n\n

\u0905\u0917\u0930 \u0906\u092a\u0915\u093e blog \u092f\u093e news site \u0939\u0948, \u0924\u094b steps simple \u0939\u0948\u0902\u0964 robots.txt \u092e\u0947\u0902 \u0926\u094b line add \u0915\u0930\u0947\u0902, User-Agent: __usage_ai-training \u0914\u0930 Disallow: \/\u0964 \u0938\u093f\u0930\u094d\u092b \u0915\u0941\u091b sections protect \u0915\u0930\u0928\u0947 \u0939\u094b\u0902 \u0924\u094b Disallow \u0915\u0947 \u092c\u093e\u0926 \u0935\u0939 path \u0932\u093f\u0916\u0947\u0902\u0964 \u0932\u0947\u0915\u093f\u0928 \u092f\u093e\u0926 \u0930\u0916\u0947\u0902, \u0905\u0938\u0930 \u0924\u0941\u0930\u0902\u0924 \u091a\u093e\u0930\u094b\u0902 labs \u092a\u0930 \u0928\u0939\u0940\u0902 \u0939\u094b\u0917\u093e, \u0939\u0930 company \u0915\u0940 timeline \u0905\u0932\u0917 \u0939\u0948\u0964 Anthropic \u0938\u092c\u0938\u0947 \u092a\u0939\u0932\u0947, Google \u0938\u092c\u0938\u0947 \u0906\u0916\u093f\u0930 \u092e\u0947\u0902\u0964 \u091c\u094b content \u092a\u0939\u0932\u0947 training datasets \u092e\u0947\u0902 \u091c\u093e \u091a\u0941\u0915\u093e \u0939\u0948, \u0935\u0939 \u0935\u093e\u092a\u0938 \u0928\u0939\u0940\u0902 \u0906\u090f\u0917\u093e\u0964 robots.txt \u0915\u094b regularly check \u0915\u0930\u0924\u0947 \u0930\u0939\u0947\u0902, \u0915\u094d\u092f\u094b\u0902\u0915\u093f draft \u0905\u092d\u0940 evolve \u0939\u094b \u0930\u0939\u093e \u0939\u0948\u0964<\/p>\n\n\n\n

FAQ<\/h2>\n\n\n\n

__usage_ai-training token \u0915\u094d\u092f\u093e \u0939\u0948?<\/h3>\n\n\n\n

robots.txt \u0915\u093e \u0928\u092f\u093e single token\u0964 30 October 2026 \u0915\u094b \u091a\u093e\u0930\u094b\u0902 \u092c\u0921\u093c\u0947 AI labs \u0928\u0947 \u0907\u0938\u0947 honor \u0915\u0930\u0928\u0947 \u0915\u093e \u0910\u0932\u093e\u0928 \u0915\u093f\u092f\u093e\u0964 \u090f\u0915 directive \u0938\u0947 publishers \u091a\u093e\u0930\u094b\u0902 labs \u0915\u094b training \u0938\u0947 \u0930\u094b\u0915 \u0938\u0915\u0924\u0947 \u0939\u0948\u0902\u0964<\/p>\n\n\n\n

\u0907\u0938\u0947 robots.txt \u092e\u0947\u0902 \u0915\u0948\u0938\u0947 \u0932\u0917\u093e\u092f\u093e \u091c\u093e\u0924\u093e \u0939\u0948?<\/h3>\n\n\n\n

\u0926\u094b line \u0932\u093f\u0916\u0947\u0902, User-Agent: __usage_ai-training \u0914\u0930 Disallow: \/\u0964 \u091a\u093e\u0939\u0947\u0902 \u0924\u094b \u0938\u093f\u0930\u094d\u092b \u0915\u0941\u091b paths \u0932\u093f\u0916\u0915\u0930 partial block \u092d\u0940 \u0915\u0930 \u0938\u0915\u0924\u0947 \u0939\u0948\u0902\u0964<\/p>\n\n\n\n

\u0915\u094d\u092f\u093e \u092f\u0939 \u092a\u0939\u0932\u0947 collect \u0915\u093f\u090f \u0917\u090f data \u0915\u094b \u0939\u091f\u093e\u090f\u0917\u093e?<\/h3>\n\n\n\n

\u0928\u0939\u0940\u0902\u0964 \u092f\u0939 \u0938\u093f\u0930\u094d\u092b future training \u0930\u094b\u0915\u0924\u093e \u0939\u0948\u0964 \u092a\u0939\u0932\u0947 collect \u0939\u0941\u090f data \u092f\u093e train \u0939\u094b \u091a\u0941\u0915\u0947 models \u092a\u0930 \u0915\u094b\u0908 \u0905\u0938\u0930 \u0928\u0939\u0940\u0902\u0964<\/p>\n\n\n\n

\u0915\u094d\u092f\u093e \u092f\u0939 crawling \u092d\u0940 \u0930\u094b\u0915\u0947\u0917\u093e?<\/h3>\n\n\n\n

\u0928\u0939\u0940\u0902\u0964 \u092f\u0939 \u0938\u093f\u0930\u094d\u092b training permission control \u0915\u0930\u0924\u093e \u0939\u0948\u0964 Search \u0914\u0930 AI agents \u0915\u0940 crawling \u0905\u0932\u0917 \u0928\u093f\u092f\u092e\u094b\u0902 \u0938\u0947 \u091a\u0932\u0924\u0940 \u0930\u0939\u0947\u0917\u0940\u0964<\/p>\n\n\n\n

\u0915\u094c\u0928 \u0938\u0940 company \u0915\u092c \u0932\u093e\u0917\u0942 \u0915\u0930\u0947\u0917\u0940?<\/h3>\n\n\n\n

Anthropic already implement \u0915\u0930 \u0930\u0939\u093e \u0939\u0948\u0964 OpenAI \u0906\u0928\u0947 \u0935\u093e\u0932\u0947 months \u092e\u0947\u0902\u0964 Microsoft \u0915\u093e target 1 February 2027 \u0939\u0948\u0964 Google \u0928\u0947 \u0938\u093f\u0930\u094d\u092b 2027 \u0915\u0939\u093e \u0939\u0948\u0964<\/p>\n\n\n\n

\u0915\u094d\u092f\u093e \u092f\u0939 \u0915\u094b\u0908 \u0915\u093e\u0928\u0942\u0928 \u0939\u0948?<\/h3>\n\n\n\n

\u0928\u0939\u0940\u0902\u0964 \u092f\u0939 voluntary standard proposal \u0939\u0948, IETF \u0915\u093e individual Internet-Draft\u0964 \u0915\u094b\u0908 enforcement \u092f\u093e penalty mechanism \u0928\u0939\u0940\u0902 \u0939\u0948\u0964<\/p>\n\n\n\n

\u0906\u0916\u093f\u0930\u0940 \u092c\u093e\u0924<\/h2>\n\n\n\n

__usage_ai-training \u091b\u094b\u091f\u093e technical change \u0932\u0917 \u0938\u0915\u0924\u093e \u0939\u0948, \u0932\u0947\u0915\u093f\u0928 \u0915\u0939\u093e\u0928\u0940 \u092c\u0921\u093c\u0940 \u0939\u0948\u0964 \u092a\u0939\u0932\u0940 \u092c\u093e\u0930 \u091a\u093e\u0930 \u0938\u092c\u0938\u0947 \u092c\u0921\u093c\u0947 AI labs publishers \u0915\u0940 \u090f\u0915 \u092e\u093e\u0902\u0917 \u092a\u0930 \u090f\u0915 \u0938\u093e\u0925 \u0916\u0921\u093c\u0947 \u0939\u0941\u090f \u0939\u0948\u0902\u0964 Standard \u0905\u092d\u0940 draft \u0939\u0948, timeline \u0905\u0932\u0917 \u0939\u0948, enforcement \u0928\u0939\u0940\u0902 \u0939\u0948, \u092b\u093f\u0930 \u092d\u0940 direction \u0938\u093e\u092b \u0939\u0948\u0964 \u0906\u0917\u0947 \u0926\u0947\u0916\u0928\u093e \u0939\u094b\u0917\u093e \u0915\u093f draft final standard \u092c\u0928\u0924\u093e \u0939\u0948 \u092f\u093e \u0928\u0939\u0940\u0902\u0964<\/p>\n","protected":false},"excerpt":{"rendered":"

Website publishers \u0915\u0947 \u0932\u093f\u090f 30 October 2026 \u090f\u0915 \u092c\u0921\u093c\u0940 \u0924\u093e\u0930\u0940\u0916 \u092c\u0928 \u0917\u0908 \u0939\u0948\u0964 OpenAI, Google, Microsoft \u0914\u0930 Anthropic \u0928\u0947 \u092e\u093f\u0932\u0915\u0930 announce \u0915\u093f\u092f\u093e \u0939\u0948 \u0915\u093f \u0935\u0947 robots.txt \u092e\u0947\u0902 \u090f\u0915 \u0928\u090f single token \u0915\u094b honor \u0915\u0930\u0947\u0902\u0917\u0947, \u091c\u093f\u0938\u0915\u093e \u0928\u093e\u092e \u0939\u0948 __usage_ai-training\u0964 Website owner \u0915\u094b \u0938\u093f\u0930\u094d\u092b \u0926\u094b line \u0932\u093f\u0916\u0928\u0940 \u0939\u094b\u0902\u0917\u0940, User-Agent: __usage_ai-training \u0914\u0930 Disallow: \/, \u0914\u0930 \u091a\u093e\u0930\u094b\u0902 companies \u0909\u0938 […]<\/p>\n","protected":false},"author":1,"featured_media":249,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[13,66,12,67,65],"class_list":["post-248","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","tag-ai-news","tag-ai-training","tag-openai","tag-publisher-rights","tag-robots-txt"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/akshaymaurya.in\/blog\/wp-content\/uploads\/2026\/10\/thumb-robots-token.jpg","_links":{"self":[{"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/posts\/248","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/comments?post=248"}],"version-history":[{"count":1,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/posts\/248\/revisions"}],"predecessor-version":[{"id":250,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/posts\/248\/revisions\/250"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/media\/249"}],"wp:attachment":[{"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/media?parent=248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/categories?post=248"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/akshaymaurya.in\/blog\/wp-json\/wp\/v2\/tags?post=248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}