After some consideration, I also applied this convention to every site I build - including content negotiation: Clients can either send an Accept header with their preference, or append an explicit extension (.md|.markdown for Markdown, .json for JSON API responses, or .html for the human HTML page). Together with the content negotiation part, it feels very much like HTTP was intended to work - especially the fact that API clients, AI agents, and humans all use the same URLs, but get the content in the shape they need.
vidarh 9 hours ago [-]
I've done this off and on for various sites over the years too, and probably should be more consistent about it. A number of sites do or used to do some variation of this, and I wish it was more widespread. E.g. Reddit will serve up a json version of a sub-dreddit if you do /r/subreddit.json
mceachen 5 hours ago [-]
Here's how to do it with more recent versions of Hugo:
(It includes the grandparent's head link suggestion, but it's not just "change .html to .md" because I'm old skool and as a wee nerd was told that URLs ending in .html or .php or whatever we're frowned on, so the above link's markdown is available by appending /index.md )
10 hours ago [-]
chrisweekly 7 hours ago [-]
It gets even more "interesting" for markdown-based systems like Astro or Obsidian Publish: author in md -> ship html && optionally serve md?
dspillett 11 hours ago [-]
The same could be said of robots.txt
And anything else that might tell them not to access something.
reddalo 8 hours ago [-]
robots.txt predates the modern web though
dspillett 5 hours ago [-]
My point was that llms.txt not working is no different from them ignoring everything else that came before and probably everything that is yet to come.
If they want it, they will take it, polite directives in text files will have no effect.
pfannl 8 hours ago [-]
To be fair, "not adopted by any major AI player" is probably the most web-standard-compliant phase of a new web standard.
sandblast 13 hours ago [-]
No, in fact I don't. But this post wouldn't be of any help anyway. It feels like it's about nothing, there is no substance, just stating some obvious facts. Without examples that lead to some real recommendations, this whole expertise claimed by the author is of no use.
amiga386 9 hours ago [-]
> expertise claimed by the author
The author is on record as trying to remove HTTP 418 "I'm a teapot" support from NodeJS, which resulted in backlash and Python adding support for it.
Back in 2018 when the internet wasn't completely fun, but still more fun than today...
AlienRobot 9 hours ago [-]
Not many people could have gotten that done. Sounds like expertise to me.
wseqyrku 12 hours ago [-]
The point of the post was that you need to add robots.txt (or similar) because it's a thing, and also tell us where they are.
Geezus_42 10 hours ago [-]
> add a robots.txt
Which bots will then ignore.
devdoshi 3 hours ago [-]
I do think it is important to have autonomous discoverability with domain-anchored trust, whether through .well-known or DNS records or DNS over HTTP. It looks like cloudflare has already added a bunch of observability into their products around this area, and I am investigating too [1]. It seems like the number of services needing these, and the amount needed per org should both go up with more agentic use cases.
I believe auth.md is also a recent example that uses .well-known
Why password-reset instead of a more generic link tree?
Why discord domain verification instead of domain-verifications with a dynamic list on entries?
Seems like a waste of time. I would just define my own spec outside of well known for my use case.
reddalo 13 hours ago [-]
Your own spec wouldn't be used by anyone else.
The password-reset well-known endpoint is used by password managers to show a "Change password..." button in their interface, which magically links to the password change page described in that well-known file.
jvuygbbkuurx 13 hours ago [-]
If the website implements it. What about email preferences? Removing account links? There are many use-cases you might want to redirect a user to, but having to make their own well known for it seems dumb instead of using a more generic one. I guess the more flexible it is, the harder adoption becomes as the usage within a spec might diverge, or it grows outside of the spec and becomes unofficial. So maybe password-reset is correct level of specification.
Anyway discord domain verification can tell in their onboarding docs to put it anywhere. It being well known does nothing. If there was a root level domain verification, then you might as well put it under that. But otherwise why go through a process?
notpushkin 12 hours ago [-]
It’s just easier for everybody to implement. Password manager opens https://<some-website>/.well-known/change-password in the user’s browser, it gets redirected to the actual page where password change form is located. You could make the password manager look it up in a link tree and then open a correct page, yes, but...
> I guess the more flexible it is, the harder adoption becomes
Yeah. If there is one account management related URL that password managers care about, it’s the change password page. You don’t really need to change email on your account that often, but it is probably a good idea to rotate your password once in a while. So I guess it’s a good idea to make it as easy as possible to adopt – which means just a single URL redirecting to another.
> If the website implements it.
That’s a good catch, though. I guess right now password managers would still have to make a “preflight” request just to see if /.well-known/change-password is implemented before showing it to the user. (But that can go away if most websites adopt it.)
masklinn 12 hours ago [-]
> That’s a good catch, though. I guess right now password managers would still have to make a “preflight” request just to see if /.well-known/change-password is implemented before showing it to the user. (But that can go away if most websites adopt it.)
It’s not really a catch? Like robots.txt it’s just something you probe if you have the capabilities to use it. You can just cache the info afterwards.
jnewton_dev 8 hours ago [-]
[dead]
arcfour 13 hours ago [-]
> Why discord domain verification instead of domain-verifications with a dynamic list on entries?
The TXT record itself is already a dynamic list of entries. It's far simpler and easier to iterate through the list and compare the start of each value with your search string until you find "discord domain verification" directly than it would be to do anything else.
Example:
;; ANSWER SECTION:
ycombinator.com. 300 IN TXT "openai-domain-verification=dv-QbhxxK0G0JK0dnyZ4YTsNAfw"
ycombinator.com. 300 IN TXT "v=spf1 include:_spf.google.com include:mailgun.org a:rsweb1-36.investorflow.com include:_spf.createsend.com include:servers.mcsv.net -all"
ycombinator.com. 300 IN TXT "MS=ms37374900"
ycombinator.com. 300 IN TXT "anthropic-domain-verification-0qe2ww=yK576oHdDgyTcXgkPfj1KXgGt"
ycombinator.com. 300 IN TXT "ZOOM_verify_2ndw8KZxSRa8PT8NmdyXvw"
ycombinator.com. 300 IN TXT "google-site-verification=KsI69Y_jEVkp4eXqSQ9R9gwxjIpZznvuvrus6UolB9Y"
ycombinator.com. 300 IN TXT "ca3-4861b957e83847c188e45d04ec314ee3"
ycombinator.com. 300 IN TXT "apple-domain-verification=WG0sP5Alm7N6h1Te"
ycombinator.com. 300 IN TXT "dropbox-domain-verification=asc63coma4mv"
ycombinator.com. 300 IN TXT "google-site-verification=GJKdQskycEclAGPua3yXB9m_nVhxbrsVps_y-t9SXV0"
ycombinator.com. 300 IN TXT "Wayback verify for support request 741082"
ycombinator.com. 300 IN TXT "google-site-verification=rivq8jKu6AADGtbbEzJhmOpcqq08B7QxIzXxYV8DtyU"
ycombinator.com. 300 IN TXT "rippling-domain-verification=a660f7a4ab77a3de"
teddyh 9 hours ago [-]
Having all those TXT records at the domain apex like that makes the TXT query reply huge, which affects, for instance, every mail recipient who merely wants to check the SPF record. This is a bad pattern to follow.
Bender 6 hours ago [-]
The domains with large numbers of TXT records are also used in DNS DDoS amplification attacks. Spoofed UDP requests to domains that have a large number of TXT records are used to slam other sites. In the past I would transparently strip the TXT records when I ran public DNS recursive resolvers nobody noticed except the botters but some here may be activated. Some domains with a lot of dangling records:
Whee, my chance to be the useless use of cat asshole.
Why the echo? "for" should handle a list of terms just fine.
Pedantic assholery aside, genuine question. Is this some sort of shell expansion injection countermeasure of which I am unfamiliar?
And for the record I quite enjoy employing the useless use of cat. It turns pumping a file into a pipeline from a screwball shell meta command into a command isometric to any other command. I sort of wish tee had a "suppress stdout flag" so it could be used more naturally as cat's counterpart.
Bender 4 hours ago [-]
Whee, my chance to be the useless use of cat asshole.
Would it be mean if I said I do that to expose cat rectum? I used to cat to tac to cat but that was too on the nose. Another fun one is mixed case HtMl elements. I miss that old dokimos site from 2001.
The better pattern is to use an underscore prefix like _discord-verification.domain.com
If your site allows user-created subdomains it shouldn't allow leading underscore. This is reserved somehow.
quotemstr 4 hours ago [-]
Why would you want a new prefix over using record types as they were meant to be used?
sandblast 13 hours ago [-]
"Domain-verifications" is an invitation for everyone else that might need it to use the same standard and convention. "Discord-domain-verification" is not, it's what feels like polluting the global namespace with the company name that might cease to exist in a few years.
At the very least, it should be "domain-verification-discord", "-google" and so on. Maybe even "-com.discord", "-com.google"? And the first part clearly standardized and registered, instead of one entity using "domain" and another one "site".
arcfour 12 hours ago [-]
Why?
zamadatix 10 hours ago [-]
Why reinvent the wheel differently 50,000 times instead? I'll usually even prefer a badly designed, but standard, format/encoding over a NIH one from each company - it's just less friction in the end. Heck - include a common format for the value too, then it opens up doors to automating generation with new sites & automatically validating this config for any site following the common format.
11 hours ago [-]
11 hours ago [-]
bombcar 10 hours ago [-]
Domain verifications leak information that they shouldn't - it should be "random key.domain.com in TXT randomkey"
13 hours ago [-]
quotemstr 4 hours ago [-]
Literally the inner platform effect. We have multiple kinds of DNS record. Let's use them instead of creating a key value store inside a key value store.
Does a change-password registry actually get used, even by bots? I don't see bots checking for a .well-known/change-password url on my sites. It seems a good place to put public configs, just to have a place for them, but not as a means of discovery.
ameliaquining 4 hours ago [-]
Some password managers, such as Chrome's, offer a "change password" button in the UI that informs the user if their password has been compromised. This is based on .well-known/change-password.
1vuio0pswjnm7 7 hours ago [-]
"This Web site requires a more modern browser to operate securely; please upgrade your browser."
SNI is good, though? I'm curious how you are running into this.
quotemstr 4 hours ago [-]
Perhaps he's trying to drive adoption of ECH. SNI is better than nothing, but it leaves the name of the destination domain in plaintext.
6 hours ago [-]
inigyou 6 hours ago [-]
The consideration about having more than one of them on a domain seems like something that's often overlooked.
momoraul 8 hours ago [-]
.well-known started tidy and quietly became the junk drawer of the web root. security.txt, ACME, app-site-association, and counting.
marcosdumay 6 hours ago [-]
What do you mean? It was explicitly designed to be a junk drawer.
ape4 6 hours ago [-]
Why not call it .junk-drawer instead ;)
marcosdumay 5 hours ago [-]
Well, it does live space for other drawers for other kinds of junk. This one is just for junk you already know it will be there.
delichon 7 hours ago [-]
A junk drawer is an improvement over scattered junk.
chrisweekly 7 hours ago [-]
exactly! in fact, that's most of the whole point of .well-known!
inigyou 6 hours ago [-]
Isn't that the point though? Keeping all the junk in a drawer labeled as junk, instead of keeping it on the kitchen counter?
jiggunjer 9 hours ago [-]
Title says uri but post only about urls, a type of uri
einpoklum 13 hours ago [-]
How well-known are those URIs though? :-\
eschatology 13 hours ago [-]
I spent 10 minutes searching for one in the article, in the RFC, in the wikipedia page, on google, to search for a .well-known example. Couldn't find one.
I did read one before while working with github oidc, and I did find it very useful.
What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example? This far from the first case I've ran into either.
deathanatos 6 hours ago [-]
> I spent 10 minutes searching for one in the article, in the RFC, in the wikipedia page, on google, to search for a .well-known example. Couldn't find one.
I don't know how that can be, since you claim to have found the RFC; the RFC straight-forwardly states,
> 5. IANA Considerations
> This specification updates the registration procedures for the "Well-Known URI" registry, first defined in [RFC5785]; see Section 3.1.
& then of course directs IANA to establish a registry. We'd expect this section, given the very nature of the RFC is that it establishes a collection of things, so that there is an IANA considerations section should be wholly unsurprising…
And there's a link to a listing of every standardized .well-known URI there is.
> What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example?
The RFC provides an example in the form of "example", but also in the form of "robots.txt" (as a "it could have used this, had this existed", but what else could it have done?).
hahajk 9 hours ago [-]
I've been setting up some federated servers (Matrix, activitypub) and I ran into .well_known/ paths in many of them. Webfinger resolver for activitypub and a more custom matrix server-to-server federation endpoint.
well-known is for programmatic access, it either namespaces something you’re told to look for (e.g. various types of domain markers) or it lets you discover a feature / endpoint.
In the latter case you just probe, for instance if you’re a password manager and you have a password for site A you hit A/.well-known/change-password and if they returns something you can surface a change password link to your user.
The one you found is for OIDC provider discovery (https://openid.net/specs/openid-connect-discovery-1_0.html#P...) so someone tells you they want to log in via Google, you hit that endpoint, and it lets you setup Google as an oidc provider rather without needing to hard-code providers. Even if you just want to support Google as a provider, you hit that and you get the entire configuration rather than have to hunt down the same information in the docs.
eschatology 12 hours ago [-]
Thank you, that it is part of OIDC provider discovery spec explains a lot.
That said, I still find it very bizzare that it's so hard to find a tangible example to see how it is in practice.
The rfc has none. Another spec including the use of it has none. In the end only completed service provider/implementers show it.
Before programmatic access happens, it needs to be written by a human. Yet the whole thing feels so human-unfriendly.
Perhaps I am biased robots.txt sets a high bar on how easy it is to find and work with?
masklinn 12 hours ago [-]
What RFC? The oidc discovery spec has an example, and for change-password it’s just a redirect. RFC 8615 is about the existence and management of the .well-know namespace, so examples don’t really make sense.
hparadiz 11 hours ago [-]
A JWKS is defined at /.well-known/jwks.json
It's a JSON array of public keys which you can use to validate a JWT which is what an OIDC token is.
Making it an array means you can rotate keys whenever but the validator is typically caching the public keys.
I agree. I was hoping for a few positive examples, but didn't see any. The only one I know of is the OIDC discovery endpoint.
asdfasdfadsfs 13 hours ago [-]
I would say acme-challenge is one of the most used ones. How else would one get SSL certificates today
echoangle 12 hours ago [-]
DNS TXT challenge for example. Also better because you can get wildcard certs.
ameliaquining 4 hours ago [-]
The great virtue of the in-band challenge types is that web servers can just handle them out of the box, without any need for a separate setup step that depends on your stack. I think this has done a heck of a lot to increase adoption of HTTPS.
sureglymop 9 hours ago [-]
Also, DNS-PERSIST-01 seems to be coming soon for Let's Encrypt, which should allow even people that can't easily dynamically update their DNS records to get wildcard certs. I assume this might become more widely used than HTTP-01 challenges.
inigyou 6 hours ago [-]
I wish someone would write a blog post about the difference between DNS registrars and DNS hosts, because I've seen people assume they need to use a registrar that has an API in order to change their DNS records programmatically. I used to assume that too.
Natfan 4 hours ago [-]
- registrars control NS records, however these can be changed
- NS records control other records
- registrars can also use their own nameservers to manage your DNS
echoangle 4 hours ago [-]
Yeah, so basically it’s fine as long as your registrar lets you set your own NS. Are there ones where you can’t do that?
Slightly less well-known than XDG directories among the developers of Linux-targeted software, it would seem.
Seriously, what an oxymoronic name. "/index.html" is a well known URL, literally: most of web-developers are aware of it. But inventing a bunch of URLs with predefined semantics and then slapping the "well-known" label on it... well, it won't magically make them actually well-known.
user3939382 8 hours ago [-]
I wish we had one for navigation layout of a site so browser chrome could render that in a consistent way. It would also be a boon for a11y.
quotemstr 4 hours ago [-]
One disappointment you can't help but feel, having worked in technology a while, is about how people solve the same problems over and over in redundant and subtly incompatible ways.
How do you associate metadata with a public name? A SRV record! No, a TXT record! No, a meta tag! No, data attributes! No, an X.509 attribute! No, a random file at top level! No, a well known file under some schema! No, ...
It goes on forever. We're left with a mishmash of mechanisms and lowest common denominator support for them all.
It would be nice if we picked an extension mechanism and maximally enhanced it rather than having everyone invent his own
Whoever decided it would be a good idea for ".well-known" to be a "hidden" directory is a complete fool. All it does is provide the opportunity for confusion, misconfiguration, skipped backups, missed git check-ins, forgotten updates and more. Literally the only people a folder like that is hidden from is the whoever is managing the web server.
Sure, if everyone knows what they're doing, it's not a problem. But we all know how long that assumption lasts.
zamadatix 10 hours ago [-]
I think the blog author is the one who wrote the original RFC. To be fair to him, there once was a time web servers were more commonly thought of as truly being remote directories of files you can view or link to, not just domains the browser hides the rest of, and dotfiles would commonly act like dotfiles in local file listings. Nowadays, the assumption is if you go to the base URL it should only ever serve the default page and if you try to go to a directory it should throw an error. Well, unless you're one of those ancient sites like https://ftp.mozilla.org/
I'm not saying it's good or bad how things turned it, but the choice of a dotfile for this sure did not pan out well as the web went the exact opposite direction it would have been relevant in.
9dev 10 hours ago [-]
The main point of consideration here probably was how to avoid conflicts with URLs of existing sites, not exactly people who aren't able to serve an endpoint with a dot within its path...
marcosdumay 6 hours ago [-]
TBF, those people are already hit with problems on their apache configuration and fixed their tooling long before the lack of .well-known gives them any problem.
Let's stop polluting the root of a domain!
[1] https://llmstxt.org/
I have created an `llms.txt` for my website anyhow. I use a fixed LLM prompt to generate it from the internal links in `index.md`.
https://code.claude.com/docs/en/overview and
https://code.claude.com/docs/en/overview.md
https://photostructure.com/coding/hugo-markdown-output/
(It includes the grandparent's head link suggestion, but it's not just "change .html to .md" because I'm old skool and as a wee nerd was told that URLs ending in .html or .php or whatever we're frowned on, so the above link's markdown is available by appending /index.md )
And anything else that might tell them not to access something.
If they want it, they will take it, polite directives in text files will have no effect.
The author is on record as trying to remove HTTP 418 "I'm a teapot" support from NodeJS, which resulted in backlash and Python adding support for it.
https://en.wikipedia.org/wiki/Hyper_Text_Coffee_Pot_Control_...
Which bots will then ignore.
I believe auth.md is also a recent example that uses .well-known
[1] https://instagrate-me.sudoscience.dev/
Why password-reset instead of a more generic link tree?
Why discord domain verification instead of domain-verifications with a dynamic list on entries?
Seems like a waste of time. I would just define my own spec outside of well known for my use case.
The password-reset well-known endpoint is used by password managers to show a "Change password..." button in their interface, which magically links to the password change page described in that well-known file.
Anyway discord domain verification can tell in their onboarding docs to put it anywhere. It being well known does nothing. If there was a root level domain verification, then you might as well put it under that. But otherwise why go through a process?
> I guess the more flexible it is, the harder adoption becomes
Yeah. If there is one account management related URL that password managers care about, it’s the change password page. You don’t really need to change email on your account that often, but it is probably a good idea to rotate your password once in a while. So I guess it’s a good idea to make it as easy as possible to adopt – which means just a single URL redirecting to another.
> If the website implements it.
That’s a good catch, though. I guess right now password managers would still have to make a “preflight” request just to see if /.well-known/change-password is implemented before showing it to the user. (But that can go away if most websites adopt it.)
It’s not really a catch? Like robots.txt it’s just something you probe if you have the capabilities to use it. You can just cache the info afterwards.
The TXT record itself is already a dynamic list of entries. It's far simpler and easier to iterate through the list and compare the start of each value with your search string until you find "discord domain verification" directly than it would be to do anything else.
Example:
In unbound.conf:
after the changes:Why the echo? "for" should handle a list of terms just fine.
Pedantic assholery aside, genuine question. Is this some sort of shell expansion injection countermeasure of which I am unfamiliar?
And for the record I quite enjoy employing the useless use of cat. It turns pumping a file into a pipeline from a screwball shell meta command into a command isometric to any other command. I sort of wish tee had a "suppress stdout flag" so it could be used more naturally as cat's counterpart.
Would it be mean if I said I do that to expose cat rectum? I used to cat to tac to cat but that was too on the nose. Another fun one is mixed case HtMl elements. I miss that old dokimos site from 2001.
Here's [1] something to play with. not my repo
[1] - https://github.com/bashfuscator/bashfuscator
If your site allows user-created subdomains it shouldn't allow leading underscore. This is reserved somehow.
At the very least, it should be "domain-verification-discord", "-google" and so on. Maybe even "-com.discord", "-com.google"? And the first part clearly standardized and registered, instead of one entity using "domain" and another one "site".
That’s on Discord. They’re not in the registry: https://www.iana.org/assignments/well-known-uris/well-known-...
> Why password-reset instead of a more generic link tree?
[edit: answered in more detail in a sibling thread https://news.ycombinator.com/item?id=48596286]
Alternative, no SNI required
https://web.archive.org/web/20260619061625if_/https://mnot.n...
I did read one before while working with github oidc, and I did find it very useful.
What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example? This far from the first case I've ran into either.
I don't know how that can be, since you claim to have found the RFC; the RFC straight-forwardly states,
> 5. IANA Considerations
> This specification updates the registration procedures for the "Well-Known URI" registry, first defined in [RFC5785]; see Section 3.1.
& then of course directs IANA to establish a registry. We'd expect this section, given the very nature of the RFC is that it establishes a collection of things, so that there is an IANA considerations section should be wholly unsurprising…
If you see the linked section…
> The "Well-Known URIs" registry is located at <https://www.iana.org/assignments/well-known-uris/>.
And there's a link to a listing of every standardized .well-known URI there is.
> What is it with technical documentations that go deep describing what it is in plenty words but refusing to give a single example?
The RFC provides an example in the form of "example", but also in the form of "robots.txt" (as a "it could have used this, had this existed", but what else could it have done?).
And https://www.rfc-editor.org/info/rfc8615/#section-3
> Applications that wish to mint new well-known URIs MUST register. them, following the procedures in Section 5.1 […]
Of course this is all voluntary, but why would you risk a name collision with another project if the process is quite straightforward?
Here's one I could find: https://accounts.google.com/.well-known/openid-configuration
But how does one even find this?
In the latter case you just probe, for instance if you’re a password manager and you have a password for site A you hit A/.well-known/change-password and if they returns something you can surface a change password link to your user.
The one you found is for OIDC provider discovery (https://openid.net/specs/openid-connect-discovery-1_0.html#P...) so someone tells you they want to log in via Google, you hit that endpoint, and it lets you setup Google as an oidc provider rather without needing to hard-code providers. Even if you just want to support Google as a provider, you hit that and you get the entire configuration rather than have to hunt down the same information in the docs.
That said, I still find it very bizzare that it's so hard to find a tangible example to see how it is in practice.
The rfc has none. Another spec including the use of it has none. In the end only completed service provider/implementers show it.
Before programmatic access happens, it needs to be written by a human. Yet the whole thing feels so human-unfriendly.
Perhaps I am biased robots.txt sets a high bar on how easy it is to find and work with?
It's a JSON array of public keys which you can use to validate a JWT which is what an OIDC token is.
Making it an array means you can rotate keys whenever but the validator is typically caching the public keys.
https://www.hanko.io/blog/understanding-jwks
Actually... found it https://datatracker.ietf.org/doc/html/rfc6749
And here's a PHP implementation that is perfect. https://github.com/thephpleague/oauth2-client
Seriously, what an oxymoronic name. "/index.html" is a well known URL, literally: most of web-developers are aware of it. But inventing a bunch of URLs with predefined semantics and then slapping the "well-known" label on it... well, it won't magically make them actually well-known.
How do you associate metadata with a public name? A SRV record! No, a TXT record! No, a meta tag! No, data attributes! No, an X.509 attribute! No, a random file at top level! No, a well known file under some schema! No, ...
It goes on forever. We're left with a mishmash of mechanisms and lowest common denominator support for them all.
It would be nice if we picked an extension mechanism and maximally enhanced it rather than having everyone invent his own
Sure, if everyone knows what they're doing, it's not a problem. But we all know how long that assumption lasts.
I'm not saying it's good or bad how things turned it, but the choice of a dotfile for this sure did not pan out well as the web went the exact opposite direction it would have been relevant in.