Robots.txt checking if path is disallowed (with *)

Vote count:

0

I'm trying to figure out what the best way is to interpret whether a particular url matches a disallow line in a robots.txt.

For example, if the disallow line url is:

/*/comments/*

And the url is

/welcome/test/comments/hello/p

Or if the disallowed line is:

/*?*sort

and the url is:

/hello?idsort

How can I know that these urls all disallowed? The only solution I had in my mind is to write a lexical analyzer. Are there any other solutions?

Please provide tips rather than entire code because I really want to do this myself.

asked 1 min ago

Lotus

10

Robots.txt checking if path is disallowed (with *)