Does robots.txt still matters? Short answer: yes. A wrong committed robots.txt and Googlebot (real bot, verified IP block) can hammer an application (Solr server CPU in the graph), until an aggressive rate limit is applied. End of the story: the application queries should be improved.