What is robots-parser?
"A specification compliant robots.txt parser with wildcard (*) matching support."
robots-parser is used to facilitate high-performance Node.js workflows. It provides a slim footprint of 45.2 KB on disk.
Installation Guide
# Install via NPM
npm install robots-parser
# Install via Yarn
yarn add robots-parser