What our crawler does
StorePrism is the crawler behind StorePrism. It reads public pages from Shopify storefronts to work out what a store runs and what it sells. If you would rather it did not read yours, one line of robots.txt stops it. This page gives you that line, says how soon it takes effect, and names who to email if you would rather talk to a person.
How to identify us
Every request we make carries this header:
User-Agent: StorePrism/1.0 (+https://storeprism.com/bot)Requests come from our own servers. We do not use a residential proxy pool, we do not run JavaScript on your storefront, and we never sign in, add to a cart or place an order.
Every request is also signed with Web Bot Auth, so you can check that it really came from us rather than from someone borrowing our name. The Signature-Agent header names https://api.storeprism.com, the key id is ehkA2qriJ8bqaQLREv6Y8srKM8nnKX2DiWRnRCNGLRU, and the public key is published at https://api.storeprism.com/.well-known/http-message-signatures-directory.
What we request, and how often
These paths and no others, and you can check that against your access log.
| Path | Why | How often |
|---|---|---|
/robots.txt | To find out whether you want us to read anything else at all. | Once a day per store, before anything else. |
/products.json | The public product list Shopify storefronts publish. It is how we track what a store sells and how stock moves over time. | Between once an hour and once a week, depending on how much interest a store gets. Most stores are at the slow end. |
/ | The homepage, read to identify what a store runs: its platform, its theme and the apps it loads. | When somebody looks the store up on one of our tools (no more than once a day), about once a month to keep that reading current, and now and then to check that our app detection still works. |
/meta.json | The small public document a Shopify storefront publishes about itself. We ask for it only when your homepage did not answer us, to confirm the store is on Shopify. | Only after a homepage read failed, as a single request. |
We ask for one page at a time and pace ourselves on our side, before a request leaves. Each of our 2 services paces itself: to any one store it averages no more than one request every 5 seconds after a burst of at most 4, and across every store together it currently averages no more than 12 requests a minute after a burst of at most 4. The services do not coordinate, so a store both are reading at the same moment can see up to twice that pace. We should never be a noticeable share of your traffic. If we are, please tell us. That is a bug on our side and we would like to know.
How to stop us
Put this in the robots.txt at the root of your store:
User-agent: StorePrism
Disallow: /StorePrism is the only name we have ever crawled under, so that one line is all you need. If we ever rename, we will keep honouring every name we have ever published, permanently. A rename adds a name to that stanza and never takes one off it, because a rule you wrote years ago should not stop working because we changed our branding.
You can also block one path rather than all of them. If you are happy for us to say what platform your store runs on but would rather we did not read your product list, disallow just that:
User-agent: StorePrism
Disallow: /products.jsonWhat happens next
We re-read your robots.txt once a day, so a line added now takes effect within a day. After that we stop requesting the disallowed path entirely. We do not keep trying and we do not check whether you meant it.
If you take the line out again, the next time anyone looks your store up on one of our tools we will notice and start reading it again. If you would rather that happened sooner, email us.
We keep the readings we already took, which we think is the honest default: they were taken while you permitted them, and a store's history is not something we can reconstruct if you change your mind back. If you would like it removed as well, ask us and we will remove it.
Talking to a person
[email protected] reaches a human. Write to us if we are reading something we should not be, if we are asking too often, if you want your store removed, or if you would like us to start reading it again after a block. You do not need an account and you do not need to be a customer.
If your question is about a revenue figure we publish about your store, a correction to it, or the data we hold about an account, contact says which address answers what and how long we take.
One thing worth saying out loud
This site publishes its own robots.txt, and we expect crawlers to honour it. It took us longer than it should have to hold ourselves to the same rule: we were asking to be read on our terms while reading others on ours. This page and the behaviour behind it are the fix.