PropertyismBot

PropertyismBot is the web crawler for Propertyism, a UK property search service in development. It visits the websites of estate agents, letting agents, auction houses and housebuilders to find homes that are publicly advertised for sale or to let, so that people searching on Propertyism can find them and go to the advertiser's own page.

How to recognise it

Every request from our crawler carries this user agent:

PropertyismBot/0.1 (+https://propertyism.co.uk/bot)

The version number may change. Other software can copy a user agent, so if you are unsure whether traffic is ours, email us the time, the address requested and the IP address, and we will check.

How it behaves

• Before visiting a website it reads robots.txt and follows the rules for PropertyismBot, or the rules for all crawlers if there are none for us.
• It reads robots.txt again at least once every 24 hours, so changes take effect within a day.
• It makes no more than one request every 5 seconds to any one website, and waits longer if your robots.txt sets a longer Crawl-delay.
• It slows down or pauses when a site says it is busy, and waits as long as the site asks.
• If a site refuses access, asks for a login or shows a security check, it stops. It does not disguise itself or try to get around a block.
• It only reads pages that anyone can see without logging in.

What we do with what it finds

• We use the facts in each advert, such as the price, type of property, number of bedrooms, location and advertiser, to list the home in our search, with a credit and a link to the advertiser's own page.
• We do not republish your full property descriptions.
• We do not keep copies of your photographs. Where we show a photograph, it is loaded from your own website.
• We never show a more precise address than the advert does, and we show branch contact details rather than the names or personal numbers of individual staff.
• We keep working copies of pages for a short time to check our results. They are never published.

How to stop or slow it

To stop it visiting your website, add this to your robots.txt:

User-agent: PropertyismBot
Disallow: /

To keep it away from part of your site, disallow just those paths. To slow it down, add a Crawl-delay in seconds under the same User-agent line.

When you block it, it stops collecting adverts from your site and we remove your existing adverts from Propertyism within 7 days.

Removing or correcting an advert

Email bot@propertyism.co.uk with your website address and the adverts concerned. If you own or live in a property shown on Propertyism and want something removed, tell us the address and what you would like removed. We will reply within 2 working days and act within 5 working days of confirming the request.

Personal data and complaints

Our privacy notice explains what personal information we handle, why, and your rights. To ask a question or make a complaint about how we use personal data, email privacy@propertyism.co.uk. You can also complain to the Information Commissioner's Office at ico.org.uk.

Working with us

Agents who would like to send us their listings directly are welcome to get in touch.