Spider/Scraper Policy

Information about our automatic data gathering programs

Some of our services involve advising our members about changed Internet content, such as Alerts notified on various websites around the world. To do this we use a custom-designed program called a "Scraper", which detects if a webpage has been changed, downloads a copy of that page, and extracts certain information from that webpage.

Note: These data gathering programs are only used by MyRedFlare to check updated information from specified Source websites. They do nothing else, and do not get involved in our member's affairs in any way. i.e. do not say "what's that, are they invading my privacy?". We are not, nor do we want to!

Our systems then analyse that extracted data for certain keywords, threshold parameters or geographic location to determine it relevence to our members. If a webpage update is deemed of interest to our members (either as a group or individually, based of user preferences and actual geographic location) we will send them a message (email and, optionally, SMS) to let them know about it.

We do not provide our users with a copy of your webpage. In fact, we actively encourage our users to visit your website for the complete information.

When we send an SMS message to a user we will append your website address to the end of it if space exists within the 160 character limit of a standard SMS text message, with a message such as "See yoursite.com for full info."

When there is not enough space for this full link the user can refer to the email message that we will send

Scraper Indentification

Our scraper programs will identify themselves in your web access logs as:     MyRedFlare/x.xx[/yyyyyy]
where:

• x.xx is the current scraper version number, and;
• [/yyyyyy] is an optional code (maximum 10 characters) to allow us to confirm that our scraper is not being impersonated by another.

Respect...

It is MyRedFlare policy to demonstrate respect for websites that we visit. This policy includes:

• Respecting your access preferences as defined in that your "robots.txt" file.
• No "overscraping". We do not want to overload your site or to become a burden to you at all. We do not envisage any situation at this stage that requires that we visit any website more often than once per minute (for time critical alert websites). In fact, for many of the sites we visit the visitation rate is substantially lower (for example, travel advisory websites may be check once every 10 minutes, once per hour etc, depending upon the content and our perceptions of its "time criticality".
• Limiting download bandwidth... we will only download a full webpage from your site for analysis if we determine that it has actually changed. That is, while we may check every 10 minutes we will not download an actual page unless it is a new page.
• Encourage user visits to source websites. The sites we have chosen to provide information about through MyRedFlare are excellent and we want to encourage further growth and enhancment of those sites. To that end, all we do is provide a message that new info is available and enough of the key data to provide our users with enough information to decide whether they need to visit your site for the full picture.

Our Scraping Is Good For You!

We believe the nature of our service is an "intelligent, near-real-time web linking service". If we visit your website it is because we have determined provide time critical safety related information which would be of significant benefit to individuals if they could be notified about it quickly. Our automatic data gathering, analysis and delivery systems will get your message our quicker to more people, and encourage more people to visit your site to get the full story. It is a thing of mutual benefit, and certainly intended to enhance rather than detract from your service.

Questions, Comments or Requests? Please Contact Us

If you, as a webmaster, owner or representative of a site we visit, have any questions, comments or requests please send us a message at admin@myredlare.com and we will respond as soon as possible.