01Who runs it
FinPay Talent Research Bot is run by FinPay Talent, trading name of Sourcific Talent Ltd (company number 11650642), registered in England and Wales. Our registered office is 203 West Street, Fareham, United Kingdom, PO16 0EN.
We're a payments recruitment business. The bot is the name our portal uses when it reads public websites, feeds and registers itself.
02How it identifies itself
Every request it makes sends this user agent:
Mozilla/5.0 (compatible; FinPayTalentResearchBot/1.0; +https://finpay-talent.co.uk/bot/)
The name to use in robots.txt is FinPayTalentResearchBot.
03What we use it for
We use it to understand which payments companies are hiring, growing or changing, so we know who may need recruitment help. For example:
- what a company does and who it sells to
- whether it has live job adverts
- company news, such as funding, new offices or senior appointments
- changes on public registers, such as a new licence or a new UK company
It's for business research about companies.
04What it may read
- Company websites: the home page, careers or jobs pages (for example /careers or /jobs) and job advert pages.
- Job board feeds a company uses for its own jobs, such as Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor, Pinpoint, BambooHR, Breezy, Workday and Rippling.
- News feeds (RSS) published by payments and fintech news sites.
- Public registers: Companies House, the FCA register, the EBA register and The Gazette.
- Our FinPay Intelligence research may also read a company's about, legal, imprint and contact pages (to confirm a website really belongs to a registered company), its sitemaps and its news pages.
It only reads pages anyone can see. It never logs in, never fills in or submits forms on your website, never gets around a paywall, captcha or bot check, and never runs page scripts.
05How it reads websites
Our everyday BD tools
- Once a night, they check careers pages and job board feeds for companies on our research lists. Each company is checked at most once every 3 days, with no more than nine requests in a visit (usually far fewer).
- Once a night, they read each news feed and the public registers.
- When we ask for it in our portal, they read a company's home page or a job advert, or check that a website address answers.
- For each company, requests go one after another. Requests to company websites and news feeds stop after 7 to 15 seconds and aren't retried.
FinPay Intelligence research
- One request at a time to your site, at least a second apart, and no more than 4 sites at once.
- If your robots.txt sets a Crawl-delay of up to 10 seconds, it waits that long. If it asks for longer, it leaves your site alone.
- If you answer 429 or 503, it backs off for as long as your Retry-After asks (up to 10 minutes), or a minute if you don't say.
- A capped number of requests per company in a visit (40 at most), 10 seconds per request, and size limits on every page.
- Secure (https) addresses on ordinary public websites only, and no more than 5 redirects.
06robots.txt
FinPay Intelligence research reads your robots.txt before anything else and follows it:
- It follows a group for FinPayTalentResearchBot if you have one, otherwise your * group.
- Allow and Disallow rules work as the robots.txt standard (RFC 9309) says, including * and $. The most specific rule wins.
- It uses a saved copy of your robots.txt for no more than a day before reading it again.
- If there's no robots.txt (a 404), it treats your site as open. If your server can't answer for it (an error or no response), it reads nothing until it can.
- Job board feeds and official registers are read through the access their providers publish for programs, so the robots.txt on your own website doesn't apply to those.
Our everyday BD tools don't check robots.txt yet. To stop those reads as well, email us.
07How to ask us to stop
Add these two lines to your robots.txt:
User-agent: FinPayTalentResearchBot
Disallow: /
and email privacy@finpay-talent.co.uk with your website address. We'll take your site out of our research by hand and reply to confirm.
You can also block only part of your site, for example Disallow: /careers/.
08Personal data
- From websites and feeds we look for company information, not information about people. Pages can still mention people, for example a news story about a new hire.
- Our everyday BD tools keep the results, not a copy of your page: a job title, date, place and link; a news headline and link; a short description of what the company does; category tags. Page text is passed to our AI provider to work these out.
- Public registers list company officers and people in approved roles. Our BD tools record those names and roles against the company, as the register publishes them.
- FinPay Intelligence research looks for company facts. It keeps a copy of the text of the public pages a finding comes from, as evidence, and that text can include names that appear on those pages, for example a senior appointment in a news story. When our business-contact research is in use, it may record a person's name and business role against their company where a public page shows it. Work contact details, such as business email addresses, come from our data provider, not from this bot. Its request log keeps the site, the page address (without any query) and the result, nothing from the page itself, and is deleted after 30 days.
- It doesn't keep cookies, and it never signs in to your website.
How we handle personal information, and your rights, are in our Privacy Policy (https://finpay-talent.co.uk/privacy-policy/). Questions: privacy@finpay-talent.co.uk.
09Other companies' crawlers
Some of our research uses outside providers, such as data providers (Apollo) and AI web search (Anthropic). They collect information with their own systems and crawlers, under their own names. They don't use our identifier, and a robots.txt rule for FinPayTalentResearchBot doesn't affect them. To control them, use the names those providers publish.
FinPay Talent