We're really happy that you are considering submitting your web scraping project on Apify! Apify is the leading web scraping and web automation platform, and we can’t wait to help you automate your workflow so you can focus on the fun tasks! 😉
As you might know, your web scraping or web automation project can be delivered through two main streams. Here is a short recap for clarity:
Apify Freelancers - the most affordable and fastest delivery option for your custom web scraping solution, suitable for smaller projects or one-time data extraction. In this case, you are basically hiring an Apify-certified web scraping developer (freelancer) through the Apify platform who will deliver your web scraping project.
Apify Enterprise - fully-customized web scraping and automation solution for any scale, ideal for companies that require data continuously. The solution will be fully developed, maintained, and managed by the internal Apify Team or our Apify Solution Providers.
How to submit a proposal
Regardless of the channel you choose, we ask you to make your proposal as detailed as possible so that we can send you an accurate price estimation. The more detailed the information, the more accurate the price estimation. Describe the process as though a human were to do this manually – click by click. Bonus points if you include screenshots or videos!
That’s why we have created a quick guide for you to submit the perfect proposal. Here is the main information we will need from you:
...plus a helpful example.
Now let’s look more closely at each question to understand how to best answer them in your project proposal.
1. What websites do you want to scrape?
Include the URLs of the websites you would like to scrape. Try to describe whether you are only interested in a specific category or subcategory of the website or whether you need to extract data from the whole online store or website. Optionally, we can also search for keywords in the store and extract only the results that come up.
Please try to describe in close detail what data you need – as if you were to extract the data yourself manually. This will help us understand the whole workflow and estimate how difficult it would be to set up scrapers or automation bots for your project.
2. What type of data do you want to extract?
What type of data do you want to extract from the page? Product prices? Article dates of publication? Movie ratings?
Let us know what information needs to be extracted and stored for each record that we get for you. Do you want to store the title, price, stock count, and shipping costs for each item, or is there any other information that you would like to extract, such as breadcrumbs? If you are familiar with data types, you can specify these for each of them. If not, don't worry about this and we'll suggest some.
3. How many pages or products do you want to scrape for each run?
Often, websites with a lot of content have the option to turn the page (e.g., you can find the number of pages at the top or at the bottom of the web page). How many pages do you want to scrape each time you run the web scraping operation?
Alternatively, you can set a limit on the number of products you want to scrape.
4. How often do you want to scrape?
Do you need to scrape the web page regularly (e.g., once an hour, once a day, twice a week,...) or only once? How often will you need new data?
If you plan to scrape more than once, then it might be useful to know whether you want to extract all the data from the webpage every time you scrape it or you only want the new data (added to the website since your last scrape). Consider that, as only extracting new data includes less data, this option will be faster and cheaper.
Example of a web scraping project proposal
Here’s an example of what a project proposal might look like:
First name: Mario
Last name: Rossi
Work e-mail: email@example.com
Project name: Notino market research
Does your project need a service level agreement (SLA)? Yes, it does/No, it doesn’t
Project description: I want to scrape https://www.notino.co.uk/, through all the menu categories, for perfume for men. For every product, I want to extract the item ID number, item URL, item name, discounted price, original price, and category. I want to scrape about 10 pages for each run, for a total of approximately 30 products. I want to scrape the website once a week, and I only want to extract new data (data that was added since the last scrape). I want to see my results in Google Sheets.
An example of what I want to extract for each product:
Product name: Armani Code Absolu Eau de Parfum for Men 110ml
Item ID: 15843458
Original price: $35
Discounted price: $28
Category: Perfume for men
Please, upload any relevant files, e.g., screenshots or a screen share recording (i.e., via Vidyard or Loom):
Final tips for the BEST project proposal:
- screen recording videos
- screenshots with highlighted/circled data attributes