Blog > Post

Web Scraping -- Scraping AJAX and JavaScript Websites

Monday, September 05, 2016



Handle AJAX and JavaScript

Speaking about handling AJAX and JavaScript while web scraping, sometimes it could be tricky, especially when you are a tech noob. (Hmm.. I’m sorry. A...What... What did you just say?)

I’ve received lots of questions about scraping AJAX and JavaScript lately. And I’ve compiled some questions I get asked the most often.


. How do I crawl an infinite-scrolling AJAX website?

. How do I scrape data and click on load button or next button?

. How do I scrape any websites with AJAX content(like Gumtree)?

. Can Octoparse be used to scrape dynamic content from websites that are using AJAX?

. Can I scrape data from website with pagination?

. Can I scrape websites which dynamically load data(like Facebook)?

. Can I crawl a website that loads content using Javascript?



Scraping Sites with AJAX Is Not Easy

Sometimes people see web pages and find AJAX content loading and think the site cannot be scraped. If you are learning Python and you are dipping your hand into building a web scraper. It is not going to be easy. If you are looking for an easy and quick way to do this, especially for large workloads, you may want to look into some third-party applications for scraping websites with AJAX.


Example: Scrape Websites with Infinite Scrolling

So as an example, what I’m going to show you is how to scrape websites with infinite scrolling. ( If you are experienced programmer and you just write your awesome web scraper, then just ignore my gibberish.)

Check out how to scrape websites with Infinite scrolling here.




Laden Sie Octoparse herunter, um mit Web-Scraping zu beginnen, oder kontaktieren Sie uns
für die Fragen über Web Scraping!

Kontaktieren Sie uns Herunterladen
Diese Website verwendet Cookies um Ihnen ein besseres Internet-Erlebnis zu ermöglichen. Lesen Sie wie wir Cookies verwenden und Sie können sie kontrollieren, indem Sie auf Cookie-Einstellungen klicken. Wenn Sie die Website weiter nutzen, akzeptieren Sie unsere Verwendung von Cookies.
Akzeptieren Ablehnen