Scrape Glassdoor Jobs using Python (Save Data to CSV)

Web Scraping Glassdoor can provide you with some insights like what salary should one expect when applying for a job.

Employers can do Glassdoor scraping to improve their hiring strategy by comparing data with their competition. Or can use this data if building a job board.

The use cases for web scraping Glassdoor are endless and here in this article, we will extract data from jobs.

In this article, we are going to use Python & design a Glassdoor scraper. At the end of this tutorial, you will be able to save this data in a CSV file too.

For this article, we will need python 3.x, and I am assuming that you have already installed it on your machine. Along with this, we have to download third-party libraries like BeautifulSoup, Selenium, and a chromium driver.

Here is how you will install them

Along with this, you will have to create a dedicated folder where you will keep the script file. Also, create a python file inside that folder. I am going to name it glassdoor.py

It is always recommended to decide what data are you going to scrape before even writing a single line of code. As you can see in the above image, we are going to scrape four items from each posted job.

1. Name of the Company

2. Job Title

3. Location

4. Salary

First, we are going to extract the raw HTML using Selenium from the website and then we are going to use .find() and .find_all() methods of BS4 to parse this data out of the raw HTML.

Chromium will be used in coordination with Selenium to load the website. You can download Chromium from here.

Before we start we are going to run a small test to check if our setup works or not.

Now, let me explain to you what we have done here step by step.

1. We have imported the libraries that were installed earlier in this article.

2. Then we have declared PATH where our chromium driver is installed.

3. An empty list and an empty object to store job data are also declared.

4. target_url holds the target page URL of glassdoor.

5. Then we created an instance using .Chrome method.

6. Using .get() method we are trying to connect with the target webpage. Chromium will load this page.

7. Using .maximize_window() _we are increasing the size of the chrome window to its maximum siz_e.

8. Then we are using .sleep() method to wait before we close down the chrome instance. This will help us to load the website completely.

9. Then using .page_source we are collecting all the raw HTML of the page.

10. Then finally we are closing down the chromium instance using the .close() method provided by the Selenium API.

Once you run this code, it should open a chrome instance, load the page and then close the browser. If this too happens with your script then we can move ahead. Our main setup is ready.

You will notice that all these jobs are under ul tag with class JobsList_jobsList__lqjTr. So, we have to find this class first. We will use .find() method of BS4 to find this tag inside our DOM.

Using .find() method we are searching for the ul tag and then using .find_all() method we are searching for all the li tags inside the ul tag.

Now, we can use a for loop to access data of all the 30 jobs available on the page.

Sometimes Glassdoor will show you a page with no jobs. All you have to do is clear the cookies and try again.

Now, let’s find the location of each target element, one by one.

As you can see the name of the company can be found under the div tag with the class EmployerProfile_profileContainer__VjVBX. Let’s parse it out from the raw HTML using BS4.

Now let’s find the job name.

You can find the name of the job in a tag with the class JobCard_jobTitle___7I6y.

In the above image, you can see the location is stored under the div tag with the class JobCard_location__rCz3x.

The last thing left is the salary and this is the most important part for obvious reasons.

You can see in the above image that the salary information can be found under the div tag with the class css-3g3psg pr-xxsm.

In the end, we have pushed the object o inside the list l. Then we declared the object o empty.

Once you run and print the list l, you will get these results.

You will get the name of the company, the name of the job, the location, and the salary in a list.

For saving this data to a CSV file all we have to do is install pandas. This is just a two-line code and we will be able to create a CSV file and store this data in that file.

First, let’s install pandas.

Then import this into our main script glassdoor.py file.

Now using DataFrame method we are going to convert our list l into a row and column format. Then using .to_csv() method we are going to convert a DataFrame to a CSV file.

You can add these two lines once your list l is ready with all the data. Once the program is executed you will get a CSV file by the name jobs.csv in your root folder.

Isn’t that simple? You were able to scrape and save the data to a CSV file in no time.

You can make more changes to scrape a little more information. But for now, the complete code will look like this.

The code is simple, neat, and straightforward.

If you want to scrape all the jobs from all the pages from Glassdoor then you will first notice URL patterns.

First-page https://www.glassdoor.com/Job/new-york-python-jobs-SRCH_IL.0,8_IC1132348_KO9,15.htm?includeNoSalaryJobs=true

Second Page — Second Page — https://www.glassdoor.com/Job/new-york-python-jobs-SRCH_IL.0,8_IC1132348_KO9,15_IP2.htm?includeNoSalaryJobs=true&pgc=AB4AAYEAHgAAAAAAAAAAAAAAAfkQ90AAUgEBAQgW%2Fr3vuIzCm5wwBSiI3WKjWOqbueSQvnI%2BGizAAsjV8NiAL80nAjkvw3vucgztbs4IIrkoqerQ462C14jLJVNRIV0ihlakU7p20hMXIG4AAA%3D%3D

Third Page-Third page-— https://www.glassdoor.com/Job/new-york-python-jobs-SRCH_IL.0,8_IC1132348_KO9,15_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAfkQ90AAdwEBAQtEzo8VunEQLF8uBoWr%2BRnCsnMFj0JNOLbRUXIkLkFAzjjZlKDW1axVwiTVV%2BbXo8%2BX471WNF8IEWPMdAwCPhbzQe1T1HHMEVPYFwQLM8h1NnGMDPcEwo7tpQ7XL65R7DMDR26n0NhBU7lFGCODAwxNTsJRAAA%3D

As you can see there is no common pattern in the above URLs apart from this part — https://www.glassdoor.com/Job/new-york-python-jobs-SRCH_IL.0,8_IC1132348_KO9,15. But this is not helpful if you want to scrape other pages. So, the only solution is to use the .click() method provided by the Selenium API. Using .click() method we are going to click the next button by scrolling down.

So, this is how you are going to scroll down the page of any Glassdoor page.

With .find_element_by_xpath() we are finding the column where all the jobs are stored. Once you scroll down this element you have to find the button and click it.

You have to find it using the same method of selenium .find_element_by_xpath(). And finally, you have to use .click() method to click it. This will take you to the next page.

Now, you have to use it in a loop to extract all the jobs from the particular location. I know it is a bit lengthy process, but unfortunately, this is the only way to scrape Glassdoor.

But while scraping Glassdoor might limit your search and restrict your IP. In this case, you have to use a Web Scraping API. Let’s see how you can avoid getting blocked with a Web Scraper API like Scrapingdog.

You can use Scrapingdog’s web scraping API to avoid getting blocked while scraping Glassdoor at scale.

You have to sign up from here for the free account to start using it. It will take just 10 seconds to get you started with Scrapingdog. In a free account, Scrapingdog offers 1000 free API calls.

Once you sign up, you will be redirected to your dashboard. The dashboard will look somewhat like this.

You have to use your own API key.

Now, you can paste your target Glassdoor target page link to the left and then select JS Rendering as No. After this click on Copy Code from the right. Now use this API in your Python script to scrape Glassdoor.

You will notice the code will remain somewhat the same as above. We just have to change one thing and that is our target URL.

As you can see we have replaced the target URL of Glassdoor with the API URL of Scrapingdog and we no longer require Selenium to scrape Glassdoor. You have to use your own API Key in order to successfully run this script.

With this script, you will be able to scrape Glassdoor with a lightning-fast speed that too without getting blocked.

In this post, we learned to scrape Glassdoor and store the data in a CSV file. Such data can also help companies refine their hiring strategy and make better talent decisions. We later discovered a way to scrape all the pages for any given location. Now, you can create your logic and scrape Glassdoor but this was a pretty straightforward way to scrape it.
I have more tutorials on Scraping Indeed Job Portal with Python, Scraping LinkedIn Jobs using Python & Scraping Google Jobs using Nodejs. You can check them out too!!

Of course, I would recommend a Web Scraping API if you are planning to scrape it at scale. With a normal script without proxy rotation, you will be blocked in no time and your data pipeline will be stuck. For scraping millions of such postings you can always use Scrapingdog.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on your social media.

Scrape the web without the hassle of getting blocked Try for Free Contact sales