How To Use A Proxy with Axios & Nodejs

TL;DR

Shows Axios + Node.js proxy setup for scraping.
Configure a proxy in axios config; add username / password for private proxies.
Rotate a list of proxies per request to cut blocks.
For scale, use Scrapingdog proxies (10M+ pool) with a 1k-credit free trial.

Nodejs offers multiple HTTP client options and Axios is one of them. Axios has by far the best community support compared to other libraries like Fetch, Unirest, etc. It provides a stable API through which you can make XMLHttpRequests. Using a middleware-like function you can customize the pattern of request. It supports GET, POST, DELETE, PUT, etc methods. Axios is built on top of Promises, making it easy to work with asynchronous code and response handling.

Axios is a very powerful tool when it comes to web scraping. In this article, we will understand how we can scrape a website with the combination of proxy and Axios.

Requirements

For this tutorial, I hope you have already installed the latest version of Nodejs on your machine. If not then you can download it from here.

Now, let’s set up our coding playground. First, we will create a folder to keep our Nodejs script.

1mkdir axiostut
2cd axiostut
3npm init

Then we have to install Axios inside this folder.

1npm i axios

Once this is done we can create a JS file in which we will learn how Axios works with proxy. I am naming the file as axiostut.js.

How Axios works?

In this section, I will explain step by step how Axios works.

How to make a Basic GET Request

Let’s first start with the basic HTTP GET request. For this entire tutorial, we are going to use this website for scraping.

1const axios = require('axios');
2 
3async function scraping() {
4    try {
5        // Make a GET request to https://httpbin.org/ip
6        const response = await axios.get('https://httpbin.org/ip');
7 
8        // Check if the request was successful (status code 200)
9        if (response.status === 200) {
10            // Parse JSON response
11            const ipAddress = response.data.origin;
12 
13            // Log or use the IP address
14            console.log('Your IP address is:', ipAddress);
15        } else {
16            console.error(`Error: ${response.status} - ${response.statusText}`);
17        }
18    } catch (error) {
19        // Handle errors
20        console.error('Error:', error.message);
21    }
22}
23 
24// Call the function to get the IP address
25scraping();

Let me explain the code in brief.

We use the axios.get method to make a GET request to https://httpbin.org/ip.
The response is checked for a successful status code 200.
If the request is successful, the IP address is extracted from the JSON response and logged.
If there’s an error during the request, it is caught and logged.

How to use a proxy with Axios?

For this example, we can use any public proxy. Take any free proxy, from this list.

1const axios = require('axios');
2 
3let config= {
4    method: 'get',
5    url: 'https://httpbin.org/ip',
6    proxy: {
7      host: '20.157.194.61',
8      port: 80
9    }
10  };
11 
12async function getIpAddress() {
13    try {
14        // Make a GET request to https://httpbin.org/ip
15        const response = await axios(config);
16 
17        // Check if the request was successful (status code 200)
18        if (response.status === 200) {
19            // Parse JSON response
20            const ipAddress = response.data.origin;
21 
22            // Log or use the IP address
23            console.log('Your IP address is:', ipAddress);
24        } else {
25            console.error(`Error: ${response.status} - ${response.statusText}`);
26        }
27    } catch (error) {
28        // Handle errors
29        console.error('Error:', error.message);
30    }
31}
32 
33// Call the function to get the IP address
34getIpAddress();

In the above code, config is an object that contains configuration options for making an HTTP request using the Axios library. Here’s a breakdown of the properties in the config object.

method: 'get' -Specifies the HTTP method for the request. In this case, it is set to ‘get,’ indicating an HTTP GET request.
url: 'https://httpbin.org/ip' -Specifies the target URL for the HTTP request. The request will be made to the ‘https://httpbin.org/ip‘ endpoint, which returns information about the requester’s IP address.
proxy: { host: '20.157.194.61', port: 80 } -Specifies a proxy configuration for the request. The proxy property is an object that includes the host (IP address) and port of the proxy server. This configuration is optional and is used here to demonstrate how to request a proxy.

How to use a password-protected proxy with Axios?

To use the proxy which is protected by a password you can simply pass the username and the password to the config object.

1const axios = require('axios');
2 
3let config= {
4    method: 'get',
5    url: 'https://httpbin.org/ip',
6    proxy: {
7      host: '94.103.159.29',
8      port: 8080,
9      auth: {
10        username: 'Your-Username',
11        password: 'Your-Password',
12    }
13    }
14  };
15 
16async function getIpAddress() {
17    try {
18        // Make a GET request to https://httpbin.org/ip
19        const response = await axios(config);
20 
21        // Check if the request was successful (status code 200)
22        if (response.status === 200) {
23            // Parse JSON response
24            const ipAddress = response.data.origin;
25 
26            // Log or use the IP address
27            console.log('Your IP address is:', ipAddress);
28        } else {
29            console.error(`Error: ${response.status} - ${response.statusText}`);
30        }
31    } catch (error) {
32        // Handle errors
33        console.error('Error:', error.message);
34    }
35}
36 
37// Call the function to get the IP address
38getIpAddress();

We have just passed an auth object with the properties username and password. Once you run this code you will get this output.

1Your IP address is: 94.103.159.29

How to rotate proxies with Axios?

Many crawler-sensitive websites like Amazon, Walmart, LinkedIn, etc will block you if you keep scraping these websites with just a single IP. Headers are important too but changing the IPs on every request is as critically important.

1const axios = require('axios');
2 
3let proxy_arr=[
4{host: '69.51.19.191',
5port: 8080,
6auth: {
7  username: 'Your-Username',
8  password: 'Your-Password',
9}},
10{host: '69.51.19.193',
11port: 8080,
12auth: {
13  username: 'Your-Username',
14  password: 'Your-Password',
15}},
16{host: '69.51.19.195',
17port: 8080,
18auth: {
19  username: 'Your-Username',
20  password: 'Your-Password',
21}},
22{host: '69.51.19.207',
23port: 8080,
24auth: {
25  username: 'Your-Username',
26  password: 'Your-Password',
27}},
28{host: '69.51.19.220',
29port: 8080,
30auth: {
31  username: 'Your-Username',
32  password: 'Your-Password',
33}}]
34 
35let config= {
36    method: 'get',
37    url: 'https://httpbin.org/ip',
38    proxy:proxy_arr[Math.floor(Math.random() * 5)]
39  };
40 
41 
42async function getIpAddress() {
43    try {
44        // Make a GET request to https://httpbin.org/ip
45        const response = await axios(config);
46 
47        // Check if the request was successful (status code 200)
48        if (response.status === 200) {
49            // Parse JSON response
50            const ipAddress = response.data.origin;
51 
52            // Log or use the IP address
53            console.log('Your IP address is:', ipAddress);
54        } else {
55            console.error(`Error: ${response.status} - ${response.statusText}`);
56        }
57    } catch (error) {
58        // Handle errors
59        console.error('Error:', error.message);
60    }
61}
62 
63// Call the function to get the IP address
64getIpAddress();

In the above code, I have created a proxy array that contains five proxy objects. Using **Math.random** function we are passing these proxies on a random basis to the config object.

Now, every request will go through a different proxy and the chances of getting your scraper getting blocked will be very low.

How to use Scrapingdog Proxy with Axios?

For small-scale scraping above methods are fine and will do the job. But if you want to scrape millions of pages then you have to go with some premium web scraping API that can take this proxy management on an autopilot mode. You simply have to send a GET request and the API will handle all these headaches for you.

In this section, I will show you how Scrapingdog proxy can be used for scraping purposes. First of all, you have to sign up for the free pack from here.

The free pack will provide you with a generous 1000 credits and that is enough for you to test the service before proceeding with the paid plan. Once you sign up you will get an API key on your dashboard.

You have to pass this API key in the below code as your proxy password. You can read more about the proxies from the documentation.

1const axios = require('axios');
2 
3 
4 
5let config= {
6    method: 'get',
7    url: 'http://httpbin.org/ip',
8    proxy:{
9    host: 'proxy.scrapingdog.com',
10    port: 8081,
11    auth: {
12      username: 'scrapingdog',
13      password: 'Your-API-key'
14    }
15  }
16}
17 
18 
19async function getIpAddress() {
20    try {
21        // Make a GET request to https://httpbin.org/ip
22        const response = await axios(config);
23 
24        // Check if the request was successful (status code 200)
25        if (response.status === 200) {
26            // Parse JSON response
27            const ipAddress = response.data.origin;
28 
29            // Log or use the IP address
30            console.log('Your IP address is:', ipAddress);
31        } else {
32            console.error(`Error: ${response.status} - ${response.statusText}`);
33        }
34    } catch (error) {
35        // Handle errors
36        console.error('Error:', error.message);
37    }
38}
39 
40// Call the function to get the IP address
41getIpAddress();

Scrapingdog has a proxy pool of more than 10M proxies which makes large-scale scraping seamless. Once you run this code after placing your API key, every run will print a new IP on the console.

It is a very economical solution for large-scale scraping. You just have to focus on data collection and the rest will be managed by Scrapingdog.

Key Takeaways:

Axios can be configured to send HTTP requests through a proxy server in Node.js.
Proxy configuration includes specifying host, port, and authentication credentials when required.
Using proxies helps bypass IP-based restrictions and geo-blocking.
Rotating proxies improves reliability when scraping or sending large volumes of requests.
Proper error handling is essential to manage timeouts, connection failures, and blocked requests.

Conclusion

Axios is a very popular choice when it comes to web scraping and in this article, we saw how Axios can be used with proxies in different scenarios. We also understood the importance of proxy rotation and how APIs like Scrapingdog can take all the hassle of proxy rotation on autopilot mode.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on social media.

Additional Resources

And there’s the list! At this point, you should feel comfortable writing your first web scraper to gather data from any website. Here are a few additional resources that you may find helpful during your web scraping journey:

Web Scraping with Scrapingdog

Scrape the web without the hassle of getting blocked Try for Free Contact sales