C# Web Scraping Login


HttpClient scrape data from website with login c# - Stack ...

HttpClient scrape data from website with login c# – Stack …

There are many ways to perform a login to a website, and that depends on the authentication method used by the specific site (Forms authentication, Basic authentication, Windows authetication etc. ). Usually websites use the FormsAuthentication.
To perform a login in a standard FormsAuthentication website using the HttpClient, you need to set the CookieContainer, because authentication data will be set on cookies.
In your specific example, the login form makes a POST to any of the page in HTTPS, i used as an example. This is the code to make the request using HttpClient:
var baseAddress = new Uri(“);
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer})
using (var client = new HttpClient(handler) { BaseAddress = baseAddress})
//usually i make a standard request without authentication, eg: to the home page.
//by doing this request you store some initial cookie values, that might be used in the subsequent login request and checked by the server
var homePageResult = tAsync(“/”);
var content = new FormUrlEncodedContent(new[]
//the name of the form values must be the name of tags of the login form, in this case the tag is
new KeyValuePair(“username”, “username”),
new KeyValuePair(“password”, “password”), });
var loginResult = Async(“/cgi-bin/WebObjects/”, content);
loginResult. EnsureSuccessStatusCode();
//make the subsequent web requests using the same HttpClient object}
However, many websites uses some javascript loaded form values or even more some captcha controls, and obviously this solution will not work. That might be done as said with a WebBrowser control (by automating the user input on form fields and then the login button click, this link has an example:).
As a general rule to inspect how works the login on your desidered website, use Fiddler: when you click the login button on your website, watch Fiddler and find the login request (usually it’s the first request just after you click the “Login” button, and usually is a POST request).
Then inspect the request data (select the request and go to the “Inspectors” – “TextView” tab) and try to replicate the request on your code.
On the left pane there are all the requests intercepted by Fiddler, on the right pane there are the request and response inspectors (on top there are request inspectors, on bottom there are the response inspectors)
Same code with old WebRequest class:
HttpWebRequest request = (HttpWebRequest)(“);
okieContainer = cookieContainer;
//set the user agent and accept header values, to simulate a real web browser
erAgent = “Mozilla/5. 0 (Windows NT 10. 0; WOW64) AppleWebKit/537. 36 (KHTML, like Gecko) Chrome/45. 0. 2454. 101 Safari/537. 36”;
= “text/html, application/xhtml+xml, application/xml;q=0. 9, image/webp, */*;q=0. 8”;
tomaticDecompression = flate |;
Console. WriteLine(“FIRST RESPONSE”);
Console. WriteLine();
using (WebResponse response = tResponse())
using (StreamReader sr = new StreamReader(tResponseStream()))
Console. WriteLine(adToEnd());}}
request = (HttpWebRequest)(“);
//set the cookie container object
//set method POST and content type application/x-www-form-urlencoded
= “POST”;
ntentType = “application/x-www-form-urlencoded”;
//insert your username and password
string data = (“username={0}&password={1}”, “username”, “password”);
byte[] bytes = (data);
ntentLength =;
using (Stream dataStream = tRequestStream())
(bytes, 0, );
Console. WriteLine(“LOGIN RESPONSE”);
//request = (HttpWebRequest)(“INTERNAL PROTECTED PAGE ADDRESS”);
//After a successful login, you must use the same cookie container for all request
//okieContainer = cookieContainer;
c# logic to login to a website, enter value and scrap the data.

c# logic to login to a website, enter value and scrap the data.

Remove From My Forums
HI EXperts,
I need to write a framework or code to perform below tasks using c# code.
1. ) Firstly from code using logic i need to connect to a website.
2. ) Enter UID and PWD to login.
3. ) Navigate to a page, enter search value and upon submit, extract the data and send the response back.
NOte:- This should be like a silent backend work, should not pop up any chrome or browser window.
I am completely new to this, can any one guide me on the above, also please share any valuable links if have.
This is needed very urgently.
Thanks in Advance.
Marked as answer by
Sunday, July 21, 2019 4:52 AM
Login to Website Programmatically Using C# Web Scraping

Login to Website Programmatically Using C# Web Scraping

Below is C# example of programmatically login to login page
Below is HTML code of Login form:
In this code you can notice there is ID for email input box that is id=”loginEmail” and password input box that is id=”loginPass”, so by taking this ID we will use below two method of webBrowser control and fill the value of each input box using following code:
After the value filled to Email and Password input box we will just call click event of submit button which is named as Sign In:
So this is very basic example how you can login to website programatically when you need to access data that is available after login to website. This is very simple way in which you can work with Web Browser control but there are some other way as well using which you can do same thing.
You can request for a FREE sample web data extraction project for a website from where you want to extract data. We will come up with sample scraped data as per your requirements and if that matched with your requirement than pay.

Frequently Asked Questions about c# web scraping login

About the author


If you 're a SEO / IM geek like us then you'll love our updates and our website. Follow us for the latest news in the world of web automation tools & proxy servers!

By proxyreview

Recent Posts

Useful Tools