Blog > Post

The Primary Use of Regular Expression in Data Processing

Monday, August 01, 2016

Previously, I introduced the usage, the pros and cons of Regular Expressions in extracting HTML content.


Using Regular Expression to Match HTML


Advanced Text - Recommendations to Handle HTML With Regular Expression


Regular expressions can be used:


  1. Data validation. (A string complies with email rule or is a phone number, etc.)


  1. Replace text. (Replace the characters we don’t want with other characters.)


  1. Extract substrings from a string based on a pattern match. (Search specific text in the document.)


In data processing, we mainly use Regular Expressions to replace text and extract substrings.


Examples: Remove all non-alphanumeric characters other than invalid characters (@-.) from the string, then return a string.


Sample code (C#)


using System;

using System.Text.RegularExpressions;

void Main()




static string CleanInput(string strIn)


   // Replace invalid characters with empty strings.

   return Regex.Replace(strIn, @"[^\w\.@-]", "");



The output







We can see the characters (* # $ %) which we believe to be invalid characters are replaced by an empty string, which removed these invalid characters


Examples: get the domain name of URL


Sample code (C#)


using System;

using System.Text.RegularExpressions;

void Main()


    string url = "http://www.octoparse.com/pricing/";

    Regex r = new Regex(@"((\w)+\.)+\w+",RegexOptions.IgnorePatternWhitespace);




The output





From the code above, we see that Regular Expression can help you extract strings with characteristics of domain names from the URL.

For hundreds of thousands of URL like this, you can extract their domain names with only a regular expression.




Author: The Octoparse Team




Download Octoparse Today



For more information about Octoparse, please click here.

Sign up today.



Author's Picks


About Octoparse

Octoparse 6.0 is Now Available

What A Price Monitor Can Help you?

Examples of Businesses Who Use Data Scraping

Collect Data from Facebook

Collect Data from Craigslist




Laden Sie Octoparse herunter, um mit Web-Scraping zu beginnen, oder kontaktieren Sie uns
für die Fragen über Web Scraping!

Kontaktieren Sie uns Herunterladen
Diese Website verwendet Cookies um Ihnen ein besseres Internet-Erlebnis zu ermöglichen. Lesen Sie wie wir Cookies verwenden und Sie können sie kontrollieren, indem Sie auf Cookie-Einstellungen klicken. Wenn Sie die Website weiter nutzen, akzeptieren Sie unsere Verwendung von Cookies.
Akzeptieren Ablehnen