How to parse emails from HTML in Python

In this tutorial we are going to get an idea of how to parse emails from HTML using Python.

2017-02-01 dataparsingpythontutorialTutorials

In this tutorial we are going to get an idea of how to parse emails from HTML using Python.

Python is a scripting language easy to get started and is perfect for tasks like parsing emails.

So let’s elaborate an approach of how parsing works:

Initialize a queue of URLs. The first item will be the initial URL.
Initialize a set of already visited URL to avoid repetitions.
Start parsing the current URL from the queue.
Add the URL to processed URLs.
Extract the whole HTML, search for an email pattern using a regex.
If one or multiple emails were found, write to CSV.
Loop through tags found.

Check if URL is relative or absolute.

Check if URL is already in the processed URLs set. If not, add to the processing queue

Repeat from step 3.

Before launching the script don’t forget to install proper libraries.

Using command line do:

pip install requests pip install urlparse pip install csv pip install beautifulsoup4pip install requests pip install urlparse pip install csv pip install beautifulsoup4

Once you have the libraries installed, you’ll be able to check the script.

from bs4 import BeautifulSoup import requests import requests.exceptions from urllib.parse import urlparse, urlsplit from collections import deque import re import csv

initialize CSV writer and filename

cw = csv.writer(open(“Singa.csv”, ‘a’, newline=”), delimiter=’,’)

a queue of urls, start

new_urls = deque([‘https://foundersgrid.com/50-singapore-startups/’])

a set of urls that we have already crawled

processed_urls = set()

a set of crawled emails

emails = set()

process urls one by one until we exhaust the queue

while len(new_urls):

extract the last one from queue

url = new_urls.popleft()

mark as visited by adding to processed URLs

processed_urls.add(url)

break down the extract the base url to resolve relative links

parts = urlsplit(url) base_url = “{0.scheme}://{0.netloc}“.format(parts) path = url[:url.rfind(’/’)+1] if ’/’ in parts.path else url

get url’s content

handle exception if any

try: response = requests.get(url) except (requests.exceptions.MissingSchema, requests.exceptions.ConnectionError):

skip pages with errors

continue

extract all email addresses and add them into the resulting set

new_emails = set(re.findall(r”[a-z0-9.-+]+@[a-z0-9.-+]+.[a-z]+”, response.text, re.I)) emails.update(new_emails) print(new_emails)

write to CSV the new mails.

alternatively you can write the emails set to CSV after parsing

for em in new_emails: cw.writerow([em,])

create a beautiful soup object as representation of the html page

soup = BeautifulSoup(response.text, “html.parser”)

walk through anchors

for anchor in soup.find_all(“a”):

extract link url from the anchor

link = anchor.attrs[“href”] if “href” in anchor.attrs else ”

resolve relative links

if link.startswith(’/’): link = base_url + link elif not link.startswith(‘http’): link = path + link

add the new url to the queue if it was not enqueued nor processed yet

if link and not link in new_urls and not link in processed_urls: new_urls.append(link)from bs4 import BeautifulSoup import requests import requests.exceptions from urllib.parse import urlparse, urlsplit from collections import deque import re import csv

initialize CSV writer and filename

cw = csv.writer(open(“Singa.csv”, ‘a’, newline=”), delimiter=’,’)

a queue of urls, start

new_urls = deque([‘https://foundersgrid.com/50-singapore-startups/’])

a set of urls that we have already crawled

processed_urls = set()

a set of crawled emails

emails = set()

process urls one by one until we exhaust the queue

while len(new_urls):

extract the last one from queue

url = new_urls.popleft()

mark as visited by adding to processed URLs

processed_urls.add(url)

break down the extract the base url to resolve relative links

parts = urlsplit(url) base_url = “{0.scheme}://{0.netloc}“.format(parts) path = url[:url.rfind(’/’)+1] if ’/’ in parts.path else url

get url’s content

handle exception if any

try: response = requests.get(url) except (requests.exceptions.MissingSchema, requests.exceptions.ConnectionError):

skip pages with errors

continue

extract all email addresses and add them into the resulting set

new_emails = set(re.findall(r”[a-z0-9.-+]+@[a-z0-9.-+]+.[a-z]+”, response.text, re.I)) emails.update(new_emails) print(new_emails)

write to CSV the new mails.

alternatively you can write the emails set to CSV after parsing

for em in new_emails: cw.writerow([em,])

create a beautiful soup object as representation of the html page

soup = BeautifulSoup(response.text, “html.parser”)

walk through anchors

for anchor in soup.find_all(“a”):

extract link url from the anchor

link = anchor.attrs[“href”] if “href” in anchor.attrs else ”

resolve relative links

if link.startswith(’/’): link = base_url + link elif not link.startswith(‘http’): link = path + link

add the new url to the queue if it was not enqueued nor processed yet

if link and not link in new_urls and not link in processed_urls: new_urls.append(link)

As you can see, parsing emails in Python is rather a simple task.

If you have any questions on this tutorial, you can contact us hello@cyberwhale.tech

Also, if you need assistance with data collection or any other digital service, please let us know.

Don’t forget to share the tutorial and visit us at cyberwhale.tech

PS. In the next tutorial we will discuss how to parse dynamic HTML content using Python.

By admin data parsing python tutorial

AI Strategy & Consulting

Build a future-ready AI strategy with trusted guidance

initialize CSV writer and filename

a queue of urls, start

a set of urls that we have already crawled

a set of crawled emails

process urls one by one until we exhaust the queue

extract the last one from queue

mark as visited by adding to processed URLs

break down the extract the base url to resolve relative links

get url’s content

handle exception if any

skip pages with errors

extract all email addresses and add them into the resulting set

write to CSV the new mails.

alternatively you can write the emails set to CSV after parsing

create a beautiful soup object as representation of the html page

walk through anchors

extract link url from the anchor

resolve relative links

add the new url to the queue if it was not enqueued nor processed yet

initialize CSV writer and filename

a queue of urls, start

a set of urls that we have already crawled

a set of crawled emails

process urls one by one until we exhaust the queue

extract the last one from queue

mark as visited by adding to processed URLs

break down the extract the base url to resolve relative links

get url’s content

handle exception if any

skip pages with errors

extract all email addresses and add them into the resulting set

write to CSV the new mails.

alternatively you can write the emails set to CSV after parsing

create a beautiful soup object as representation of the html page

walk through anchors

extract link url from the anchor

resolve relative links

add the new url to the queue if it was not enqueued nor processed yet

Recent Posts

Archive

Tags

AI Strategy & Consulting