Using Gmail as a Processing Queue with VB.NET and IMAP

18 May 2012

Whilst at work I hit a problem where we had a legacy website containing a contact form would be used for requesting free samples of a product. The business had a need to streamline this because the contact form would submit over email and then somebody would have to manually process the email into a spreadsheet that is manually processed by another team.

The problem with the website is that it cannot be easily modified and a quick solution was needed. Therefore I came up with the following outline solution:

  • Change the form to submit to a Gmail/Google Apps email address.
  • Create three labels in Gmail of:
    • Not Yet Processed
    • Processed Successfully
    • Processing Failed
  • Set up a filter so all emails from the contact form get the "Not Yet Processed" label
  • Turn on IMAP in Gmail.
  • Create a console application to:
    • Read all emails from Gmail via IMAP with the label of "Not Yet Processed"
    • For each email validate and add to a database
    • If added to the database fine then change the email label to "Processed Successfully"
    • If failed adding to the database then change the email label to "Processed Failed" and send an alert email

This means that the website can be tweaked to send to a different email address and then this application can be scheduled to run every so often and batch process the requests for samples and only the one's that can't be processed automatically are alerted to be dealt with manually

For IMAP I used MailSystem.NET

Some key things I learned when creating this:

  • Gmail sometimes respond with a System Error, this needs handling so I added in a second retry after waiting a bit.
  • When moving between labels, if you move whilst looping the messages then the message numbers change at Gmail so you will end up with synchronisation problems. I changed the application to store what message to move and then move them all after looping the emails.

The Basic Code

Imports ActiveUp.Net.Mail

Module GmailImapReaderAndProcessor

    Sub Main()

        Try

            ' This stores where to move emails to (String being the label name to move to)
            Dim toMoveFromInbox As New SortedList(Of Integer, String)

            Dim imapClient As New Imap4Client()

            imapClient.ConnectSsl(AppSettings("Imap4.Server"),
                                  CInt(AppSettings("Imap4.Port")))

            TryToLoginToImap(imapClient)

            Log("Authorised. Loading list of mailboxes.")

            imapClient.LoadMailboxes()

            Dim notYetProcessedBox As Mailbox = imapClient.SelectMailbox("Not Yet Processed")

            Log(String.Format("Found {0} emails in label.", notYetProcessedBox.MessageCount.ToString))

            Try

                ' Loop through all the emails in the mailbox
                For emailNumber As Integer = 1 To notYetProcessedBox.MessageCount

                    Log(String.Format("Processing email number {0}...", emailNumber.ToString))

                    ' Fetch the message and print the details out to the log for information
                    Dim msg As Message = notYetProcessedBox.Fetch.MessageObject(emailNumber)
                    Log(String.Format("Email Subject: {0}.", msg.Subject))
                    Log(String.Format("Email From: {0}.", msg.From.Email))
                    Log(String.Format("Email Received At: {0}.", msg.Date.ToString))

                    Dim success As Boolean = processEmail(msg)

                    If success Then

                        LogFile.Log("Storing instruction to move email to label ""Processed Successfully""...")
                        toMoveFromInbox.Add(emailNumber, "Processed Successfully")

                        ' Mark as read to clean up in Gmail
                        Dim readFlagCollection As New FlagCollection()
                        readFlagCollection.Add("Seen")
                        notYetProcessedBox.AddFlags(emailNumber, readFlagCollection)

                        LogFile.Log("Moved email.")

                    Else

                        LogFile.Log("Storing instruction to move email to label ""Processing Failed""...")
                        toMoveFromInbox.Add(emailNumber, "Processing Failed")

                    End If

                Next

                Log("Finished.")

            Catch ex As Exception

                LogAndEmail(ex.ToString)

            End Try


            Log("Starting to process emails that need moving...")

            ' Only if we have emails to move
            If toMoveFromInbox IsNot Nothing AndAlso toMoveFromInbox.Count > 0 Then

                Log(String.Format("Found {0} emails to move.", toMoveFromInbox.Count))

                ' Loop each instruction and try to do each one - backwards because the email number changes when it is moved
                For i As Integer = toMoveFromInbox.Count - 1 To 0 Step -1

                    Dim currentNumber As Integer = toMoveFromInbox.Keys(i)

                    Log(String.Format("Moving email {0} to ""{1}"".", currentNumber, toMoveFromInbox(currentNumber)))

                    notYetProcessedBox.MoveMessage(currentNumber, toMoveFromInbox(currentNumber))

                Next

            End If

            Log("Finished moving any emails that needed moving.")


            ' If connected then disconnect
            If imapClient IsNot Nothing AndAlso imapClient.IsConnected Then
                imapClient.Disconnect()
            End If

        Catch ex As Exception

            LogAndEmail(msg)

        End Try

    End Sub

    Private Sub TryToLoginToImap(ByRef imap As Imap4Client,
                                 Optional tryAgain As Boolean = True)

        Try

            imap.Login(AppSettings("Imap4.Username"), AppSettings("Imap4.Password"))

        Catch imapException As Imap4Exception

            Log(imapException.ToString)

            ' This error occasionally occurs at Google. So wait and try again in a little bit before a big exception.
            If imapException.Message.Contains("System error (Failure)") And tryAgain Then

                Dim waitForInMinutes As TimeSpan = TimeSpan.FromMinutes(30D)

                Log("Gmail gave a system error so am going to wait 30 minutes before trying again.")

                Threading.Thread.Sleep(waitForInMinutes)

                TryToLoginToImap(imap, False)

            Else

                Log("Gmail gave a system error again so am re-throwing to be handled like a big exception.")

                Throw

            End If

        End Try

    End Sub

End Module
vb.net   imap  
Posted by makit
Last revised 18 May 2012 01:28 PM

makit / Martyn Kilbryde

Professional software developer, ponderer and eccentric.

flickr

Github

LinkedIn

Stack Overflow

Twitter