Exception when getting HTML from URL

0

I'm trying to get HTML from a URL so I can strip it down using Boilerpipe. However, I keep on getting an exception. I am using the NewsAPI to get my URLs. Here is the relevant code snippet:

foreach (var article in articlesResponse.Articles)
{
    string html;
    string url = article.Url;

    using (WebClient client = new WebClient())
    {
        html = client.DownloadString(url);
    }

    string text = CommonExtractors.DefaultExtractor.GetText(html);
    System.IO.File.AppendAllText(fileName, "Title: " + article.Title + "\n");
    System.IO.File.AppendAllText(fileName, "Author: " + article.Author + "\n");
    System.IO.File.AppendAllText(fileName, "Description: " + article.Description + "\n");
    System.IO.File.AppendAllText(fileName, "URL: " + article.Url + "\n");
    System.IO.File.AppendAllText(fileName, "Published at: " + article.PublishedAt + "\n");
    System.IO.File.AppendAllText(fileName, "Text: " + text + "\n\n");
}

And this is the details from the exception:

System.Net.WebException
  HResult=0x80131509
  Message=The remote server returned an error: (404) Not Found.
  Source=System
  StackTrace:
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at System.Net.WebClient.DownloadString(String address)
   at newsapi_take_two.Program.Main(String[] args) in ...\source\repos\newsapi console\newsapi take two\Program.cs:line 53
c#
boilerpipe
asked on Stack Overflow May 28, 2020 by esb5415 • edited May 28, 2020 by Danny

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0