I'm trying to get HTML from a URL so I can strip it down using Boilerpipe. However, I keep on getting an exception. I am using the NewsAPI to get my URLs. Here is the relevant code snippet:
foreach (var article in articlesResponse.Articles)
{
string html;
string url = article.Url;
using (WebClient client = new WebClient())
{
html = client.DownloadString(url);
}
string text = CommonExtractors.DefaultExtractor.GetText(html);
System.IO.File.AppendAllText(fileName, "Title: " + article.Title + "\n");
System.IO.File.AppendAllText(fileName, "Author: " + article.Author + "\n");
System.IO.File.AppendAllText(fileName, "Description: " + article.Description + "\n");
System.IO.File.AppendAllText(fileName, "URL: " + article.Url + "\n");
System.IO.File.AppendAllText(fileName, "Published at: " + article.PublishedAt + "\n");
System.IO.File.AppendAllText(fileName, "Text: " + text + "\n\n");
}
And this is the details from the exception:
System.Net.WebException
HResult=0x80131509
Message=The remote server returned an error: (404) Not Found.
Source=System
StackTrace:
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at System.Net.WebClient.DownloadString(String address)
at newsapi_take_two.Program.Main(String[] args) in ...\source\repos\newsapi console\newsapi take two\Program.cs:line 53
User contributions licensed under CC BY-SA 3.0