问题在于浏览器通常会执行javascript,并且会生成更新的DOM。除非您可以分析JavaScript或拦截其使用的数据,否则您将需要像浏览器一样执行代码。在过去,我遇到了同样的问题,我利用selenium和PhantomJS渲染页面。呈现页面后,我将使用WebDriver客户端浏览DOM并检索所需的内容,然后发布AJAX。
从高层次上讲,这些步骤是:
这是phantomjs网络驱动程序的示例用法:
var options = new PhantomJSOptions();
options.AddAdditionalCapability("IsJavaScriptEnabled",true);
var driver = new RemoteWebDriver( new URI(Configuration.SeleniumServerHub),
options.ToCapabilities(),
TimeSpan.FromSeconds(3)
);
driver.Url = "http://www.regulations.gov/#!documentDetail;D=APHIS-2013-0013-0083";
driver.Navigate();
//the driver can Now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement = driver.FindElementById("some-id");
有关硒,phantomjs和webdriver的更多信息,可以在以下链接中找到:
http://docs.seleniumhq.org/projects/webdriver/
似乎有一个适用于phantomjs的nuget程序包,因此您不需要集线器(我使用集群以这种方式进行大量报废):
安装网络驱动程序:
Install-Package Selenium.WebDriver
安装嵌入式exe:
Install-Package phantomjs.exe
更新的代码:
var driver = new PhantomJSDriver();
driver.Url = "http://www.regulations.gov/#!documentDetail;D=APHIS-2013-0013-0083";
driver.Navigate();
//the driver can Now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement = driver.FindElementById("some-id");