One key features of Darcy Ripper make it essential to it users, it has HTTP / HTTPS Support.
Maybe you are wondering for what do you need http support at a web crawler?
First of all, I would like to introduce you the basic of HTTP and HTTPS:
Http can uphold the user requesting a specific document to be sent just assuming that it has been redesigned after a certain date and time. This might be utilized if the user has recently recovered a duplicate of an index by that name from the server, yet needs to verify whether it has been overhauled since then. The server reacts either with the upgraded index, with a message to say the index has not been modified, or with a message that the index no more drawn out exists.
Https (Hypertext Transfer Protocol over Secure Socket Layer or Http over SSL) is a Web methodology advanced by Netscape and incorporated with its browser that encrypts and decrypts client page solicits and additionally the pages that are returned by the Web server.
I guess you already understood for what is in need the HTTP support at Darcy Ripper. The Http Support is the backend of State, Progress, Status Code, etc of our web crawler.
Hopping I made you better understand the functionality of Darcy Ripper, happy Crawling!