htcap, our super powerful web crawler/security scanner, has been updated with lots of cool features, such as:
- heuristic engine to detect duplicated contents
- the ability to crawl the DOM even if no ajax requests are triggered
- improvements on the synchronous DOM crawler
- url deduplication based on the request pattern
Check’em out on github!