Machine Learning for Smart Automation
July 15th, 2020
Comment to help prioritize content.
Comments not loaded in development.
Data Engineering focuses storing and moving data between processes, just like house water pipelines transport water from streets to showers and back. If you're tackling complex problems and you're not using appropriate tooling, your risk failure increases. Some processes may be senstive to data quality while others may be sensitive to speed, volume, or other properties. In this post you will learn the top tools and techniques for different use cases, and trade-offs between them from a business perspective.
In this free and short course we show how to build a crawler that detects broken links in a website. We designed it to help us find broken links in our own DELTA LAB website, and figured other people will benefit from it as well, so we decided to make into a small free course. The task at hand is to go through every link in a website and keep track of those that come back as errors when visited, while keeping control of not going into external sites in a way that we end up crawling the whole internet.