So I’m pretty much making this up as we go along since I have no data to publicly share, nor do I want to use Contoso. Therefore, let’s follow the journey of Rexco! We’ll go into Rexco more later.
Today’s interest du jour is “Webscraping”. I guess this is the idea of just trawling the internet and pulling together data that’s out there for us to consume. Luckily, I don’t need to be a rocket scientist. I just need to use Power BI Desktop and –
- Get Data -> Web
- Use the website I want and connect anonymously
- See which suggested table works or Add Table Using Examples
- Through the magic of Power Query, figure out the data that comes across and transform them accordingly
For this demo, let’s use discogs.com (https://www.discogs.com/sell/list?q=the+smiths&page=1) and try to gather data on The Smiths! Mind you I’m sharing my learnings as we go along, and any helpful tutorial links I come across.
Tips: Make sure to get the url that gives the structure for the pages (try going to the next page and then going back to the first page). Also check your search parameters! Having ‘the’ in the search query as above muddies it up, and maybe use any advanced search parameters to really hone in on your search results. Do this BEFORE transforming in Power Query.
Test out scraping a few pages first! I put in 100 pages and it took 15-20 minutes to load… But check it out – we now have data that we can visualize!
Tutorials:
