Paulo

Author Archives: Paulo

Guide to Web Scraping with JavaScript

Web scraping – also referred to as either web harvesting or web data extraction – is the action of extracting large quantities of data from various websites and saved as a file in your computer or to a database in a spreadsheet or other table format.

When we browse the web, most websites can only be viewed by using an internet browsing application (Internet Explorer, Firefox, Chrome, etc.) and do not offer an option to save a copy of the website’s data for private purposes. In this case, the only thing left to do is to manually copy and paste the data. As you may already know, going about this task manually is labor-intensive and can take you many hours or days to complete, depending on how much data you need to compile.

Web scraping simplifies the task for us by automating the process, so instead of sitting in front a computer all day and copy-pasting website data into a spreadsheet, Web Scraping scripts will do the task for us and complete it in only a fraction of the time. In this tutorial, we’re going to take learn the basics of automating and scraping the web using JavaScript. To get this done, we’re going to need Puppeteer, a Node library API which allows us to use headless Chrome. Headless Chrome is basically a method of running chrome without actually running it. In a nutshell, we’ll be writing JavaScript code that will run Chrome for us automatically.

Prerequisites

Before we begin, take a moment to make sure that you have the following technologies ready on your machine:

  • NodeJS
  • ExpressJS: infamous node framework everyone uses
  • Request: makes the HTTP calls
  • Cheerio: traverses the DOM and extracts data

Setup

The simple is pretty straightforward. After getting familiar with Node.js and all its intricacies, you can go ahead and include Express, Request, and Cheerio into your setup as the dependencies. The following lines of code will set up of the dependencies:

{

“name” : “node-web-scrape”,

“version” : “0.0.1”,

“description” : “Scrape le web.”,

“main” : “server.js”,

“author” : “authorsname”,

“dependencies” : {

“express” : “latest”,

“request” : “latest”,

“cheerio” : “latest”,

}

}

With the .json file ready to run, install the dependencies with:

npm install

Now let’s see what our setup will be making. In this guide, we’ll make a singular request to IMDB to get:

  • Movie title
  • Year of its release
  • IMDB community score

After compiling this information, we’ll save it into a .json file on your computers.

Application

Our web scraping application will be very simplistic. The things it’ll do include:

  • Launch web server
  • Visit a website on our server which will activate the web scraper
  • The web scraper will request information from the website that it wants to extract
  • The request will record the HTML of the website and save it into our server
  • We will traverse the DOM and take the requested information
  • The extracted data will be formatted into the form we need
  • Finally, the formatted information will be saved into a .json file on our computers

 

Now let’s set the logic in our server.js file:

var express = require(‘express’);

var fs = require(‘fs’);

var request = require(‘request’);

var cheerio = require(‘cheerio’);

var app = express();

app.get(‘/scrape’, function(req, res){

//Web scraping processes will be done here

})

app.listen(‘8081’)

console.log(‘Done on port 8081’);

exports = module.exports = app;

Requesting Information

Now that the application is ready to run, we’ll have to make the request to external URLS containing the information we want to scrape:

var express = require(‘express’);

var fs  = require(‘fs’);

var request = require(‘request’);

var cheerio = require(‘cheerio);

var app = express();

app.get(‘/scrape’, function(req, res){

//The website we want to scrape info from – The Wolf of Wall Street (2013)

url = ‘http://www.imdb.com/title/tt0993846/’;

 

//Structure of request

//First parameter is our URL

//Callback function needs 3 parameters, an error, response status code and the html

request(url, function(error, response, html){

//Checking to make sure no errors occur

If(!error){

//Utilize cheerio library on the returned HTML

Var $ = cheerio.load(html);

//Lastly, define the variables we want to take

var title, release, rating;

var json = {title : “”, release : “”, rating : “”};

}

})

})

app.listen(‘8081’)

console.log(‘Done on port 8081’);

exports = module.exports = app;

The request function requires two parameters (URL and a callback). For the URL, we set the link of the IMDB movie. In the callback, we’ll take three parameters (error, response and html).

Traversing the DOM

First, we’ll need the movie title. Open the IMDB website for the movie, open Developer Tools and check the movie title element. We’re looking for a unique element which will assist in locating the title of the movie (header). Search for <hl class=”header”>.

var express = require(‘express’);

var fs = require(‘fs’);

var request = requite(‘request’);

var cheerio = require(‘cheerio’);

var app = express();

 

app.get(‘/scrape, function(req, res){

url = ‘http://www.imdb.com/title/tt0993846/’;

request(url, function(error, response, html){

if(!error){

var $ = cheerio.load(html);

var title, release, rating;

var json = { title : “”, release : “”, rating : “”};

 

//Using unique header as a starting pint

$(‘.header).filter(function(){

//Store filtered data into a variable

var data = $(this);

//Title rests within the first child element of header tag

//Utilize jQuery for easy navigation

title = data.children().first().text();

//After getting the title, store it to json object

json.title = title;

})

}

})

})

 

app.listen(‘8081’)

console.log(‘Done on port 8081’);

exports = module.exports = app;

Next we’ll need the release year of the movie. We’ll need to repeat the process, this time finding a unique element in the DOM for release year (<h1> tag).

var express = require(‘express’);

var fs = require(‘fs’);

var request = require(‘request’);

var cheerio = require(‘cheerio’);

var app = express();

 

app.get(‘/scrape’, fuction(req, res){

url = ‘http://www.imdb.com/title/tt0993846/’;

request(url, function(error, response, html){

if(!error){

var $ = cheerio.load(html);

var title, release, rating;

var json = { title : “”, release : “”, rating : “”};

$(‘.header’).filter(function(){

var data = $(this);

title = data.children().first().text();

 

//Repeat the same process but for release year

//Find exact location of release year

 

release = data.children().last().children().text();

json.title = title;

//Extract and save to json object

json.release = release;

})

})

})

})

 

app.listen(‘8081’)

console.log(‘Done on port 8081’);

exports = module.exports = app;

Finally, we repeat the process for community rating. The unique class name is .star-box-giga-star.

var express = require(‘express’);

var fs = require(‘fs’);

var request = require(‘request’);

var cheerio = require(‘cheerio’);

var app = express();

 

app.get(‘/scrape’, function(req, res){

url = ‘http://www.imdb.com/title/tt0993846/’;

request(url, function(error, response, html){

if(!error){

var $ = cheerior.load(html);

 

var title, release, rating;

var json = { title : “”, release : “”, rating : “”};

 

$(‘.header’).filter(function(){

var data = $(this);

title = data.children().first().text();

release = data.children().last().children().text();

json.title = title;

json.release = release;

})

//Community rating is found in a separate section of the DOM so write

  new jQuery

$(‘.star-box-giga-star’).filter(function(){

var data = $(this);

//.star-box-giga-star class exactly where we wanted

//To get the rating, simply get the text

rating = data.text();

json.rating = rating;

})

)

})

})

 

app.listen(‘8081)

console.log(‘Done on port 8081’);

export = module.exports = app

That’s all there is to retrieving information from the website. In a nutshell, the steps mentioned above are:

  1. Fine unique element or attribute on the DOM to assist in singling out data
  2. If no unique element exists on a certain tag, then find the closest tag as your starting point
  3. If necessary, traverse the DOM

Formatting and Extracting

Not that we’ve successfully extracted the information, it’s time to format it to a project folder. Everything has been stored to a variable named json. If you’re unfamiliar with what the fs library is for, it gives access to our computer’s file system. The following code will write the files to the file system:

var express = require(‘express’);

var fs = require(‘fs’);

var request = require(‘request’);

var cheerio = require(‘cheerio);

var app = express();

app.get (‘/scrape’, function(req, res){

url = ‘http://www.imdb.com/title/tt0993846/’;

 

request(url, function(error, response, html){

if(!error){

var $ = cheerio.load(html);

var title, release, rating

var json = { title : “”, release : “”, rating : “”};

$(.header’).filter(function(){

var data = $(this);

title = data.children().first().text();

release = data.children().last().children().text();

json.title = title;

json.release = release;

})

 

$(‘.star-box-giga-star’).filter(function(){

var data = $(this);

rating = data.text();

json.rating = rating;

})

}

 

//Use default ‘fs’ library to write to the system

//Pass 3 parameters to the writeFile function

//Parameter 1: output.json – filename

//Parameter 2: JSON.stringify(json, null, 4) – data to write, JSON.stringify makes JSON easier to read

//Parameter 3: callback function – tells the status of function

 

fs.writeFile(‘output.json’, JSON.stringify(json, null, 4), function(err){

console.log(‘File successfully recorded. Check project directory for the output .json file’);

})

 

//Finally, send a message to your browser to remind you that the app has no UI

res.send(‘Check console’)

});

})

 

app.listen(‘8081)

console.log(‘Done on port 8081’);

exports = module.exports = app;

With the node, we are ready to scrape and store the extracted information. To start up the node server, open http://localhost:8081/scrape and see what happens.

CleanMyPc Review 2017

Our rating: 4.0 out of 5                            Cost: $$

Pros

  • Easy to use interface
  • Effective tool which improves boot time and overall performance
  • Comes with program installer and file shredder

Cons

  • No backup tool
  • Doesn’t provide full details of files and apps that will be uninstalled

If your computer has slowed down tremendously or just doesn’t seem to run as smoothly as you like, then you should consider getting yourself a tune-up tool like CleanMyPC by MacPaw. The software allows you to have complete control over your computer and easily manage applications and files to ensure your computer runs smoothly.

CleanMyPC has three different types of subscription offers. A single license for 1 computer, Double for 2, and Family pack for up to 5. It is compatible with Windows 7,8, and 10 and is a fairly light application.

The Interface

The overall interface of the program is quite elegant and easy to navigate. It allows you to access files/applications on your computer and uninstall or shred them. The interface is easy to navigate through due to the big icons and labels of exactly what each icon is for. You should have no trouble getting things done. It provides you easy cleaning option such as “Scan” which filters out files and programs you don’t really use to get rid of which will free up space on your hard disc. However, it doesn’t show you the exact details of each program and file which can be annoying for those that like complete control of what is being discarded. But it works efficiently to clean up your PC ensuring all your important files go untouched.

Did performance actually improve after using CleanMyPC?

We ran a performance check on the Geekbench system before cleaning our PC with CleanMyPC. It gave us a performance score of 5,900 while booting up in 51 seconds. This is on an Intel Core i7 with 2 GHz, 4GB RAM, and 125 Intel SSD drive. After running CleanMyPC, the score rose to 6,095 and boot up time also decrease to 46 seconds.

Now to the average user that type of improvement won’t seem like much. However, when you run heavier programs and they load up faster and perform much better, you will notice and appreciate the difference. As wonderful as CleanMyPC is, there are other programs that are better at improving overall PC performances.

Our verdict

While CleanMyPC works well at improving PC performance and also clearing up unused files and apps. It doesn’t work as well as programs like SlimCleaner Plus. If you are a normal PC user that doesn’t really care much about what is being deleted and want to run a simple clean then CleanMyPC is a great option. If not, then you should consider a more in-depth cleaner that provides you with information about every file that is being cleaned and or deleted.

What to do if my SD Card Won’t Read

Technology can be a bit tricky, while it makes life easier, it can also make it much more difficult. As we rely more and more on technology, we tend to put a lot more trust in it since they make daily tasks easier. However, there are occasions when technology breaks down on us. If you use SD cards on a normal basis then you probably know exactly what we mean. While SD cards make it easy for us to store data and transfer it, there are times it just stops working for one reason or another. If that happens to you then we have some tips for you.

Identify the problem

If your computer refuses to detect the SD card then you first need to figure out where the problem is. Is it the reader on the computer or the SD card itself. The best way to figure out if it is the SD card is to insert the card back into the device you normally use it on be it a phone, digital camera, etc. If the device reads the card then that means the card is fine and there is something wrong with the reader on your computer. If it doesn’t, then there is something wrong with the card.

It’s a fairly easy fix if the reader on the computer is not working. All you have to do is purchase an external reader and hook it up to your computer with the help of a USB. The card should work then.

What if the card doesn’t work?

There are a few steps you can try if the card isn’t working properly. First, you want to start off with the basic checks. Make sure the card is slotted in properly, push it in until you can’t anymore. Sometimes people make the mistake of not pushing it enough so the reader can get a read on the sensor on the card.

If the problem still persists, then you want to check the write protection lock on the card. Most cards now come with a write protection option, once enabled, it doesn’t allow anyone to write on it.

The final thing you can do is format the card altogether. Phones and computer require an SD card to be in a particular format so they can read it. If it isn’t formatted properly then the device would not be able to read it. You want to format the card in a compatible format for your computer and the device you normally use the card on, so it is readable on both. Note that, when you format your card all data will be lost on it. So this is the final step you should take.

If you have tried all the above solution and your card still refuses to work then it is recommended you look for a replacement. SD card after excessive use can become non-functional especially if they have been treated badly. So it is important you take proper care of the SD card, keep it away from humid and hot places and make sure not to get any scratches on the sensors. This will ensure you don’t have any problems with the SD card.

How to Download Pictures from Google Drive

Like Dropbox, Google Drive is a service from Google that allows you to store your data via a cloud storage technique. For beginners, Google offers every user a total of 15GB storage space at no cost. This kind of space can be quite useful for storing documents, photos, audios, and also videos. If you feel this space is insufficient, you can choose to upgrade your space by subscribing to a monthly plan to upscale your space.

Paid subscription

For example, you have an option to purchase 100GB storage space for $1.99 per month or a bigger package at $299.99/month for 30TR space. These figures can change as and when Google deems fit. The point here is to encourage you to save your items in the most secure way. Photos create beautiful memories that need to be safely backed up in order for you not to lose them. You might lose an SD card or hard disk and if you had no backup for your photos or documents, you might lose very crucial and sentimental items.

In addition, Google Drive allows you to transfer the items to other locations and devices. For instance, you can manually transfer photos to your computer or to your phone gallery.  At the moment, you are limited to transfer items that are not more than 2GB from Google drive to your preferred locations.  In case the selected items exceed 2GB, you will see a message prompting you to select fewer files that do not exceed the threshold.

Why Google Drive

There are many different benefits of the Google cloud-based storage system. First, the cost is affordable and the first 15GB is offered freely by Google. Other additional storage spaces are also comparatively affordable. Second, since the system is cloud-based you can give access to your clients or family as they can access your account remotely. Besides, you can access your account from any system as long as there is an internet connection. You simply have to log into your account and view the file uploads. The system also allows you to make downloads from any location. Lastly, Google Drive is as good as other popular office applications that bring real benefits that can be used by individual businessmen and also small businesses.

Downloading Pictures from Google Drive

Google Drive allows users not only to upload pictures but also to download them and transfer them to other devices such as android phones, iPad, or even your laptop. The process of downloading photos involves simple steps and taps. For all devices, you should start by downloading the Google Drive app on your device. Though you may choose to access the drive offline, it is highly recommended that you download the app directly on your device so that you may closely monitor your account. However, if your files are primarily for offline use, you necessarily don’t need the app on your phone.

Below are some quick steps to follow if you want to download pictures from Google Drive.

Downloading photos into your phone Gallery

  • Download the Google Photos app to your device and open it
  • Click on the photo that you would like to download. Kindly note that Google drive does not allow you to download multiple photos simultaneously, you have to click on the photos one after the other.
  • Click on the Options Menu and select download, then select the exact location you would like to save your photo. In this case, choose gallery and your picture will appear in your gallery in no time.

Downloading pictures to your iPhone device

  • Open the ‘photo transfer’ app on your iDevice and click on the receive option
  • Then swipe left to reveal the pluggins
  • Click on the Google Drive button
  • Tap the select and download option and the google drive folders will pop
  • Select the photos you desire to download and follow the prompts and save them in your desired location
  • Repeat the same procedure if you are using an iPad

Downloading photos from Google Drive to your PC  

  • Type https://drive.google.com on your URL and sign in to access your account. For those who have not signed up, follow the link and go to my drive section. Go to folders and files section and upload your items, you may also choose to sync. Upload files to Google Drive and save them. You can also edit them and format them the way you prefer. You can share your folders and files and name other people as file or folder owners.
  • Next, to the photos or images, you will see a checkbox, click on it
  • Right click on the photo you would like to transfer and select the download option. You will receive a prompt asking you to save the file selected (comes as a zip file), click on it and specify the location of the image. The zip file selected will contain the image you selected to download
  • Click on the download button on the zip file and there you have your picture/s.

Conclusion

There are many cloud services today but there is none like Google Drive, which is simple to use and also securely created to suit your needs. You need to create a free account using the simple sign-up procedure so that you start saving your work on Google drive. Google Drive offers users some free storage space for their files and folders. Nonetheless, if the storage space runs out, you can make a small monthly fee to access a bigger storage.

With the up-scale, you can know save as many audios, videos, documents, and pictures safely without any fear of losing them. Google drive keeps them very safe for you. Additionally, Google makes it easy for parties to transfer files and folders to other locations including PCs or phones. Google Drive could not have come at a better time when the online craze is on the high. Finally, Google Drive is a flexible cloud system that is specifically designed to make life even easier and enjoyable.

How to Repair Damaged SD Card in Android

It can be devastating when you are trying to take videos or photos only to get an error message that there is no valid storage or the SD card has damage. The worst scenario is when you are told to reformat your card and fear to lose your favorite videos and photos that were stored on the SD cards. Do not worry though; this article will give you real quick solutions to your damaged or corrupted SD card.

Android devices

When you have an Android phone, you are likely to boost your phone memory by inserting an SD card due to the limitless downloads for both images and videos that you will probably make. In other words, you need an additional device that increases your memory capacity. Consequently, SD cards are popular since they provide a convenient way to store your mobile data. Besides, SD card read-and-write speeds are incredible and may be a true hypothesis that every Android owner has an SD card. Furthermore, an SD card is affordable.

Apart from the benefits that you might enjoy in an SD card, it is also important to highlight that you might encounter some challenges along the way. When the SD card gets damaged, destroyed, or corrupted, you risk losing all stored data including videos, photos, and other data that you might have saved.

Signs of a damaged SD card

Here are some of the signs of a corrupted SD card.

  • Getting an error message that your memory card is corrupted or SD card damaged
  • If most of your files, photos, or videos are missing, it means that you have a corrupted SD card
  • Additionally, when no folder appears when you read your SD card from other devices is a sign of damage. In this case, you get a read-and-write error. The device used might even be a PC but you find no folder when you query the SD card
  • On the other hand, you might be able to view the files on camera, PC, or any other devices but a ‘write-protected’ error appears when you try to copy the contents

At this point, the SD card can get damage due to two main ways: physical damage or software error. If your SD card suffers physical damage, it is difficult to fix it. Nonetheless, if it is a software issue, you can get that fixed in some simple ways, as outlined below.

Solution 1

If your SD is not reading on your Android device, try to connect to another device and see if the problem recurs.  From there, you can safely diagnose whether the problem is rooted in your device. Sometimes, it might be a simple case of an incompatible SD with your Android device. Solve other cases by cleaning the metal area around the card with either water or alcohol and inserting the card when the slot is dry and this will fix your problem instantly.

Solution 2

Connect the damaged SD card to a card reader in a system, if there is no new drive letter assigned to it, the SD card is not being read. In other instances, the reader is assigned to a drive letter, however, you might still get a message prompting you to insert the disk into drive E; this means your card is not being read. To resolve the problem, a new drive letter needs to be assigned through the operating system settings.

Solution 3

Insert the damaged card into a card reader and connect it to a system. Search for the card reader through the Windows taskbar and note the drive letter assigned to your affected SD. Click on the start menu and search for ‘’cmd”. Right-click the ‘’cmd’’ and then go to the ‘’run as administrator’. A command window will open after that type “chkdsk” followed by the corresponding drive letter, followed by a colon, then /f and you press enter. All possible errors in your card will be discovered here and your corrupted card is fixed at this point. All you do is to follow the prompts and lastly check if you have recovered your lost files and data

Solution 4

If you right click on properties in the drive letter section and an empty space appears, it means that your directory might be deleted or erased. In this case, you will use another option such as the Sandisk inbuilt solution option. The option requires that you use RescuePro Deluxe and RescuePro that can be downloaded online. Download a demo version before you purchase it, this way you can know how it works prior to buying it. After you have recovered your data, you can format your SD and use it again.

Solution 5

If your card can read files but you can’t save them, it is in write protected mode. Try unlocking the switch and modify or save the files on the SD. If the problem persists, then you may have lost your data or your card has become inaccessible. You can store all files and copy to another device and format your card so that you can access it. Alternatively, you need a disk diagnostic tool to solve the problem such as SD Recovery software. Choosing the right diagnostic tool will enable you to access your SD card, and to recover lost or deleted data or files.

Conclusion

A damaged SD card can be fixed and you will consequently recover your files that you think you may have lost. In case you have physically damaged your SD, unfortunately, you may not be able to recover your lost files. Nonetheless, if there the damage is due to software issues, it you can solve them amicably. More importantly, the above quick solutions will go a long way to solving your problems with your corrupted SD. If you do not handle the problem correctly, you risk losing your files permanently. In other words, you should know how to fix your SD card in case of software damages.