Github Trending Web Scraper

DAY 9 👾

I’m back 💙

Github Trending Repos.

The JSON API can be found here.

It gives repo description & the link to the repo.

The API is in the following format

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
{ 
  {
    "name": "cs-video-courses",
    "url": "https://github.com/Developer-Y/cs-video-courses",
    "description": "List of Computer Science courses with video lectures."
  },
  {
    "name": "Keyframes",
    "url": "https://github.com/facebookincubator/Keyframes",
    "description": "A library for converting Adobe AE shape based animations to a data format and play it back on Android and iOS devices."
  },
  {
    "name": "pipfile",
    "url": "https://github.com/pypa/pipfile",
    "description": "Description Not Provided"
  },
  {
    "name": "md2googleslides",
    "url": "https://github.com/googlesamples/md2googleslides",
    "description": "Generate Google Slides from markdown"
  },
  {
    "name": "easy-application",
    "url": "https://github.com/j-delaney/easy-application",
    "description": "351 software engineering companies that are easy to apply to"
  },
  {
    "name": "pix2pix",
    "url": "https://github.com/phillipi/pix2pix",
    "description": "Image-to-image translation using conditional adversarial nets"
  },
  {
    "name": "FreeCodeCamp",
    "url": "https://github.com/FreeCodeCamp/FreeCodeCamp",
    "description": "The https://FreeCodeCamp.com open source codebase and curriculum. Learn to code and help nonprofits."
  },
  {
    "name": "channels",
    "url": "https://github.com/andrew--r/channels",
    "description": "A collection of useful YouTube channels for javascript developers and web designers"
  },
  {
    "name": "react-draft-wysiwyg",
    "url": "https://github.com/jpuri/react-draft-wysiwyg",
    "description": "A Wysiwyg editor build on top of ReactJS and DraftJS. https://jpuri.github.io/react-draft-wysiwyg"
  },
  {
    "name": "katana-swift",
    "url": "https://github.com/BendingSpoons/katana-swift",
    "description": "Swift Apps in a Swoosh! A modern framework for creating iOS apps, inspired by React/Redux."
  },
  {
    "name": "BeeHive",
    "url": "https://github.com/alibaba/BeeHive",
    "description": " BeeHive is a solution for iOS Application module programs, it absorbed the Spring Framework API service concept to avoid to coupling between modules."
  },
  {
    "name": "ArcLayout",
    "url": "https://github.com/florent37/ArcLayout",
    "description": "With Arc Layout explore new styles and approaches on material design"
  },
  {
    "name": "jstraining",
    "url": "https://github.com/ruanyf/jstraining",
    "description": "全栈工程师培训材料"
  },
  {
    "name": "faust",
    "url": "https://github.com/grame-cncm/faust",
    "description": "Description Not Provided"
  },
  {
    "name": "go-audit",
    "url": "https://github.com/slackhq/go-audit",
    "description": "go-audit is an alternative to the auditd daemon that ships with many distros"
  },
  {
    "name": "awesome-mac",
    "url": "https://github.com/jaywcjlove/awesome-mac",
    "description": " This repo is a collection of awesome Mac applications and tools for developers and designers."
  },
  {
    "name": "ENViews",
    "url": "https://github.com/codeestX/ENViews",
    "description": "A cool dynamic view library"
  },
  {
    "name": "Diff.swift",
    "url": "https://github.com/wokalski/Diff.swift",
    "description": "The fastest Diff and patch library in Swift. Includes UICollectionView/UITableView utils."
  },
  {
    "name": "poisontap",
    "url": "https://github.com/samyk/poisontap",
    "description": "Exploits locked/password protected computers over USB, drops persistent WebSocket-based backdoor, exposes internal router, and siphons cookies using Raspberry Pi Zero & Node.js."
  },
  {
    "name": "vue",
    "url": "https://github.com/vuejs/vue",
    "description": "Simple yet powerful library for building modern web interfaces."
  },
  {
    "name": "Eve",
    "url": "https://github.com/witheve/Eve",
    "description": "Better tools for thought"
  },
  {
    "name": "tensorflow",
    "url": "https://github.com/tensorflow/tensorflow",
    "description": "Computation using data flow graphs for scalable machine learning"
  },
  {
    "name": "Crescento",
    "url": "https://github.com/developer-shivam/Crescento",
    "description": "Add curve at bottom of image views and relative layouts."
  },
  {
    "name": "superset",
    "url": "https://github.com/airbnb/superset",
    "description": "Superset is a data exploration platform designed to be visual, intuitive, and interactive"
  },
  {
    "name": "LolliPin",
    "url": "https://github.com/OrangeGangsters/LolliPin",
    "description": "A Material design Android pincode library. Supports Fingerprint."
  }
]

This project looks very simple, but it was very tricky because it was my first time using Nightmare JS. Nightmare is a High Level Automation Library made using Electron. It is same as Phantom JS, but it is fast & a little modern.

(1) Clone this repo

1
$ git clone git@github.com:deadcoder0904/github-trending-json-api.git

(2) Go into the directory

1
$ cd github-trending-json-api

(3) Type yarn to install all dependencies

1
$ yarn

(4) Type yarn start to start scraping the web

1
$ yarn start

In a minutes time, depending on the speed of your internet, a new file will be created according to today’s date in json folder.

Voila, we have scraped the web together. It has many different applications. Further we can login to Facebook and Twitter using Nightmare. We can send Automated Emails using Nightmare & many more things . . .

Till the next time 👻